| | |
| | |
Stat |
Members: 3643 Articles: 2'488'730 Articles rated: 2609
29 March 2024 |
|
| | | |
|
Article overview
| |
|
Entities as Experts: Sparse Memory Access with Entity Supervision | Thibault Févry
; Livio Baldini Soares
; Nicholas FitzGerald
; Eunsol Choi
; Tom Kwiatkowski
; | Date: |
15 Apr 2020 | Abstract: | We focus on the problem of capturing declarative knowledge in the learned
parameters of a language model. We introduce a new model, Entities as Experts
(EaE), that can access distinct memories of the entities mentioned in a piece
of text. Unlike previous efforts to integrate entity knowledge into sequence
models, EaE’s entity representations are learned directly from text. These
representations capture sufficient knowledge to answer TriviaQA questions such
as "Which Dr. Who villain has been played by Roger Delgado, Anthony Ainley,
Eric Roberts?". EaE outperforms a Transformer model with $30 imes$ the
parameters on this task. According to the Lama knowledge probes, EaE also
contains more factual knowledge than a similar sized Bert. We show that
associating parameters with specific entities means that EaE only needs to
access a fraction of its parameters at inference time, and we show that the
correct identification, and representation, of entities is essential to EaE’s
performance. We also argue that the discrete and independent entity
representations in EaE make it more modular and interpretable than the
Transformer architecture on which it is based. | Source: | arXiv, 2004.7202 | Services: | Forum | Review | PDF | Favorites |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser claudebot
|
| |
|
|
|
| News, job offers and information for researchers and scientists:
| |