|
| | | | |
| | | |
| Stat |
Members: 3645
Articles: 2'504'585
Articles rated: 2609
25 April 2024 |
|
| | | | |
|
Article overview
| |
|
|
Pretrained Language Model Embryology: The Birth of ALBERT | | David C. Chiang
; Sung-Feng Huang
; Hung-yi Lee
; | | Date: |
6 Oct 2020 | | Abstract: | While behaviors of pretrained language models (LMs) have been thoroughly
examined, what happened during pretraining is rarely studied. We thus
investigate the developmental process from a set of randomly initialized
parameters to a totipotent language model, which we refer to as the embryology
of a pretrained language model. Our results show that ALBERT learns to
reconstruct and predict tokens of different parts of speech (POS) in different
learning speeds during pretraining. We also find that linguistic knowledge and
world knowledge do not generally improve as pretraining proceeds, nor do
downstream tasks’ performance. These findings suggest that knowledge of a
pretrained model varies during pretraining, and having more pretrain steps does
not necessarily provide a model with more comprehensive knowledge. We will
provide source codes and pretrained models to reproduce our results at
this https URL | | Source: | arXiv, 2010.02480 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
|
| |
|
|
|
|
News, job offers and information for researchers and scientists:
| |
REGISTER