|
| | | | |
| | | |
| Stat |
Members: 3645
Articles: 2'504'928
Articles rated: 2609
25 April 2024 |
|
| | | | |
|
Article overview
| |
|
|
Learning from Imperfect Annotations | | Emmanouil Antonios Platanios
; Maruan Al-Shedivat
; Eric Xing
; Tom Mitchell
; | | Date: |
7 Apr 2020 | | Abstract: | Many machine learning systems today are trained on large amounts of
human-annotated data. Data annotation tasks that require a high level of
competency make data acquisition expensive, while the resulting labels are
often subjective, inconsistent, and may contain a variety of human biases. To
improve the data quality, practitioners often need to collect multiple
annotations per example and aggregate them before training models. Such a
multi-stage approach results in redundant annotations and may often produce
imperfect "ground truth" that may limit the potential of training accurate
machine learning models. We propose a new end-to-end framework that enables us
to: (i) merge the aggregation step with model training, thus allowing deep
learning systems to learn to predict ground truth estimates directly from the
available data, and (ii) model difficulties of examples and learn
representations of the annotators that allow us to estimate and take into
account their competencies. Our approach is general and has many applications,
including training more accurate models on crowdsourced data, ensemble
learning, as well as classifier accuracy estimation from unlabeled data. We
conduct an extensive experimental evaluation of our method on 5 crowdsourcing
datasets of varied difficulty and show accuracy gains of up to 25% over the
current state-of-the-art approaches for aggregating annotations, as well as
significant reductions in the required annotation redundancy. | | Source: | arXiv, 2004.3473 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
|
| |
|
|
|
|
News, job offers and information for researchers and scientists:
| |
REGISTER