|
| | | | |
| | | |
| Stat |
Members: 3645
Articles: 2'506'133
Articles rated: 2609
26 April 2024 |
|
| | | | |
|
Article overview
| |
|
|
Reducing overfitting in challenge-based competitions | | Elias Chaibub Neto
; Bruce R Hoff
; Chris Bare
; Brian M Bot
; Thomas Yu
; Lara Magravite
; Andrew D Trister
; Thea Norman
; Pablo Meyer
; Julio Saez-Rodrigues
; James C Costello
; Justin Guinney
; Gustavo Stolovitzky
; | | Date: |
1 Jul 2016 | | Abstract: | Over-fitting is a dreaded foe in challenge-based competitions. Because
participants rely on public leaderboards to evaluate and refine their models,
there is always the danger they might over-fit to the holdout data supporting
the leaderboard. The recently published Ladder algorithm aims to address this
problem by preventing the participants from exploiting willingly or
inadvertently minor fluctuations in public leaderboard scores during model
refinement. In this paper, we report a vulnerability of the Ladder that induces
severe over-fitting of the leaderboard when the sample size is small. To
circumvent this attack, we propose a variation of the Ladder that releases a
bootstrapped estimate of the public leaderboard score instead of providing
participants with a direct measure of performance. We also extend the scope of
the Ladder to arbitrary performance metrics by relying on a more broadly
applicable testing procedure based on the Bayesian bootstrap. Our method makes
it possible to use a leaderboard, with the technical and social advantages that
it provides, even in cases where data is scant. | | Source: | arXiv, 1607.0091 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
|
| |
|
|
|
|
News, job offers and information for researchers and scientists:
| |
REGISTER