|
| | | | |
| | | |
| Stat |
Members: 3669
Articles: 2'599'751
Articles rated: 2609
22 March 2025 |
|
| | | | |
|
Article overview
| |
|
|
Implicit biases in multitask and continual learning from a backward error analysis perspective | | Benoit Dherin
; | | Date: |
1 Nov 2023 | | Abstract: | Using backward error analysis, we compute implicit training biases in
multitask and continual learning settings for neural networks trained with
stochastic gradient descent. In particular, we derive modified losses that are
implicitly minimized during training. They have three terms: the original loss,
accounting for convergence, an implicit flatness regularization term
proportional to the learning rate, and a last term, the conflict term, which
can theoretically be detrimental to both convergence and implicit
regularization. In multitask, the conflict term is a well-known quantity,
measuring the gradient alignment between the tasks, while in continual learning
the conflict term is a new quantity in deep learning optimization, although a
basic tool in differential geometry: The Lie bracket between the task
gradients. | | Source: | arXiv, 2311.00235 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
|
| |
|
|
|
REGISTER