| | |
| | |
Stat |
Members: 3643 Articles: 2'487'895 Articles rated: 2609
29 March 2024 |
|
| | | |
|
Article overview
| |
|
A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks | Umut Simsekli
; Levent Sagun
; Mert Gurbuzbalaban
; | Date: |
18 Jan 2019 | Abstract: | The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is
often considered to be Gaussian in the large data regime by assuming that the
classical central limit theorem (CLT) kicks in. This assumption is often made
for mathematical convenience, since it enables SGD to be analyzed as a
stochastic differential equation (SDE) driven by a Brownian motion. We argue
that the Gaussianity assumption might fail to hold in deep learning settings
and hence render the Brownian motion-based analyses inappropriate. Inspired by
non-Gaussian natural phenomena, we consider the GN in a more general context
and invoke the generalized CLT (GCLT), which suggests that the GN converges to
a heavy-tailed $alpha$-stable random variable. Accordingly, we propose to
analyze SGD as an SDE driven by a L’{e}vy motion. Such SDEs can incur ’jumps’,
which force the SDE transition from narrow minima to wider minima, as proven by
existing metastability theory. To validate the $alpha$-stable assumption, we
conduct extensive experiments on common deep learning architectures and show
that in all settings, the GN is highly non-Gaussian and admits heavy-tails. We
further investigate the tail behavior in varying network architectures and
sizes, loss functions, and datasets. Our results open up a different
perspective and shed more light on the belief that SGD prefers wide minima. | Source: | arXiv, 1901.6053 | Services: | Forum | Review | PDF | Favorites |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser claudebot
|
| |
|
|
|
| News, job offers and information for researchers and scientists:
| |