|
| | | | |
| | | |
| Stat |
Members: 3645
Articles: 2'504'928
Articles rated: 2609
25 April 2024 |
|
| | | | |
|
Article overview
| |
|
|
All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation | | Di Xie
; Jiang Xiong
; Shiliang Pu
; | | Date: |
6 Mar 2017 | | Abstract: | Deep neural network is difficult to train and this predicament becomes worse
as the depth increases. The essence of this problem exists in the magnitude of
backpropagated errors that will result in gradient vanishing or exploding
phenomenon. We show that a variant of regularizer which utilizes orthonormality
among different filter banks can alleviate this problem. Moreover, we design a
backward error modulation mechanism based on the quasi-isometry assumption
between two consecutive parametric layers. Equipped with these two ingredients,
we propose several novel optimization solutions that can be utilized for
training a specific-structured (repetitively triple modules of Conv-BNReLU)
extremely deep convolutional neural network (CNN) WITHOUT any shortcuts/
identity mappings from scratch. Experiments show that our proposed solutions
can achieve 4% improvement for a 44-layer plain network and almost 50%
improvement for a 110-layer plain network on the CIFAR-10 dataset. Moreover, we
can successfully train plain CNNs to match the performance of the residual
counterparts. Besides, we propose new principles for designing network
structure from the insights evoked by orthonormality. Combined with residual
structure, we achieve comparative performance on the ImageNet dataset. | | Source: | arXiv, 1703.1827 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
|
| |
|
|
|
|
News, job offers and information for researchers and scientists:
| |
REGISTER