| | |
| | |
Stat |
Members: 3643 Articles: 2'488'730 Articles rated: 2609
29 March 2024 |
|
| | | |
|
Article overview
| |
|
Improving performance of recurrent neural network with relu nonlinearity | Sachin S. Talathi
; Aniket Vartak
; | Date: |
12 Nov 2015 | Abstract: | In recent years significant progress has been made in successfully training
recurrent neural networks (RNNs) on sequence learning problems involving long
range temporal dependencies. The progress has been made on three fronts: (a)
Algorithmic improvements involving sophisticated optimization techniques, (b)
network design involving complex hidden layer nodes and specialized recurrent
layer connections and (c) weight initialization methods. In this paper, we
focus on recently proposed weight initialization with identity matrix for the
recurrent weights in a RNN. This initialization is specifically proposed for
hidden nodes with Rectified Linear Unit (ReLU) non linearity. We offer a simple
dynamical systems perspective on weight initialization process, which allows us
to propose a modified weight initialization strategy. We show that this
initialization technique leads to successfully training RNNs composed of ReLUs.
We demonstrate that our proposal produces comparable or better solution for
three toy problems involving long range temporal structure: the addition
problem, the multiplication problem and the MNIST classification problem using
sequence of pixels. In addition, we present results for a benchmark action
recognition problem. | Source: | arXiv, 1511.3771 | Services: | Forum | Review | PDF | Favorites |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser claudebot
|
| |
|
|
|
| News, job offers and information for researchers and scientists:
| |