|
| | | | |
| | | |
| Stat |
Members: 3645
Articles: 2'501'711
Articles rated: 2609
19 April 2024 |
|
| | | | |
|
Article overview
| |
|
|
On the Expressive Power of Deep Learning: A Tensor Analysis | | Nadav Cohen
; Or Sharir
; Amnon Shashua
; | | Date: |
16 Sep 2015 | | Abstract: | It has long been conjectured that hypothesis spaces suitable for data that is
compositional in nature, such as text or images, may be more efficiently
represented with deep hierarchical architectures than with shallow ones.
Despite the vast empirical evidence, formal arguments to date are limited and
do not capture the kind of networks used in practice. Using tensor
factorization, we derive a universal hypothesis space implemented by an
arithmetic circuit over functions applied to local data structures (e.g. image
patches). The resulting networks first pass the input through a representation
layer, and then proceed with a sequence of layers comprising sum followed by
product-pooling, where sum corresponds to the widely used convolution operator.
The hierarchical structure of networks is born from factorizations of tensors
based on the linear weights of the arithmetic circuits. We show that a shallow
network corresponds to a rank-1 decomposition, whereas a deep network
corresponds to a Hierarchical Tucker (HT) decomposition. Log-space computation
for numerical stability transforms the networks into SimNets.
In its basic form, our main theoretical result shows that the set of
polynomially sized rank-1 decomposable tensors has measure zero in the
parameter space of polynomially sized HT decomposable tensors. In deep learning
terminology, this amounts to saying that besides a negligible set, all
functions that can be implemented by a deep network of polynomial size, require
an exponential size if one wishes to implement (or approximate) them with a
shallow network. Our construction and theory shed new light on various
practices and ideas employed by the deep learning community, and in that sense
bear a paradigmatic contribution as well. | | Source: | arXiv, 1509.5009 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
|
| |
|
|
|
|
News, job offers and information for researchers and scientists:
| |
REGISTER