| | |
| | |
Stat |
Members: 3667 Articles: 2'599'751 Articles rated: 2609
07 February 2025 |
|
| | | |
|
Article overview
| |
|
Learning Decorrelated Representations Efficiently Using Fast Fourier Transform | Yutaro Shigeto
; Masashi Shimbo
; Yuya Yoshikawa
; Akikazu Takeuchi
; | Date: |
4 Jan 2023 | Abstract: | Barlow Twins and VICReg are self-supervised representation learning models
that use regularizers to decorrelate features. Although they work as well as
conventional representation learning models, their training can be
computationally demanding if the dimension of projected representations is
high; as these regularizers are defined in terms of individual elements of a
cross-correlation or covariance matrix, computing the loss for $d$-dimensional
projected representations of $n$ samples takes $O(n d^2)$ time. In this paper,
we propose a relaxed version of decorrelating regularizers that can be computed
in $O(n dlog d)$ time by the fast Fourier transform. We also propose an
inexpensive trick to mitigate the undesirable local minima that develop with
the relaxation. Models learning representations using the proposed regularizers
show comparable accuracy to existing models in downstream tasks, whereas the
training requires less memory and is faster when $d$ is large. | Source: | arXiv, 2301.01569 | Services: | Forum | Review | PDF | Favorites |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
|
| |
|
|
|