|
| | | | |
| | | |
| Stat |
Members: 3645
Articles: 2'506'133
Articles rated: 2609
26 April 2024 |
|
| | | | |
|
Article overview
| |
|
|
Compressing Transformer-based self-supervised models for speech processing | | Tzu-Quan Lin
; Tsung-Huan Yang
; Chun-Yao Chang
; Kuang-Ming Chen
; Tzu-hsun Feng
; Hung-yi Lee
; Hao Tang
; | | Date: |
18 Nov 2022 | | Abstract: | Despite the success of Transformers in self-supervised learning with
applications to various downstream tasks, the computational cost of training
and inference remains a major challenge for applying these models to a wide
spectrum of devices. Several isolated attempts have been made to compress
Transformers, prior to applying them to downstream tasks. In this work, we aim
to provide context for the isolated results, studying several commonly used
compression techniques, including weight pruning, head pruning, low-rank
approximation, and knowledge distillation. We report wall-clock time, the
number of parameters, and the number of multiply-accumulate operations for
these techniques, charting the landscape of compressing Transformer-based
self-supervised models. | | Source: | arXiv, 2211.09949 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
|
| |
|
|
|
|
News, job offers and information for researchers and scientists:
| |
REGISTER