|
| | | | |
| | | |
| Stat |
Members: 3665
Articles: 2'599'751
Articles rated: 2609
25 January 2025 |
|
| | | | |
|
Article overview
| |
|
|
RepCodec: A Speech Representation Codec for Speech Tokenization | | Zhichao Huang
; Chutong Meng
; Tom Ko
; | | Date: |
1 Sep 2023 | | Abstract: | With recent rapid growth of large language models (LLMs), discrete speech
tokenization has played an important role for injecting speech into LLMs.
However, this discretization gives rise to a loss of information, consequently
impairing overall performance. To improve the performance of these discrete
speech tokens, we present RepCodec, a novel speech representation codec for
semantic speech tokenization. In contrast to audio codecs which reconstruct the
raw audio, RepCodec learns a vector quantization codebook through
reconstructing speech representations from speech encoders like HuBERT or
data2vec. Together, the speech encoder, the codec encoder and the vector
quantization codebook form a pipeline for converting speech waveforms into
semantic tokens. The extensive experiments illustrate that RepCodec, by virtue
of its enhanced information retention capacity, significantly outperforms the
widely used k-means clustering approach in both speech understanding and
generation. Furthermore, this superiority extends across various speech
encoders and languages, affirming the robustness of RepCodec. We believe our
method can facilitate large language modeling research on speech processing. | | Source: | arXiv, 2309.00169 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
|
| |
|
|
|
REGISTER