StepAudio researchers published a technical report on StepAudio 3 Gen, an audio generation model. It handles text-to-speech without prior voice samples, voice design, singing, sound effects and music. It can also produce mixtures of several audio types in one output. The model predicts discrete audio codes one step at a time, rather than using the diffusion methods common in recent audio systems. Its tokenizer compresses audio into a shared code space that keeps both meaning and sound detail in every layer. The authors name three design choices as central, including a staged pretraining recipe that preserves the underlying language model's text skills. They report state-of-the-art results on text-to-speech and voice design. Audio samples are posted on a project site. The report does not state release or licensing terms.
What changed
Recent general audio models generate sound with diffusion Transformers over continuous representations.
What it unlocks
Generating speech, singing, sound effects and music from one model, including mixtures of audio types.
- 16 x 2048 residual code space
- 12.5 Hz audio token rate
- 16 codebooks, 15 completed causally
Sources