TTBA: Spatial Prompted Text to Binaural Audio Generation Using Transformer
Figure 1. Diagrams of the overall structure of our proposed text to binaural audio model (TTBA).
Model Configuration
- Transformer-ALM: Autoregressive Transformer with num_layers = 24, dim = 1536, num_heads = 24.
- Text Encoder: T5-large with max_length = 128, cond_dim = 1536.
- Audio Encoder/Decoder: EnCodec 16 kHz with 4 codebooks × 2048 bins.
-
Optimizer: Adam with learning rate = 5e-5, betas = (0.9, 0.999), weight decay = 1e-3;
Learning rate scheduler = InverseLR (inv_gamma = 1e6, power = 0.5, warmup = 0.99). -
Teacher Model: Autoregressive Transformer with num_layers = 48, dim = 1536, num_heads = 24;
Audio Encoder/Decoder: EnCodec 16 kHz with 4 codebooks × 2048 bins.