CSAVocoder: mel adaptor beats pose 2.6×

CSAVocoder is a streaming spatial audio vocoder that sits downstream of any mel-spectrogram generator — TTS, music diffusion, codec decoder — and synthesises binaural or first-order ambisonics (FOA) waveforms conditioned on a live 7D pose stream.

The ablation tells the story. Remove the Attentional Mel Adaptor and ANG COS drops from 62.11 to 42.60 — a 19.51-point hit. Remove pose conditioning entirely and the loss is 7.33 points. The mel adaptor matters more than 2.6× as much as the position input.

What it is

CSAVocoder is a research artifact described in the preprint arxiv 2608.25404v1. No code, weights, or licence are stated in the paper.

Under the hood: a modified HiFi-GAN V1, total upsampling factor 8×5×4×2 = 320, running at 48 kHz with a 320-sample hop. Transposed convolutions are gone; ShuffleUpsampleBlocks take their place. These are causal channel-to-time pixel-shuffle ops — no temporal mixing, so causality is preserved for free. RTF is reported on a single NVIDIA RTX 4090 only.

One shared backbone, channel-free generator, handles both binaural (2-channel) and FOA (4-channel) output. Per-format discriminator heads carry the format-specific objectives. Extending to 5.1 or 7.1 surround would need new objectives and heads but no changes to the generator core.

Training data (per the paper): approximately 600 hours / 350k samples of binaural speech — MRSSpeech subset of MRSAudio and EasyCom, both denoised with ClearVoice — plus approximately 900 hours / 310k samples of FOA speech from Spatial LibriSpeech, supplemented by simulated data from SoundSpaces/Habitat-Sim on MP3D indoor scenes. All stored at 48 kHz, 16-bit PCM. Test set: 700 randomly sampled segments from all datasets.

Fact	Value	Condition / source
Output formats	Binaural (2-ch), FOA (4-ch)	Paper, section 3.5
Sample rate	48,000 Hz	Paper, A.1
Mel bins / hop / FFT	128 / 320 / 1024	Paper, A.1
Generator backbone	HiFi-GAN V1, causal-modified	Paper, A.2
Total upsampling factor	320 (8×5×4×2)	Paper, A.2
Pose representation	7D: 3D Cartesian + 4D quaternion	Paper, section 3.1
Pose rate	20 Hz	Paper, C.4
Binaural training data	~600 h / ~350k samples	Paper, Dataset
FOA training data	~900 h / ~310k samples	Paper, Dataset
Test set size	700 segments	Paper, Dataset
RTF	0.1587	RTX 4090 only; paper, section 4.2
MOS-P / MOS-Q	4.25 ± 0.16 / 4.09 ± 0.21	Subjective eval; paper, section 4.5
Code / weights	Not stated	—
Licence	Not stated	—

Performance claims and their conditions

RTF = 0.1587 on a single NVIDIA RTX 4090. Batch size and streaming chunk size: not stated. The authors call this “well below unity.” No other hardware is benchmarked.

Spatial quality on the binaural test set: ANG COS = 62.11, DIS COS = 77.05. Computed with Spatial-AST embeddings. Each clip is partitioned into 1-second segments — Spatial-AST only produces position estimates for static sources — and segment-level cosine similarities are averaged. The metric resolves at 1-second granularity. Fast dynamic trajectories may be under- or overscored relative to perceptual quality.

Audio quality on the same set: MRSTFT = 1.223, PESQ = 2.109, MCD = 2.153, Periodicity = 0.107. PESQ 2.109 is below the ITU-T “good” threshold of ≥ 3.0. The training corpus is predominantly speech, and the paper makes no wideband quality claim.

MOS-P = 4.25 ± 0.16, MOS-Q = 4.09 ± 0.21. CSAVocoder scores highest MOS-P among compared models. Evaluator count, stimuli count and listener recruitment are not stated.

Ablation results on binaural ANG COS and DIS COS (all figures from the paper, section 4.6):

Ablation	ANG COS	DIS COS	Δ ANG COS vs full
Full model	62.11	77.05	—
w/o Mel Adaptor	42.60	65.39	−19.51
w/o Position Adaptor	54.78	70.63	−7.33
w/o SCD	58.82	74.63	−3.29
Mel Adaptor 2 heads	61.03	76.55	−1.08
Mel Adaptor 8 heads	61.50	76.70	−0.61

The interpretation: the Mel Adaptor’s cross-channel attention picks up spectral cues already present in the training BRIRs, and the Position Adaptor refines with explicit geometry. Nobody has independently verified these numbers.

Reported by 2608.25404 m· scroll →

Conditions around the comparison

scroll →

600 hours

binaural training data

48 kHz

sampling rate

0.1587

RTF on RTX 4090

62.11

ANG COS binaural

What it is and is not useful for

Useful for: streaming TTS or generative audio pipelines where spatial rendering is the last stage. If upstream already emits multi-channel mel-spectrograms at 48 kHz, CSAVocoder drops in as the vocoder and handles HRTF rendering without a separate convolution reverb unit. Frame-by-frame processing works because each StreamingResBlock holds a fixed left-context buffer sized to its dilation — constant memory overhead.

Binaural speech synthesis for head-tracked playback. The 7D pose input (position + orientation quaternion) at 20 Hz maps to what headphone motion APIs already expose — see the earlier note on CMHeadphoneMotionManager capabilities. One caveat: 20 Hz is coarser than the 100 Hz device IMUs typically deliver, and the paper does not evaluate mismatched pose rates.

FOA encoding from arbitrary mono speech sources with a spatial trajectory. The distribution is read-aloud English; music and non-speech are out of distribution.

Not useful for: pipelines that don’t already produce mel-spectrograms. It’s a vocoder, not a synthesis model — no text encoder, diffusion backbone, or codec decoder. Edge or CPU deployment: RTF is measured only on an RTX 4090. Room geometries outside the training distribution: source-listener distances outside 1–10 m horizontal and height differences above 2 m were excluded during data construction. Outdoor or large reverberant spaces are not represented. High-fidelity music or wideband content: PESQ 2.109 is below typical speech quality thresholds, and the metric is speech-targeted.

If you want to probe spatial rendering behaviour interactively before committing to an integration, the Head-Tracked Stereo Pan Lab lets you drive azimuth and elevation directly.

What the paper does not settle

RTF is measured at unspecified batch and chunk size on one GPU class. Whether 0.1587 survives realistic streaming — small chunks, CPU-GPU round trips, concurrent sessions — is not established. Parameter count: not reported. Memory footprint comparison against lighter vocoders is impossible. MOS evaluator pool size is not stated; with CIs of ±0.16 to ±0.21, rankings between closely scored models may be within noise.

---

Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from a single source — the arXiv abstract for 2608.25404. The claims and quantities it worked from were extracted from that document by qwen/qwen3.8-27b. Before publication an automated gate re-checked every extracted claim against the source document (119 claim(s) and 76 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (9 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run 2608.25404’s benchmarks; figures attributed to it are its own.

FAQ

Does this replace a convolution reverb or HRTF pipeline?

It replaces the rendering step for trained distributions. The model absorbed BRIR/RIR convolution implicitly from training data. It does not expose a parameterisable room model — you cannot specify a novel geometry at inference time. For controlled acoustic simulation, a physics-based pipeline still applies. CSAVocoder's advantage is low-latency neural rendering of positions and rooms represented in training.

What does RTF = 0.1587 actually mean for buffer latency?

RTF = 0.1587 means roughly 6.3× faster than real-time on an RTX 4090 (1 / 0.1587 ≈ 6.3). Chunk size is not stated. A favourable RTF with a large chunk still imposes latency equal to the chunk duration regardless of throughput. Algorithmic delay is not reported and cannot be inferred from RTF alone.

Is the 4-head attention choice in the Mel Adaptor meaningful?

The ablation shows 4 heads (ANG COS 62.11) beats 2 heads (61.03) and 8 heads (61.50). Margins are small — 1.08 points for 2 heads, 0.61 for 8 — and both still substantially outperform removing the adaptor entirely (42.60). If you're targeting a lower-budget variant, 2 or 8 heads are viable substitutes with modest regression.

Can this handle non-speech audio?

The paper does not test non-speech content. The binaural training set is entirely speech (MRSSpeech, EasyCom); the FOA set is Spatial LibriSpeech from read-aloud English. PESQ is speech-targeted too. Behaviour on music, ambiences, or sound effects is undocumented — treat as out-of-distribution until tested independently.