# CSAVocoder: mel adaptor beats pose 2.6× CSAVocoder renders binaural and FOA audio from mel-spectrograms and a 7D pose stream. Canonical page: https://makerportal.ai/blog/auralinter-arxiv-260825404v1 Author: Joshua Hrisko, Principal Engineer — MakerPortal Published: 2026-08-27 Section: Readout / Audio DSP · 6 min read Tags: auralinter, dsp-audio --- CSAVocoder is a streaming spatial audio vocoder that sits downstream of any mel-spectrogram generator — TTS, music diffusion, codec decoder — and synthesises binaural or first-order ambisonics (FOA) waveforms conditioned on a live 7D pose stream. The ablation tells the story. Remove the Attentional Mel Adaptor and ANG COS drops from 62.11 to 42.60 — a 19.51-point hit. Remove pose conditioning entirely and the loss is 7.33 points. The mel adaptor matters more than 2.6× as much as the position input. ## What it is CSAVocoder is a research artifact described in the preprint [arxiv 2608.25404v1](http://arxiv.org/abs/2608.25404v1). No code, weights, or licence are stated in the paper. Under the hood: a modified HiFi-GAN V1, total upsampling factor 8×5×4×2 = 320, running at 48 kHz with a 320-sample hop. Transposed convolutions are gone; ShuffleUpsampleBlocks take their place. These are causal channel-to-time pixel-shuffle ops — no temporal mixing, so causality is preserved for free. RTF is reported on a single NVIDIA RTX 4090 only. One shared backbone, channel-free generator, handles both binaural (2-channel) and FOA (4-channel) output. Per-format discriminator heads carry the format-specific objectives. Extending to 5.1 or 7.1 surround would need new objectives and heads but no changes to the generator core. **Training data** (per the paper): approximately 600 hours / 350k samples of binaural speech — MRSSpeech subset of MRSAudio and EasyCom, both denoised with ClearVoice — plus approximately 900 hours / 310k samples of FOA speech from Spatial LibriSpeech, supplemented by simulated data from SoundSpaces/Habitat-Sim on MP3D indoor scenes. All stored at 48 kHz, 16-bit PCM. Test set: 700 randomly sampled segments from all datasets. | Fact | Value | Condition / source | |---|---|---| | Output formats | Binaural (2-ch), FOA (4-ch) | Paper, section 3.5 | | Sample rate | 48,000 Hz | Paper, A.1 | | Mel bins / hop / FFT | 128 / 320 / 1024 | Paper, A.1 | | Generator backbone | HiFi-GAN V1, causal-modified | Paper, A.2 | | Total upsampling factor | 320 (8×5×4×2) | Paper, A.2 | | Pose representation | 7D: 3D Cartesian + 4D quaternion | Paper, section 3.1 | | Pose rate | 20 Hz | Paper, C.4 | | Binaural training data | ~600 h / ~350k samples | Paper, Dataset | | FOA training data | ~900 h / ~310k samples | Paper, Dataset | | Test set size | 700 segments | Paper, Dataset | | RTF | 0.1587 | RTX 4090 only; paper, section 4.2 | | MOS-P / MOS-Q | 4.25 ± 0.16 / 4.09 ± 0.21 | Subjective eval; paper, section 4.5 | | Code / weights | Not stated | — | | Licence | Not stated | — | ## Performance claims and their conditions RTF = 0.1587 on a single NVIDIA RTX 4090. Batch size and streaming chunk size: not stated. The authors call this "well below unity." No other hardware is benchmarked. Spatial quality on the binaural test set: ANG COS = 62.11, DIS COS = 77.05. Computed with Spatial-AST embeddings. Each clip is partitioned into 1-second segments — Spatial-AST only produces position estimates for static sources — and segment-level cosine similarities are averaged. The metric resolves at 1-second granularity. Fast dynamic trajectories may be under- or overscored relative to perceptual quality. Audio quality on the same set: MRSTFT = 1.223, PESQ = 2.109, MCD = 2.153, Periodicity = 0.107. PESQ 2.109 is below the ITU-T "good" threshold of ≥ 3.0. The training corpus is predominantly speech, and the paper makes no wideband quality claim. MOS-P = 4.25 ± 0.16, MOS-Q = 4.09 ± 0.21. CSAVocoder scores highest MOS-P among compared models. Evaluator count, stimuli count and listener recruitment are not stated. Ablation results on binaural ANG COS and DIS COS (all figures from the paper, section 4.6): | Ablation | ANG COS | DIS COS | Δ ANG COS vs full | |---|---|---|---| | Full model | 62.11 | 77.05 | — | | w/o Mel Adaptor | 42.60 | 65.39 | −19.51 | | w/o Position Adaptor | 54.78 | 70.63 | −7.33 | | w/o SCD | 58.82 | 74.63 | −3.29 | | Mel Adaptor 2 heads | 61.03 | 76.55 | −1.08 | | Mel Adaptor 8 heads | 61.50 | 76.70 | −0.61 | The interpretation: the Mel Adaptor's cross-channel attention picks up spectral cues already present in the training BRIRs, and the Position Adaptor refines with explicit geometry. Nobody has independently verified these numbers.
Reported by 2608.25404 m· scroll →
maximum horizontal 10 m maximum height 2 m listener height 1.5 m minimum horizontal 1 m

Conditions around the comparison

scroll →
600 hours
binaural training data
48 kHz
sampling rate
0.1587
RTF on RTX 4090
62.11
ANG COS binaural
Figure: values as 2608.25404 reports them, drawn from quantities extracted and quote-verified for this post. Bars are proportional within their shared unit; condition cells are exact values and are not scaled against one another. The studio has not re-measured them.
## What it is and is not useful for **Useful for:** streaming TTS or generative audio pipelines where spatial rendering is the last stage. If upstream already emits multi-channel mel-spectrograms at 48 kHz, CSAVocoder drops in as the vocoder and handles HRTF rendering without a separate convolution reverb unit. Frame-by-frame processing works because each StreamingResBlock holds a fixed left-context buffer sized to its dilation — constant memory overhead. Binaural speech synthesis for head-tracked playback. The 7D pose input (position + orientation quaternion) at 20 Hz maps to what headphone motion APIs already expose — see the earlier note on [CMHeadphoneMotionManager capabilities](/blog/motionlink-headphone-motion-api). One caveat: 20 Hz is coarser than the 100 Hz device IMUs typically deliver, and the paper does not evaluate mismatched pose rates. FOA encoding from arbitrary mono speech sources with a spatial trajectory. The distribution is read-aloud English; music and non-speech are out of distribution. **Not useful for:** pipelines that don't already produce mel-spectrograms. It's a vocoder, not a synthesis model — no text encoder, diffusion backbone, or codec decoder. Edge or CPU deployment: RTF is measured only on an RTX 4090. Room geometries outside the training distribution: source-listener distances outside 1–10 m horizontal and height differences above 2 m were excluded during data construction. Outdoor or large reverberant spaces are not represented. High-fidelity music or wideband content: PESQ 2.109 is below typical speech quality thresholds, and the metric is speech-targeted. If you want to probe spatial rendering behaviour interactively before committing to an integration, the [Head-Tracked Stereo Pan Lab](/lab/head-tracked-stereo-pan) lets you drive azimuth and elevation directly. ## What the paper does not settle RTF is measured at unspecified batch and chunk size on one GPU class. Whether 0.1587 survives realistic streaming — small chunks, CPU-GPU round trips, concurrent sessions — is not established. Parameter count: not reported. Memory footprint comparison against lighter vocoders is impossible. MOS evaluator pool size is not stated; with CIs of ±0.16 to ±0.21, rankings between closely scored models may be within noise. --- *Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from a single source — the arXiv abstract for [2608.25404](http://arxiv.org/abs/2608.25404v1). The claims and quantities it worked from were extracted from that document by qwen/qwen3.8-27b. Before publication an automated gate re-checked every extracted claim against the source document (119 claim(s) and 76 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (9 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run 2608.25404's benchmarks; figures attributed to it are its own.* ## Questions this note answers ### Does this replace a convolution reverb or HRTF pipeline? It replaces the rendering step for trained distributions. The model absorbed BRIR/RIR convolution implicitly from training data. It does not expose a parameterisable room model — you cannot specify a novel geometry at inference time. For controlled acoustic simulation, a physics-based pipeline still applies. CSAVocoder's advantage is low-latency neural rendering of positions and rooms represented in training. ### What does RTF = 0.1587 actually mean for buffer latency? RTF = 0.1587 means roughly 6.3× faster than real-time on an RTX 4090 (1 / 0.1587 ≈ 6.3). Chunk size is not stated. A favourable RTF with a large chunk still imposes latency equal to the chunk duration regardless of throughput. Algorithmic delay is not reported and cannot be inferred from RTF alone. ### Is the 4-head attention choice in the Mel Adaptor meaningful? The ablation shows 4 heads (ANG COS 62.11) beats 2 heads (61.03) and 8 heads (61.50). Margins are small — 1.08 points for 2 heads, 0.61 for 8 — and both still substantially outperform removing the adaptor entirely (42.60). If you're targeting a lower-budget variant, 2 or 8 heads are viable substitutes with modest regression. ### Can this handle non-speech audio? The paper does not test non-speech content. The binaural training set is entirely speech (MRSSpeech, EasyCom); the FOA set is Spatial LibriSpeech from read-aloud English. PESQ is speech-targeted too. Behaviour on music, ambiences, or sound effects is undocumented — treat as out-of-distribution until tested independently.