AutoSubs star ratings hide different training

The models in AutoSubs’ model table were trained on different corpora, for different language sets, with different architectures and quantization levels, yet they share a single star-rating scale. The only performance figure the README supplies—a 58–110% WER for Whisper large-v3 on Central Asian languages—is a number that exists specifically to justify one model’s presence in the table, not to establish a common benchmark across all of them.

What AutoSubs is

AutoSubs is a subtitle generator that runs on the machine doing the editing. You point it at a timeline or a file, it transcribes the speech, and the result goes back where you need it: subtitles imported as caption tracks in Premiere Pro, SRT entries created as text layers in After Effects, straight into a DaVinci Resolve timeline, or exported as SRT from the standalone app. The README’s line for all of this is “no cloud, no subscription, no data leaving your machine.”

That is also why the table below exists at all. Running transcription locally means the model is something you choose and download — from an in-app Model Manager — rather than something a vendor picks for you behind an API. The choice is part of the product, and this table is how AutoSubs presents that choice to the person making it.

The figures

AutoSubs presents its transcription options in a table with columns for model size, RAM, language coverage, and a star rating (1–4). No external benchmark is cited for any model. The star ratings are AutoSubs’ own qualitative assessment, not a measured score on a shared test set.

Model	Size (AutoSubs reports)	RAM (AutoSubs reports)	Quantization	Languages (AutoSubs reports)	Star rating (AutoSubs’ own)	Architecture / notes
Whisper tiny / tiny.en	80 MB	1 GB	not stated	Multilingual / English	★	via whisper-rs
Whisper base / base.en	150 MB	1 GB	not stated	Multilingual / English	★	via whisper-rs
Whisper small / small.en	480 MB	2 GB	not stated	Multilingual / English	★★	via whisper-rs
Whisper medium / medium.en	1.5 GB	5 GB	not stated	Multilingual / English	★★★	via whisper-rs
Whisper large-v3-turbo	1.6 GB	6 GB	not stated	Multilingual	★★★	via whisper-rs
Whisper large-v3	3.1 GB	10 GB	not stated	Multilingual	★★★★	via whisper-rs
Moonshine tiny	60 MB	1 GB	quantized	English	★	via ONNX Runtime
Moonshine tiny-ar/zh/ja/ko/uk/vi	120 MB each	1 GB	float-precision	one language each	★★ or ★★★	via ONNX Runtime
Moonshine base	200 MB	1 GB	float-precision	English	★★	via ONNX Runtime
Parakeet-TDT-0.6B-v3	700 MB	2 GB	int8 ONNX	25 EU + RU + UK	★★★★	0.6B params
SenseVoice	230 MB	1 GB	int8 ONNX	zh, en, ja, ko, Cantonese	★★★	compact, CJK-focused
Canary-1B-v2	1 GB	3 GB	int8 ONNX	25 EU + RU + UK	★★★★	1B params, encoder-decoder, native translation
Cohere Transcribe	2 GB	4 GB	int4 ONNX	14 languages	★★★★	“highest-accuracy option” per AutoSubs
GigaAM v3	225 MB	2 GB	int8 ONNX	Russian, English	★★★★	Conformer, 700k hours Russian speech training
GigaAM Multilingual	592 MB	3 GB	int8 ONNX	RU, KK, KG, UZ, EN	★★★★	600M params, character-wise CTC head, no punctuation output

The only measured performance figure in the entire README is the one attached to GigaAM Multilingual’s section: AutoSubs states that “Whisper large v3 scores 58–110% WER on Kazakh, Kyrgyz and Uzbek.” This is presented as motivation for GigaAM Multilingual’s existence. No WER is given for GigaAM Multilingual on those same languages.

Where the comparison holds

Model size and RAM are directly comparable. They are file sizes and memory footprints. A 60 MB Moonshine tiny file is 60 MB regardless of what it was trained on. If your constraint is disk space or available RAM on a target machine, the table answers the question.

Language coverage is comparable in the sense that you can check whether your target language is listed. Parakeet covers 25 European languages plus Russian and Ukrainian. Cohere covers a specific set of 14. GigaAM Multilingual covers five. These are categorical facts about what each model can be asked to transcribe.

Reported by tmoroney/auto-subs GB· scroll →

Conditions around the comparison

scroll →

80 MB

model size for Whisper tiny

150 MB

model size for Whisper base

480 MB

model size for Whisper small

1.5 GB

model size for Whisper medium

Where it stops holding

The star ratings are not a shared scale. AutoSubs gives Whisper tiny one star and Parakeet four stars, but these ratings were not produced by running both models on the same audio corpus and comparing WER. The README does not state what criterion the stars represent, what test set was used, or what the baseline is for each star level. A four-star rating for Cohere on its 14 languages and a four-star rating for GigaAM v3 on Russian and English are not the same claim. They are AutoSubs’ editorial judgment applied under different conditions.

The quantization levels make a direct speed or accuracy comparison between rows invalid without further information. Cohere is int4, GigaAM v3 is int8, Moonshine tiny is quantized (method not stated), and the Whisper models’ quantization is not stated at all. int4 vs int8 is not a small difference in accuracy or latency, and the README does not provide a quantization-matched comparison between any two models.

The training data differences are the largest gap. GigaAM v3 was trained on 700k hours of Russian speech. AutoSubs does not state the training data size for Whisper, Parakeet, Canary, Cohere, SenseVoice, or Moonshine. A model trained on 700k hours of a single language and a model trained on an unstated multilingual corpus are not competing on the same axis. The 58–110% WER figure for Whisper on Central Asian languages is a data point about Whisper’s out-of-domain performance on languages it was not specifically trained for. It does not tell you how GigaAM Multilingual performs on those languages, because that number is absent from the README.

This is structurally the same problem as gpt-oss-20b scoring between 0% and 87% depending on the harness: the number you get is a property of the model and the evaluation setup, and presenting numbers from different setups in one table without flagging the mismatch implies a comparability that does not exist.

What cannot be concluded

You cannot rank these models by accuracy from this table. The star ratings are not derived from a shared benchmark. The single WER figure is a negative result (Whisper fails on Central Asian languages) used to motivate one model, not a positive result for that model. No model’s WER on its home language is stated. No latency numbers are given. No A/B comparison between any two models on the same audio is provided.

Who this is for

The person this lands on is whoever is standing in front of the Model Manager with a job to deliver. The table is the only thing they have to choose from, and two of its three decision columns are facts while the third is not.

Use RAM. It is a memory footprint, it is comparable across every row, and on a laptop it is the constraint that decides whether a model runs at all. Use the language list the same way: check that the language you are captioning is listed, and treat a model that does not list it as unavailable rather than untested.

Do not rank by stars. A four-star rating on Cohere’s 14 languages and a four-star rating on GigaAM v3’s Russian and English are not the same claim, and the README does not state what either one measures. The substitute costs two minutes: caption the same clip with the largest model your RAM allows and with the smallest one you would be willing to ship, and read the difference yourself. That is a worse instrument than a published WER table and a better one than a star.

---

Method: this note was drafted by qwen/qwen3.8-27b from a single source — the published README of tmoroney/auto-subs. Before publication an automated gate re-checked every extracted claim against the source document (25 claim(s) and 50 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (0 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run tmoroney/auto-subs’s benchmarks; figures attributed to it are its own.

FAQ

Can I use the star ratings to pick a model for my language?

No. The stars are AutoSubs' qualitative assessment and the README does not state the evaluation protocol behind them. Use the language list to confirm your language is covered, then treat accuracy as unknown until you test on your own audio.

Why is GigaAM v3 only 225 MB but uses 2 GB of RAM?

The README lists both figures but does not explain the discrepancy. The model is int8 ONNX, so the 225 MB is the on-disk weight size. The 2 GB RAM figure likely includes runtime allocations (buffers, KV cache, intermediate activations) that the README does not break down. The same pattern appears across all rows: RAM is 2–10× the file size depending on the model.

Does the "no data leaving the machine" claim cover the model downloads?

The README states the application is "local-first" with "no cloud, no subscription, no data leaving your machine." It also states that models are downloaded (from a Model Manager) and that the speaker diarization model (~40 MB) and Silero VAD model are downloaded separately. The download step is a one-time fetch of model weights; whether that is consistent with "no data leaving the machine" depends on your definition, because the audio you transcribe is processed locally after the weights are in place.

Is the MMS forced-alignment option a commercial-use blocker?

Yes, if your use is commercial. AutoSubs states the MMS weights are licensed under CC BY-NC 4.0 for noncommercial use. The transcription models themselves are not described with a commercial restriction in the README, but the alignment feature specifically is limited.