# AutoSubs star ratings hide different training AutoSubs' model table uses one star scale across models with different corpora, quantization levels, and language sets. Canonical page: https://makerportal.ai/blog/on-device-ai-github-614149835 Author: Joshua Hrisko, Principal Engineer — MakerPortal Published: 2026-09-06 Section: Comparison / On-Device AI · 8 min read Tags: on-device-ai --- The models in AutoSubs' model table were trained on different corpora, for different language sets, with different architectures and quantization levels, yet they share a single star-rating scale. The only performance figure the README supplies—a 58–110% WER for Whisper large-v3 on Central Asian languages—is a number that exists specifically to justify one model's presence in the table, not to establish a common benchmark across all of them. ## What AutoSubs is AutoSubs is a subtitle generator that runs on the machine doing the editing. You point it at a timeline or a file, it transcribes the speech, and the result goes back where you need it: subtitles imported as caption tracks in Premiere Pro, SRT entries created as text layers in After Effects, straight into a DaVinci Resolve timeline, or exported as SRT from the standalone app. The README's line for all of this is "no cloud, no subscription, no data leaving your machine." That is also why the table below exists at all. Running transcription locally means the model is something you choose and download — from an in-app Model Manager — rather than something a vendor picks for you behind an API. The choice is part of the product, and this table is how AutoSubs presents that choice to the person making it. ## The figures AutoSubs presents its transcription options in a table with columns for model size, RAM, language coverage, and a star rating (1–4). No external benchmark is cited for any model. The star ratings are AutoSubs' own qualitative assessment, not a measured score on a shared test set. | Model | Size (AutoSubs reports) | RAM (AutoSubs reports) | Quantization | Languages (AutoSubs reports) | Star rating (AutoSubs' own) | Architecture / notes | |---|---|---|---|---|---|---| | Whisper tiny / tiny.en | 80 MB | 1 GB | not stated | Multilingual / English | ★ | via whisper-rs | | Whisper base / base.en | 150 MB | 1 GB | not stated | Multilingual / English | ★ | via whisper-rs | | Whisper small / small.en | 480 MB | 2 GB | not stated | Multilingual / English | ★★ | via whisper-rs | | Whisper medium / medium.en | 1.5 GB | 5 GB | not stated | Multilingual / English | ★★★ | via whisper-rs | | Whisper large-v3-turbo | 1.6 GB | 6 GB | not stated | Multilingual | ★★★ | via whisper-rs | | Whisper large-v3 | 3.1 GB | 10 GB | not stated | Multilingual | ★★★★ | via whisper-rs | | Moonshine tiny | 60 MB | 1 GB | quantized | English | ★ | via ONNX Runtime | | Moonshine tiny-ar/zh/ja/ko/uk/vi | 120 MB each | 1 GB | float-precision | one language each | ★★ or ★★★ | via ONNX Runtime | | Moonshine base | 200 MB | 1 GB | float-precision | English | ★★ | via ONNX Runtime | | Parakeet-TDT-0.6B-v3 | 700 MB | 2 GB | int8 ONNX | 25 EU + RU + UK | ★★★★ | 0.6B params | | SenseVoice | 230 MB | 1 GB | int8 ONNX | zh, en, ja, ko, Cantonese | ★★★ | compact, CJK-focused | | Canary-1B-v2 | 1 GB | 3 GB | int8 ONNX | 25 EU + RU + UK | ★★★★ | 1B params, encoder-decoder, native translation | | Cohere Transcribe | 2 GB | 4 GB | int4 ONNX | 14 languages | ★★★★ | "highest-accuracy option" per AutoSubs | | GigaAM v3 | 225 MB | 2 GB | int8 ONNX | Russian, English | ★★★★ | Conformer, 700k hours Russian speech training | | GigaAM Multilingual | 592 MB | 3 GB | int8 ONNX | RU, KK, KG, UZ, EN | ★★★★ | 600M params, character-wise CTC head, no punctuation output | The only measured performance figure in the entire README is the one attached to GigaAM Multilingual's section: AutoSubs states that "Whisper large v3 scores 58–110% WER on Kazakh, Kyrgyz and Uzbek." This is presented as motivation for GigaAM Multilingual's existence. No WER is given for GigaAM Multilingual on those same languages. ## Where the comparison holds Model size and RAM are directly comparable. They are file sizes and memory footprints. A 60 MB Moonshine tiny file is 60 MB regardless of what it was trained on. If your constraint is disk space or available RAM on a target machine, the table answers the question. Language coverage is comparable in the sense that you can check whether your target language is listed. Parakeet covers 25 European languages plus Russian and Ukrainian. Cohere covers a specific set of 14. GigaAM Multilingual covers five. These are categorical facts about what each model can be asked to transcribe.
Reported by tmoroney/auto-subs GB· scroll →
Whisper large-v3 RAM 10 GB Whisper large-v3-turbo RAM 6 GB Whisper medium RAM 5 GB Cohere RAM 4 GB Whisper large-v3 model size 3.1 GB

Conditions around the comparison

scroll →
80 MB
model size for Whisper tiny
150 MB
model size for Whisper base
480 MB
model size for Whisper small
1.5 GB
model size for Whisper medium
Figure: values as tmoroney/auto-subs reports them, drawn from quantities extracted and quote-verified for this post. Bars are proportional within their shared unit; condition cells are exact values and are not scaled against one another. The studio has not re-measured them.
## Where it stops holding The star ratings are not a shared scale. AutoSubs gives Whisper tiny one star and Parakeet four stars, but these ratings were not produced by running both models on the same audio corpus and comparing WER. The README does not state what criterion the stars represent, what test set was used, or what the baseline is for each star level. A four-star rating for Cohere on its 14 languages and a four-star rating for GigaAM v3 on Russian and English are not the same claim. They are AutoSubs' editorial judgment applied under different conditions. The quantization levels make a direct speed or accuracy comparison between rows invalid without further information. Cohere is int4, GigaAM v3 is int8, Moonshine tiny is quantized (method not stated), and the Whisper models' quantization is not stated at all. int4 vs int8 is not a small difference in accuracy or latency, and the README does not provide a quantization-matched comparison between any two models. The training data differences are the largest gap. GigaAM v3 was trained on 700k hours of Russian speech. AutoSubs does not state the training data size for Whisper, Parakeet, Canary, Cohere, SenseVoice, or Moonshine. A model trained on 700k hours of a single language and a model trained on an unstated multilingual corpus are not competing on the same axis. The 58–110% WER figure for Whisper on Central Asian languages is a data point about Whisper's out-of-domain performance on languages it was not specifically trained for. It does not tell you how GigaAM Multilingual performs on those languages, because that number is absent from the README. This is structurally the same problem as [gpt-oss-20b scoring between 0% and 87% depending on the harness](/blog/itria-hackernews-49523381): the number you get is a property of the model *and* the evaluation setup, and presenting numbers from different setups in one table without flagging the mismatch implies a comparability that does not exist. ## What cannot be concluded You cannot rank these models by accuracy from this table. The star ratings are not derived from a shared benchmark. The single WER figure is a negative result (Whisper fails on Central Asian languages) used to motivate one model, not a positive result for that model. No model's WER on its home language is stated. No latency numbers are given. No A/B comparison between any two models on the same audio is provided. ## Who this is for The person this lands on is whoever is standing in front of the Model Manager with a job to deliver. The table is the only thing they have to choose from, and two of its three decision columns are facts while the third is not. Use RAM. It is a memory footprint, it is comparable across every row, and on a laptop it is the constraint that decides whether a model runs at all. Use the language list the same way: check that the language you are captioning is listed, and treat a model that does not list it as unavailable rather than untested. Do not rank by stars. A four-star rating on Cohere's 14 languages and a four-star rating on GigaAM v3's Russian and English are not the same claim, and the README does not state what either one measures. The substitute costs two minutes: caption the same clip with the largest model your RAM allows and with the smallest one you would be willing to ship, and read the difference yourself. That is a worse instrument than a published WER table and a better one than a star. --- *Method: this note was drafted by qwen/qwen3.8-27b from a single source — the published README of [tmoroney/auto-subs](https://github.com/tmoroney/auto-subs). Before publication an automated gate re-checked every extracted claim against the source document (25 claim(s) and 50 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (0 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run tmoroney/auto-subs's benchmarks; figures attributed to it are its own.* ## Questions this note answers ### Can I use the star ratings to pick a model for my language? No. The stars are AutoSubs' qualitative assessment and the README does not state the evaluation protocol behind them. Use the language list to confirm your language is covered, then treat accuracy as unknown until you test on your own audio. ### Why is GigaAM v3 only 225 MB but uses 2 GB of RAM? The README lists both figures but does not explain the discrepancy. The model is int8 ONNX, so the 225 MB is the on-disk weight size. The 2 GB RAM figure likely includes runtime allocations (buffers, KV cache, intermediate activations) that the README does not break down. The same pattern appears across all rows: RAM is 2–10× the file size depending on the model. ### Does the "no data leaving the machine" claim cover the model downloads? The README states the application is "local-first" with "no cloud, no subscription, no data leaving your machine." It also states that models are downloaded (from a Model Manager) and that the speaker diarization model (~40 MB) and Silero VAD model are downloaded separately. The download step is a one-time fetch of model weights; whether that is consistent with "no data leaving the machine" depends on your definition, because the audio you transcribe is processed locally after the weights are in place. ### Is the MMS forced-alignment option a commercial-use blocker? Yes, if your use is commercial. AutoSubs states the MMS weights are licensed under CC BY-NC 4.0 for noncommercial use. The transcription models themselves are not described with a commercial restriction in the README, but the alignment feature specifically is limited.