Hy4 preview: 770B MoE, open weights, gaps

Tencent’s Hy Team has released Hy4 preview — a 770B-parameter MoE text model — as Apache 2.0 weights on Hugging Face, deployable today through vLLM and SGLang. The model card lists four residual streams per layer: not the single stream that standard transformer implementations and inference kernels assume, and a structural detail the card provides without any explanation.

What it is

A text-generation instruct model available in two variants: a full-precision release and an FP8 quantised version (Hy4 preview-FP8). The licence is Apache 2.0 with no additional use restrictions stated in the model card. No predecessor model is named as replaced; no cloud endpoint or API is listed. The card labels this a “preview” without defining what that implies for stability or long-term availability.

Property	Value	Condition / source
Total parameters	770B	Tencent model card; backbone only
Activated per token	49B	Tencent model card
MTP layer	10B total, 0.7B activated	Tencent model card; excluded from backbone parameter count
Total layers	78	Tencent model card
Dense FFN layers	1 (first layer only)	Tencent model card
MoE layers	77	Tencent model card
Routed experts per MoE layer	256	Tencent model card
Shared experts per MoE layer	1	Tencent model card
Experts activated per token	top-8 routed + 1 shared	Tencent model card
Attention type	Gated DSA	Tencent model card
Hidden size	6,144	Tencent model card
Attention heads	64	Tencent model card
Query compression dimension	2,048	Tencent model card
KV compression dimension	512	Tencent model card
Indexer heads / head dimension	32 / 128	Tencent model card
Indexer top-k	2,048	Tencent model card
Residual streams	4	Tencent model card
MoE intermediate size	2,048	Tencent model card
Dense FFN intermediate size (layer 1)	18,432	Tencent model card
Context length	1M tokens	Tencent model card; quality at length not stated
Vocabulary size	120,832	Tencent model card
Licence	Apache 2.0	Tencent model card
FP8 variant	Available (Hy4 preview-FP8)	Tencent model card
Reference deployment	TP=8; GPU type not stated	vLLM and SGLang sections, Tencent model card

Evaluation figures and their conditions

The model card’s entire quantitative case is one internal study. The Tencent model card reports that 163 Tencent employees rated outputs on 203 engineering tasks in a blind side-by-side format. No task list, scoring rubric, or rater instructions are published. No external benchmark numbers appear.

Against GLM 5.3, the Tencent model card reports:

• Average scores: Hy4 preview 2.99 vs. GLM 5.3 2.92
• 46.8% wins / 12.8% ties / 40.4% losses; net win rate: 46.8 − 40.4 = 6.4 pp

Against Kimi K3, the Tencent model card reports:

• Average scores: Hy4 preview 2.99 vs. Kimi K3 2.94
• 51.2% wins / 7.9% ties / 40.9% losses; net win rate: 51.2 − 40.9 = 10.3 pp

Both comparisons use raters employed by the team releasing the model, on tasks that team selected. The card describes both results as “slightly ahead.”

The model card recommends temperature=0.9, top_p=1.0 for inference. The reasoning mode defaults to “high” (deep chain-of-thought), which the card states suits math, coding, and reasoning.

Reported by tencent/Hy4-preview B· scroll →

Conditions around the comparison

scroll →

770B parameters

Total parameters

78 layers

Backbone layers

77 layers

MoE layers

256 routed experts

Number of MoE layer

Actual utility, by workload

The operative compute-per-token figure is 49B activated parameters. At any given token, 770 − 49 = 721B parameters are dormant experts. Per-forward-pass compute is in the range of a dense 49B model; routing overhead across 256 experts with top-8 selection adds latency the card does not quantify.

The Gated DSA attention mechanism compresses KV representations to dimension 512 against a hidden size of 6,144, and uses an indexer selecting from a top-k of 2,048 positions. That design is architecturally consistent with managing memory at very long contexts. The model card does not report retrieval accuracy, perplexity degradation, or throughput at any specific context length; the 1M figure is an architectural specification, not a verified quality claim.

The native MTP layer (0.7B activated from a 10B block) enables speculative decoding without a separate draft model. The vLLM deployment example specifies 3 speculative tokens via the MTP method; the SGLang example specifies 3 speculative steps and 4 draft tokens via the NEXTN algorithm. Neither section reports acceptance rate or throughput uplift for any hardware configuration or request distribution.

Try it if: you need an Apache 2.0 model at this weight class for long-context coding, mathematical reasoning, or structured analysis, and you have TP=8 or more across high-memory GPUs already provisioned. The FP8 variant and native speculative decoding reduce practical infrastructure burden compared to a naive full-precision deployment.

Hold off if: you need third-party benchmark coverage before deploying, your inference budget does not reach TP=8, or your task distribution falls outside the code/math/reasoning cluster the model card targets. The 203-task evaluation is not externally reproducible.

What the card leaves open

The four residual streams are listed in the specification table and never explained. A standard transformer accumulates all layer outputs into one residual stream per layer; four streams require a fundamentally different computation graph — branching paths, some form of ensemble residual, or something else the card does not describe. vLLM and SGLang inference runs through the documented flags. Any work that touches model internals — fine-tuning, continued pre-training, custom kernel development, or gradient analysis — requires understanding how those four streams interact before an implementation can be correct. No technical report is linked.

For context on how deployment configuration flags can mask what a model actually does at runtime, see the vLLM serving conditions note.

---

Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from a single source — the model card for tencent/Hy4-preview. Before publication an automated gate re-checked every extracted claim against the source document (31 claim(s) and 36 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (4 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run tencent/Hy4-preview’s benchmarks; figures attributed to it are its own.

FAQ

Does Apache 2.0 mean commercial use is unrestricted?

The Tencent model card states "Apache License 2.0" with no additional clauses listed. Apache 2.0 permits commercial use with standard attribution and patent notice requirements. Organisations with compliance constraints on Tencent-originated assets should verify separately; the licence text itself is permissive, but jurisdiction and internal policy are distinct from licence type.

Is TP=8 a hard floor or just the documented example?

It is the only configuration the card provides for both vLLM and SGLang, with no discussion of smaller tensor-parallel sizes. The backbone is 770B parameters; the GPU type is not stated for the TP=8 examples. Engineers targeting a smaller GPU footprint will need to characterise support and throughput independently.

The win margins are slim — what does that actually establish?

A 6.4 pp net win rate over GLM 5.3 and a 10.3 pp net win rate over Kimi K3, from 163 Tencent raters on 203 Tencent-selected tasks, establishes that the model is not visibly weaker than those two on that task distribution. It does not establish superiority on external benchmarks, does not account for rater familiarity bias, and cannot be reproduced from the information provided. Treat it as a weak lower bound on quality, not a ranking.

Does the MTP layer activate automatically, or does it need explicit configuration?

It requires explicit flags in both frameworks. The vLLM example passes --speculative-config with method: mtp and 3 speculative tokens; the SGLang example passes --speculative-algorithm NEXTN with 3 steps and 4 draft tokens. Without those flags, the MTP layer is not engaged. The model card reports no acceptance rate or throughput benefit for any workload or hardware configuration.