# Hy4 preview: 770B MoE, open weights, gaps Tencent's Hy4 ships Apache 2.0 weights with 49B activated parameters and 1M-token context, but four unexplained residual streams block anything beyond… Canonical page: https://makerportal.ai/blog/field-note-hf-tencent-hy4-preview Author: Joshua Hrisko, Principal Engineer — MakerPortal Published: 2026-08-28 Section: Readout / Trend-grounded · 6 min read Tags: readout --- Tencent's Hy Team has released Hy4 preview — a 770B-parameter MoE text model — as Apache 2.0 weights on Hugging Face, deployable today through vLLM and SGLang. The model card lists four residual streams per layer: not the single stream that standard transformer implementations and inference kernels assume, and a structural detail the card provides without any explanation. ## What it is A text-generation instruct model available in two variants: a full-precision release and an FP8 quantised version (Hy4 preview-FP8). The licence is Apache 2.0 with no additional use restrictions stated in the model card. No predecessor model is named as replaced; no cloud endpoint or API is listed. The card labels this a "preview" without defining what that implies for stability or long-term availability. | Property | Value | Condition / source | |---|---|---| | Total parameters | 770B | Tencent model card; backbone only | | Activated per token | 49B | Tencent model card | | MTP layer | 10B total, 0.7B activated | Tencent model card; excluded from backbone parameter count | | Total layers | 78 | Tencent model card | | Dense FFN layers | 1 (first layer only) | Tencent model card | | MoE layers | 77 | Tencent model card | | Routed experts per MoE layer | 256 | Tencent model card | | Shared experts per MoE layer | 1 | Tencent model card | | Experts activated per token | top-8 routed + 1 shared | Tencent model card | | Attention type | Gated DSA | Tencent model card | | Hidden size | 6,144 | Tencent model card | | Attention heads | 64 | Tencent model card | | Query compression dimension | 2,048 | Tencent model card | | KV compression dimension | 512 | Tencent model card | | Indexer heads / head dimension | 32 / 128 | Tencent model card | | Indexer top-k | 2,048 | Tencent model card | | Residual streams | 4 | Tencent model card | | MoE intermediate size | 2,048 | Tencent model card | | Dense FFN intermediate size (layer 1) | 18,432 | Tencent model card | | Context length | 1M tokens | Tencent model card; quality at length not stated | | Vocabulary size | 120,832 | Tencent model card | | Licence | Apache 2.0 | Tencent model card | | FP8 variant | Available (Hy4 preview-FP8) | Tencent model card | | Reference deployment | TP=8; GPU type not stated | vLLM and SGLang sections, Tencent model card | ## Evaluation figures and their conditions The model card's entire quantitative case is one internal study. The Tencent model card reports that 163 Tencent employees rated outputs on 203 engineering tasks in a blind side-by-side format. No task list, scoring rubric, or rater instructions are published. No external benchmark numbers appear. Against GLM 5.3, the Tencent model card reports: - Average scores: Hy4 preview 2.99 vs. GLM 5.3 2.92 - 46.8% wins / 12.8% ties / 40.4% losses; net win rate: 46.8 − 40.4 = 6.4 pp Against Kimi K3, the Tencent model card reports: - Average scores: Hy4 preview 2.99 vs. Kimi K3 2.94 - 51.2% wins / 7.9% ties / 40.9% losses; net win rate: 51.2 − 40.9 = 10.3 pp Both comparisons use raters employed by the team releasing the model, on tasks that team selected. The card describes both results as "slightly ahead." The model card recommends temperature=0.9, top_p=1.0 for inference. The reasoning mode defaults to "high" (deep chain-of-thought), which the card states suits math, coding, and reasoning.

Reported by tencent/Hy4-preview B· scroll →

Conditions around the comparison

scroll →

770B parameters
Total parameters: 78 layers
Backbone layers: 77 layers
MoE layers: 256 routed experts
Number of MoE layer

Figure: values as tencent/Hy4-preview reports them, drawn from quantities extracted and quote-verified for this post. Bars are proportional within their shared unit; condition cells are exact values and are not scaled against one another. The studio has not re-measured them.

## Actual utility, by workload The operative compute-per-token figure is 49B activated parameters. At any given token, 770 − 49 = 721B parameters are dormant experts. Per-forward-pass compute is in the range of a dense 49B model; routing overhead across 256 experts with top-8 selection adds latency the card does not quantify. The Gated DSA attention mechanism compresses KV representations to dimension 512 against a hidden size of 6,144, and uses an indexer selecting from a top-k of 2,048 positions. That design is architecturally consistent with managing memory at very long contexts. The model card does not report retrieval accuracy, perplexity degradation, or throughput at any specific context length; the 1M figure is an architectural specification, not a verified quality claim. The native MTP layer (0.7B activated from a 10B block) enables speculative decoding without a separate draft model. The vLLM deployment example specifies 3 speculative tokens via the MTP method; the SGLang example specifies 3 speculative steps and 4 draft tokens via the NEXTN algorithm. Neither section reports acceptance rate or throughput uplift for any hardware configuration or request distribution. **Try it if:** you need an Apache 2.0 model at this weight class for long-context coding, mathematical reasoning, or structured analysis, and you have TP=8 or more across high-memory GPUs already provisioned. The FP8 variant and native speculative decoding reduce practical infrastructure burden compared to a naive full-precision deployment. **Hold off if:** you need third-party benchmark coverage before deploying, your inference budget does not reach TP=8, or your task distribution falls outside the code/math/reasoning cluster the model card targets. The 203-task evaluation is not externally reproducible. ## What the card leaves open The four residual streams are listed in the specification table and never explained. A standard transformer accumulates all layer outputs into one residual stream per layer; four streams require a fundamentally different computation graph — branching paths, some form of ensemble residual, or something else the card does not describe. vLLM and SGLang inference runs through the documented flags. Any work that touches model internals — fine-tuning, continued pre-training, custom kernel development, or gradient analysis — requires understanding how those four streams interact before an implementation can be correct. No technical report is linked. For context on how deployment configuration flags can mask what a model actually does at runtime, see the [vLLM serving conditions note](/blog/on-device-ai-github-1116516142). --- *Method: this note was drafted by us.anthropic.claude-sonnet-4-6 from a single source — the model card for [tencent/Hy4-preview](https://huggingface.co/tencent/Hy4-preview). Before publication an automated gate re-checked every extracted claim against the source document (31 claim(s) and 36 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (4 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run tencent/Hy4-preview's benchmarks; figures attributed to it are its own.* ## Questions this note answers ### Does Apache 2.0 mean commercial use is unrestricted? The Tencent model card states "Apache License 2.0" with no additional clauses listed. Apache 2.0 permits commercial use with standard attribution and patent notice requirements. Organisations with compliance constraints on Tencent-originated assets should verify separately; the licence text itself is permissive, but jurisdiction and internal policy are distinct from licence type. ### Is TP=8 a hard floor or just the documented example? It is the only configuration the card provides for both vLLM and SGLang, with no discussion of smaller tensor-parallel sizes. The backbone is 770B parameters; the GPU type is not stated for the TP=8 examples. Engineers targeting a smaller GPU footprint will need to characterise support and throughput independently. ### The win margins are slim — what does that actually establish? A 6.4 pp net win rate over GLM 5.3 and a 10.3 pp net win rate over Kimi K3, from 163 Tencent raters on 203 Tencent-selected tasks, establishes that the model is not visibly weaker than those two on that task distribution. It does not establish superiority on external benchmarks, does not account for rater familiarity bias, and cannot be reproduced from the information provided. Treat it as a weak lower bound on quality, not a ranking. ### Does the MTP layer activate automatically, or does it need explicit configuration? It requires explicit flags in both frameworks. The vLLM example passes `--speculative-config` with `method: mtp` and 3 speculative tokens; the SGLang example passes `--speculative-algorithm NEXTN` with 3 steps and 4 draft tokens. Without those flags, the MTP layer is not engaged. The model card reports no acceptance rate or throughput benefit for any workload or hardware configuration.