Tencent’s WeChat AI team has published a detailed account of a model-scaling technique it calls Hidden Decoding, using it to build two large mixture-of-experts models: WeLM-HD4-80B and WeLM-HD4-617B. The work is not a consumer product launch. It is a technical argument that language models can perform more internal computation for each token without expanding the main Transformer backbone in the usual way.
The team’s official WeLM research blog says the 80B model has 3 billion activated parameters and the 617B model has 23 billion activated parameters. Both use an n=4 Hidden Decoding configuration. In the team’s tests, the Hidden Decoding versions exceeded their matched autoregressive baselines on nine shared benchmarks after the same early supervised fine-tuning recipe.
Training and serving a large model requires expensive computing resources. A method that improves reasoning or coding performance while keeping the core backbone unchanged could be useful if its performance gains hold up in broader testing and if its inference costs remain manageable.
Hidden Decoding adds internal computation through parallel streams
The central idea is to expand one input token into several internal streams. In the WeLM team’s description, Hidden Decoding does this without adding parameters to the main Transformer backbone. The system uses several representations of a token inside a single forward pass. Only the final stream is directly trained to predict the next token, while earlier streams act as internal computational states.
The team says this creates more effective computation per token. Rather than asking a model to predict one representation immediately, the approach lets it carry intermediate representations through the network before producing the next-token output. The claimed benefit is especially relevant to problems that require multi-step reasoning, code generation, or detailed factual comparison.
The research post compares WeLM-HD4-80B with a matched WeLM-80B baseline and WeLM-HD4-617B with a matched WeLM-617B baseline. The 80B base activates 3 billion parameters and the 617B base activates 23 billion. Those activated-parameter figures are different from total model size, an important distinction in mixture-of-experts architectures. Such models can contain a large overall parameter pool while using only part of it for each token.
TechNode reported the work as a new scaling approach from Tencent’s WeChat AI team and summarized the two models’ tested results. The publication should still be read as company research. The benchmark comparisons are from the WeLM team’s own evaluation, not an independent ranking. That does not make the results irrelevant, but it means they should not be treated as a final verdict on global model leadership.
Tencent has already made model availability and application integration central to its AI strategy. EastFrontier covered Tencent’s decision to take Hy3 global with a free enterprise offer. WeLM’s technical paper offers a different view of the company’s AI work: it focuses less on distribution and more on the architecture and systems work needed to improve model capability.
The 617B model is a test of scaling without a new backbone
WeLM’s 617B model is the paper’s largest example. According to the official post, the team compared the Hidden Decoding model with its matched autoregressive base under the same early supervised fine-tuning conditions and without reinforcement learning in that comparison. The team says the Hidden Decoding versions outperformed their matched baselines across all nine shared benchmarks.
The reported gains covered benchmarks including GPQA Diamond, HLE, MMMLU, FrontierMath, PHYBench, MathArena Apex, HMMT, IMO-AnswerBench, and SciCode. The blog presents results for the 617B model that are higher than its matched base on each listed shared test. It also includes an external comparison column for Kimi K2.6, but the most controlled comparison in the paper is between each WeLM model and its own baseline.
That distinction is the one that deserves most attention. External benchmark tables are influenced by data choices, prompts, evaluation settings, and reporting methods. A matched model comparison can more directly test whether a specific architectural change helps, because the authors hold more variables constant. Even then, the findings need replication before they can establish a general rule for large-model design.
The WeLM post says that the method’s gains are not free. Training the 80B Hidden Decoding configuration took 5.1 times the single-batch cost of the baseline, while the 617B configuration took 4.4 times. The authors argue that this is close to linear scaling for an n=4 method and much lower than the quadratic attention cost that could arise if every stream attended densely to every other stream.
That cost discussion is not a minor footnote. Chinese AI companies are increasingly trying to balance capability with access to compute. A system that improves tests but multiplies training and serving requirements too aggressively may be difficult to deploy. Tencent’s paper positions Stream-Factorized Attention as the engineering mechanism intended to keep the method practical at large scale.
Stream-Factorized Attention is the paper’s key efficiency claim
The WeLM team’s proposed solution is Stream-Factorized Attention. Most layers process information within the same stream, while a smaller number of layers mix information across streams. The goal is to avoid treating the expanded sequence as one fully dense attention problem, which the paper says would drive costs toward a quadratic increase.
The authors report that the method allowed them to train the larger configurations with costs nearer to linear growth. They also describe a WeLM-specific KV-mirror design intended to reduce work in later layers. These engineering details matter because model research is increasingly inseparable from systems design. A new training method has little commercial significance if it cannot fit within the memory, throughput, and hardware limits of a real serving environment.
The paper’s results also include model self-evaluations against external systems, but those should be treated cautiously. The official post uses Kimi K2.6 and Qwen models as reference points in some tables. The useful conclusion is not that WeLM has definitively surpassed every competitor. It is that Tencent is showing a concrete alternative to simply enlarging a conventional backbone: add structured internal computation and control the cost through attention design.
This direction aligns with the broader Chinese emphasis on squeezing more work from limited computing resources. EastFrontier’s coverage of Tencent’s open-source Hy-MT2 translation models highlighted a separate effort to make models useful in constrained deployment settings. Hidden Decoding addresses a different scale, but the strategic question is similar: how can a company improve AI capability without relying only on ever-larger hardware budgets?
The next evidence will need to come outside Tencent’s own paper. Developers will want to know whether the models are released, how they behave under real workloads, what hardware they require, and whether the benchmark improvements translate into better agent, coding, or WeChat-specific applications. For now, WeLM-HD4-80B and WeLM-HD4-617B demonstrate that Tencent is investing in a serious architectural experiment. The experiment’s value will depend on whether its internal-computation approach produces a practical advantage when models leave the benchmark table.