China’s AI industry is confronting a less visible constraint than model training: the machinery needed to serve models at scale after they have been built. South China Morning Post reports that companies are redesigning software to manage a sharp rise in inference demand while their access to high-end Nvidia processors remains limited. The dividing line is not simply whether a Chinese accelerator can run an AI workload. It is whether it can serve the higher-value work for which customers will pay, especially difficult coding requests.
That distinction matters because inference is where an already-trained model processes a prompt and produces a response. It is the recurrent cost of a chatbot, coding tool, search assistant, or agent. Citing the National Data Administration, SCMP put China’s daily token requests above 140 trillion in March, a more than 1,000-fold increase since early 2024. A token is the basic unit of information processed by a model, so that increase describes a substantial change in the volume of work that must be scheduled, routed, and delivered to users.
The immediate effect is to make software design a commercial issue rather than a back-office engineering task. The same question sits behind EastFrontier’s report on Chinese firms using Southeast Asian Nvidia clouds: where can a company obtain enough reliable capacity for the tasks that cannot easily move to local hardware? The new SCMP report shows that the answer is increasingly a mixed environment, in which domestic chips may handle some inference while scarce Nvidia capacity is reserved for workloads that demand more consistent performance.
Inference Demand Turns Software Efficiency Into a China AI Constraint
Training and inference place different demands on compute infrastructure. The SCMP report notes that inference can be adapted to domestic hardware more readily than training, because it follows the earlier phase in which a model is built. That does not mean every inference request is equivalent. A short, low-stakes answer and a long coding task can both be generated by an AI model, yet they differ in quality expectations, latency requirements, and the economic value attached to an accurate result.
The report’s 140-trillion daily-token figure is important in that context. It is not a count of users or of models. It is a measure of model activity. As that activity rises, companies have to decide which hardware receives which requests, how much capacity to reserve, and whether a response is worth running on a constrained pool of advanced processors. This is related to the inference-serving problem examined in EastFrontier’s coverage of TensorCast, which focused on a Chinese startup addressing performance limits in AI-agent inference.
SCMP linked part of the surge to more agentic AI, where systems carry out practical tasks rather than only return text. Such use expands the operational importance of inference. An agent that takes several steps through a workflow can create repeated requests for compute, and each weak answer can impose a cost through a failed task, a user retry, or a human review. The economics therefore depend on both the quantity of tokens and the quality of the output they produce.
Coding Workloads Preserve a Premium for Nvidia Capacity
Guan Jiawei, vice president of inference-optimization startup Approaching.AI, described a polarized market to SCMP. Demand for high-quality tokens, he said, far exceeds supply. Guan’s point was that domestic processors can handle a lower tier of work, but complex tasks, particularly coding, still place requirements on chips that local hardware does not consistently meet. He said users will pay more for coding scenarios, which turns hardware quality into a monetization question.
This is more precise than saying China’s domestic chip sector is either succeeding or failing. The report describes a layered market. At one layer are tasks that can be served on local processors. At another are the workloads that require stricter performance, where companies see limited Nvidia capacity as the more commercially useful resource. If a provider is forced to allocate a premium task to hardware that produces weaker results, it risks serving a category with low demand and low monetization rather than one with clear willingness to pay.
The structure also explains why software optimization has gained urgency. Companies cannot treat every prompt as identical when their processor pool is uneven. They need systems that can match a request to available capacity, protect the limited supply of high-end compute, and preserve response quality where the economic stakes are highest. Those requirements echo the earlier push by Alibaba to open-source an AI-chip software stack challenging CUDA, although the SCMP account is about a wider market problem rather than a single company release.
A Mixed Hardware Strategy Replaces Simple Substitution
The implication is not that Chinese companies will abandon domestic chips. SCMP explicitly reports that inference can be adapted to local hardware. The more consequential question is how companies divide work between those chips and a constrained supply of advanced Nvidia processors. A business that uses local hardware for eligible requests can preserve higher-end capacity for coding or other workloads where performance matters most to revenue.
That approach also changes the role of optimization firms such as Approaching.AI. Their relevance is not limited to reducing a model’s computational appetite in the abstract. In a market where high-quality-token demand outruns supply, an optimization layer can influence which service tier a customer receives and whether the underlying capacity produces enough economic return. The reported shortage is therefore not just a procurement headache. It reaches product pricing, service design, and the ability to scale agent features without degrading the tasks customers value most.
For China’s AI providers, the next contest will be measured in the quality of service delivered under those constraints. The country can expand domestic inference capacity and still face a premium bottleneck if the most demanding tasks remain dependent on a small pool of Nvidia processors. SCMP’s reporting makes clear that the answer companies are pursuing is not one-for-one hardware replacement. It is a software-led effort to decide where each unit of compute produces the most useful response.