The hard part of training a large language model is not limited to chips and algorithms. It also includes the work of collecting, cleaning, organizing, tracing, and revisiting enormous volumes of data. That operational layer is the focus of OmniTable, a system associated with Ant Group that won the Best Industry Paper award at VLDB 2026. The VLDB conference named the OmniTable paper, which addresses petabyte-scale curation and exploration of LLM training data, as this year’s winning industry submission.
The award gives independent confirmation that the paper was recognized at the 52nd International Conference on Very Large Data Bases, held in Boston from August 31 to September 4. The system’s scale and performance descriptions come from the paper itself and from company-supplied material reproduced by QbitAI, so they should be read as implementation and evaluation results reported by the authors rather than as a general benchmark for every AI data pipeline.
The paper describes a data-management design built for an environment in which a training corpus is repeatedly transformed. A dataset may be parsed, cleaned, deduplicated, scored, labeled, filtered, tokenized, and assembled into a training set. Each stage can produce new fields and new intermediate results. At petabyte scale, tracking those steps becomes an engineering problem in its own right.
OmniTable Treats Data and Features as One Managed System
The central design idea is what the paper calls logical unity with physical separation. Instead of requiring users to keep track of a growing collection of physical tables, OmniTable presents a logical wide table while allowing the underlying data to be stored and organized in separate ways. The paper says this approach is intended to give engineers a stable view of data even as the physical layout evolves.
That architecture matters for LLM curation because a single data item can acquire many attributes over time. A web document, code file, PDF, or supervised fine-tuning example may have processing status, quality scores, safety markers, duplicate signatures, and domain labels. If each new attribute is handled through a separate ad hoc pipeline, locating a record or re-running a changed feature becomes increasingly difficult.
The OmniTable paper says the production system manages more than 35 petabytes and more than 305 billion records across web, code, PDF, and supervised fine-tuning data. It also describes catalog functions that record data batches, feature definitions, execution state, and lineage. These are reported system figures, not a claim that Ant Group has disclosed the entire composition of its model-training corpus.
The design resembles a concern already visible across China’s AI infrastructure sector. Data is becoming a production input that needs its own governance, tools, and repeatable workflows. EastFrontier’s report on China’s AI data providers moving beyond annotation work described the market pressure to make data services more valuable than basic labeling alone. OmniTable addresses a related question inside a large model operation: how to make a massive, evolving dataset manageable for the teams that use it.
The Paper Reports Faster Preparation and Fewer Manual Steps
The most striking performance result in the paper concerns a real supervised fine-tuning data-preparation task. The authors report that an end-to-end cycle was reduced from about 14 days to about 2.5 days, a 5.6-fold speedup. They also report that the task’s manual steps fell from 45 to 12.
Those results do not mean every data-preparation task will see the same improvement. The paper’s comparison is tied to a particular workload, data setup, and implementation. It does, however, identify where the gains came from: fewer separate pipelines to coordinate, feature calculations that can be planned through a dependency graph, and less need to reconstruct how an output was produced.
The paper also describes record-level fault isolation. In its example, if a small number of records cause a user-defined function to time out or exhaust memory, those records can be recorded as failures while the rest of the batch continues. This can avoid a situation in which a few problematic items force a large job to be restarted. Again, the reported performance is specific to the paper’s tested scenarios rather than a guarantee for an outside user.
Such details help explain why the award is relevant to AI infrastructure. Model developers usually receive attention for parameter counts, benchmarks, and product launches. Yet the ability to prepare and revise data at scale can determine how quickly a lab can test new training recipes, investigate a quality issue, or add safety checks. The system does not replace a model or a data center. It changes the layer that connects raw inputs to training-ready material.
That connection is increasingly valuable as China builds broader AI infrastructure. The country’s new data-annotation pilot across 32 cities reflects a policy interest in strengthening the data supply chain. OmniTable provides a more specialized view from inside a model-development environment, where the challenge is not simply obtaining data but keeping a complex data lifecycle intelligible.
A VLDB Award Highlights an Often Invisible AI Bottleneck
VLDB’s award recognizes a database and systems contribution, not a consumer AI product. That is precisely why OmniTable is important. Training-data curation is often invisible to users, but errors, duplicated work, unclear lineage, and inflexible storage structures can slow every subsequent model experiment.
The paper’s logical-table approach is meant to let a user work with stable data semantics while the system chooses a physical arrangement that suits different types of queries or processing. That can be useful when some teams need large batch exports, others need quick record-level inspection, and others need to recompute a feature after its definition changes.
The system also raises questions about reproducibility and oversight. A catalog that records feature versions, dependencies, and lineage can make it easier to understand why a sample was included or how it was processed. The paper does not claim that the design resolves all data-governance problems, and it does not disclose every operational rule used in production. It does show an attempt to treat data transformation as a traceable system rather than a loose collection of scripts.
For Ant Group, the Best Industry Paper award offers a credible academic and engineering validation of a systems project tied to large-model data operations. For the broader China AI industry, it is a reminder that progress depends on more than faster accelerators and more capable models. Data has to be prepared, versioned, inspected, and rebuilt at a scale where a single manual error can become expensive.
The reported 5.6-fold reduction in a specific preparation cycle is not a universal promise. It is evidence from a real implementation that a different data-management model can materially change the pace of model development. As AI training sets keep growing, the less visible systems behind them may become just as strategically important as the models that eventually reach users.