Foundation Model Labs: A New Paradigm for Enterprise AI

Why the foundation model labs training physical AI systems need a data infrastructure partner — and why that partner doesn’t exist yet

By Pradeep — Founder, XpertSystems.ai

Eight years ago, Covariant proved something most of the robotics industry is still learning: in physical AI, the data flywheel is the business. The architecture is a research contribution. The data operation is the durable advantage.

Eight years later, nobody has built the data infrastructure company the physical AI ecosystem needs. Scale AI exists for web and language data. There is no equivalent for contact-rich, multi-modal, failure-annotated physical-world data. The market is large, the need is acute, and the structural moat is real. The window is open.

This essay is an argument about why that gap exists, why attempts to close it with better simulation have failed, and what the actual solution looks like. It is also a pitch — we have been building pieces of this solution at XpertSystems.ai, and we are looking for the foundation model labs, enterprise AI teams, and robotics companies serious enough to partner on the rest.

The simulation premise, and why it fails

The conventional wisdom in robotics has been seductive: real-world data collection is slow and expensive, so reduce 500 real demonstrations to 50 and close the gap with compute. Simulate millions of scenarios in a physics engine, domain-randomize across parameters, train a policy that generalizes, deploy. If the simulator is good enough, reality becomes a rounding error.

Nvidia has invested more in this premise than any other company on earth. Their Isaac Sim and Omniverse platforms are genuine engineering achievements. And their own documentation continues to identify the reality gap as the core unsolved problem. This is not a resourcing failure. It is a structural one.

The reason is worth stating plainly. Physics does not care about render quality. For manipulation specifically — the class of problems where a robot must make contact with the physical world and produce an outcome — the governing dynamics are contact-rich, stiff, and extraordinarily sensitive to parameters that are fundamentally unobservable. Friction coefficients vary with humidity, dust, micro-scratches on the gripper, and deformation history. Tactile sensor responses are generated by sensor physics that compound on top of contact physics. Failure modes are long-tail — a cable snags, a box is slightly crushed, a label is sticky, a gripper finger has worn down three percent. Every one of these is a real signal. None of them exists in a simulator unless a human explicitly codes it in.

A simulator generates plausible versions of force signals, contact dynamics, and failure modes. A deployed robot finds the difference in the first hour. This is not a claim about current simulator quality. It is a claim about the identifiability of the underlying physical parameters. Even a perfect physics engine requires parameters that can only be measured from real data.

The companies that got it right

Two companies have navigated this correctly. Covariant, founded in 2017, built its business on real-world manipulation data from deployed systems — warehouse logistics, industrial pick-and-place — and used that operational data advantage to train models that genuinely generalized. Physical Intelligence, the more recent Levine-and-Finn venture, has taken a similar posture at the foundation-model scale: architecture matters, but the data operation is the moat.

Both companies publish research papers that focus on architectures. Both companies, privately, will tell you the architectures are not the hard part. The hard part is the teleoperation pipeline, the tactile sensor calibration, the failure annotation taxonomy, the quality assurance on every labeled trajectory. The hard part is the operation.

These companies have built their data operations in-house. That choice made sense when there was no credible third-party alternative. It makes less sense now, as more foundation model labs enter the physical AI space and face the same build-versus-buy decision. None of them want to operate a physical data factory. All of them need the data such a factory would produce.

What the infrastructure partner actually needs to be

The partner the physical AI ecosystem needs is not a web-scraping operation with a robotics label. It is not a pure simulation vendor with a better renderer. It is not a teleoperation farm alone. It is a hybrid data factory that combines three capabilities most vendors offer in isolation.

Capability one: simulation for coverage. Simulation is genuinely valuable for generating distributional coverage — variations of scenarios, edge conditions, parametric sweeps — at a scale and cost that real-world collection cannot match. A serious infrastructure partner runs production-grade simulation for every domain it operates in, calibrated to the physics that are tractable and honest about the physics that are not.

Capability two: real-world anchoring through calibration. Simulation without calibration is generic. Calibration takes a sample of real operational data from the target environment — a customer’s specific robot, sensors, objects, conditions — and uses system identification techniques to fit simulator parameters against it. A residual model then learns the systematic gap between calibrated simulation and real observations, and applies that correction to every downstream synthetic sample.

Capability three: validation against held-out real data. The customer gives the infrastructure partner 80% of a real sample; the partner calibrates; the partner ships synthetic data validated against the 20% holdout. Distributional similarity metrics, uncertainty quantification per feature, explicit flagging of extrapolation regions — all of it delivered as part of the dataset. The customer’s ML team knows what they can trust and what they should weight skeptically.

These three capabilities together produce a fundamentally different product than any component in isolation. Pure simulation is cheap but wrong in the tail. Pure real-data collection is right but slow and expensive. Hybrid is the only approach that scales while maintaining fidelity where fidelity matters.

Why this is structurally defensible

The defensibility argument has three parts, each of which compounds.

Operational difficulty. Running teleoperation rigs, maintaining sensor calibration across fleets, training operators to annotate failure modes consistently, building QA pipelines for physical signal integrity — this is not a technical problem a smart team solves in a weekend. It is an operations problem that requires capital investment, physical infrastructure, and institutional learning. The barrier to entry is high enough to deter casual competition and low enough for a serious operator to cross in 18 to 24 months.

Data compounding. Every deployed robot hour generates more data. If the infrastructure partner is in the loop for calibration and feedback, that operational data flows back into the partner’s corpus, improving simulation fidelity, residual models, and failure-mode taxonomies for every other customer in the same domain. This is the flywheel Covariant demonstrated internally. At the infrastructure level, it compounds across customers.

Switching cost. A customer that has completed calibration, validation, and a deployment-feedback cycle with one infrastructure partner has built a dataset that is tuned to their environment and improves with each iteration. Switching means starting the calibration process over, losing the accumulated residual corrections, and paying a lead-time penalty while the new vendor catches up. For any serious production ML team, the switching cost quickly exceeds the price delta between vendors.

Together, these three produce the structural moat that Scale AI built in a different domain. The difference is that physical AI data is harder to produce than web data, which means the moat is deeper and the ceiling on pricing power is higher.

The market that needs this now

Three customer segments are converging on the need for this infrastructure.

Foundation model labs training physical AI systems. These organizations have enormous compute, world-class ML talent, and no appetite for operating physical data factories. They need training data at foundation-model scale — millions of contact-rich, multi-modal, failure-annotated trajectories — and they need it to actually reflect the environments their models will deploy into. They will pay premium prices to the infrastructure partner that can deliver this reliably.

Enterprise ML teams in industrial verticals. Pipeline operators, power utilities, manufacturers, logistics operators — all of them are deploying ML for anomaly detection, predictive maintenance, and process optimization. All of them are discovering that generic synthetic data does not generalize to their specific infrastructure. Calibrated synthetic, validated against their own historical data, is the product they actually need. Many of them will pay for it once they understand what it is.

Robotics startups at the series A and B stage. These companies have raised enough to ship product but not enough to build a data operation rivaling Covariant’s. An infrastructure partner that can deliver calibrated manipulation data as a service is an enabler for an entire cohort of companies that would otherwise stall at the sim-to-real wall.

What we are building at XpertSystems

At XpertSystems, we have built simulation engines and validation infrastructure across oil and gas, healthcare, cybersecurity, ERP/finance, robotics, and other verticals — production-grade synthetic data products validated to Grade A+ on internal statistical rigor. That is our starting point, not our thesis.

The thesis is hybrid. We are building the ingestion, calibration, residual-modeling, and validation layer that transforms baseline synthetic data into customer-specific calibrated data, validated against the customer’s own held-out real data, refined continuously through deployment feedback. The same platform architecture serves oil and gas, healthcare, cybersecurity, and — at the high-value end of the market — physical AI and robotics.

We are looking for a small number of early partners to build this out with us. Foundation model labs that need calibrated manipulation data. Enterprise ML teams that need synthetic data tuned to their specific operational environment. Robotics companies ready to move past the sim-to-real wall.

The companies that get data operations right will win the next five years of AI. The infrastructure partner that serves them — the Scale AI for physical AI — is structurally more defensible than the original Scale AI ever was, because the data it produces is structurally harder to make.

The window is open. It will not stay open forever.

• https://www.xpertsystems.ai/synthetic-data-factory.html#catalog
• ———

Pradeep is the founder of XpertSystems.ai, a synthetic data platform serving industrial, healthcare, cybersecurity, and physical AI verticals. For partnership inquiries, reach out directly on LinkedIn.