The race to build world models from game data has a dirty secret: video is the easy part, and the easy part is not the valuable part. The real prize is clean action labels, who did what, when, and why, and most of the public datasets are scraping by with approximations. A developer running a deterministic multiplayer strategy game recently laid out the hard questions that follow, and they cut to the core of how we should think about training data for AI agents. If you are building agents that need to act in complex environments, the provenance of your training data matters more than its volume, and that has direct implications for the When your AI agent follows instructions but forgets the procedure challenges we keep seeing in production.
The numbers tell the story. OpenAI's Video PreTraining (VPT) used roughly 2,000 hours of contractor play with recorded keyboard and mouse inputs to train an inverse dynamics model, which then labeled 70,000 hours of YouTube footage. Google DeepMind's Muse trained on years of first-party play with controller inputs preserved. Nvidia reportedly extracted inputs from on-screen controller overlays on streamer videos. These are clever workarounds, but they are workarounds. The developer's game, by contrast, logs every action with exact labels, who acted, what they told other players, and the full deterministic world state. That is not a marginal improvement. It is a fundamentally different category of data, and it forces us to ask whether the field has been optimizing for the wrong metric.
The practical consequence for anyone training these models is that raw video volume may be misleading you. If your agent needs to learn causal relationships, that moving the pointer to a button and clicking causes a specific outcome, then action-conditioned prediction requires knowing the action, not inferring it. The developer poses two specific questions that deserve attention: whether high-level strategic actions (attack this region, break this pact) are useful for prediction, or whether low-level inputs (pointer, camera) are what you actually need. Our view is that both matter, but the field has been starved of the high-level labels because they are expensive to collect. That is why the Six Architectural Patterns for Scaling Reliable AI Agents we covered recently keep circling back to the same bottleneck: reliable agents need reliable training signals, and you cannot get those from inferred labels.
The most provocative question the developer raises is about multi-agent data with language attached. Messages sent between players, alongside the actions that followed, could be the missing ingredient for world models that understand intent and negotiation. Most training pipelines strip this context out because it is messy. But if your goal is an agent that can collaborate, deceive, or coordinate, rather than just react, then that conversational data is not noise. It is the signal. The developer is sitting on a dataset that has all of it: exact actions, full world state, and the language exchanged right before each move. The question for the rest of the field is whether we will continue building models on approximations, or start demanding data with provenance we can trust. The answer will determine whether your next agent follows instructions because it understands the procedure, or because it got lucky.