Swansenreport

Large Language Models (LLMs)

LLMs like Claude, GPT-4, and Gemini are trained on text. They learn statistical patterns across enormous amounts of written language — books, websites, code, conversations — and become extraordinarily good at predicting what text should come next. Their “understanding” of the world is entirely mediated through language. They have no direct experience of physical reality, causality, time, or space. They can describe how a ball rolls down a hill but have never experienced physics directly.

Key limitations:

• No persistent memory across sessions
• No grounded understanding of physical causality
• Can hallucinate facts because their “knowledge” is pattern matching, not world simulation
• Passive — they respond, they don’t plan and act across time

World Models

World models are a fundamentally different architecture. Rather than predicting the next token in a sequence, they learn to predict the next state of the world — essentially building an internal simulation of how reality works. The concept comes partly from neuroscience: the human brain doesn’t just process language, it maintains a constantly updated model of the physical and social environment.

Key distinctions:

• Causality over correlation — world models learn cause and effect, not just co-occurrence
• Spatial and temporal reasoning — they understand that objects persist, physics applies, time moves forward
• Planning capability — because they can simulate future states, they can plan multi-step actions
• Grounded in perception — trained on video, sensor data, and physical interaction, not just text

Who’s building them

Yann LeCun at Meta has been the most vocal advocate, arguing LLMs are fundamentally limited and that world models are the path to human-level AI. His JEPA (Joint Embedding Predictive Architecture) is the leading research framework. Google DeepMind’s Genie 2 can generate interactive 3D environments from a single image. OpenAI’s Sora — while marketed as a video generator — is essentially a world model in embryonic form. Tesla’s Autopilot and Waymo’s self-driving systems are real-world world models running right now, just in the narrow domain of driving.

The practical upshot

LLMs are brilliant conversationalists and knowledge retrievers that have no real model of physical reality. World models will be the foundation of robots, autonomous agents, and AI systems that can actually do things in the physical world reliably — not just talk about doing them.

The convergence coming is multimodal world models that also have language capability — systems that understand the world like a world model and can communicate about it like an LLM. That’s where the frontier is headed, and it’s probably 3-5 years from production-scale deployment.