EVERWORLD / THE GUIDE
EverWorld vision and architecture
A world you can interact with, compose and shape together with its characters.
EverWorld brings real-time interactive spatial video to livestreams, playable films and immersive entertainment. Players explore, act and converse inside scenes that keep responding.
1.0 deepens interaction and continuous generation; 2.0 expands spatial scale and control; 3.0 adds autonomous agents and real-time multimodal interaction.
The three versions extend one shared foundation. Follow each stage from the experience it enables to its architecture, then explore memory, open-ended generation and the other core mechanisms below.
1.0CAPABILITIES & ARCHITECTURE
Interactive scene generation
Make a room or small connected scene continuously playable: input drives coherent scene dynamics, audiovisual generation continues, and identities and consequences persist across views.
What makes it work
- 01Playable scene interaction
- Bind player input to the actor, target, action phase and event. Semantic conditioning and fast control drive coherent scene dynamics and audiovisual responses.
- 02Continuous, open-ended AV generation
- Extend fixed clips with block-causal generation and self-history continuation. Bounded context, long-term retrieval and cache management support long streams; few-step generation targets lower interaction latency.
- 03Spatiotemporal consistency across views
- Add entity-level 3D perception, multi-view reconciliation and target-view reprojection. Treat static space and dynamic foregrounds separately, using depth, masks, tracks and pose evidence.
- 04Persistent interaction consequences
- Record changes in versioned state and event memory, and retrieve history valid for the current state. Prevent old appearance from overwriting new state or one-shot sounds from replaying.
Model focus
Retain the H3 U—MoT—G backbone and strengthen state, geometry and history conditioning. Looped refinement updates only the MoT condition bank; G has no internal loops. Initially qualify rooms or small connected scenes, with open-ended duration and spatial extent evaluated separately.
2.0CAPABILITIES & ARCHITECTURE
Large-scale spatial orchestration
Make connected spaces and scene dynamics plannable, executable and controllable. Expand beyond room-scale interaction through hierarchical organization while retaining the shared AV generation core.
What makes it work
- 01Larger connected spaces
- A coding MLLM plans regions, buildings, rooms and entities hierarchically. Global topology, local coordinates, regional loading and event-driven updates preserve state across exploration.
- 02Create and edit scenes with natural language
- A coding agent generates executable C/E/A representations: composition C, dynamics E, and appearance/sound A. Execution previews, constraint checks and local revisions improve reliability.
- 03Instant generative objects and entity-level control
- Coarse geometry, collision proxies, joints and functional regions carry execution; generation adds appearance. Functional properties become rules and state changes; appearance stays bound to entity IDs and local coordinates.
- 04Execution-constrained audiovisual generation
- Use two conditioning paths: hierarchical state encoding → MoT for entities, relations and events; geometry/motion/appearance → G for position, occlusion, movement and visual identity.
Model focus
Extend 1.0 with an external coding agent, an executable scene runtime and execution-conditioned encoders. Keep program p, live state S and appearance memory M separate. The executor advances modeled dynamics; the generator presents AV. High-quality imagery still depends on generation; the key challenge is sustained fidelity to execution.
INSTANT 3D OBJECT + SCENE SIMULATION
From a coarse shape to an interactive object
Code stores a lightweight recipe for how an object is generated and how it acts. Rich appearance is synthesized around a persistent physical proxy.
Coarse geometry
Cuboid · coordinates · grip region
Executable properties
Rigid body · contact · heat rules
Generation conditions
Steel · runes · persistent identity
Runtime state
Pose · velocity · temperature
CreateCoding agent → C/E/A program → execute, verify and revise
RunInput → executor state S → state/spatial encoders + memory M → AV generation
Program p, live state S and appearance memory M share identity and coordinates. Functional effects must execute; image errors cannot rewrite state.
3.0CAPABILITIES & ARCHITECTURE
Agent-driven scene evolution
Let agents converse, reason and act autonomously in 2.0’s executable spaces. Players can interrupt or revise intentions while agents with independent memories and goals keep the scene evolving.
What makes it work
- 01Online full-duplex multimodal interaction
- Add streaming audio understanding, incremental multimodal context, dialogue control and speech decoding. Listen, speak and act concurrently; handle interruptions, corrections and cancellation while aligning lips, actions and sound.
- 02Autonomous characters with memory and goals
- Combine a shared multimodal MLLM with independent role context for knowledge, experiences, goals and relationships. Output dialogue, skills and action parameters; controllers execute high-frequency motion.
- 03Compare consequences before acting
- Build isolated branches from each role’s available knowledge. Reuse the 2.0 executor plus uncertain behavior predictions to evaluate candidate actions, mainly in state/coarse-model space to reduce cost.
- 04Ongoing multi-agent interaction and scene evolution
- Actions change tasks, resources and relationships; experiences inform later decisions. Shared time, action scheduling, conflict handling and hierarchical updates support continued evolution after the player leaves.
Model focus
Add a streaming multimodal action model with memory, goals and rollout-based reasoning, reusing 2.0’s executor and generation core. Maintain live facts, role beliefs and hypothetical branches separately. State-bound 3D/4DGS reuse remains an optional way to reduce long-running presentation cost.
ONLINE · FULL DUPLEX · EVOLVING
Conversation changes what happens next.
Three coordinated loops share time, identity, intent versions and events. Slow planning does not block listening.
Live interaction
Continuous perception, incremental understanding, speech and interruption.
Agent decisions
Memory and goals, simulated alternatives, planning and revision.
Scene execution
Skills, physical state, shared events and spatial audiovisual feedback.
- Player
Bring me the sword.
- Agent · speaking + listening
On my way to get it…
- Player · interrupts
Wait—bring the shield instead.
- Agent · revises the task
The plan changes. Other agents keep acting.
Stop speech, cancel a task and stop an action are distinct. Consequences already committed remain.
A CLOSER LOOK
Inside the shared foundation
Explore the mechanisms behind coherent interaction, persistent memory and continuous generation.
01From input to a coherent next moment
Meaning, space, sound and control guide one audiovisual generator. Validated output and state are committed together before the next interaction.
02A world that remembers what changed
State, appearance and sound read the same world version. Returning to a scene preserves its consequences without replaying an event that has already happened.
03Continuous interaction, consistent history
Generate and commit one interval at a time. New instructions affect work that is still changeable, while committed history remains consistent.
04Refine context before generation continues
Looped MoT refines condition tokens using fixed inputs and a read-only generation summary. The generator and live world state stay outside the loop.
05Connect understanding with generation
Ten matched connections carry understanding and state into generation, with condition refinement at ports 25 and 50 and the original U50 route preserved.
EXPLORE FURTHER