en.Wedoany.com Reported - The Wujie·RoboBrain Orca Team at the Beijing Academy of Artificial Intelligence (BAAI) has released a technical report titled "Orca: The World is in Your Mind," aiming to explore a pathway where models first learn a unified representation of world states, and then derive capabilities for understanding, prediction, and action from it. The project homepage is https://orca-wm.github.io, and the full technical report is available at https://arxiv.org/abs/2606.30534. This release has garnered attention from the overseas research community, with discussions focusing on "multimodal representation world models." It is believed that Orca attempts to learn the common states and evolutionary laws behind different modalities within a unified world latent space. Some commentators noted that Orca is closer to the form of an early general world model, which first learns how the world changes and then applies this representation to downstream tasks. Orca has also been featured on the Daily Papers monthly list.

The core concept of Orca is "Next-State Prediction," where the model focuses on the current state of the world and how that state transitions to another under natural evolution, event conditions, or external intervention. The team abstracts learning into two types: Unconscious Learning, which learns natural, dense state changes from continuous video; and Conscious Learning, which introduces language and events, making state transitions subject to semantic constraints. Both types of learning jointly serve to construct world latent variables that can express world states and support state transition modeling.

In the first type, Unconscious Learning, the model learns state changes from continuous video without relying on explicit language annotations, such as object movement, hand-object contact, and scene evolution over time. The second type, Conscious Learning, introduces language and event constraints, enabling the model to establish connections between linguistic conditions and current observations, learning sparse but more meaningful state changes.

Both types of learning jointly serve to construct a world latent variable capable of expressing world states and supporting state transition modeling.

The Orca Team built an automated filtering and annotation pipeline, processing internet data to obtain a database of 125,000 hours of video, 160 million event annotations, and 11.5 million Visual Question Answering (VQA) items. The data covers various sources, including first-person view interactions, third-person object manipulation, robot execution videos, natural dynamic scenes, event-level state transitions, and general visual question answering, all used to learn a unified world latent space from real-world signals.

In terms of training infrastructure, the BAAI team performed system-level refactoring based on their self-developed FlagScale framework, including FSDP2 upgrades, Chunked Cross-Entropy Loss, and forward/backward prefetching optimizations. On an H100 cluster, these optimizations increased training throughput to 2.91 Samples/Sec/GPU, achieving a 4.4x speedup over the StarVLA baseline.

To validate the effectiveness of the world representation, the Orca team froze the backbone network during the downstream phase, training only lightweight readout modules. The experiments designed three types of readouts: Text Readout, to verify whether the model can convert world representation into understanding and reasoning capabilities; Image Readout, to verify whether the model can predict future visual states based on the current state and conditions; and Action Readout, to verify whether the model can transfer world representation to real robot control. These three capabilities correspond to understanding, prediction, and action, respectively.

Experimental results show that as the scale of pre-training data increases, Orca's training loss consistently decreases, and checkpoints from different training stages indicate synchronized improvements in text, image, and action readout performance. All comparison results are derived from the same set of pre-trained backbone checkpoints, without using benchmark-boosting data.

In text generation and visual question answering tasks, Orca achieved higher average performance compared to various vision-language models and world models at the 4B scale, with improvements concentrated in dimensions such as state transition, event evolution, and dynamic motion understanding.

In the Image Readout experiments, Orca's image readout emphasizes the plausibility of future states, better maintaining robot morphology, object layout, scene consistency, and instruction constraints.

In real robot action readout experiments, Orca did not use robot trajectories with action labels during the pre-training phase. For downstream tasks, the Orca backbone was frozen, and only a DiT-style action expert trained from scratch was attached, with post-training conducted using 200 in-domain trajectories per task. Results show that Orca still brings significant gains in out-of-distribution (OOD) tasks involving object and scene generalization.

Ablation studies indicate that the three training objectives—unconscious state transition, conscious event transition, and VQA language supervision—each play distinct roles.

The Orca Team discussed current limitations in the technical report. The report points out that a world model should not be defined by a single output modality but should learn how the world is represented, how it changes, and how this representation can support understanding, prediction, and action. This pathway could also serve fields such as physical systems, life processes, the macroscopic universe, microscopic quantum mechanics, and scientific experiments in the future. Different domains require different data and modeling approaches, but the question of "learning how the world state changes" holds a more fundamental significance.