en.Wedoany.com Reported - New York University, in collaboration with Yann LeCun's startup AMI, has released the latest JEPA series achievement, AdaJEPA, proposing a world model capable of continuous learning during the deployment phase.

Unlike previous world models that freeze parameters after pre-training, AdaJEPA employs a test-time adaptation mechanism, adjusting the encoder and predictor parameters of the world model in real-time during interaction with the environment, thereby enabling continuous learning.

The core process of AdaJEPA is a closed loop consisting of planning, execution, observation, update, and re-planning. The system only executes the first action planned by model predictive control, then uses the next frame state fed back from the real environment as a self-supervised signal to update the world model online. Thus, the model called upon in the next planning cycle is no longer the frozen version at deployment, but a model calibrated by the current environment.

This approach is similar to the Dyna architecture in classic reinforcement learning, where the model is not trained in one go but continuously refines its understanding of the environment through real interaction.

Traditionally, latent space world models based on the JEPA approach freeze parameters after training. The entire process is: the model first learns to compress high-dimensional images into a latent space from offline trajectories, then predicts future states within this latent space. During testing, model predictive control calls this frozen model to "imagine" the future by rolling out in the latent space, optimizing the action sequence, and only executing the first action in the real environment.

The challenge with this method is that when the environment changes, the frozen world model can become inaccurate. Faced with test-time distribution shifts, actions that seem to reach the goal in the latent space may deviate in the real environment. Since model predictive control relies on short-horizon rollout planning, single-step errors can be amplified during subsequent rollouts.

To address these issues, the paper proposes the AdaJEPA framework, whose core judgment is that a world model should not be fixed after training but, like an agent in real deployment, should calibrate itself with new experiences while acting.

The AdaJEPA cycle is specifically divided into four steps. The first step is planning: the model encodes the current observation into a latent state, uses the current world model for model predictive control, rolls out predictions in the latent space, and finds a sequence of actions closest to the target state. The second step is execution: the model executes only the first action, and the real environment returns the next observation. The third step is update: the system stores this real state transition in an online buffer. The model predicts the next latent state based on the observation and action, aligns it with the latent state encoded from the real next observation, and updates parameters via gradient computation. The fourth step is re-planning: the updated world model immediately enters the next cycle of model predictive control.

In terms of implementation details, AdaJEPA's foundation remains the Joint Embedding Predictive Architecture. Unlike traditional pixel-level predictive world models, the Joint Embedding Predictive Architecture does not directly predict future images but compresses images into a more compact latent space and predicts future states only within this latent space. The entire model consists of three core components: a state encoder, an action encoder, and a predictor.

Online updates are performed in the latent space. After each action execution, the system stores the real state transition in an online buffer, which by default holds the most recent N transitions. During an update, the model predicts the next latent state based on the current observation and action, and aligns it with the latent state encoded from the real next observation. To prevent online updates from destroying the original representation space, the paper sets two constraints: using stop-gradient on the target representation and updating only a small number of parameters. In experiments, by default, only the last few layers of the visual encoder and predictor are updated, and each model predictive control re-planning performs only one step of gradient descent.

To verify the effectiveness of test-time adaptation, the paper conducted experiments on two benchmarks: PushT/PushObj and PointMaze. In the unseen shape test of PushObj, AdaJEPA nearly doubled the planning success rate. In the unseen layout test of PointMaze, the GD planning success rate increased from 53.3% to 78.7%, and the CEM planning success rate increased from 49.3% to 70.7%.

Notably, the additional latency introduced by this online update is low. Since only the last few layers of the visual encoder and predictor are updated, and each re-planning performs only one step of gradient descent, the additional latency is only 0.01 to 0.03 seconds. This indicates that AdaJEPA does not trade success rate improvement for heavy online training but adds a lightweight "deployment-time self-calibration" mechanism to the original world model.

The paper's author, Ying Wang, is currently a PhD student at the CILVR Lab of NYU's Center for Data Science, researching world models under the supervision of Mengye Ren and Yann LeCun. Author Oumayma Bounou is a postdoctoral researcher at New York University, with research interests including world models, control, and optimization, currently collaborating with LeCun on world model research. Mengye Ren, Assistant Professor of Computer Science and Data Science at New York University, and Turing Award winner Yann LeCun serve as co-supervising authors.

Reference link: https://arxiv.org/pdf/2606.32026