en.Wedoany.com Reported - NVIDIA GEAR, in collaboration with the Fei-Fei Li team, Georgia Institute of Technology, and other institutions, has released the Real2Sim system SimFoundry, which can automatically generate interactive, trainable, and evaluable robot simulation environments from a single real-world video.
SimFoundry can automatically replace objects, adjust scene layouts, and generate new manipulation tasks while preserving object functionality and affordances, thereby expanding a nearly infinite space for simulation data generation from a single video. The system establishes a complete Real-to-Sim loop encompassing scene generation, data generation, policy evaluation, and policy training.
Robot policy training has long relied on high-cost real-world data, making simulation environments a scalable alternative. However, building simulation environments with realistic geometric and physical interaction capabilities still requires extensive manual modeling. SimFoundry leverages 3D reconstruction and generative models to rapidly convert the real world into Sim-ready environments that support physical interaction.
Robot policies trained on data generated by SimFoundry can be deployed zero-shot to real robots, achieving real-world transfer in tasks such as multi-step manipulation, dual-arm collaboration, and manipulation of articulated objects.
SimFoundry's pipeline is divided into three stages: Extraction, Generation, and Augmentation. In the Extraction stage, the system takes an RGB video as input and uses tools such as depth estimation and vision-language models (VLMs) to identify and segment objects in the scene. The Generation stage uses 2D-to-3D models to generate 3D meshes, combined with models like FoundationPose to recover object poses, and derives joint structures for articulated objects. This ultimately exports simulation scenes compatible with physics engines like IsaacLab, completing the construction of a Digital Twin.
Augmentation is the core innovation of SimFoundry. It automatically generates Digital Cousins based on the Digital Twin, expanding along three dimensions: changing object appearance and geometry while preserving functionality (Object Cousins); adjusting object layouts or adding new objects to generate new scenes (Scene Cousins); and automatically deriving new manipulation tasks based on the affordances of scene objects (Task Cousins).
The study validated SimFoundry's effectiveness across two robot platforms and seven types of manipulation tasks. In policy evaluation experiments, robot performance in SimFoundry closely matched the real world, with an average Pearson correlation coefficient of 0.911 and an average Maximum Rank Violation (MMRV) of 0.018. After introducing Digital Cousins, the average task success rate in the real world improved by 17% (Object Cousins), 21% (Scene Cousins), and 40% (Task Cousins) compared to using only the Digital Twin.
The authors of this research paper are from NVIDIA GEAR, Georgia Institute of Technology, Stanford University, UT Austin, and the University of Toronto. Key researchers include first authors Nadun Ranawaka Arachchige, Josiah Wong, Jiangyun Fan, Tianyuan Dai, Masoud Moghani, Hang Yin, as well as Jim Fan, Fei-Fei Li, Danfei Xu, Yuke Zhu, Ajay Mandlekar, Ruohan Zhang, Wenbowen, and others.
