Researchers at Cornell University have recently developed an AI pipeline called DRAWER that can automatically transform short videos of rooms into interactive 3D simulations—highly accurate "digital twins." Within this digital space, users can freely open drawers and cabinets, touch items on countertops, and enjoy an immersive interactive experience. DRAWER technology can be used not only to create more realistic video games but also to virtually train robots to operate in specific real-world spaces, opening new possibilities for applications that require realistic, interactive room models.

"Existing technologies can synthesize different perspectives of the real world, but they lack immersive experience and true interactivity," said Weiqui Ma, assistant professor of computer science at Cornell University. "Thanks to advances in generative artificial intelligence, we have taken a small step toward creating interactive digital twins." On June 15, project collaborator Hongchi Xia, a doctoral student at the University of Illinois Urbana-Champaign, presented the project "DRAWER: Digital Reconstruction and Articulation With Environment Realism" at the IEEE/CVF Conference on Computer Vision and Pattern Recognition in Nashville, Tennessee.

The process of creating a digital twin with DRAWER is simple and can begin with just a few minutes of footage. Xia said: "The input is just a casually shot video of a kitchen—no need to interact with cabinet doors or objects, and an iPhone is sufficient; no advanced video equipment is required." To turn the video into a realistic interactive digital room, the researchers integrated multiple AI models, combined two rendering methods, added perception modules to identify movable parts, and filled in unseen areas. Xia noted that developing DRAWER required integrating all modules into a unified framework; he has already used this method to redesign kitchens, bathrooms, and offices. The digital twins generated by the technology work seamlessly with game engines, and the research team has demonstrated this by creating games. Additionally, DRAWER can virtually train robots to operate in real environments through a "real-to-sim-to-real" transfer process.

Currently, DRAWER technology is limited to rigid objects, but the researchers plan to extend it to soft or deformable objects. At the same time, they hope to expand the work to entire buildings and create digital twins of outdoor spaces for use in urban design or optimizing agricultural yields. Xia said: "Our ultimate goal is to try to build digital twins of everything in the world—there is a lot to explore in the future."