Create 3D interactive digital rooms from simple videos

Cornell researchers have developed an AI-driven process that automatically converts short videos in rooms into interactive 3D simulations of space.

Within this highly accurate “digital twin” the user can open drawers and cabinets and handle countertop objects. This technology can be used to develop more realistic video games and effectively train robots to operate within a specific real world space. Essentially, it is an application that requires a realistic and interactive model of the room.

“Existing techniques allow us to synthesize what the world looks like from different perspectives, but sometimes this ability of immersion is lacking. “Because of advances in generator AI techniques, there are enough tools to ultimately create baby steps to create digital twins that can now be interacted with.”

His collaborators include PhD Hong Kong Xia. Computer Science student at the University of Illinois Urbana-Champaign. Xia announced its project “Drawers: Digital Reconstruction and Clarification with Environmental Realism” at the IEEE/CVF Conference on Computer Vision and Pattern Recognition in Nashville, Tennessee on June 15th.

The process of using a drawer to create a digital twin for your room begins with just a few minutes of shooting.

“Our input is a video that casually captures in the kitchen. There’s no need to interact with cabinet doors or objects,” says Xia. “It’s just holding your iPhone. You don’t need advanced video devices or expensive cameras.”

To turn the video into a photorealistic and interactive digital room, researchers have put together several AI models. We combined two ways to render digital images. The first is what looks attractive and the second one that recreates the scene with very accurate dimensions. We have also added a concept module. This determines how the fridge door swings, thus determining which parts of the scene are mobile and how they should move. Finally, we included a model that fills the invisible inside of the drawer.

However, development of development was not just about linking modules, Xia said. He had to integrate them into a unified framework. Once completed, he used this method to develop recreational kitchens, bathrooms and even offices.

The digital twins generated by this approach work seamlessly with the game engine used to create video games, Xia said. The researchers demonstrated this by creating a game in which users shoot balls to knock over objects in the kitchen, such as kettles and soap bottles.

This framework can also be applied to effectively train a robot to operate in a substantial environment. Researchers effectively trained robotic arms in digital twins in the kitchen, showing that they managed to clean up the objects in real-world drawers.

They believe that in the near future, someone can order a robot, upload a video of their home, and train the robot using the digital twins of their homes, before they leave the box. Simulation is a cheaper, faster, and safer way to train a robot, MA said.

Currently, drawers only work with rigid objects like kettles, but ultimately they plan to include soft or deformable objects like fabrics and windows that can break.

Additionally, while the drawers are currently replicating a single room, Ma and Xia want to expand this work to encompass the entire building. It also envisions creating digital twins for outdoor spaces that can use technology to design cities and optimize agricultural yields.

“Our final goal is to try and build every digital twin in the world,” Xia said. “There’s a lot to explore in the future.”

Additional authors on this study include Enton Su, Marius Memel, Ahan Jain, Raymond Yu, Moofo Mbijiu Tiapo, Ali Farhadi (Allen Institute for Artificial Intelligence) and Abhishek Gupta, as well as Shenlong Wang from the Illinois Urbana Kampang family.