New AI model turns photos into explorable 3D worlds, with caveats

On Tuesday, Tencent released HunyuanWorld-Voyager, a new open-weights AI model that generates 3D-consistent video sequences from a single image, allowing users to pilot a camera path to "explore" virtual scenes. The model simultaneously generates RGB video and depth information to enable direct 3D reconstruction without the need for traditional modeling techniques. However, it won't be replacing video games anytime soon.

The results aren't true 3D models, but they achieve a similar effect: The AI tool generates 2D video frames that maintain spatial consistency as if a camera were moving through a real 3D space. Each generation produces just 49 frames—roughly two seconds of video—though multiple clips can be chained together for sequences lasting "several minutes," according to Tencent. Objects stay in the same relative positions when the camera moves around them, and the perspective changes correctly as you would expect in a real 3D environment. While the output is video with depth maps rather than true 3D models, this information can be converted into 3D point clouds for reconstruction purposes.

The system works by accepting a single input image and a user-defined camera trajectory. Users can specify camera movements like forward, backward, left, right, or turning motions through the provided interface. The system combines image and depth data with a memory-efficient "world cache" to produce video sequences that reflect user-defined camera movement.

A major limitation of all AI models based on the Transformer architecture is that they fundamentally imitate patterns found in training data, which limits their ability to "generalize," which means to apply those patterns to novel situations not found in the training data. To train Voyager, researchers used over 100,000 video clips, including computer-generated scenes from Unreal Engine—essentially teaching the model to imitate how cameras move through 3D video game environments.