Understanding holistic panoramic 3D scenes using spherical harmonics

Advanced artificial intelligence (AI) technology has been developed that allows you to extract 3D (3D) spatial structure and object information within an indoor environment using a single 360-degree panoramic photograph. This breakthrough is expected to have a significant impact on fields that require accurate spatial understanding, such as augmented reality (AR), mixed reality (MR), and digital twin applications.

A research team led by Professor Kyungdon Joo of the Graduate School of Artificial Intelligence at Unist introduced Hush (understanding the overall panoramic 3D scene using spherical harmonics).

Integrating digital content with real space requires AI systems to accurately interpret and represent information such as the location of walls and furniture, as well as distances between objects. Traditionally, achieving this level of understanding requires multiple images from different angles, such as depth sensors, and from expensive equipment.

The Hush model goes beyond these limits by utilizing only a single 360-degree panoramic image to derive this information. Panoramic images can capture a wider scene with a single shot, but their spherical distortion makes accurate analysis difficult. Traditional methods attempt to mitigate this by segmenting images and repeatedly applying standard AI models, but this often results in loss of information or inefficiency in computation.

To address these issues, the researchers adopted Spherical Harmonics (SH), a mathematical technique that accurately models the spherical properties of panoramic images. This method breaks down the scene into frequency components. Low-frequency components effectively represent a wide range of flat areas such as ceilings and floors, while high-frequency components capture detailed structures such as furniture and objects, thereby improving accuracy.

Jung-seung Lee, the first author of the study, said, “Spherical harmonics are traditionally used in virtual view synthesis to represent the color and illumination of objects and scenes. Recognizing the ability to analyze data on spherical surfaces, we have applied SH in the first place innovate.

The Hush model showed superior accuracy in depth prediction and other spatial understanding tasks compared to existing 3D scene reconstruction models. Surprisingly, multiple spatial details can be inferred from a single image, providing both high performance and computational efficiency.

Professor Joo said, “This technology has a wide range of applications in real-world scenarios where accurate understanding of indoor spaces is essential, such as AR and MR environments, or create immersive media that allows users to interact with one image.”

This study was presented at CVPR 2025 (Conference on Computer Vision and Pattern Recognition) held in Nashville from June 11th to 15th, 2025.