This article is cited in 3 papers

Efficient methods and algorithms to synthesize 360-degree video based on cubemap projection of virtual environment

P. Yu. Timokhin, M. V. Mikhaylyuk, E. M. Vozhegov

Scientific Research Institute for System Analysis of RAS

Abstract: The paper deals with the task of creation and playback of panoramic video with 360-degree overview, which allows the researcher to be immersed in virtual medium outside parent virtual environment system (VES). To solve the task, the extension of the cubemap method is proposed, in which cubemap resolution is determined taking into account viewer camera field of view and screen resolution (Adequate Cubemap Projection, ACMP). The paper studies the influence of the camera orientation inside the cube on the “cubemap pixel /~screen pixel” ratio determining panorama visualization quality. Based on this, a method to calculate cubemap resolution for high-quality panorama visualization for all possible camera orientations is proposed. The paper considers an efficient method and algorithm to create ACMP-video on the GPU using render-to-texture technology, which allow to synthesize panoramas with constant orientation or bound to the observer's view direction. In the research efficient methods and algorithms to play ACMP-video are also proposed, which are based on the visualization of visible cube faces and adaptive frame buffering. The obtained methods and algorithms are implemented in ACMP-video synthesis program complex (С++, OpenGL, FFmpeg) including frame capture module (embeddable into VES) and the player. The developed solution was tested in system «Virtual Earth» designed for training to observe Earth objects from the International Space Station (ISS). Using the capture module, an ACMP-video of the flight along the ISS orbit track was created. When playing this video, the trainee flies in orbit above virtual 3D surface of the Earth and can explore it by means of camera rotation. Testing of the complex confirmed the adequacy of the developed methods and algorithms to the task. The obtained scientific and practical results expand the capabilities and scope of application of VES, scientific visualization systems, video simulators and virtual laboratories; provide effective exchange of experience between researchers, etc.

Keywords: 360-degree video, omnidirectional video, cubemap projection, visualization, virtual environment.

DOI: 10.15514/ISPRAS-2020-32(4)-5