/ILLUMINATE

ILLUMINATE

Representation and streaming technologies for hyper-realistic, 6-degrees-of-freedom immersive experiences

Back to overview

About the project

Project information

About the project

Project information

Immersive experiences and applications have become increasingly common, enabled by a growing range of sensor, camera and display technologies. Indeed, some of these experiences are now part of everyday life. For instance, many players in the property market offer virtual visits to 3D-reconstructions of houses with a variety of static 360° image viewpoints. Similarly, AR/XR applications (for monitoring and / or assistance) are used in healthcare, manufacturing, retail and automotive. And 360° movies and augmented/virtual experiences are also available in several social networks.

However, these new technologies are still quite restricted in how they represent and process data. They are also limited in the freedom of movement and associated immersiveness a user can experience while interactively consuming the multimedia content.

ILLUMINATE sought to significantly go beyond these limitations by focusing on technologies that enable hyper-realistic immersive experiences where the viewer can change their viewpoint as if they were actually in the location. The project also targeted a natural way of guiding the viewer towards the underlying storyline and what visually matters most in the scene.

ILLUMINATE had several specific goals. The first was to create hyper-realistic datasets and a movie staging a compelling story. Secondly, it aimed to develop a multimedia data representation that would be optimal for processing and streaming immersive experiences while offering high-quality and interactive visualization, something which is currently nonexistent. It also sought to create a streaming protocol that can handle low-latency viewpoint changes and requests, along with rendering and visualization proof-of-concepts for its new representations. Last but not least, the project investigated protocols for assessing the quality of experience (QoE) of interactive immersive experiences.

THE OUTCOMES

1. End-to-end proof-of-concept demonstrators

The work conducted in the ILLUMINATE project resulted in an end-to-end proof-of-concept demonstrating the full experience including streaming, along with seven other proofs-of-concept each focusing on a particular aspect of the end-to-end delivery chain. As yet, no commercial solution delivers immersive video streaming with the same degrees of freedom and hyper-realism as those demonstrated in the ILLUMINATE proofs-of-concept. In particular, the project delivered: - specific guidelines for storytelling in 6 DoF experiences - improved depth estimation for light-field cameras - AI-based view synthesis for free viewpoint generation - novel real-time depth-image-based rendering techniques - multi-plane images for view-dependent 6 DoF - a novel learning-based representation format for 6 DoF content that allows for real-time light-field rendering

2. Tools for assessing the quality of immersive experiences

In addition, the project created a new Quality of Experience model and procedure to assess the quality of both the immersion and storytelling. This protocol provides insights into which dimensions contribute most to the QoE and which system, context or user factors lead to significant fluctuations in these dimensions and, thus, the general perception of quality.

3. Resources for further research

ILLUMINATE also led to numerous research publications and new insights in the domain. These publications included eight journal papers and eleven conference papers. The project also generated datasets for the research community as well as Intellectual Property Rights (IPR) – in particular, a patent was filed. regarding a novel way to represent and interactively render content with 6 degrees of freedom.