Enhancing Capacity and Research Excellence in Holographic Telepresence Systems as a Catalyst of Digitalization

✦ National Program "European Scientific Networks"

Technical University of Sofia, through the Faculty of Telecommunications, is a beneficiary of the new National Program "European Scientific Networks", developed by the Ministry of Education and Science.

The aim of the program is to support universities and research organizations in the country, whose project proposals under the measure "Twinning" of the Framework Program for Research and Innovation of the European Union "Horizon 2020" are highly valued, but have not received funding due to lack of funds.

The program will be implemented in the period 2020-2022 and will enable 13 Bulgarian state universities and research organizations to receive funding for their project proposals. Through national funding, Bulgarian organizations aim to maintain the established partnerships, build new ones and participate in existing scientific networks.

In the program TU-Sofia, through the Faculty of Telecommunications, participates with the project "HOLOTWIN" in the field of "multisensory" communication or so-called holoportation. Holoportation is a new technology based on augmented, mixed and virtual reality technologies, which offers the opportunity to include all five human senses in the communication process to make it more realistic for all participants.

✦ Bee Cube Architecture for Holographic communication

Conceptual architecture for a two-way real time holoportation system

Conceptual model of the Bee-cube environment for holoporation including all sensors deployment for two-way communication where participants from both locations where VR/AR glasses and data is sent both ways

Holographic Telepresence

✦ What is holographic telepresence?

Holographic telepresence is a complete human-computer-machine interface combining AR and VR telepresence thus creating real-life digital scans and realistic 3D avatars of subjects, displayed in MR environment, used for real time communication and interaction between remote users.

Immersive AR/VR telepresence offers a shared virtual space that people can join and experience together from anywhere.

Two are the essential requirements for successful telepresence system: a shared person space and a shared task space.

These spaces must be coherently integrated in order to provide a realistic communication environment for geographically distributed users.

An ideal 3D telepresence system should offer eye contact among participants, mutual gaze awareness and the sense of identity of the speaker ("me") and the speaker's location ("here") or "spatial faithfulness".

✦ Current state of the art

Limitations of the communication channel when streaming the huge amounts of data generated by the capture process.
Even with broadband channels, streaming big amounts of data will lead to a latency which will influence the natural perception of the communication.
In the initial holographic telepresence system presented in 2016, a capture of a 3D frame including mesh geometry is about 5MB with compression so for real time performance, an average per frame transmission of 2-3Gbit/sec with overhead for compression (< 10ms) is a must.
TO DO: Developing an intelligent way to "compress" the data and exploit the potential of 5G and 6G are the main challenges of long distance real time holoportation.

✦ VR and AR: connectivity requirements

More capacity, lower cost: Increased throughput per user as quality of immersion improves, and more simultaneous usage;
Low latency: Reduces throughput requirements, buffering requirements, and lag for interactive content like tactile Internet and 6 DoF* (Six degrees of freedom);
Uniform experience: Full immersion everywhere requires consistent throughput, even at the cell edge;
Richer visual content: Higher resolution, higher frame rate, Stereoscopic, High Dynamic Range (HDR), 360^o spherical content, 6 DoF.

The "Quality" of a hologram involves not just color depth, resolution, and frame rate as in video, but it also involves the transmission of volumetric data from multiple viewpoints to account for shifts in tilt, angle, and position of the observer relative to the hologram ("Six Degrees of Freedom").

The streaming of underlying volumetric data and image arrays imposes additional synchronization requirements to ensure smooth viewing transitions for the user.

VR and AR: connectivity requirements

Low latency: critical for immersion

The Project

✦ Main Goal

HOLOTWIN aims to strengthen and stimulate the research excellence and innovation capacity of the Technical University of Sofia (TUS) in technologies for holographic telepresence systems (HTPS).

The twinning will activate a process of knowledge transfer towards TUS (Bulgaria), which leverages on the academic and research excellence at AU (Denmark), UNiS (UK) and Hellenic American University (Greece) in the chosen area of research with the possibilities to extend beyond the consortium organizations.

The knowledge transfer will be enabled by mentoring, access to research results, networks and experimental infrastructure available at the three supporting organizations with the objective to maximize joint and individual scientific success and to jointly identify, develop, and assess good and transitional practices.

The plan of activities includes: research visits, joint brain-storm meetings, research and entrepreneurship workshops, cross-functional mentoring and training.

✦ Planned activities

Joint Workshops;
High-impact journal publications;
Expert visits to Bulgaria;
Entrepreneurial workshops;
Conference papers;
Participation in conferences and Special Session Chairing.

✦ Scientific goals

✔ 3D human body model acquisition and human avatar creation

Design a calibration procedure that allows us to obtain reliable data from the RGB-D devices, and to use multiple devices in a coherent reference frame;
Reconstruction and parameterization of a 3D human body model from RGB -D data;
Multi-view deformable 3D models of face and hands;
Design and simulation of realistic clothing.

✔ Real-time 3D avatar animation and rendering

Methods for semantic annotation and prediction of human activities;
Integration of sematic aspects into strategies for synthesis of biometric data;
Methods and algorithms for real-time photorealistic animation of the avatar's body and head movement;
Deep Neural network-based rendering of the 3D avatar.

✔ Context-dependent semantic models for holographic communication

Measurement of semantic information and compression of semantic data;
Use case scenario - Bee cube holographic telepresence.

Partners