First demonstrations of the capabilities of real-time 3D modeling for virtual reality applications were quite poor on the immersion side. People were able to interact with objects with their own body but they had to look at themselves on a screen. Interactions were difficult. Users had poor sensation of depth and space and their point of view on the scene had to be controlled separately. We decided to incorporate a HMD inside our setup to have first person point of view.
During our first experiments with HMD and 3D Modeling we came to realize that while the quality of 3D reconstruction is good enough for 3rd person visualization it is not perfect for 1st person. In fact, we perform most of our actions with our hands and we tend to look at them while doing those actions. The quality of the hand 3D reconstruction and of the texture mapped onto it strongly depends on the configuration of the cameras around the user, their resolutions or their lenses. To avoid those facts and to enhance the user's feeling of seeing his own body we decided to mount a camera on top of our HMD and to use it for texture mapping and 3D reconstruction.
Having the ability to reconstruct in 3D and in real-time anyone inside our 3D reconstruction platform opened the way to telepresence applications. We wanted to join 2 persons located in distant sites in a common virtual world where they can see themselves in 3D and interact together with a common virtual world.
Interaction with 3D reconstruction were highly limited to detecting collisions inside a physics simulation engine and perform actions according to those collisions. Furthermore, at each reconstruction step we have a new mesh as input of the simulation which is completely independent from the previous one: there is no temporal consistency between two iterations. During this project we are trying to get some motion information in real-time in order to have more accurate simulation and to be able to track points in time. To do so we implemented a method to recover dense 3D surface flow from 2D and 3D features correspondances and dense normal flow.
