Anatomical Mirroring: Real-time User-specific Anatomy in Motion Using a Commodity Depth Camera

Armelle Bauer, Ali Hamadi Dicko, François Faure, Olivier Palombi, Jocelyne Troccaz

Motion in Games (MIG'16), October 2016, San Francisco, United States

Abstract : This paper presents a mirror-like augmented reality (AR) system to display the internal anatomy of a user. Using a single Microsoft V2.0 Kinect , we animate in real-time a user-specific internal anatomy according to the user's motion and we superimpose it onto the user's color map, as shown in Fig.e. The system inputs are the Kinect SDK outputs (the user's : color map (Fig.a), body traking skeleton (Fig.b), point cloud (Fig.c)) and a 3d generic anatomical model (Fig.d). The output is a user-specific 3d anatomy following the user motion. The user can visualize his anatomy moving as if he was able to look inside his own body in real-time.

A new calibration procedure to set up and attach a users-pecific anatomy to the Kinect body tracking skeleton is introduced. At calibration time, the bone lengths are estimated using a set of poses. By using Kinect data as input, the practical limitation of skin correspondance in prior work is overcome. The generic 3D anatomical model is attached to the internal anatomy registration skeleton, and warped on the depth image using a novel elastic deformer, subject to a closest-point registration force and anatomical constraints. The noise in Kinect outputs precludes any realistic human display. Therefore, a novel filter to reconstruct plausible motions based on fixed length bones as well as realistic angular degrees of freedom (DOFs) and limits is introduced to enforce anatomical plausibility. Anatomical constraints applied to the Kinect body tracking skeleton joints are used to maximize the physical plausibility of the anatomy motion, while minimizing the distance to the raw data. At run-time, a simulation loop is used to attract the bones towards the raw data, and skinning shaders eficiently drag the resulting anatomy to the user's tracked motion.

Our user-specific internal anatomy model is validated by comparing the skeleton with segmented MRI images. A user study is established to evaluate the believability of the animated anatomy.

Documents : slides, hal-01366704, video

Keywords : User-specific anatomy, Augmented Human, Real-Time, Motion Capture, Augmented Reality, Markerless Device.