Direction des Relations Européennes et Internationales (DREI)

Programme INRIA "Equipes Associées"

I. DEFINITION

This is the first year of this new collaboration. Based on previous research activoties, exchanges have been mainly devoted to assign cross participation of senior researchers as co-supervisor of PhD students. The main outcomes of this joint teams agreement have been to collaborate on the following projects :

1. Re-use of 3D facial animation
Participants: Jacobo Bibliowicz (PhD student, DGP), Karan Singh (DGP), Christine Depraz (EVASION) and Lionel Reveret (EVASION)

The mapping of live facial motion capture data to the animation of a 3D character can be very difficult when there are large differences of morphology between the human performer and the virtual actor. The goal of this project is to explore a new approach to solve this problem by automatically learning correspondences between motion capture data collected on a human perfomer and facial motion created by a professional animator for a 3D character. The DGP has collaborated on the animated short movie Ryan (oscar awarded in 2004) for the making of some special effects. Thanks to this participation, the DGP has access to large sequence of facial animation of 3D character. During summer 2005, we used the motion capture equipement available at DGP to record a human performer dubbing some part of the facial animation sequences. We thus have collected both real data from the performer and corresponding artistic data from the original animation. Based on work done by Reveret et Essa, 2001, some parameterization of facial motion will be extracted for both sources of data in order to latter learn a high-level bridge between these two sources. For the moment, only the parameterization of facial motion dedicated to speech production is considered. Works are under progress to learn a separate parameterization for the emotional part of the facial motion. Christine Depraz, who is an engineer working at EVASION, participated in this project by providing additional sequences of facial animation of a virtual character.

2. 3D mesh decomposition using anatomically constrained hierarchy of ellipsoids
Participants: Laurent Favreau (PhD student, EVASION), Patricio Simari (PhD student, DGP), Karan Singh (DGP) and Lionel Reveret (EVASION)

Laurent Favreau is doing a PhD under the co-supervision of Lionel Reveret and Marie-Paule Cani. His work is dedicated to the 3D modeling and animation of animals in motion from video. In 2004, some results have been obtained to automatically extract kinematics information from video during standard locomotion (Favreau et al., 2004). In order to address dynamic aspects of animal motion such as jumps or influence of the animals' weight on locomotion, it is now necessary to incorporate information about inertia of body parts. Recently, Patricio Simari and Karan Singh have proposed a method to optimally approximate any 3D mesh with a set of 3D ellipsoids. When applied to 3D animal meshes, this decomposition turned out to provide interesting cues to compute inertia of body parts, since ellipsoids offer simplification to the computation of volume inertia. We are currently working on this method to add more constrains on a hierarchy of elliposids so that it follows better anatomical structure related to the animals body shapes.

3. Shape analysis by automatic detection of 3D symmetries
Participants: Patricio Simari (PhD student, DGP), Karan Singh (DGP), Marie-Paule Cani (EVASION)

The topic of Patrico Simari's PhD thesis, under the joint supervision of  Karan Singh and Marie-Paule Cani, is the search for symetries and for structural correspondances in 3D geometric models. The general approach, set up during our meeting in june in Boston (at the international conference "IEEE Shape modeling and applications"), is to unfold geometric models based on skeletons to recover structural symetries. Then, model edition and correspondances will be computed in the unfolded space.  Patrico is currently working on a first version of this method in 2D, before generalisation to 3D. Such methods will provide fundamental improvement in the compression of 3D geometry or in 3D surface representation for automatic texturing.

4. 3D modeling and animation of trees and wind from analysis of video data
Participants: Julien Diener (PhD, EVASION), Lionel Reveret (EVASION), Fabrice Neyret (EVASION), Eugene Fiume (DGP)

The goal of this project is to make use of video data to automatically extract both shape and motion information to model and animate 3D trees. At DGP, Eugene Fiume has previously investigated a method based on physical model to evaluate from the video of moving trees, cues on the wind velocity. This approach gives good information about the environnement constraint, allowing to realistically introduce new virtual trees composited on top of the orignal video, moving with a motion controlled by the wind velocity automatically extracted. Recently, Julien Diener and Lionel Reveret explored another approach based on statistical analysis of video of trees moving with the wind, which allowed to automatically extract geometrical information on the tree 3D structure in terms of shape and degrees of freedom. Julien Diener is now starting a PhD under the supervision of Lionel Reveret and Fabrice Neyret. For this project, the collaboration between DGP and EVASION will be to combine both approaches to automatically learned some physical properties of the trees and wind. A publication is currently under preparation with Eugene Fiume on the work Julien Diener achieved during his master.

5. 3D reconstruction of animals skin surface by learning fur reflectance from video
Participants: Lionel Reveret (EVASION), Aaron Hertzmann (DGP)

This project is not supported by any specific PhD student yet. However, Aaron Hertzmann has developped some impressive methods to automatically learn the reflectance properties of fur material such as velvet in order to recover 3D shape from changes of illumination. This problem is particularly challenging as fur material present a lot of specularities, which make imossible to apply standard shape-from-shading or stereographics methods where the material is supposed to be lambertian. These results are particullarly well adapted to the current projet of the EVASION group with the Museum National d'Histoire Naturelle on the measurement and the 3D modeling of real animals to collect information for 3D animation of virtual creatures.

6. Automatic creation of 3D models from hand-drawn sketches
Participants: Alexis Angelidis (post-doc, DGP), Jamie Wither (PhD, EVASION), Karan Singh (DGP), Marie-Paule Cani (EVASION)

This project is handled by Alexis Angelidis who did his PhD under the co-supervision of Marie-Paule Cani. Thanks to the co-support of the joint teams agreement I-MAGE for mission expense and a grant from the University of Toronto for the salary, Alexis is now starting a postdoctoral stay at the DGP to work on the rapid 3D prototyping of 3D models from the automatic analysis of hand drawn sketches. The approach is to learn the correspondences between a database of 3D models and main sketch strokes which can be intuitively associated. In adition, Jamie Wither will start a PhD under the supervision of Marie-Paule Cani on this subject. Jamie Wither will actively participate in this project relevant to the I-MAGE objectives. He is sponsored by a european Marie-Curie PhD grant from the VISITOR programm.

7. Interactive shape modeling
Participants: Alexis Angelidis (post-doc, DGP), Karan Singh (DGP), Marie-Paule Cani (EVASION)

Karan Singh and Marie-Paule Cani have been working, in collaboration with Marc Alexa from TUD, Germany, on a joint post-grade course on Interactive Shape Modeling, which includes both sculpting and sketch based approaches. Set up to be held at international conferences and at summer schools, this course already took place twice: a long 5 days version of it took place in Darmstadt, Germany, in july, and a shorter one day venue was within he tutorial programm of the EUROGRAPHICS'05 conference (Dublin, Ireland, sept 2005). Alexis Angelidis has been one of the speakers of both issues. The plans are now to submitt a new version of this course to the SIGGRAPH tutorials programm (deadline in december). This course project has been partly supported by the European Network of Excellence Aim@Shape to sponsor a summer school on surface parametrization at Darmstaad in Germany where both Marie-Paule Cani and Karan Singh have been invited. The NoE specifically helped to partially fund one of the visit of Pr. Karan Singh in Europe. This has been an opportunity for the joint team I-MAGE to host Pr. K. Singh for one week in Grenoble, covering only staying expenses, travel being covered by the NoE Aim@Shape.

III. PREVISIONS 2006

Programme de travail

1. Objectives

During year 2005, several projects have been initiated and all will be continued during year 2006. We have searched to assign each project to a dedicated PhD student or postdoctoral researcher. This has been succesfull for 6 out of the 7 projects listed in the previuous section. People involved in each project are motivated in the continuation of these projects and are candidates for continuating exchanges between DGP and EVASION. This motivates us to sollicitate a contiuation of the joint teams agreement I-MAGE for the year 2006. As a context framework, we recall here the main objectives and the guideline methodology originally described for the goal of this joint teams agreement.

Our teams share the belief that intuitive design interfaces for 3D tools will lead to substantially improved productivity and usability by artists. We believe that more design tools can be created by extracting 3D shapes and motions information from video sequences of real scenes and hand-drawn sketches. This project aims at designing more intuitive and expressive human-computer interface for the creating of 3D virtual worlds. In particular, our main contribution will be to combine automatic video processing with hand-drawn sketching as annotations. By creating new interfaces for sketching 3D shapes and motions – and leveraging the rich information present in existing video footage – we will create interfaces that allow fast and expressive creation of 3D virtual worlds, especially well adapted to the design of complex natural scenes.

2. Guideline methodology
The main idea of this project is to integrate sketches and annotations from video sequences in order to extract shapes and motions from video and to create new ones. Sketches and annotations of video will be used as input for the automatic system in two ways:
- to provide outlines of shape and motion directly (the user gives a 2D contour and directions of motion, the computer automatically deduces the 3D shape and animation),
- to serve as focal cues for a video processing system (the user roughly designates the moving object on the first frame, the computer automatically segments object shape and tracks its 3D motion from video).
The guideline methodology will be to ask the user to provide only 2D information. A priori 3D models of shape and motion will be instantiated by automatic inference from 2D information cues. A 2D projection of the 3D models and animations will be proposed to the user through a rendering phase. Automatic change of 3D view or advance in time after tracking will be applied – change of view in video sequence could also be supported by tracking. The user will be allowed afterwards to provide new annotations in 2D to correct models and results from the automatic inference phase. This way, our approach allows a user to balance interactivity and automation as necessary. With this methodology, we formulate the creation of 3D worlds as a converging dialogue between the user and the automatic system.

3. Long term plans

Some long term plans have been discussed for the future of this collaboration in additon to the on-going projects listed in the previous sections. In particular, two main projects have been mentionned.

3.1. gestural interface from standard video camera
The goal of this project is to combine intuitive 3D modelling tools with the tracking of hand gestures. We plan to rely on learning a priori class of hand motion gestures recorded by motion capture equipment available at DGP, on a subject performing sculpting on real material. These classes of gestures will be later used to constrain and enhance robustness of the video analysis of hand gestures, with no markers. Alexis Angelidis has already work on intuitive interface for 3D modelling and will investigate this project.

3.2. evaluating the realism of 3D animation by statistical comparaison of real and synthetic images
The goal of this project is to investigate how the realism of a 3D animation can be quantified in terms of perceptual signifcance by comparison with real images. This idea is inspired by some results of Favreau et al., 2004 where it has been shown that common high-level statistical properties can be extracted both from real and snthetic images of animals in motion. These properties could be used as cues to quantify the main required features of a perceived motion. We plan to investigate this idea to evaluate the generated motion of a synthetic controller for 3D animation of a swimmer done by Yang et al. at DGP.