Toward Natural Gesture Synthesis: Gesture Modeling and Animation Based on a Probabilistic Recreation of Speaker Style

Animated characters that move and gesticulate appropriately with spoken text are useful in a wide range of applications. Unfortunately, this class of movement is very difficult to generate, even more so when a unique, individual movement style is required. We present a system that, with a focus on arm gestures, is capable of producing full-body gesture animation for given input text in the style of a particular performer. Our process starts with video of a person whose gesturing style we wish to animate. A tool-assisted annotation process is performed on the video, from which a statistical model of the person's particular gesturing style is built. Using this model and input text tagged with theme, rheme and focus, our generation algorithm creates a gesture script. As opposed to isolated singleton gestures, our gesture script specifies a stream of continuous gestures coordinated with speech. This script is passed to an animation system, which enhances the gesture description with additional detail. It then generates either kinematic or physically simulated motion based on this description. The system is capable of generating gesture animations for novel text that are consistent with a given performer's style, as was successfully validated in an empirical user study.

A cooperation with MPI Informatik and UC Davis. This research was partially funded by DAAD under the US-PPP programme.

For follow-up work on this project we won the Best Paper Award at the IVA 2007 conference. Bild der Wissenschaft reported about our gesture research in the 05/2008 issue.

Related publications:

Luo, P., Kipp, M., Neff, M. (2009) Augmenting Gesture Animation with Motion Capture Data to Provide Full-Body Engagement. In: Proceedings of the 9th International Conference on Intelligent Virtual Agents (IVA-09), Springer.

Neff, M., Kipp, M., Albrecht, I. and Seidel, H.-P. (2008) Gesture Modeling and Animation Based on a Probabilistic Recreation of Speaker Style. In: ACM Transactions on Graphics 27 (1), ACM Press, pp. 1-24.

Kipp, M., Neff, M., Kipp, K.H. und Albrecht, I. (2007) Toward Natural Gesture Synthesis: Evaluating gesture units in a data-driven approach. In: Proceedings of the 7th International Conference on Intelligent Virtual Agents, LNAI 4722, Springer, pp.15-28.

Overview video (.mp4, 30MB): Shows animation from annotation and gesture synthesis on novel text input.
Style comparison (.mp4, 5MB) Shows two variations of synthesized gestures on the same text.

Back to projects