Sashariri - A Talking Face

Sashariri is being developed as a plugin to Art of Illusion - a 3-D modeling and animation package. It is developed in Java.

The aim was to create a Text-To-Speech-Animation framework in Art of Illusion (AOI). Till my tenure at CDAC, we achieved the following :-

Linked Two Text-to-Speech (TTS) engines with our package. One was for English and the other for Indian Languages (Indic Script - Unicode). We also extracted the phoneme symbols used in these TTS systems to create the mapping scheme.
Created a generalized phoneme to viseme mapping scheme that the loads visemes listed by the animator and allows them to match the visemes with the phonemes of the language. This mapping scheme is saved as an XML file. This tool is run independent of Art of Illusion.
Created a generalized viseme to gesture mapping scheme. This tool maps the visemes codes to the actual gestures of the model created by the animator. It is embedded in Art of Illusion to retrieve the different gestures of a model created by the animator.
Created an in-built component in Art of Illusion that allows the user to type a sentence for a given face (or any object created by the animator with multiple gestures). Once executed, this system used the mapping schemes' XML files to keyframe the gestures corresponding to the visemes/phonemes in the sentence. These gestures are placed in a timeline based on the timing information received from the TTS.
Run a preview of the resultant animation, synced with sound. The Java Sound API was used for this.

For testing purposes we used existing facial model. The movement of the lips to signify various gestures was done by placing 'skeletons' (a feature in AOI that allows the animator to control and deform the mesh). Several gestures were created by deforming the mesh using this. We created a set of 14 gestures, corresponding to 14 visemes - the MPEG 4 standard. However, using the mapping scheme tool, the animator can specify and create his/her own gestures and visemes.

The path taken...

My first task was to understand how Art of Illusion works. I played around with a few modeling tutorials. Using the screwdriver tutorial, I managed to create something 'reasonable' :-

I then moved on to learning the internals of the software. It has a scripting language built-in called Beanshell, which is a simple Java-like scripting language. I played around with it a bit and along with my mentor, Mr Vinod Kumar, we created a tutorial. It was a very basic tutorial that discussed the many ways one could simulate a bouncing ball animation in Art of Illusion, both procedurally and using the scripting language. We posted the tutorial on the AOI Wiki. Click the image to be redirected to the site.

We then moved on to the task at hand. We took an existing facial model and created several gestures for it using AOI's skeleton system.

Finally the tasks above were implemented to create a working prototype of the project.

My Role

The tasks I completed were

Implemented both the mapping schemes. We used JformDesigner to create the GUI. I worked along with a summer intern in this part.
Linked the third party TTS systems
Plugged-in our package to Art of Illusion.
Implemented the ability to preview the resultant animation in Art of Illusion. This step involved caching the images before playing them, in order to synchronize the animation with sound.

A research paper, which explains our work was submitted to the Computer Society of India and was subsequently accepted.

My partner in this project, Mayank Madhav presented this paper. Unfortunately, I had joined RIT by the time this event occurred.

Here a few screenshots of the preview animation window. The input sentence is "How are you."

Back to Programming page