Kenny Bellew’s Lip Synching Method for Streaming Audio

This document copyright © 2005, 2006 by Kenny Bellew of Cowfly.Com Design, kennybellew@hotmail.com

Kenny Bellew’s Lip Synching Method for Streaming Audio

Overview:
This method supplies the actionscript needed to create lip synch code that is produced by the tapping the mouse in synch with the desired mouth movement. You will listen to the audio and tap the mouse as it plays. Each tap will correspond to a movement of the mouth (each syllable). This will produce trace code in the output panel. You will cut and paste this code onto the movie that contains the mouth animation. Using this method, you can literally create lip-synch animation for a 2-minute song in about 5 minutes.

There are several ways to synchronize mouth movement to audio and several types of lip-synch animation. One method uses various mouth shapes that are displayed in relation to specific consonants and vowel sounds. Another method was popularized by Monty Python animated skits. It moves the lower lip and chin as one block. It is this method that will be described in this tutorial.

To see an example produced using this tutorial, view the following link:

http://www.kennybellew.com/tutorial/lipSynch

This tutorial will cover the following:

1. The code needed to create the trace that will be cut and pasted onto the mouth animation movie.

2. How to build a sample mouth animation.

3. Tips on preparing for the project and using the lip-synch tool described here.

4. How to reset the code so that it can be played over and over.
   

This tutorial deals with lip-synching to streaming audio that is loaded externally. This is significant because a popular method of synchronization involves embedding the audio in the timeline and activating the mouth animation at various points in a timeline that spans the length of the audio. The advantage to the embedded-audio approach is that you can see the audio position in relation to the timeline. The disadvantage is that audio can easily get out of synch with the animation. Also, the user has to wait for the audio to load.

Using my method, the animation cannot get out of synch because it uses the audio’s position (in milliseconds) to determine when the animation begins.

Building the Lip-Synch Tool

Note: The lip-synch tool is not the animation. It is the code to make the tool (Flash file) that will generate the code needed to animate the mouth.

1. Create a new Flash document large enough for a few buttons and text boxes (e.g., 600 x 400).

2. Create an empty movie clip (press CTRL-F8 to insert a new movie clip symbol) and name it whatever you like. I suggest emptyMc.

3. Drag an instance of this empty movie on stage. Place it wherever you like. I suggest in the upper left corner or off stage. You will place code on this movie.

4. Click the empty movie so that you can see its properties.

Paste the following into the movie’s Action panel. Be careful that you paste the code into the Action panel and not onto a timeline frame.

In the above code, a sound object is created called _root.soundObject. Into that sound object, we are loading sound.mp3 and setting it to streaming (that is what “true” does in the loadSound method).

Next, we set two variables to equal the position and the duration of the audio. Since these are measured in milliseconds, we divide it by 1000 to get seconds.

Next, we create code that runs each time the mouse is clicked. The basic function of the code tells the mouth animation to play if the audio position is greater than the position in seconds listed in the line of code. Obviously, you only want the animation to play once at that audio time position. To keep it from playing again, a unique variable is declared. The code is instructed to run only if the variable does not equal 1. As soon as the code runs, the variable changes to 1. A unique variable is obtained upon each mouse click by incrementing the value of “n” and adding n's value to the variable.

Notice that the value of “n” is incremented by 2 on each mouse click. The reason I suggest this is to allow for fine tuning later. You may wish to add a line, and this provides the flexibility in the numbering sequence. The following is what the trace will look like. I am only showing a few lines. There will probably end up being three or four lines of code per second of audio, depending on how fast the singer/orator is singing/speaking. Rap music will be fun.

I suggest that you use a higher than default (12 FPS) frame rate. A frame rate of 24 FPS works well.

Here is a sample trace that results from the code provided above:

In the first line of the above, it instructs that – If the audio position is greater than 2.168 seconds and root.doOnce4 does not equal one, the mouth animation should play. Then, it immediately sets the variable to 1 to keep it from playing the next time the code determines that the audio position is greater than 2.168. Later, if you want the user to be able to replay the animation, you will need to reset the _root.doOnce variable with a “for loop,” which I will describe.

Setup the Stage to See the Audio’s Position

This section continues building the lip-synch tool (versus the actual animation). The text boxes will be used to fine tune the animation.

1. Place three text boxes on stage. Make the first one an Input Text box and the other two Dynamic Text boxes.

2. Give the Input text box the variable name of “inputText” (without the quotes).

3. Give the first dynamic text box the instance name of “posText”.

4. Give the second dynamic text box the instance name of “durationText”.

5. Place two buttons on stage that can be used for starting and stopping the audio. Alternatively, draw a green circle and a red circle, press F8 when each is selected (separately) and make the images buttons.
   

6. On the Start Audio button, paste the following code.
   

   on(press) { if (_root.inputText == nul) { _root.soundObject.start(0, 0); // Start at position 0 and repeat 0 times. } else { _root.soundObject.start(_root.inputText, 0); // Start at the position that the user inputs into the input box } }//END

    

7. On the Stop Audio Button, paste the following code.
   

   on(press) { _root.soundObject.stop(); }// END

    

8. Place your mp3 file in the same folder as your SWF or change the path in the code where the audio is loaded. Also, make sure the name of the audio file matches the name in the code.

9. Test your lip-synching tool by pressing the play button.

Note: The main feature of the tool above is to produce a trace in Flash's output panel. This trace will be cut and pasted onto the movie that contains the mouth animation. Therefore, you will not be able to review the full functionality of the tool by viewing it outside of the Flash development space.

Tips on Using the Lip-Synching Tool

1. Listen to the song with a pen in hand before using the mouse to create the trace code. Record the start times of vocals that occur after lengthy interludes or time gaps between the vocals. This will help keep you from pressing the mouse button too soon.

2. You will need to produce the trace without stopping the audio. If you need to stop the audio, you will need to take into account the value of “n”, making sure that it is unique all the way through the code. I have found that most audio can be done in one shot with a minimal amount of practice.

3. Generally, you will click the mouse once for each syllable you hear. The more dexterous you are, the better your results.

4. Songs that have sections in which the singer holds a note for several counts can be handled by quickly clicking the mouse during the note hold. This gives the impression of a singer trying to produce vibrato during the note hold.

5. Cut and paste the code onto your animation (as I will show you) and test it. If you have more than 5 major errors, just redo the trace.

6. Watch the animation with the lip-synching tool you have created (text boxes and buttons on stage). When you see an error, press stop. This will freeze the audio position in the posText text box. Find that position in the code to make the correction. Corrections usually include deleting a line of code as well as adding one. To add one, copy the line above it and massage the position number up or down as needed. Make sure that the “doOnce” variable has a unique number. Notice that the “doOnce” variable is used twice in the line of code. Make sure you change it in both locations.

7. If you will be synching multiple animations, place the code for each on different movies or frames. Flash seems to have a hard time crunching that many lines of code on one movie at 24 FPS. At least, that was my experience. I ran into some really bizarre problems until I separated the code onto multiple movies.

Creating the Animation

To create the animation, you will need at least two images: the head and the lower part of the mouth and chin. To create the example, I brought a photo of my friend, GrimCity, into Photoshop and extracted the head. I used the marquee tool to select the rectangle that is the lower mouth and chin. I cut the mouth to a new layer and saved the head and mouth as separate PNG files with transparency.

Next, I imported these two files into a movie, placing each on its own layer. I selected the mouth and pressed F8 to turn it into a movie (the timeline shown below). I also added animated eyes with random blink, which I will cover briefly.

I made the mouth animation 6 frames, but for many animations, 3 frames is probably sufficient. The following images show the simple animation. In the "head" movie, which contains the "mouth" movie, I drew a dark area on the layer below the mouth. This way, when the mouth opens, you see the dark area instead of the background.

In the above example, there is a stop() on frame one. This is the timeline for the "mouth" movie which exists as a movie on the timeline of the "head" movie, which is on the _root timeline.

The following is the timeline for my "head" movie.

A brief note about the eye animations. To make these, each eye is its own movie. I probably could have made them a single movie just as effectively. I used eyes with good contrast so you can see the change at low resolutions (when the head is very small). They are big like manga eyes for this reason. Below is the timeline for the eye.

In Photoshop, I cloned the skin tone over his eye so the real eye would not peak through (no pun intended). You might chose to use a real photo of an eye open and closed, etc.

The following is the code I used on the "eye" movies. I placed the code directly on the "head" movie, which contains the "eye" movies.

In the above code, if you lower the number from 60, the eyes will blink more often. If you increase it, they blink less often. Keep in mind that this is affected by framerate.

Now you are ready to add the code (generated by the lip-synch tool) to the movie that controls your lip-synch animation (in my example, it is _root.head). I place my code on an empty movie clip. By habit, I always place these empty movies that have code on them just off stage in the upper left corner (see blue arrows below). In my timeline below, the _root timeline only has two frames. The preloader is one frame one, and all of the other movies are on frame two.

Place an empty movie on stage (or off stage), select the movie so you can see its properties, and then insert the following code onto the empty movie.

Run your lip-synch tool, copy the resulting code in trace and insert the code between the lines shown above. Below is what it will look like after inserting the code, however, I'm only inserting a small section of the code (the first 8 seconds) to save space.

Test your movie.

Replaying the Movie
In order to replay the movie, you must reset all of the _root.doOnce variables. To reset all of the variables, use a for loop that will loop at least as many times as you have variables. If your highest _root.doOnce is _root.doOnce979, then your Replay button might have the following code on it.

In the example, I use the same technique to activate the various other animations and events. This ensures that everything is in synch - down to the last millisecond. See the FLA for the code used to activate the other animations in synch with the audio.

This document copyright © 2006 by Kenny Bellew of Cowfly.Com Design, kennybellew@hotmail.com