AI Lip Sync Explained: Make a Still Photo Talk

By the end of this post you'll know how AI lip sync works, which of two routes suits your project, and how to prepare a photo and a script so the mouth actually matches the words.

What AI lip sync actually does

AI lip sync means generating mouth movement (and ideally the rest of the face) that matches a piece of speech. The input is audio. The output is video where someone appears to say it.

There are two broad families, and knowing the difference saves a lot of wasted runs.

• Lip-sync models that fix the mouth. They take an existing video and new audio, then redraw the mouth area to match. The classic example is Wav2Lip.
• Audio-driven portrait models. They take one still image plus audio and generate the whole moving head: lips, jaw, brows, blinks, head tilt. Research systems like EMO and OmniHuman work this way.

A third route has become common for creators: video models that generate speech and picture together from a prompt. More on that below.

How the older approach works

The Wav2Lip paper (Prajwal et al., ACM Multimedia 2020) is a good starting point because it explains why early lip sync often looked wrong. The authors argue that earlier methods struggled to match lips for arbitrary people in unconstrained video, so parts of a clip ended up out of sync with the audio.

Their fix was to train the generator against a strong, pretrained lip-sync judge, rather than relying on weaker signals. A separate quality discriminator then helps keep the face from going blurry. The authors report lip-sync accuracy that's almost as good as real synced video, but note that this is measured on benchmarks and metrics they introduced themselves. Treat it as a research claim, not a promise about your footage.

What this means in practice: this family is good at one job, matching the mouth to speech in a video that already exists. It doesn't invent a performance from a single photo.

How a still photo gets to talk

For a single photo, the model has to invent everything that moves. EMO from Alibaba takes a single reference image and a vocal audio clip and generates the video directly, without intermediate 3D models or facial landmarks. The video length follows the audio length. The authors say it keeps identity consistent and works on styles such as anime and 3D even though it was trained on realistic video.

OmniHuman-1 pushes this further. Per its abstract, it supports talking and singing, several image styles, and framings from face close-up to full body, and it accepts audio, video or both as driving signals. The authors claim more realistic output than earlier end-to-end audio-driven methods.

Both papers are honest about trade-offs, and it's worth knowing them:

• EMO's authors say head-motion speed control only approximates the target, and that very fast settings can look jittery.
• They also say generation is slow: about 15 seconds for one batch of 12 frames in their setup.
• On their internet-data test, EMO didn't reach the top SyncNet lip-sync score. They note Wav2Lip scored higher on that measure. Expressive, natural motion and tight lip accuracy can pull against each other.
• OmniHuman's abstract reports no numbers, so its "more realistic" claim is the authors' own.

Two practical routes for creators

You don't need to run research code. Which route fits depends on what you're making.

Route 1: generate speech and picture together

Some video models can produce spoken dialogue as part of the clip. Models such as Veo 3.1 are in this group, and the studio lists a range of video models that change over time, so check which ones you can pick. Sound is covered in detail in Veo 3.1 Native Audio: How to Prompt Sound and Speech.

A prompt in this style:

Medium close-up of a woman in a bright kitchen, facing the camera. She says: "Three ingredients, ten minutes, and no cleanup." Soft window light, steady camera, natural gestures.

Keep the spoken line short and put it in quotes. Whether the result matches your exact words varies run to run, so plan on a few attempts.

Route 2: start from a still, then animate

If you already have the exact portrait you want, begin with image to video. The start frame matters a lot, and Image to Video: Pick a Start Frame That Animates Well goes through what to look for.

In Mediafy Studio you can make the portrait with an image model, then send it to a video model and describe the speech and motion. If you plan to reuse one face, Keep a Consistent AI Character Across Images and Video covers how.

Prepare the photo

Most bad talking-photo results come from the source image, not the model.

1. Face the camera, or nearly. Front-facing or a slight three-quarter angle gives the model the most information about the mouth.
2. Closed or gently parted lips. A wide grin or a mid-word expression gives the model a strange starting point.
3. Visible mouth. No hand, microphone, scarf or heavy shadow across the lips.
4. Clear, even lighting. Strong shadows under the nose or chin can flicker as the jaw moves.
5. Enough resolution on the face. If the face is tiny in the frame, the mouth detail will be mushy. A close-up or medium close-up beats a wide shot. AI Image Upscaling: What It Fixes and What It Can't explains the limits of fixing a soft face afterwards.
6. One person. Group photos make it unclear who should speak.

Prepare the audio or script

• Keep it short. Short lines are easier to keep in sync and easier to redo. Cut long scripts into separate clips and join them in an editor.
• Clean recording. Background music or room noise makes audio-driven models guess. Use a dry voice track and add music afterwards.
• Natural pacing. Very fast or monotone delivery tends to give fast or flat faces. Read it aloud before you generate.
• Write for the ear. Contractions, short sentences, no tongue twisters.

For timing a spoken ad, you'll want to know roughly how many seconds your words take. Read the script aloud with a timer and note it before you generate.

Prompt the performance, not just the words

The audio sets the mouth shape. Your prompt shapes everything else. Describe the delivery:

Facing the camera, warm and relaxed, small nods on key words, blinks naturally, shoulders still, camera locked off.

A locked camera helps a lot. Moving the camera and the face at once makes it harder to tell where drift comes from. If you do want movement later, see Camera Movement Prompts for AI Video: Dolly, Pan, Orbit.

Common mistakes

• Over-long clips. Face identity and sync tend to wander the longer a clip runs. Shorter takes are safer.
• Quoting a paragraph. One or two sentences per clip.
• Asking for big action at the same time. Walking, turning and talking together raises the odds of a warped mouth.
• Judging with the sound off. Always check lips against audio at full speed, then frame by frame on any mouth shape that looks wrong.
• Expecting exact words every time. Where a model generates its own speech, it may change or garble a word. Re-run, or shorten the line.
• Skipping disclosure. A talking face of a real person who didn't say those words can mislead viewers. Use your own likeness or people who've agreed, and follow each platform's rules on synthetic media.

Using it for marketing videos

If your goal is a presenter-style ad rather than a one-off experiment, UGC Factory in Mediafy Studio handles the chain for you: pick a character and a product, and it writes a short script, makes a selfie-style image, lip-syncs it into a talking video and adds subtitles. You still need to review the result and regenerate if the mouth or script isn't right.

For a custom pipeline (portrait, then video, then voice), Flows in Mediafy Studio: Chain Image, Video and Voice shows how to link the steps on one canvas.

Each model has its own credit cost, shown on the Generate button before you run it, and failed generations are refunded automatically. Since talking clips often need several takes, check the cost first. How Mediafy Credits Work: Costs, Packs and Refunds explains it.

A simple workflow to try

1. Write one or two spoken sentences, under about ten seconds when read aloud.
2. Make a front-facing, well-lit portrait with a neutral mouth.
3. Animate it with a locked camera and a short delivery description.
4. Check with sound on. If the mouth drifts, shorten the line or change the start frame.
5. Make the next line as a separate clip, then edit them together.

What lip sync can't do (yet)

It can't guarantee identical results twice, can't rescue a blurry face, and tends to struggle with profile views, hands near the mouth and fast head turns. The research above shows the same pattern: sync accuracy, expression and speed are each being traded against one another.

Ready to try it? Sign in at /signin and run your first talking clip.

Sources

• Prajwal, Mukhopadhyay, Namboodiri, Jawahar. arXiv. "A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild" (arXiv:2008.10010). https://arxiv.org/abs/2008.10010
• Tian et al., Alibaba Group. arXiv. "EMO: Emote Portrait Alive: Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions" (arXiv:2402.17485, v3). https://arxiv.org/html/2402.17485v3
• arXiv. OmniHuman-1 abstract (arXiv:2502.01061). https://arxiv.org/abs/2502.01061