Activity, emotion, mouth motion and gaze solve different problems. Plan how they work together so an interrupted reply or failed connection leaves the character in a clear, usable state.
Written for: AI startup founders, conversational product designers and technical leads.
Define the conversation lifecycle first
Start with what your product can actually report: recording started, recording ended, a request is pending, audio playback began, playback stopped, a connection failed. Those are application signals. “The character looks thoughtful” is the visual interpretation you design around them.
Keep visual feedback consistent with the actual product. A speaking pose should not imply that audio is playing before playback starts. A listening pose should not imply that the microphone is active when it is unavailable. Essential recording and connection information also needs accessible UI text.
Use an activity table to expose missing decisions
| Activity | Visual purpose | Exit or recovery condition |
|---|---|---|
| Ready | A calm default | A new interaction starts |
| Listening | Show that input is being received | Recording ends or is cancelled |
| Thinking | Acknowledge a pending response | A response, timeout or failure arrives |
| Speaking | Accompany active audio playback | Playback ends or is interrupted |
| Connecting or error | Explain availability | Connection recovers or a retry begins |
| Sleeping | Communicate inactivity if the product needs it | An agreed wake event occurs |
This table is an illustrative planning model. It does not prescribe input names, enum numbers or a fixed package of states for every assistant.
Keep emotion, speech and gaze independently useful
A character may speak while remaining happy, concerned or empathetic. If a single “talking” clip resets every facial feature, the expression may change unintentionally whenever speech starts. Separate behavior layers or properties where the selected runtime and design support them.
The Kids AI demo breakdown documents expression and mouth-shape animations present in a real export. It is useful evidence for discussing character structure, not proof of a complete AI backend or universal speech-provider integration.
Choose mouth motion around the data you have
If the app can supply only audio level, scope amplitude-driven mouth motion. If the voice pipeline supplies timed viseme events, agree the mouth mapping and playback clock. A repeated talking loop is a third option, but it should not be described as phoneme-synchronized speech.
Before commissioning a large mouth library, share a representative audio response and the actual timing format if available. A short technical review can reveal whether the visual ambition matches the information your app can provide. The speech comparison guide explains the decision in detail.
Design cancellation and failure paths deliberately
Speaking-to-listening interruption is a common example: the user starts a new request while an old reply is playing. Decide who stops playback, who clears queued mouth updates and which activity appears next. The visual system should not continue reading obsolete timing events.
Also review a connection lost during thinking, a response that arrives after cancellation and a wake event received while sleeping. For each, document a fallback. Unknown values should resolve to agreed neutral behavior rather than leaving the character stuck in the last animated pose.
Agree what the animation service owns
The animation scope covers the character or face, agreed expressions, State Machine or data-binding controls and developer handoff. Your application team owns AI requests, speech recognition, audio playback and the bridge to those controls unless additional integration work is included explicitly.
Start with the conversation events and artwork you have. For an app, see AI companion animation. For a screen-based physical device, use the separate AI robot face service, where the device, renderer and testing responsibilities need their own review.