Interactive Avatar

A production platform pairing a real-time animated avatar with a full conversational AI runtime — the largest and longest-running of my products, and the one the others grew out of.

A multi-provider runtime spanning OpenAI, Gemini and Groq, with provider-specific message formatting behind one interface and automatic model fallback when a provider errors. A complete tool-calling loop covers built-in web search, image generation and vision, plus user-defined external API calls and internal logic functions.

Streaming and non-streaming paths both carry token accounting, and cancellation runs through AbortController so an abandoned response stops costing money the moment the user moves on.

Speech-to-text and text-to-speech over WebRTC and WebSocket, with voice cloning, voice-activity detection, and handling for the ways transcription fails rather than only the way it succeeds. Facial tracking runs through the MediaPipe SDK.

Latency is the product here. A conversational avatar that is accurate but slow is not usable, so the streaming, cancellation and provider-fallback paths all exist to protect time-to-first-word.

The avatar itself is a Three.js talking head driven by streamed audio. Lipsync runs on ARKit and Oculus viseme morph targets, and the loader supports Ready Player Me and Avaturn models — or works it out from the URL when the source is not specified, because asking a non-technical user which avatar vendor they are using is a question they should never have to answer.

On top of the speech there are moods, emojis, gestures and poses, each individually switchable, with an option to revert to neutral between utterances. Skin tone can be overridden at runtime, including a proportional shift of the subsurface-scattering attenuation colour so a changed tone still looks like skin rather than paint.

A browser will not play audio until the user has interacted with the page, and an AudioContext that starts cold clips the first fraction of a second. So the avatar streams a tenth of a second of silence as soon as it loads. By the time there is something to say, the pipe is already warm and the first word arrives whole.

GLB models and animation clips do not always agree about which bones exist. Rather than let a mismatched rig throw, the loader filters every animation track against the nodes actually present in the scene and rebuilds the clip from what survives. A slightly wrong model degrades to a slightly stiller avatar instead of a blank canvas.

The avatar can be held invisible until the host page has applied its styling, which avoids a flash of the default look — but a hidden element that never reveals is a worse bug than the flash. So there is a five-second timeout that reveals it anyway and logs why.

Camera limits are exposed as percentages rather than radians and converted internally. The people configuring this are Bubble developers building products, not 3D programmers, and an API that demands they think in π is an API that gets configured wrong.

A Retrieval-Augmented Generation system built end to end: client-side PDF parsing, a custom token-aware chunker with configurable size and overlap, OpenAI embeddings, and cosine-similarity retrieval with threshold filtering and top-K selection.

Backends are swappable between a Pinecone vector database and embeddings stored in a standard database, with namespace and per-user scoping so one tenant's documents can never surface in another's answers.

An iframe and SDK embed library, a WebRTC facial-tracking SDK, live web-search and site-scraping tools, conversation summaries, lead capture, Google and Calendly integrations, and a published developer API with documentation.