Interactive Avatar

A live platform that combines a real-time animated avatar with a full conversational AI system. It's the biggest and longest-running of my products.

It works with OpenAI, Gemini and Groq, formatting messages the way each one expects, and switches to another model automatically if one fails. The AI can use tools, including web search, image generation and image recognition, as well as calling external APIs and custom logic that users set up themselves.

Both streaming and standard responses keep track of token usage, and a response can be cancelled part way through, so if a user moves on it stops straight away and stops costing money.

Speech-to-text and text-to-speech run over WebRTC and WebSocket, with voice cloning, detection of when someone is speaking, and proper handling for when transcription goes wrong as well as when it goes right. Face tracking uses the MediaPipe SDK.

Speed matters most here. An avatar that gives good answers slowly isn't much use in a conversation, so the streaming, cancelling and model fallback are all there to get the first word out as quickly as possible.

The avatar is a 3D talking head built with Three.js and driven by the audio as it streams in. Lip-sync uses ARKit and Oculus viseme shapes, and it supports Ready Player Me and Avaturn models. If you don't say which one you're using, it works it out from the web address, because most users wouldn't know or care.

On top of the speech there are moods, emojis, gestures and poses, which can each be switched on or off, with an option to go back to a neutral expression between replies. Skin tone can be changed on the fly, and it also adjusts how light passes through the skin, so a new tone still looks like skin and not paint.

Browsers won't play audio until someone has interacted with the page, and audio that starts cold tends to cut off the first fraction of a second. So the avatar plays a tenth of a second of silence as soon as it loads, which means the first word comes through complete.

3D models and their animations don't always agree on which bones exist. Instead of crashing when they don't match, the loader drops any animation for bones that aren't there and keeps the rest. A slightly mismatched model just ends up a bit stiller, instead of not showing at all.

The avatar can stay hidden until the page around it has finished styling, so you don't see it flash up in its default look first. There's also a five-second fallback that shows it anyway, in case the styling never arrives.

Camera limits are set as percentages instead of angles, and converted behind the scenes. The people setting it up are Bubble developers building apps, not 3D programmers, so percentages are much easier to get right.

It includes a full RAG setup. PDFs are read in the browser, split into chunks with a size and overlap you can adjust, turned into OpenAI embeddings, and then searched by similarity, with a minimum score and a limit on how many results come back.

The data can be stored in Pinecone or in an ordinary database, and each user's documents are kept separate, so one customer's files can never turn up in someone else's answers.

An embed library for iframes and apps, a WebRTC face tracking SDK, live web search and website scraping, conversation summaries, lead capture, Google and Calendly integrations, and a public developer API with documentation.