The animation library was audited
It used to be 56 clips. It is 27 now. Every one was played back solo on the model and kept on the strength of the performance, and each cut was recorded with a reason attached.
Half of the library turned out to be broken in ways a list of filenames won’t show you. Seven hit reactions had finger tracks pinned at identity, so the character flinched with flat splayed shovel hands. hit_head was a byte-identical copy of gratitude. And a run of things filed as “gestures” never moved more than seven degrees in any joint, which makes them idles with ambitious names.
Nothing was thinned out to make room. Where a mood pool emptied it was restocked from what survived, and every chat keyword pointing at a cut clip was re-pointed. A companion who used to wave at “hi” still waves at “hi”.
Crossfades are sized by measured distance
A crossfade is a straight line drawn between two poses, so how long it should take depends on how far apart they are. Every clip’s first and last frame gets measured against the idle underneath it once, on load, and the fade is sized to hold a constant angular rate.
greet opens 139° from a stand, since it starts mid-bow with one leg already back. It gets 0.70s to get there, which spares you the sight of a character dragged into the pose at 460°/s. joy opens within 32° and gets 0.19s, so a gesture that should snap does.
A single fixed blend time is most of what makes avatar work look like a puppet. The same 0.3s is a lurch on one transition and a smear on another.
The lip-sync is a phoneme pipeline
Text goes through a rule-based grapheme-to-phoneme stage. The phones become a timed track of overlapping envelopes. The render loop samples that track every frame.
The property that matters is that each envelope rises before its own phone, so the lips are already rounding for the oo of “soon” while the s is still hissing. Coarticulation and crossfading both come out of that one mechanism, with nothing special-cased per phoneme pair. Lips close properly on m, b and p, diphthongs travel through two shapes, and the whole utterance is one timeline, so word-boundary events only correct its drift and teach it the voice’s real tempo. That last part is what keeps the mouth working on engines that never fire those events at all.
With the local speech service there is nothing to guess. It returns a real viseme track alongside the audio, fifteen Oculus shapes folded onto VRM’s five, and every chunk is scheduled on one continuous audio clock. The mouth runs on that clock and never on wall time.
Expressions were given mouthless twins
A VRM emotion preset is one blendshape covering the whole face, so happy is a wide smile as much as it is narrowed eyes. Hold that while a viseme track is shaping vowels and the two fight over the same geometry. The louder the emotion, the deader the speech.
So on load, every expression that shapes the mouth is given a twin with the mouth taken out of it, and they crossfade onto it as they start talking. Brow, eyes and blush keep playing at full strength while the lips are handed to the viseme track.
Which morphs count as “the mouth” is measured per model, and names are never trusted for it. The model’s own viseme expressions say which vertices the mouth is made of, and a combined morph that moves eyes and mouth gets a masked copy so that none of it is lost.
They take their lips back off the viseme track’s own clock. Browsers lose the engine’s end event: Chrome can leave speechSynthesis.speaking stuck true forever, and a mouth waiting for permission to come back never comes back.
A reply has three phases
- Thinking. They look away and up, the way people do while retrieving something, cycle thinking faces, and play the pondering clips.
- Speaking. The reply gets parsed into a timeline of face beats, one per clause, keyed off punctuation, keywords and mood, then walked against the lip-sync playhead so the expression lands with the word.
- Idle. The cursor gets their eyes back, but only while it is moving. Leave the mouse alone and they drift off it and onto you, meaning the camera projected onto their gaze plane. Orbit around them and they hold eye contact from wherever you are watching. And they fidget.
They start speaking before the reply is finished
The model takes seconds to write an answer and the voice takes seconds more to say it. Done one after the other that is the sum of both waits. So as soon as the opening is worth starting on, a speech session opens and the rest of the reply is fed in behind it on a second connection held open for the whole turn.
The hard part is where to cut, because the text arriving is markdown and the text they say is not. Cut inside a code fence and they read out the code. Cut inside *waves at you* and a stage direction becomes dialogue. Cut before “Sources:” lands and they recite a bibliography. Cut after “Dr.” and they say half a name.
So a boundary is only taken when nothing still to come could change what it means: sentence ends that a lowercase word does not contradict, block starts, balanced markdown. When in doubt it waits. A boundary held back costs a second, and a boundary taken early is audible.
Memory is typed facts
Each memory is a fact on five independent axes: whose it is, which companion may recall it, who it is about, which attribute it asserts, and which lifecycle it follows. There is no transcript anywhere in it, and no vector blob.
Setting the attribute is the part that matters. With a predicate, works at Globex supersedes works at Acme and the old value stays reachable as history. Without one the two coexist forever and the companion contradicts itself a week later. That is the failure mode of nearly every “AI that remembers” you have used.
What gets recalled carries the store’s own confidence into the prompt in words, so a weak match is offered as something half-remembered and never stated flat. A fact they have already raised is annotated as such, so they refer back to it and don’t announce it twice.
The case that justified going further was a shellfish allergy. Mentioned in one conversation, and dinner asked about in another. A pre-turn search that happens not to surface it gives you a companion cheerfully recommending prawns. So on the Claude brain they can stop and look, mid-sentence, through one read-only tool of ours. One, not the service’s own twenty. The lookup shows up as a note on the message, the same way a web search does.
When two facts contradict each other the store refuses to pick and keeps both. Your companion raises it once, in their own words, and a small chooser appears under the reply with both memories side by side, where each came from and roughly when. Pick one, keep both, or carry on talking and the newer one stands. A birthday, or anything private to one companion, they will not decide on their own whatever you do.
Two processors, one of them off the latency path
The main one answers you, and is the only latency anybody feels. The sub one does everything they do unasked: putting an arriving notification into their own words, and reading a finished exchange for things worth remembering. It defaults to the small model, since it runs several times a conversation without anyone asking for it. Either job can be told not to ask a model at all, in which case a table of about thirty sentence shapes does the remembering for free and misses a great deal. That trade is yours to make.
The desktop build is more than the web build in a window
Steady-state frame rate is identical, since it is the same Chromium. Everything around the render loop is different:
- They keep drawing at their configured idle rate when unfocused and when another window covers them. Chromium’s own request-animation-frame throttle and its native occlusion detection both override the app’s frame budgeting, so both are switched off.
- 50MB of models come off local disk, with the default companion warm in RAM before the window has finished being created.
- The chat bridge is in-process. Nothing to start, no open port.
- The API key is encrypted to your Windows account and never enters the page.
- Voice input is whisper.cpp on your own processor: offline, no session cap.
- Their voice and their memory are both in the box, and the memory store is open before the first message. That is the one thing a companion has to have ready by the time you say hello.