The animation library was audited, not accumulated
It used to be 56 clips. It is 27. Every one was played back solo on the model and kept on the strength of the performance, and the cuts were recorded with the reason attached rather than quietly dropped.
Half of the library turned out to be broken in ways a list of filenames cannot show you. Seven hit reactions had finger tracks pinned at identity, so the character flinched with flat splayed shovel hands. hit_head was a byte-identical copy of gratitude. And a run of things filed as “gestures” never moved more than seven degrees in any joint — they were idles with ambitious names.
Nothing was thinned out to make room. Where a mood pool emptied it was restocked from what is actually good, and every chat keyword that pointed at a cut clip was re-pointed at a survivor rather than deleted. A companion who used to wave at “hi” still waves at “hi”.
Crossfades are sized by measured distance
A crossfade is a straight line drawn between two poses, so how long it should take depends on how far apart they are. Every clip’s first and last frame is measured against the idle underneath it once, on load, and the fade is sized to hold a constant angular rate.
greet opens 139° from a stand — it starts mid-bow with one leg already back — and gets 0.70s to get there instead of being dragged into the pose at 460°/s. joy opens within 32° and gets 0.19s, so a gesture that should snap does.
A single fixed blend time is what makes most avatar work look like a puppet: the same 0.3s is a lurch on one transition and a smear on another. This is the same fix as sizing an easing curve to a distance rather than to a preference.
The lip-sync is a phoneme pipeline, not a jaw flap
Text goes through a rule-based grapheme-to-phoneme stage. The phones become a timed track of overlapping envelopes. The render loop samples that track every frame.
The important property is that each envelope rises before its own phone. So the lips are already rounding for the oo of “soon” while the s is still hissing. Coarticulation and crossfading both fall out of that one mechanism rather than being special-cased per phoneme pair. Lips genuinely close on m, b and p; diphthongs travel through two shapes; and the whole utterance is one timeline, so word-boundary events only correct its drift and teach it the voice’s real tempo — which means the mouth still works on engines that never fire those events at all.
With the local speech service there is nothing to guess: it returns a real viseme track alongside the audio, fifteen Oculus shapes folded onto VRM’s five, and every chunk is scheduled on one continuous audio clock. That clock — not wall time — is what the mouth runs on.
Expressions were given mouthless twins
A VRM emotion preset is one blendshape covering the whole face, so happy is a wide smile as much as it is narrowed eyes. Hold that while a viseme track is shaping vowels and the two fight over the same geometry: the louder the emotion, the deader the speech.
So on load, every expression that shapes the mouth is given a twin with the mouth taken out of it, and they crossfade onto it as they start talking. Brow, eyes and blush keep playing at full strength while the lips are handed to the viseme track.
Which morphs count as “the mouth” is measured per model, not guessed from names. The model’s own viseme expressions say which vertices the mouth is made of, and a combined morph that moves eyes and mouth gets a masked copy rather than being dropped.
They take their lips back off the viseme track’s own clock rather than off the engine’s end event, because browsers lose that event — Chrome can leave speechSynthesis.speaking stuck true forever, and a mouth that waits for permission to come back never comes back.
A reply has three phases and they behave differently
- Thinking. They look away and up, the way people do while retrieving something, cycle thinking faces, and play the pondering clips.
- Speaking. The reply is parsed into a timeline of face beats — one per clause, keyed off punctuation, keywords and mood — walked against the lip-sync playhead so the expression lands with the word rather than after the sentence.
- Idle. The cursor gets their eyes back, but only while it is actually moving. Park the mouse and they drift off it and back onto you — the camera projected onto their gaze plane, so orbit around them and they keep eye contact from wherever you are watching. And they fidget.
They start speaking before the reply has finished being written
The model takes seconds to write an answer and the voice takes seconds more to say it. Done one after the other, that is the sum of both waits. So as soon as the opening is worth starting on, a speech session opens and the rest of the reply is fed in behind it on a second connection held open for the whole turn.
The hard part is where to cut, because the text arriving is markdown and the text they say is not. Cut inside a code fence and they read out the code. Cut inside *waves at you* and a stage direction becomes dialogue. Cut before “Sources:” lands and they recite a bibliography. Cut after “Dr.” and they say half a name.
So a boundary is only taken when nothing still to come could change what it means — sentence ends a lowercase word does not contradict, block starts, balanced markdown — and when in doubt it waits. A boundary held back costs a second. A boundary taken early is audible.
Memory is typed facts, and the type is what makes it work
Not a transcript, and not a vector blob. Each memory is a fact on five independent axes: whose it is, which companion may recall it, who it is about, which attribute it asserts, and which lifecycle it follows.
Setting the attribute is the part that matters. With a predicate, works at Globex supersedes works at Acme and the old value stays reachable as history. Without one, the two coexist forever and the companion contradicts itself a week later — which is the failure mode of nearly every “AI that remembers” you have used.
What gets recalled carries the store’s own confidence into the prompt in words, so a weak match is offered as something half-remembered rather than stated as fact. A fact they have already raised is annotated as such, so they refer back to it instead of announcing it twice.
The case that justified going further: a user mentions a shellfish allergy in one conversation and asks about dinner in another. A pre-turn search that happened not to surface it is a companion cheerfully recommending prawns. So on the Claude brain they can stop and look, mid-sentence, through one read-only tool of ours rather than the service’s own twenty — and the lookup shows up as a note on the message, the same way a web search does.
When two facts genuinely contradict, the store refuses to pick and keeps both. They raise it once, in their own words, and a small chooser appears under the reply with both memories side by side, where each came from and roughly when. Pick one, keep both, or carry on talking and the newer one stands. A birthday, or anything private to one companion, they will not decide on their own whatever you do.
Two processors, because one of them is not on the latency path
The main one answers you, and is the only latency anybody feels. The sub one does everything they do unasked: putting an arriving notification into their own words, and reading a finished exchange for things worth remembering. It defaults to the small model because it runs several times a conversation without anyone asking for it, and either job can be told not to ask a model at all — there is a table of about thirty sentence shapes that does the remembering for free and misses a great deal, and that trade is yours to make.
The desktop build is not just the web build in a window
Steady-state frame rate does not change — it is the same Chromium. Everything around the render loop does:
- They keep drawing at their configured idle rate when unfocused and when another window covers them. Chromium’s own request-animation-frame throttle and its native occlusion detection both override the app’s frame budgeting, and both are switched off.
- 50MB of models come off local disk, with the default companion warm in RAM before the window has finished being created.
- The chat bridge is in-process. Nothing to start, no open port.
- The API key is encrypted to your Windows account and never enters the page.
- Voice input is whisper.cpp on your own processor: offline, no session cap.
- Their voice and their memory are both in the box, and the memory store is open before the first message rather than after it — the one thing a companion has to have ready by the time you say hello.