project · astro-kbve · axum-kbve · vrm

Persistent VRM concierge forkbve.com

An always-available concierge that ships inside the right-side dock on every kbve.com page — persistent across ClientRouter navigation, deferred until opened, with the heavy VRM model, Three.js scene, and AudioContext only landing in the browser when the user opts in.

Architecture Public surface

No React, no webcam

The chat panel is vanilla DOM because it lives inside a transition:persist node the ClientRouter swap-mutates — breaking React reconciliation. Yuki never requests camera access; pose is canned or audio-derived from lipsync.

Vanilla panel — saves the React payload on first open.
Audio-driven — lipsync from the speak source, not a mirror.

Vanilla DOMFrontend

three-vrmAvatar

SSEStream

axum-kbveBackend

Persistent corner dock

A FAB plus collapsed panel that survives ClientRouter navigation via transition:persist, restoring state from localStorage.

Lazy-loaded heavy parts

The VRM model, Three.js scene, and AudioContext only load in the browser once the user opens the dock and opts in.

VRM avatar + lipsync

A three-vrm scene with idle/blink animation and aa/ih/ou lipsync driven by an AnalyserNode tap on the audio.

SSE-streamed replies

The panel streams tokens from GET /api/v1/yuki/chat into the bubble, then hands the assembled text to speak() for lipsync.

Roadmap

At a glance

Phase	Status	What ships
A — Dock shell	✅ merged (#11474)	Persistent FAB + collapsed panel, vanilla driver, lazy-mount contract, localStorage state, ESC-to-close, reduced-motion respect
B — VRM avatar	✅ merged (#11475)	`@pixiv/three-vrm@^3.5.3` scene, idle/blink animation, `aa`/`ih`/`ou` lipsync via AnalyserNode tap, render-loop visibility + fps gates
C — SSE stream	✅ merged (#11476)	`GET /api/v1/yuki/chat?q=<…>` returns `text/event-stream` chunks; panel streams tokens into the bubble + hands assembled text to `speak()` for lipsync
D — Organic + float	✅ merged (#11492)	Manual rest pose (no T-pose), portrait camera framing, `vrm.lookAt` cursor tracking, spine/hips micro-sway, draggable float-detach layer that persists across `<ClientRouter />` swaps
D.1 — Vtuber polish	✅ merged (#11507)	Spring-bone reset + body-twist on extreme gaze; first-open wave, input-focus look, nod-on-send, happy-on-done, click-to-wave, blendshape lerp; idle FPS drop to 15, render pause when dock collapsed + not floating; page-wide cursor follow + autonomous wander when floating
D.2 — Followups	🚧 this PR	Strip browser SpeechSynthesis (sounded awful); swap deprecated `THREE.Clock` for `THREE.Timer`; cache humanoid bone refs once instead of per-tick lookup; dock script uses dynamic `import()` for ClientRouter robustness; Adsense components standardised on `data-astro-rerun` push so ads re-init on every nav swap
E — Real backend	🗓 next	Swap the canned reply in `transport/yuki.rs` for a jedi/q-routed LLM call. Prompt template, conversation compaction, rate limit. Front-end unchanged.
F — Voice in	🗓 later	Push-to-talk mic capture → on-device VAD → server STT → same SSE response path
G — Tools / actions	🗓 later	Function-call schema so Yuki can navigate the site, open the wallet card, queue a forum post, etc.

How it fits together

Architecture

┌─────────────────────────────────────────────────────────────────┐
│ AstroYukiDock.astro                                             │
│   • transition:persist="kbve-yuki-dock"                         │
│   • static HTML/CSS — zero JS on first paint                    │
└───────────────┬─────────────────────────────────────────────────┘
                │   user clicks FAB → first time only
                ▼
┌─────────────────────────────────────────────────────────────────┐
│ yuki-dock.ts (vanilla)                                          │
│   • toggle + localStorage state                                 │
│   • dynamic import('./YukiPanel')                               │
└───────────────┬─────────────────────────────────────────────────┘
                │   panel mounts inside the dock body
                ▼
┌─────────────────────────────────────────────────────────────────┐
│ YukiPanel.ts                                                    │
│   • chat log + input                                            │
│   • "Show 3D Yuki" toggle → dynamic import('./YukiVRM')         │
│   • submit → EventSource('/api/v1/yuki/chat?q=…')               │
│   • SSE chunks stream into the bubble; on `done` → speak(text)  │
└─────────┬──────────────────────────────────┬────────────────────┘
          │                                  │
          ▼ (optional)                        ▼
┌──────────────────────────┐    ┌─────────────────────────────────┐
│ YukiVRM.ts               │    │ axum-kbve transport/yuki.rs     │
│  Three + three-vrm       │    │  GET /api/v1/yuki/chat          │
│  idle / blink / lipsync  │    │  text/event-stream, 15s keep-   │
│  destroy() releases all  │    │  alive, `event: done` terminator│
│  resources               │    └─────────────────────────────────┘
└──────────────────────────┘

Why no React for the panel?

The chat layer needs to live inside a transition:persist node that the rest of the dock shell wraps. React’s reconciliation tree is allergic to having its root DOM swap-mutated under it by Astro’s ClientRouter, so the safer move is vanilla DOM that owns its own state. Bundle-wise we save the React + ReactDOM payload on first dock open — and the VRM scene that does use Three.js never crosses into React’s tree either.

Why no Mediapipe / webcam?

Yuki’s pose data is either ours (canned animations, server-driven flags) or audio-derived (lipsync from the speak source). We deliberately don’t ask the visitor for camera access — the avatar is a concierge, not a mirror.

API & storage

Public surface

Frontend

// Loaded only after first dock expand.
import { mountYukiPanel } from '@/components/jay/dock/YukiPanel';

// Lazily imported by the panel when "Show 3D Yuki" is on.
import { mountYukiVRM } from '@/components/jay/dock/YukiVRM';
//   const handle = await mountYukiVRM({ host });
//   handle.setState('happy');
//   handle.speak(audioElement);   // analyser tap → mouth blendshapes
//   handle.destroy();             // releases WebGL ctx + AudioContext

Backend

GET /api/v1/yuki/chat?q=<utf8 prompt>

Content-Type: text/event-stream
:keepalive                 ← every 15s while idle
data: <chunk>              ← repeats per token / sentence fragment
data: <chunk>
event: done                ← terminator
data:

The handler trims and rejects empty prompts with a 400. There is no authentication on the route in Phase C; Phase D will gate it on a Supabase JWT once the real LLM call costs money to run.

Storage

Key	Purpose
`kbve:yuki-dock:state`	`'collapsed'` or `'expanded'` — restores last view across reloads
`kbve:yuki-dock:avatar-mode`	`'text'` or `'3d'` — restores 3D toggle
`kbve:yuki-dock:history`	Last 24 chat entries, `{role, text, ts}`
`kbve:yuki-dock:float-mode`	`'1'` or `'0'` — restores float-detach state across reloads
`kbve:yuki-dock:float-pos`	`{x, y}` viewport coords of the float layer — restores last drag position
`kbve:yuki-dock:greeted`	`'1'` once Yuki has fired her first-open wave + greet on this device

All six are localStorage-only, never sent to the server.

What’s next

Phase E: replace compose_reply in apps/kbve/axum-kbve/src/transport/yuki.rs with a real LLM call. Likely q-routed so we keep our existing tracing + Supabase JWT verification. Front-end stays identical.
Site-context: feed the current page slug into the prompt so Yuki can answer “where am I?” without a tool call.
Tool calls: small JSON-schema function set (navigate, open_dock_card, compose_forum_post) so Yuki can act on the page, not just talk about it.
Voice input behind a push-to-talk gate, on-device VAD to keep the mic OFF except while the button is held.
Wander: replace the static float layer with a slow drift loop so Yuki actually walks the viewport edges between turns.

If you have feedback, drop it in the forum — the link is literally what the Phase C stub tells you to do.

What is Yuki on kbve.com?

Yuki is an always-available concierge that ships inside the right-side dock on every kbve.com page. She is persistent across ClientRouter navigation, deferred until the user opens her, and architected so the heavy parts (VRM model, Three.js scene, AudioContext) only load in the browser when the user opts in.

How does Yuki stream chat replies?

The panel submits to GET /api/v1/yuki/chat?q=… on axum-kbve, which returns a text/event-stream. SSE chunks stream token fragments into the chat bubble, and on the done event the assembled text is handed to speak() for lipsync.

Why does Yuki avoid React and webcam access?

The chat panel is vanilla DOM because it lives inside a transition:persist node that Astro's ClientRouter swap-mutates, which breaks React's reconciliation, and it saves the React payload on first open. Yuki also never requests camera access — pose is canned or audio-derived from lipsync.