Pocket Character: a Desktop Digital Human in One Native Process

Every frame above is the renderer's actual output — the exact pixels the transparent widget window presents, alpha and all. The process drawing them uses 118 MB of memory and 4 % of one CPU core.

A desktop companion is a lovely idea: a small animated character that lives on your screen, blinks, breathes, watches your cursor, and — when you wire up an LLM — talks back. airi is one of the most complete open-source takes on that idea, and we mean that as a compliment: it's a whole companion platform, with providers, voice, plugins, and a stage that can render Live2D, VRM, even Godot.

But run it and look at Activity Monitor. On our M3 Max, airi's out-of-the-box stage — one idle character in a transparent window — is 8 processes, 1.7 GB of RSS, and 91 % of a CPU core, continuously. That is not an airi bug. That is what an Electron window with an uncapped rAF loop, a 2× supersampled canvas, and a helper-process tree costs before the AI does anything.

We kept asking a narrower question: what does the character itself actually need? A skinned mesh, one looping animation, two schedulers, a physics chain, a transparent window. That is not an app platform's worth of work. That is a fixed-function player — and fixed-function players are exactly the shape the Pocket runtime family was built for.

If you are new here: PocketJS runs real Solid and Vue Vapor components on a 2004 Sony PSP at a locked 60 FPS, and the same architecture — a native core that owns the frame, a QuickJS guest that owns policy, a spec-pinned surface between them — has since carried an FPS, a Figma viewer, and YouTube over a USB cable. This post is about pointing it at a desktop widget instead of a handheld.

So we rebuilt airi's 3D stage as a Pocket runtime: same character model, same idle animation, same blink and eye behavior, same spring-bone physics, same transparent always-on-top window — as one native process. Inside: what a character costs per tick when you only compute what changed, a VRM crate that learned the difference between +Z and −Z the hard way, blinks that upload vertices only while eyelids move, and a measured 10–20× drop on every resource axis, screenshots included.

The stage, itemized

airi's default character is actually Live2D — Momose Hiyori, the official Cubism sample. We couldn't chase that one even if we wanted to: rendering .moc3 requires the proprietary Live2D Cubism Core, which an open runtime can't vendor. The 3D digital human everyone pictures is airi's VRM mode: AvatarSample_A, the official VRoid sample model, driven by an idle_loop.vrma animation. That's the parity target, and it is delightfully concrete:

• One VRM 0.x model: 40,406 vertices, 3 skins, 273 joints, 14 blend-shape expressions, four 4096² textures.
• One looping clip: idle_loop.vrma, 10.375 s, 28 humanoid channels.
• A blink: a 0.2 s sine envelope, fired every 1–6 s, driving one morph target (mesh 0, target 13 — we checked).
• Idle eye saccades: a fixation jitter of ±0.25 m re-aimed on airi's exact interval table, 800–4800 ms.
• Spring bones: 10 groups — Bust, Hood, HoodString, and seven Hair chains — against 22 sphere colliders, all from the model file.
• A 450×600 transparent, undecorated, always-on-top window you can drag.

Nothing in that list wants a DOM. Nothing in it wants eight processes. Here is what it runs on instead:

The split is the same one OpenStrike proved out: the native core owns everything that happens every frame, the QuickJS guest owns policy — which clip plays, which expression fires, whether the eyes track your mouse. The guest is the personality, and it stays a hot-swappable JS bundle. The airi-parity personality is deliberately boring: set tracking to none, loop idle_loop, let the native schedulers breathe. A different character is a different bundle, not a different binary.

Blinks should cost only while blinking

The engine work that made this possible landed upstream in #125, and the piece we care most about is the morph-target design, because it encodes the whole philosophy: an idle character should cost almost exactly nothing.

VRM faces animate through blend shapes — per-vertex position deltas layered on the skinned mesh. The obvious GPU implementation binds every morph target and accumulates them in the vertex shader, every vertex, every frame, forever. But look at what a blink actually is: 0.2 seconds of eyelid movement every one to six seconds. On the other ~96 % of frames, every weight is identical to the last frame.

So pocket3d stores morph targets as sparse CPU deltas — only the vertices a target actually moves — and each character instance owns a small overlay vertex buffer. When a weight changes, the affected primitives are recomputed on the CPU (a few thousand fused multiply-adds) and re-uploaded; when nothing changed, a dirty mask says so and the render path doesn't touch a byte. The draw call redirects morphing primitives to the overlay with draw_indexed's base_vertex offset, so the shared index buffer never needs rebasing and non-morphing instances of the same asset keep reading the original vertices.

Frames 294–309 of the sequence above, three ticks apart: one blink, as rendered. These are the only frames of the ten-second loop where morph vertices were uploaded at all.

The rest of the per-tick pipeline follows the same rule — compute exactly what the frame needs, in one pass, in one process:

Two more engine details earned their keep. VRoid rigs carry ~270 joints across three skins, so the joint-palette window grew from 128 to 512 matrices — before that, most of the body silently skinned against garbage. And the widget window itself became a first-class AppConfig mode: transparent surface alpha, no decorations, always-on-top, drag-anywhere, and a max_fps pacing loop that sleeps between frames instead of spinning on vsync — on a ProMotion display, "just render on rAF" quietly means 120 Hz, and airi's stage does exactly that.

The bug worth confessing

The first full-body render came out of the headless harness looking almost right — model loaded, textures resolved, transparent background clean — except the character held both arms straight up like a referee signaling a touchdown.

Every piece of that pose was individually correct. The .vrma parsed. All 22 humanoid channels retargeted onto the right bones. The quaternions were bit-faithful. The bug was a convention: VRMC_vrm_animation poses live in VRM 1.0's humanoid space, where characters face +Z — but VRM 0.x models, and AvatarSample_A is one, face −Z. three-vrm absorbs that 180° silently inside its normalized-rig indirection, which is exactly the kind of kindness that hides a spec detail until you reimplement it. In a raw channel copy, every rotation is conjugated wrong by half a turn of yaw, and the T-pose-relative arm rotations that should bring the hands down to the hips instead raise them to the sky.

The fix is one line of quaternion algebra — conjugate every rotation by the yaw-π between the spaces, (x, y, z, w) → (−x, y, −z, w), negate the hips' X/Z translation — and it is now a documented, tested behavior of pocket-vrm's retarget, not tribal knowledge. Specs travel between ecosystems; conventions don't. Render your output and look at it.

Measured, same machine, same ruler

Methodology first, because the numbers are the headline and headlines deserve receipts. Both apps, same M3 Max, steady idle, hands off: ≥60 s of ps samples at 5 s intervals across the entire process tree, medians reported, plus macOS footprint for physical memory (it counts the GPU allocations RSS misses). Every CPU number below is the standard per-process convention — percent of one core. airi was measured in both stage modes; the VRM stage is the apples-to-apples row.

Activity Monitor tells the same story in screenshots. Here is airi's VRM stage at idle — the GPU helper and stage renderer between them holding ~70 % of a core and 1.5 GB:

airi, VRM stage, idle. The same character sits in the corner of the screen; the two highlighted helpers are what it costs to keep her there.

The memory tab's two biggest airi rows — and these are just two of the eight processes.

And here is the whole of pocket-character, one row, drawing the same model in the same kind of window:

One process, freshly launched (the 6.2 % includes model-decode startup; it settles to 2.4–2.8 % at 30 fps). The %GPU column reads 3.5.

One aside we didn't put in the ledger because the compositor is shared infrastructure: with airi's stage running, macOS's WindowServer sat at ~51 % of a core; with only pocket-character on screen it reads ~9 %. An uncapped, 2×-supersampled, full-window rAF repaint doesn't just spend its own processes' time.

The numbers

• 1 process instead of 8 · 118 MB RSS instead of 2184 MB · 3.9 % of one core at 60 fps instead of 44.4 % — and 2.1 % at 30 fps, against the 90.7 % of airi's out-of-the-box Live2D stage.
• Physical footprint 518 MB vs ~1870 MB — and 454 MB of ours is Metal texture memory; the CPU heap is ~16 MB dirty. Capping the model's four 4096² authoring textures at 2048² (invisible in a 450×600 window) was worth 413 MB on its own.
• The whole per-tick pipeline — schedulers, clip sampling, look-at, 34 spring joints, 273-joint palette, guest turn — costs ~0.03 ms of CPU.
• An 11 MB binary plus 27 MB of downloaded model assets, vs a 1.8 GB installed app from an 822 MB DMG.
• pocket-vrm ships with 21 tests against the real fixture, including retarget math and 600-step spring determinism (two runs, bitwise-equal quaternions).
• Behavior parity is parameter-exact: blink sin(π·t/0.2 s) at uniform 1–6 s; saccades on airi's own 400 ms-step interval CDF; springs from the model's 10 groups and 22 colliders; tracking mode none by default, mouse mode wired.

What this doesn't claim

Fairness matters more than the ratio. airi is a platform — providers, voice pipelines, VAD, plugins, a settings surface, multiple stage backends — and this project reimplements exactly one slice of it: the idle character stage. The comparison holds because at idle, the stage is what's running; it does not make pocket-character an airi replacement. airi's literal default (Live2D Hiyori) stays out of reach of any open runtime for licensing reasons, not technical ones. On rendering: airi tone-maps through ACES with an HDR environment; we draw an MToon approximation with cutout alpha — side by side it reads as a subtle grade difference, and the honest place to close that gap is a proper MToon pass, not this post. Lip sync only activates with a TTS stack on both sides, so neither ledger includes it. And every number here is one machine, one OS, measured over minutes, not weeks — the full methodology and raw tables are in the repo for anyone who wants to re-run them.

Try it

git clone --recurse-submodules https://github.com/pocket-stack/pocket-character
cd pocket-character
bun run setup        # vendored install + model assets (not committed)
bun run widget       # build + launch the widget; Ctrl-C to quit

# once built:
target/release/pocket-character --max-fps 30            # the 2 % version
target/release/pocket-character --headless-shot s.png   # CI-friendly render, alpha intact
bun tools/measure.ts                                  # reproduce the table above

The personality lives in app/main.ts — a policy bundle over the character surface. Change it, bun tools/build-ui.ts, relaunch: no Rust rebuild. The engine halves — morph targets, pose injection, widget windows, pocket-vrm — shipped in PocketJS 0.6.0.

---

pocket-character is open source at pocket-stack/pocket-character, on the same engine as everything else in the family. Follow @pocket_js — the pocket now has someone living in it.