Where it started
Every night before sleep I close my eyes and let my mind build the place I am listening to. Audiobooks get close and then stop: the story never notices you are there. InnerPlay is the attempt to make one that does — a story that hears an answer and goes somewhere because of it.
Removing the screen is the whole design constraint. The AI games I had used were still a chat box with atmosphere around it. The interesting question is what a story has to become when there is nothing to look at, nothing to click, and no menu to explain itself with.
What runs today
Two things had to be true before any voice work was worth starting: the story had to be a real machine, and the room had to be worth sitting in. Both of those exist. The voice does not.
The story engine is not the model
A story is a YAML file — phases, beats, choices, endings — loaded and schema-checked at startup. A pure state machine applies changes along dot paths. A rules engine matches the player’s intent against the choices the current beat actually declares, and their prerequisites. A style tracker scores how the player is behaving across the session, and that score, not a roll, decides which of the four revelations the story ends on. Sound cues are parsed out of the generated line and routed to the channel that would carry them; the sound library itself has not been recorded, so nothing plays yet.
The model sits behind an interface, and a mock sits in it
The replies you read in this build are pre-written. A mock story engine and a mock intent parser run in the browser beside the orchestrator, in the place a provider adapter will take. What is live is everything that chooses between those lines: your answer is classified, checked against the beat, allowed to change state, and the story advances or refuses to. A session in this build reaches a real ending by a real route.
There is no speech-to-text and no voice for her
Answers are typed. The microphone is real and is opened with permission, but nothing transcribes it: it feeds an analyser, and the level moves the light in the middle of the screen. The onboarding checks are real Web Audio — a panned tone across the stereo field, rain to set a level against, a meter that only settles when the microphone actually hears you. The ending screen is generated from the session record: counted facts about what you did, never a score.
The architecture being built
The browser could hold an ephemeral token and talk to Gemini directly, which is a legitimate pattern and one hop faster. We chose against it. InnerPlay needs one authoritative owner of story state, and a direct client connection either moves the rules into the browser or adds a second control path beside the audio one. One same-origin WebSocket to our own Node server keeps state, credentials and provider choice in a single place — and gives Gemini and Nova the same browser contract, which is the only reason a second provider is cheap.
PLANNED — not yet built
┌──────────────────────┐
│ Browser │ mic → 16 kHz PCM16, 32 ms binary frames
│ PlaySurface │ ───────────────────────────────▶
│ LivingForm │ same-origin WebSocket
│ PcmPlayer │ ◀───────────────────────────────
└──────────────────────┘ 24 kHz PCM16 + JSON transcripts,
cues, state
│
▼
┌─────────────────────────────────────────────────────────┐
│ server.ts + LiveSessionController │
│ owns story state · rules · transcript · credentials │
└───────────────┬─────────────────────────┬───────────────┘
│ Gemini adapter │ Nova adapter
▼ ▼
Gemini 3.1 Flash Live Amazon Nova 2 Sonic
┌──────────────────────────────────────────────────────────┐
│ TODAY: the mock engine sits where the adapters will, │
│ in the browser, and the server is not on the run path. │
└──────────────────────────────────────────────────────────┘Audio goes up as raw 16 kHz mono PCM16 in 32 ms binary frames, and comes back at 24 kHz on the same socket; transcripts, cues, state and errors travel as JSON beside it. Every turn carries an id, because a transcript or a cue that arrives after a reconnect must not be allowed to mutate the wrong caption or the wrong beat.
Four decisions taken before writing the adapter
Story authority stays in code
A structured story needs phases, escalation and an ending. A conversational model wants to be helpful and follow the player anywhere. The resolution we have designed is a blocking tool: the model must call resolve_player_turn with what it heard before it is allowed to speak, and the server answers with the current beat, the validated choice and the state snapshot. The model improvises inside a beat; it never decides which beat it is in. If a provider will not honour that gate under test, it does not get to own world physics — it stays an experimental voice mode.
The story advances when the player has heard the line
The current orchestrator records the response, applies its state changes and moves to the next beat the moment generation returns. That is correct for a text build and wrong for buffered audio: a model can generate a whole turn faster than it plays, and an interruption two seconds in would otherwise commit a revelation the player never heard. Live turns will be staged under a turn id and committed on playback completion, not on generation.
Interruption has to be handled locally first
Barge-in cannot wait for the provider to confirm it. By the time the interruption event arrives, the audio it refers to is already queued in the browser. The plan clears the local playback queue from our own speech detection — four loud analyser frames, roughly 67 ms — and treats the provider event as confirmation that arrives afterwards.
Sound tags must never reach a speech model
Cues are parsed out of generated text as [SOUND:x] today, which is safe when the text is only ever read. Hand that same string to a native audio model and it will say the bracket out loud. The live path sends structured cue identifiers on the control channel instead, timed from audible playback rather than from a character offset in a sentence.
The stack
Running
App Router; every surface in the product
End-to-end types, story schema through to UI
Phases, beats, choices, endings — validated at load
Ambience, sound cues, and the analyser behind the form
The living form: noise-displaced, driven by a real level
Pre-written replies, real intent classification
Planned
Exists as app/server.ts; not the dev or start entrypoint yet
Binary PCM plus JSON control, one contract for both providers
Primary voice; preview model, server-side only
Secondary provider behind the same browser contract
16 kHz capture out, 24 kHz playback in
First deployment target for the custom server
What has gone wrong so far
All of it in the deterministic half, which is the argument for having a deterministic half. The living form drew concentric halo arcs at every size — a radial gradient under a blur, stepping into visible bands; the fix was to delete the gradient and compute the falloff analytically. The mock adapter echoed the player’s own words back at them. The orchestrator stalled mid-phase when a choice resolved, and sat there. The session transcript quietly dropped the lines where a choice was made, which is exactly the part of a session worth keeping.
None of these needed a model to find, and none of them would have been visible if the model had been improvising the same behaviour.
What comes next
In order, with a gate on each: freeze the local message contract and make the mock emit it, so an interrupted response provably cannot advance a beat. Build browser audio in and out against a local fixture. Put the custom server on the actual run path and deploy it before a paid provider is involved. Then the Gemini adapter, checked against a simple invariant — fifty mixed turns in which the model never speaks before the story tool answers. Nova last, through the same contract, changing no browser code.
Until that first adapter lands, the honest description of this build is the one on the story page: you type, and her replies are placeholder text on a live engine.