pi-asr

Speech-to-text for the pi coding agent, backed by an NVIDIA Speech NIM realtime streaming ASR endpoint (Riva-style realtime protocol, Parakeet/Nemotron models).

Zero runtime dependencies: https, the Node 22 global WebSocket, and ffmpeg for container/codec decoding. No Python, no virtualenvs, no local model weights.

Install

# from git over ssh (uses your ~/.ssh/config)
pi install git:git@git.iaminsane.online:iulianp/pi-asr.git

# or from a local checkout
pi install /path/to/pi-asr

# or try it without installing
pi -e /path/to/pi-asr

Configure

The API key is never read from source. Priority: explicit option, then env var, then config file.

export ASR_API_KEY=sk-...
export ASR_NIM_URL=https://your-host/openai/streaming-asr   # optional, has a default

or drop a config file (mode 0600 recommended) at ~/.pi/agent/asr.json (override with $ASR_CONFIG):

{
  "apiKey": "sk-...",
  "baseUrl": "https://your-host/openai/streaming-asr"
}

baseUrl is the HTTPS base. The WebSocket URL is derived from it: wss:// + the same host/path + /realtime?intent=transcription.

What you get

surface	what it does
asr_transcribe	transcribe a local audio/video file; live partials while it streams; optional per-word timings
asr_health	health/live + health/ready + a minted session, so you see the model/language/sample-rate the server will actually use
/asr <file> [--lang xx-YY] [--words] [--speed N] [--start S] [--out f.txt]	human-facing transcript viewer

pi -p "use the asr_transcribe tool on ./meeting.m4a with words=true and save to meeting.txt"

Accepted input: anything ffmpeg can decode (wav s16le/f32le, mp3, m4a, aac, ogg, opus, flac, mp4, mov, mkv), resampled to the 16 kHz mono PCM16 the NIM mandates. Without ffmpeg, 16 kHz mono wav still works.

Layout

extensions/asr.ts   pi registration: tools, TUI rendering, /asr command
src/client.ts       RealtimeAsrClient: mint, WS subprotocol handshake, event loop, decode

Protocol notes (things the published API reference gets wrong)

Hard-won while building this against a real deployment. The client handles all of it.

1. input_audio_buffer.done does not flush the decoder — you must send input_audio_buffer.commit first, otherwise the server returns zero results.
2. conversation.item.input_audio_transcription.delta carries the cumulative segment text, not an increment — rendering it incrementally duplicates every word.
3. A segment emits two .completed frames; the second has an empty words_info. The documented is_last_result flag is never true on this model, so completion is detected by event quiescence (no new event for ~700 ms after done) rather than by a terminal flag.
4. Auth failures surface as HTTP 403 at the upgrade handshake, not WS close 1008. A tokenless handshake gets 101 Switching Protocols with no echoed subprotocol, which Node's global WebSocket treats as fatal — so the minted token is mandatory, not optional.
5. The documented ?token= query param is not rejected (the docs claim it is). Do not put tokens in URLs: they leak into access logs. The token goes in an extra Sec-WebSocket-Protocol value.
6. A session.update that omits model silently resets it to a model the server has not loaded (conformer), which yields an empty transcript. Echo the minted session config back verbatim.

Latency

mint ~120 ms, handshake ~20 ms, first partial ~360 ms, final ≈ audio duration + 0.5 s. The realtime decoder is real-time bound, so speed=2 (ffmpeg atempo) is the only way to beat wall-clock.

Dev

pi install /path/to/pi-asr     # registers the checkout; edit, then restart pi
pi --version                   # fast smoke test: prints extension load errors, if any