# pi-asr

Speech-to-text for the **pi** coding agent, backed by an NVIDIA Speech NIM realtime streaming ASR
endpoint (Riva-style realtime protocol, Parakeet/Nemotron models).

Zero runtime dependencies: `https`, the Node 22 global `WebSocket`, and `ffmpeg` for container/codec
decoding. No Python, no virtualenvs, no local model weights.

## Install

```bash
# from git over ssh (uses your ~/.ssh/config)
pi install git:git@git.iaminsane.online:iulianp/pi-asr.git

# or from a local checkout
pi install /path/to/pi-asr

# or try it without installing
pi -e /path/to/pi-asr
```

## Configure

The API key is never read from source. Priority: explicit option, then env var, then config file.

```bash
export ASR_API_KEY=sk-...
export ASR_NIM_URL=https://your-host/openai/streaming-asr   # optional, has a default
```

or drop a config file (mode 0600 recommended) at `~/.pi/agent/asr.json` (override with `$ASR_CONFIG`):

```json
{
  "apiKey": "sk-...",
  "baseUrl": "https://your-host/openai/streaming-asr"
}
```

`baseUrl` is the HTTPS base. The WebSocket URL is derived from it: `wss://` + the same host/path +
`/realtime?intent=transcription`.

## What you get

| surface | what it does |
| --- | --- |
| `asr_transcribe` | transcribe a local audio/video file; live partials while it streams; optional per-word timings |
| `asr_health` | `health/live` + `health/ready` + a minted session, so you see the model/language/sample-rate the server will actually use |
| `/asr <file> [--lang xx-YY] [--words] [--speed N] [--start S] [--out f.txt]` | human-facing transcript viewer |

```bash
pi -p "use the asr_transcribe tool on ./meeting.m4a with words=true and save to meeting.txt"
```

Accepted input: anything ffmpeg can decode (`wav` s16le/f32le, `mp3`, `m4a`, `aac`, `ogg`, `opus`,
`flac`, `mp4`, `mov`, `mkv`), resampled to the 16 kHz mono PCM16 the NIM mandates. Without ffmpeg,
16 kHz mono wav still works.

## Layout

```text
extensions/asr.ts   pi registration: tools, TUI rendering, /asr command
src/client.ts       RealtimeAsrClient: mint, WS subprotocol handshake, event loop, decode
```

## Protocol notes (things the published API reference gets wrong)

Hard-won while building this against a real deployment. The client handles all of it.

1. `input_audio_buffer.done` does **not** flush the decoder — you must send
   `input_audio_buffer.commit` first, otherwise the server returns zero results.
2. `conversation.item.input_audio_transcription.delta` carries the **cumulative** segment text, not
   an increment — rendering it incrementally duplicates every word.
3. A segment emits **two** `.completed` frames; the second has an empty `words_info`. The documented
   `is_last_result` flag is never `true` on this model, so completion is detected by event
   quiescence (no new event for ~700 ms after `done`) rather than by a terminal flag.
4. Auth failures surface as **HTTP 403 at the upgrade handshake**, not WS close 1008. A tokenless
   handshake gets `101 Switching Protocols` with no echoed subprotocol, which Node's global
   `WebSocket` treats as fatal — so the minted token is mandatory, not optional.
5. The documented `?token=` query param is **not** rejected (the docs claim it is). Do not put tokens
   in URLs: they leak into access logs. The token goes in an extra `Sec-WebSocket-Protocol` value.
6. A `session.update` that omits `model` silently resets it to a model the server has not loaded
   (`conformer`), which yields an empty transcript. Echo the minted session config back verbatim.

## Latency

mint ~120 ms, handshake ~20 ms, first partial ~360 ms, final ≈ audio duration + 0.5 s. The realtime
decoder is real-time bound, so `speed=2` (ffmpeg `atempo`) is the only way to beat wall-clock.

## Dev

```bash
pi install /path/to/pi-asr     # registers the checkout; edit, then restart pi
pi --version                   # fast smoke test: prints extension load errors, if any
```