feat: speech-to-text for pi via NVIDIA Speech NIM realtime ASR
- asr_transcribe / asr_health tools + the /asr command, with live partials, per-word timings, transcript export and TUI rendering - zero-dependency realtime client (https + Node 22 global WebSocket): mint transcription session -> subprotocol handshake -> event loop -> quiescence - decodes anything ffmpeg supports (and native 16k mono/stereo pcm16/pcm_f32le wav) to the 16 kHz mono PCM16 the NIM mandates - credentials via ASR_API_KEY / ASR_NIM_URL or ~/.pi/agent/asr.json; never in source - README documents the six places the published API reference diverges from the deployed protocol (commit vs done, cumulative deltas, double .completed, HTTP 403 at upgrade, ?token= not rejected, model reset on session.update)
This commit is contained in:
commit
19faa8b088
5 files changed
+893
No files matched your search
@@ -0,0 +1,95 @@
|
||||
# pi-asr
|
||||
|
||||
Speech-to-text for the **pi** coding agent, backed by an NVIDIA Speech NIM realtime streaming ASR
|
||||
endpoint (Riva-style realtime protocol, Parakeet/Nemotron models).
|
||||
|
||||
Zero runtime dependencies: `https`, the Node 22 global `WebSocket`, and `ffmpeg` for container/codec
|
||||
decoding. No Python, no virtualenvs, no local model weights.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
# from git over ssh (uses your ~/.ssh/config)
|
||||
pi install git:git@git.iaminsane.online:iulianp/pi-asr.git
|
||||
|
||||
# or from a local checkout
|
||||
pi install /path/to/pi-asr
|
||||
|
||||
# or try it without installing
|
||||
pi -e /path/to/pi-asr
|
||||
```
|
||||
|
||||
## Configure
|
||||
|
||||
The API key is never read from source. Priority: explicit option, then env var, then config file.
|
||||
|
||||
```bash
|
||||
export ASR_API_KEY=sk-...
|
||||
export ASR_NIM_URL=https://your-host/openai/streaming-asr # optional, has a default
|
||||
```
|
||||
|
||||
or drop a config file (mode 0600 recommended) at `~/.pi/agent/asr.json` (override with `$ASR_CONFIG`):
|
||||
|
||||
```json
|
||||
{
|
||||
"apiKey": "sk-...",
|
||||
"baseUrl": "https://your-host/openai/streaming-asr"
|
||||
}
|
||||
```
|
||||
|
||||
`baseUrl` is the HTTPS base. The WebSocket URL is derived from it: `wss://` + the same host/path +
|
||||
`/realtime?intent=transcription`.
|
||||
|
||||
## What you get
|
||||
|
||||
| surface | what it does |
|
||||
| --- | --- |
|
||||
| `asr_transcribe` | transcribe a local audio/video file; live partials while it streams; optional per-word timings |
|
||||
| `asr_health` | `health/live` + `health/ready` + a minted session, so you see the model/language/sample-rate the server will actually use |
|
||||
| `/asr <file> [--lang xx-YY] [--words] [--speed N] [--start S] [--out f.txt]` | human-facing transcript viewer |
|
||||
|
||||
```bash
|
||||
pi -p "use the asr_transcribe tool on ./meeting.m4a with words=true and save to meeting.txt"
|
||||
```
|
||||
|
||||
Accepted input: anything ffmpeg can decode (`wav` s16le/f32le, `mp3`, `m4a`, `aac`, `ogg`, `opus`,
|
||||
`flac`, `mp4`, `mov`, `mkv`), resampled to the 16 kHz mono PCM16 the NIM mandates. Without ffmpeg,
|
||||
16 kHz mono wav still works.
|
||||
|
||||
## Layout
|
||||
|
||||
```text
|
||||
extensions/asr.ts pi registration: tools, TUI rendering, /asr command
|
||||
src/client.ts RealtimeAsrClient: mint, WS subprotocol handshake, event loop, decode
|
||||
```
|
||||
|
||||
## Protocol notes (things the published API reference gets wrong)
|
||||
|
||||
Hard-won while building this against a real deployment. The client handles all of it.
|
||||
|
||||
1. `input_audio_buffer.done` does **not** flush the decoder — you must send
|
||||
`input_audio_buffer.commit` first, otherwise the server returns zero results.
|
||||
2. `conversation.item.input_audio_transcription.delta` carries the **cumulative** segment text, not
|
||||
an increment — rendering it incrementally duplicates every word.
|
||||
3. A segment emits **two** `.completed` frames; the second has an empty `words_info`. The documented
|
||||
`is_last_result` flag is never `true` on this model, so completion is detected by event
|
||||
quiescence (no new event for ~700 ms after `done`) rather than by a terminal flag.
|
||||
4. Auth failures surface as **HTTP 403 at the upgrade handshake**, not WS close 1008. A tokenless
|
||||
handshake gets `101 Switching Protocols` with no echoed subprotocol, which Node's global
|
||||
`WebSocket` treats as fatal — so the minted token is mandatory, not optional.
|
||||
5. The documented `?token=` query param is **not** rejected (the docs claim it is). Do not put tokens
|
||||
in URLs: they leak into access logs. The token goes in an extra `Sec-WebSocket-Protocol` value.
|
||||
6. A `session.update` that omits `model` silently resets it to a model the server has not loaded
|
||||
(`conformer`), which yields an empty transcript. Echo the minted session config back verbatim.
|
||||
|
||||
## Latency
|
||||
|
||||
mint ~120 ms, handshake ~20 ms, first partial ~360 ms, final ≈ audio duration + 0.5 s. The realtime
|
||||
decoder is real-time bound, so `speed=2` (ffmpeg `atempo`) is the only way to beat wall-clock.
|
||||
|
||||
## Dev
|
||||
|
||||
```bash
|
||||
pi install /path/to/pi-asr # registers the checkout; edit, then restart pi
|
||||
pi --version # fast smoke test: prints extension load errors, if any
|
||||
```
|
||||
Reference in new issue
Block a user