feat: speech-to-text for pi via NVIDIA Speech NIM realtime ASR
- asr_transcribe / asr_health tools + the /asr command, with live partials, per-word timings, transcript export and TUI rendering - zero-dependency realtime client (https + Node 22 global WebSocket): mint transcription session -> subprotocol handshake -> event loop -> quiescence - decodes anything ffmpeg supports (and native 16k mono/stereo pcm16/pcm_f32le wav) to the 16 kHz mono PCM16 the NIM mandates - credentials via ASR_API_KEY / ASR_NIM_URL or ~/.pi/agent/asr.json; never in source - README documents the six places the published API reference diverges from the deployed protocol (commit vs done, cumulative deltas, double .completed, HTTP 403 at upgrade, ?token= not rejected, model reset on session.update)
This commit is contained in:
iulianp committed
commit
19faa8b088
5 files changed
+893
No files matched your search
+10
@@ -0,0 +1,10 @@
|
|||||||
|
node_modules/
|
||||||
|
package-lock.json
|
||||||
|
yarn-error.log
|
||||||
|
*.wav
|
||||||
|
*.pcm
|
||||||
|
*.mp3
|
||||||
|
*.mp4
|
||||||
|
asr.json
|
||||||
|
.DS_Store
|
||||||
|
*.log
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# pi-asr
|
||||||
|
|
||||||
|
Speech-to-text for the **pi** coding agent, backed by an NVIDIA Speech NIM realtime streaming ASR
|
||||||
|
endpoint (Riva-style realtime protocol, Parakeet/Nemotron models).
|
||||||
|
|
||||||
|
Zero runtime dependencies: `https`, the Node 22 global `WebSocket`, and `ffmpeg` for container/codec
|
||||||
|
decoding. No Python, no virtualenvs, no local model weights.
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# from git over ssh (uses your ~/.ssh/config)
|
||||||
|
pi install git:git@git.iaminsane.online:iulianp/pi-asr.git
|
||||||
|
|
||||||
|
# or from a local checkout
|
||||||
|
pi install /path/to/pi-asr
|
||||||
|
|
||||||
|
# or try it without installing
|
||||||
|
pi -e /path/to/pi-asr
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configure
|
||||||
|
|
||||||
|
The API key is never read from source. Priority: explicit option, then env var, then config file.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export ASR_API_KEY=sk-...
|
||||||
|
export ASR_NIM_URL=https://your-host/openai/streaming-asr # optional, has a default
|
||||||
|
```
|
||||||
|
|
||||||
|
or drop a config file (mode 0600 recommended) at `~/.pi/agent/asr.json` (override with `$ASR_CONFIG`):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"apiKey": "sk-...",
|
||||||
|
"baseUrl": "https://your-host/openai/streaming-asr"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`baseUrl` is the HTTPS base. The WebSocket URL is derived from it: `wss://` + the same host/path +
|
||||||
|
`/realtime?intent=transcription`.
|
||||||
|
|
||||||
|
## What you get
|
||||||
|
|
||||||
|
| surface | what it does |
|
||||||
|
| --- | --- |
|
||||||
|
| `asr_transcribe` | transcribe a local audio/video file; live partials while it streams; optional per-word timings |
|
||||||
|
| `asr_health` | `health/live` + `health/ready` + a minted session, so you see the model/language/sample-rate the server will actually use |
|
||||||
|
| `/asr <file> [--lang xx-YY] [--words] [--speed N] [--start S] [--out f.txt]` | human-facing transcript viewer |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pi -p "use the asr_transcribe tool on ./meeting.m4a with words=true and save to meeting.txt"
|
||||||
|
```
|
||||||
|
|
||||||
|
Accepted input: anything ffmpeg can decode (`wav` s16le/f32le, `mp3`, `m4a`, `aac`, `ogg`, `opus`,
|
||||||
|
`flac`, `mp4`, `mov`, `mkv`), resampled to the 16 kHz mono PCM16 the NIM mandates. Without ffmpeg,
|
||||||
|
16 kHz mono wav still works.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```text
|
||||||
|
extensions/asr.ts pi registration: tools, TUI rendering, /asr command
|
||||||
|
src/client.ts RealtimeAsrClient: mint, WS subprotocol handshake, event loop, decode
|
||||||
|
```
|
||||||
|
|
||||||
|
## Protocol notes (things the published API reference gets wrong)
|
||||||
|
|
||||||
|
Hard-won while building this against a real deployment. The client handles all of it.
|
||||||
|
|
||||||
|
1. `input_audio_buffer.done` does **not** flush the decoder — you must send
|
||||||
|
`input_audio_buffer.commit` first, otherwise the server returns zero results.
|
||||||
|
2. `conversation.item.input_audio_transcription.delta` carries the **cumulative** segment text, not
|
||||||
|
an increment — rendering it incrementally duplicates every word.
|
||||||
|
3. A segment emits **two** `.completed` frames; the second has an empty `words_info`. The documented
|
||||||
|
`is_last_result` flag is never `true` on this model, so completion is detected by event
|
||||||
|
quiescence (no new event for ~700 ms after `done`) rather than by a terminal flag.
|
||||||
|
4. Auth failures surface as **HTTP 403 at the upgrade handshake**, not WS close 1008. A tokenless
|
||||||
|
handshake gets `101 Switching Protocols` with no echoed subprotocol, which Node's global
|
||||||
|
`WebSocket` treats as fatal — so the minted token is mandatory, not optional.
|
||||||
|
5. The documented `?token=` query param is **not** rejected (the docs claim it is). Do not put tokens
|
||||||
|
in URLs: they leak into access logs. The token goes in an extra `Sec-WebSocket-Protocol` value.
|
||||||
|
6. A `session.update` that omits `model` silently resets it to a model the server has not loaded
|
||||||
|
(`conformer`), which yields an empty transcript. Echo the minted session config back verbatim.
|
||||||
|
|
||||||
|
## Latency
|
||||||
|
|
||||||
|
mint ~120 ms, handshake ~20 ms, first partial ~360 ms, final ≈ audio duration + 0.5 s. The realtime
|
||||||
|
decoder is real-time bound, so `speed=2` (ffmpeg `atempo`) is the only way to beat wall-clock.
|
||||||
|
|
||||||
|
## Dev
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pi install /path/to/pi-asr # registers the checkout; edit, then restart pi
|
||||||
|
pi --version # fast smoke test: prints extension load errors, if any
|
||||||
|
```
|
||||||
@@ -0,0 +1,303 @@
|
|||||||
|
/**
|
||||||
|
* pi ASR extension — speech-to-text through the NVIDIA Speech NIM realtime WebSocket API.
|
||||||
|
*
|
||||||
|
* Tools:
|
||||||
|
* asr_transcribe transcribe an audio/video file (wav, mp3, m4a, aac, ogg, flac, opus, mp4, mov, mkv)
|
||||||
|
* asr_health pre-flight: health/live + health/ready + the model the NIM actually advertises
|
||||||
|
* Command:
|
||||||
|
* /asr <file> [--lang xx-YY] [--words] [--speed N] [--start S] [--out transcript.txt]
|
||||||
|
*
|
||||||
|
* Config via env (built-in defaults otherwise):
|
||||||
|
* ASR_NIM_URL https://ai.iaminsane.online/openai/streaming-asr
|
||||||
|
* ASR_API_KEY key sent as `Authorization: Bearer ...` on every HTTPS/WS call
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { existsSync, statSync } from "node:fs";
|
||||||
|
import { mkdir, writeFile } from "node:fs/promises";
|
||||||
|
import { dirname, isAbsolute, relative, resolve } from "node:path";
|
||||||
|
import type { AgentToolResult, ExtensionAPI, ExtensionContext } from "@mariozechner/pi-coding-agent";
|
||||||
|
import { Text } from "@mariozechner/pi-tui";
|
||||||
|
import { Type } from "@sinclair/typebox";
|
||||||
|
import {
|
||||||
|
DEFAULT_BASE_URL, RealtimeAsrClient, hasFfmpeg, loadPcm16,
|
||||||
|
type TranscriptResult, type WordInfo,
|
||||||
|
} from "../src/client.js";
|
||||||
|
|
||||||
|
interface TranscribeParams {
|
||||||
|
path: string;
|
||||||
|
language?: string;
|
||||||
|
model?: string;
|
||||||
|
words?: boolean;
|
||||||
|
punctuation?: boolean;
|
||||||
|
start?: number;
|
||||||
|
speed?: number;
|
||||||
|
pace?: boolean;
|
||||||
|
out?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface AsrDetails {
|
||||||
|
path: string;
|
||||||
|
text: string;
|
||||||
|
seconds: number;
|
||||||
|
wallMs: number;
|
||||||
|
model: string | null;
|
||||||
|
sessionId: string;
|
||||||
|
wordCount: number;
|
||||||
|
segments: { transcript: string; isLast: boolean }[];
|
||||||
|
words?: WordInfo[];
|
||||||
|
outPath?: string;
|
||||||
|
partial?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
const wordCount = (t: string): number => (t.trim() ? t.trim().split(/\s+/).length : 0);
|
||||||
|
const rel = (p: string, cwd: string): string => {
|
||||||
|
const r = relative(cwd, p);
|
||||||
|
return r && !r.startsWith("..") && !isAbsolute(r) ? r : p;
|
||||||
|
};
|
||||||
|
|
||||||
|
export default function asrExtension(pi: ExtensionAPI) {
|
||||||
|
async function transcribeFile(
|
||||||
|
rawPath: string,
|
||||||
|
params: TranscribeParams,
|
||||||
|
cwd: string,
|
||||||
|
signal: AbortSignal | undefined,
|
||||||
|
onDelta?: (t: string) => void,
|
||||||
|
): Promise<{ result: TranscriptResult; wallMs: number; absPath: string }> {
|
||||||
|
const absPath = resolve(cwd, rawPath.replace(/^@+/, ""));
|
||||||
|
if (!existsSync(absPath)) throw new Error(`no such file: ${absPath}`);
|
||||||
|
if (statSync(absPath).isDirectory()) throw new Error(`${absPath} is a directory, expected an audio file`);
|
||||||
|
|
||||||
|
const client = new RealtimeAsrClient({
|
||||||
|
language: params.language ?? "en-US",
|
||||||
|
model: params.model || null,
|
||||||
|
punctuation: params.punctuation ?? true,
|
||||||
|
wordOffsets: params.words ?? false,
|
||||||
|
});
|
||||||
|
|
||||||
|
const t0 = Date.now();
|
||||||
|
const pcm = await loadPcm16(absPath, {
|
||||||
|
sampleRate: 16000, channels: 1,
|
||||||
|
start: params.start ?? 0, speed: params.speed ?? 1,
|
||||||
|
});
|
||||||
|
if (!pcm.length) throw new Error(`decoded ${absPath} to zero bytes of audio (corrupt or empty track?)`);
|
||||||
|
|
||||||
|
const result = await client.transcribe(pcm, { pace: params.pace ?? true, frameMs: 100, onDelta, signal });
|
||||||
|
if (signal?.aborted) throw new Error("transcription cancelled");
|
||||||
|
if (result.failed.length) {
|
||||||
|
throw new Error(`server reported a transcription failure: ${JSON.stringify(result.failed[0]).slice(0, 300)}`);
|
||||||
|
}
|
||||||
|
return { result, wallMs: Date.now() - t0, absPath };
|
||||||
|
}
|
||||||
|
|
||||||
|
async function runTranscribe(
|
||||||
|
params: TranscribeParams,
|
||||||
|
ctx: ExtensionContext,
|
||||||
|
onUpdate?: (r: AgentToolResult<AsrDetails>) => void,
|
||||||
|
signal?: AbortSignal,
|
||||||
|
): Promise<AgentToolResult<AsrDetails>> {
|
||||||
|
if (!params.path?.trim()) throw new Error("`path` is required");
|
||||||
|
if (signal?.aborted) throw new Error("transcription cancelled");
|
||||||
|
let lastPartial = "";
|
||||||
|
const started = Date.now();
|
||||||
|
|
||||||
|
const { result, wallMs, absPath } = await transcribeFile(params.path.trim(), params, ctx.cwd, signal, (t) => {
|
||||||
|
if (!t || t === lastPartial) return;
|
||||||
|
lastPartial = t;
|
||||||
|
onUpdate?.({
|
||||||
|
content: [{ type: "text", text: t }],
|
||||||
|
details: {
|
||||||
|
path: rel(absPath, ctx.cwd), text: t, seconds: 0, wallMs: Date.now() - started,
|
||||||
|
model: null, sessionId: "", wordCount: wordCount(t), segments: [], partial: t,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
const outPath = params.out ? resolve(ctx.cwd, params.out.replace(/^@+/, "")) : undefined;
|
||||||
|
if (outPath) { await mkdir(dirname(outPath), { recursive: true }); await writeFile(outPath, `${result.text}\n`, "utf-8"); }
|
||||||
|
|
||||||
|
const details: AsrDetails = {
|
||||||
|
path: rel(absPath, ctx.cwd),
|
||||||
|
text: result.text,
|
||||||
|
seconds: result.seconds,
|
||||||
|
wallMs,
|
||||||
|
model: result.model,
|
||||||
|
sessionId: result.sessionId,
|
||||||
|
wordCount: wordCount(result.text),
|
||||||
|
segments: result.segments,
|
||||||
|
...(params.words ? { words: result.words } : {}),
|
||||||
|
...(outPath ? { outPath: rel(outPath, ctx.cwd) } : {}),
|
||||||
|
};
|
||||||
|
|
||||||
|
const summary = `${result.text || "(empty transcript)"}\n\n— ${details.wordCount} words, ${result.seconds}s audio, ${wallMs}ms wall, model=${result.model ?? "?"}${outPath ? `, wrote ${details.outPath}` : ""}`;
|
||||||
|
const timings = params.words && result.words.length
|
||||||
|
? `\n\nword timings:\n${result.words.map(w => ` ${(+w.start_time).toFixed(2).padStart(7)} → ${(+w.end_time).toFixed(2).padStart(7)} ${w.word}${w.confidence != null ? ` (${(+w.confidence).toFixed(2)})` : ""}`).join("\n")}`
|
||||||
|
: "";
|
||||||
|
return { content: [{ type: "text", text: summary + timings }], details };
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------------------------------------------------------------- tools -- */
|
||||||
|
pi.registerTool({
|
||||||
|
name: "asr_transcribe",
|
||||||
|
label: "ASR transcribe",
|
||||||
|
description: "Speech-to-text a local audio or video file via the NVIDIA Speech NIM realtime streaming endpoint (Parakeet/Nemotron streaming ASR). Accepts wav, mp3, m4a, aac, ogg, opus, flac, mp4, mov, mkv - anything ffmpeg can decode; non-16kHz/mono input is resampled automatically. Returns the final transcript, optionally with per-word timings.",
|
||||||
|
promptSnippet: "Transcribe an audio or video file to text (speech-to-text) with the streaming ASR server",
|
||||||
|
promptGuidelines: [
|
||||||
|
"Use asr_transcribe for any speech-to-text request (voice notes, meeting recordings, podcast clips, screen recordings) instead of hand-rolling ffmpeg/python.",
|
||||||
|
"asr_transcribe is real-time bound: budget roughly the duration of the audio. Pass speed=2 to halve wall time when slight accuracy loss is fine, or start=N to skip leading silence.",
|
||||||
|
"If asr_transcribe returns an empty transcript, run asr_health to confirm the server is up and the language/model are right.",
|
||||||
|
],
|
||||||
|
parameters: Type.Object({
|
||||||
|
path: Type.String({ description: "Audio/video file, relative to cwd or absolute (a leading @ is stripped)" }),
|
||||||
|
language: Type.Optional(Type.String({ description: "BCP-47 language tag, default en-US" })),
|
||||||
|
model: Type.Optional(Type.String({ description: "Override the ASR model name (default: whatever the server advertises)" })),
|
||||||
|
words: Type.Optional(Type.Boolean({ description: "Include per-word timings + confidence (default false)" })),
|
||||||
|
punctuation: Type.Optional(Type.Boolean({ description: "Server-side automatic punctuation (default true)" })),
|
||||||
|
start: Type.Optional(Type.Number({ description: "Skip the first N seconds" })),
|
||||||
|
speed: Type.Optional(Type.Number({ description: "Playback rate 0.5..4 (ffmpeg atempo); >1 is faster, slightly less accurate" })),
|
||||||
|
pace: Type.Optional(Type.Boolean({ description: "Stream at 1x realtime pacing (default true)" })),
|
||||||
|
out: Type.Optional(Type.String({ description: "Also write the plain transcript to this path" })),
|
||||||
|
}),
|
||||||
|
|
||||||
|
async execute(_id, params: TranscribeParams, signal, onUpdate, ctx): Promise<AgentToolResult<AsrDetails>> {
|
||||||
|
return runTranscribe(params, ctx, onUpdate as any, signal);
|
||||||
|
},
|
||||||
|
|
||||||
|
renderCall(args: any, theme: any) {
|
||||||
|
const bits = [
|
||||||
|
theme.fg("toolTitle", theme.bold("asr_transcribe ")),
|
||||||
|
theme.fg("muted", String(args?.path ?? "")),
|
||||||
|
];
|
||||||
|
if (args?.language && args.language !== "en-US") bits.push(theme.fg("dim", ` ${args.language}`));
|
||||||
|
if (args?.speed && args.speed !== 1) bits.push(theme.fg("dim", ` ${args.speed}x`));
|
||||||
|
if (args?.words) bits.push(theme.fg("dim", " +words"));
|
||||||
|
if (args?.out) bits.push(theme.fg("dim", ` → ${args.out}`));
|
||||||
|
return new Text(bits.join(""), 0, 0);
|
||||||
|
},
|
||||||
|
|
||||||
|
renderResult(result: any, options: any, theme: any) {
|
||||||
|
const d: AsrDetails | undefined = result?.details;
|
||||||
|
if (!d) return new Text("", 0, 0);
|
||||||
|
if (options?.isPartial && d.partial) return new Text(theme.fg("warning", `… ${d.partial}`), 0, 0);
|
||||||
|
const head = theme.fg("muted", `${d.path} ${d.seconds}s audio → ${d.wallMs}ms ${d.model ?? ""}`.trimEnd());
|
||||||
|
const body = d.text
|
||||||
|
? theme.fg("success", d.text)
|
||||||
|
: theme.fg("warning", "(empty transcript — check --lang / model, or run asr_health)");
|
||||||
|
let out = `${head}\n${body}`;
|
||||||
|
if (options?.expanded && d.words?.length) {
|
||||||
|
out += "\n" + d.words.slice(0, 50).map(w => theme.fg("dim", ` ${(+w.start_time).toFixed(2).padStart(7)} → ${(+w.end_time).toFixed(2).padStart(7)} ${w.word}`)).join("\n");
|
||||||
|
} else if (d.wordCount) {
|
||||||
|
out += " " + theme.fg("dim", `(${d.wordCount} words)`);
|
||||||
|
}
|
||||||
|
if (options?.expanded && d.outPath) out += "\n" + theme.fg("dim", ` wrote ${d.outPath}`);
|
||||||
|
return new Text(out, 0, 0);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
pi.registerTool({
|
||||||
|
name: "asr_health",
|
||||||
|
label: "ASR health",
|
||||||
|
description:
|
||||||
|
"Check the ASR server before/while debugging: GET health/live + health/ready, and mint a realtime session to " +
|
||||||
|
"report the model, language and sample rate the server will actually use. Also verifies ffmpeg availability " +
|
||||||
|
"and reports the API key being used (masked).",
|
||||||
|
promptSnippet: "Check ASR server health and the currently advertised model/sample-rate config",
|
||||||
|
parameters: Type.Object({}),
|
||||||
|
|
||||||
|
async execute(_id, _params, _signal, _onUpdate, ctx): Promise<AgentToolResult<any>> {
|
||||||
|
const client = new RealtimeAsrClient();
|
||||||
|
const [live, ready, minted, ffmpeg] = await Promise.all([
|
||||||
|
client.health("live", 10000).catch((e: any) => ({ error: e?.message ?? String(e) })),
|
||||||
|
client.health("ready", 10000).catch((e: any) => ({ error: e?.message ?? String(e) })),
|
||||||
|
client.mint(10000).catch((e: any) => ({ error: e?.message ?? String(e) })),
|
||||||
|
hasFfmpeg(),
|
||||||
|
]);
|
||||||
|
const ok = live?.status === "live" && ready?.status === "ready" && !minted?.error;
|
||||||
|
const key = client.cfg.apiKey;
|
||||||
|
const secs = minted?.client_secret ? Math.max(0, (minted.client_secret.expires_at ?? 0) - Math.floor(Date.now() / 1000)) : 0;
|
||||||
|
const text = [
|
||||||
|
`baseUrl ${process.env.ASR_NIM_URL ?? DEFAULT_BASE_URL}`,
|
||||||
|
`apiKey ${key ? `${key.slice(0, 5)}…${key.slice(-3)} (${key.length} chars)` : "NOT SET"}`,
|
||||||
|
`health/live ${JSON.stringify(live)}`,
|
||||||
|
`health/ready ${JSON.stringify(ready)}`,
|
||||||
|
minted?.error
|
||||||
|
? `mint ERROR ${minted.error}`
|
||||||
|
: [
|
||||||
|
`mint ${minted.id}`,
|
||||||
|
` model ${minted.input_audio_transcription?.model}`,
|
||||||
|
` language ${minted.input_audio_transcription?.language}`,
|
||||||
|
` audio ${minted.input_audio_format}, ${minted.input_audio_params?.sample_rate_hz}Hz x${minted.input_audio_params?.num_channels}`,
|
||||||
|
` ws token single-use, expires in ${secs}s (mint immediately before each connect)`,
|
||||||
|
].join("\n"),
|
||||||
|
`ffmpeg ${ffmpeg ? "available (any container/codec accepted)" : "MISSING — only 16kHz mono wav files will work"}`,
|
||||||
|
`note the realtime decoder runs at ~1x realtime; a 10min file takes ~10min (speed=2 halves it)`,
|
||||||
|
].join("\n");
|
||||||
|
if (!ok) throw new Error(`ASR server is not usable:\n${text}`);
|
||||||
|
return { content: [{ type: "text", text }], details: { ok, live, ready, minted, ffmpeg } };
|
||||||
|
},
|
||||||
|
|
||||||
|
renderResult(result: any, _options: any, theme: any) {
|
||||||
|
const d: any = result?.details;
|
||||||
|
if (!d) return new Text("", 0, 0);
|
||||||
|
const m = d.minted?.input_audio_transcription;
|
||||||
|
const p = d.minted?.input_audio_params;
|
||||||
|
return new Text([
|
||||||
|
d.ok ? theme.fg("success", "✓ ASR server live") : theme.fg("error", "✗ ASR server not usable"),
|
||||||
|
theme.fg("muted", ` model ${m?.model ?? d.minted?.error ?? "?"}`),
|
||||||
|
theme.fg("dim", ` ${p?.sample_rate_hz ?? "?"}Hz x${p?.num_channels ?? "?"} ${d.minted?.input_audio_format ?? ""} · ${m?.language ?? "?"} · ffmpeg ${d.ffmpeg ? "ok" : "MISSING"}`),
|
||||||
|
].join("\n"), 0, 0);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
/* --------------------------------------------------------------- /asr --- */
|
||||||
|
pi.registerCommand("asr", {
|
||||||
|
description: "Transcribe an audio/video file: /asr <file> [--lang xx-YY] [--words] [--speed N] [--start S] [--out file.txt]",
|
||||||
|
handler: async (args: string, ctx: ExtensionContext) => {
|
||||||
|
const toks = args.trim().split(/\s+/).filter(Boolean);
|
||||||
|
if (!toks.length) {
|
||||||
|
ctx.ui.notify("usage: /asr <file> [--lang xx-YY] [--words] [--speed N] [--start S] [--out file.txt]", "info");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const p: TranscribeParams = { path: "" };
|
||||||
|
for (let i = 0; i < toks.length; i++) {
|
||||||
|
const t = toks[i];
|
||||||
|
if (t === "--lang") p.language = toks[++i];
|
||||||
|
else if (t === "--words") p.words = true;
|
||||||
|
else if (t === "--speed") p.speed = Number.parseFloat(toks[++i]) || 1;
|
||||||
|
else if (t === "--start") p.start = Number.parseFloat(toks[++i]) || 0;
|
||||||
|
else if (t === "--out") p.out = toks[++i];
|
||||||
|
else if (!p.path) p.path = t;
|
||||||
|
}
|
||||||
|
if (!p.path) { ctx.ui.notify("no file given", "error"); return; }
|
||||||
|
|
||||||
|
let last = "";
|
||||||
|
ctx.ui.setStatus("asr", `transcribing ${p.path} …`);
|
||||||
|
try {
|
||||||
|
const r = await runTranscribe(p, ctx);
|
||||||
|
const d = r.details;
|
||||||
|
last = d.text || "(empty transcript)";
|
||||||
|
ctx.ui.notify(`${d.path}: ${d.wordCount} words, ${d.seconds}s audio in ${d.wallMs}ms`, "info");
|
||||||
|
if (ctx.hasUI) {
|
||||||
|
await ctx.ui.custom<void>((_tui, theme, _kb, done) => {
|
||||||
|
const body = new Text([
|
||||||
|
theme.fg("toolTitle", theme.bold(` transcribed ${d.path} `)),
|
||||||
|
theme.fg("muted", `${d.seconds}s audio → ${d.wallMs}ms · ${d.model ?? "?"} · ${d.wordCount} words`),
|
||||||
|
"",
|
||||||
|
theme.fg(d.text ? "success" : "warning", d.text || "(empty transcript)"),
|
||||||
|
...(d.words?.length ? [theme.fg("dim", `\n${d.words.slice(0, 12).map(w => `${(+w.start_time).toFixed(2).padStart(7)} → ${(+w.end_time).toFixed(2).padStart(7)} ${w.word}`).join("\n")}`)] : []),
|
||||||
|
"",
|
||||||
|
theme.fg("dim", " enter/esc to close"),
|
||||||
|
].join("\n"), 1, 1);
|
||||||
|
body.onKey = (key: string) => { if (key === "return" || key === "escape" || key === " ") { done(undefined); return true; } return true; };
|
||||||
|
return body;
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
console.log(last);
|
||||||
|
}
|
||||||
|
} catch (e: any) {
|
||||||
|
ctx.ui.notify(`ASR failed: ${e?.message ?? e}`, "error");
|
||||||
|
} finally {
|
||||||
|
ctx.ui.setStatus("asr", undefined);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
{
|
||||||
|
"name": "pi-asr",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"description": "Speech-to-text for the pi coding agent: asr_transcribe / asr_health tools and the /asr command, backed by an NVIDIA Speech NIM realtime streaming ASR endpoint (Parakeet/Nemotron).",
|
||||||
|
"keywords": [
|
||||||
|
"pi-package",
|
||||||
|
"pi-extension",
|
||||||
|
"asr",
|
||||||
|
"speech-to-text",
|
||||||
|
"transcription",
|
||||||
|
"nvidia-nim",
|
||||||
|
"streaming-asr"
|
||||||
|
],
|
||||||
|
"type": "module",
|
||||||
|
"peerDependencies": {
|
||||||
|
"@mariozechner/pi-ai": "*",
|
||||||
|
"@mariozechner/pi-coding-agent": "*",
|
||||||
|
"@mariozechner/pi-tui": "*",
|
||||||
|
"@sinclair/typebox": "*"
|
||||||
|
},
|
||||||
|
"pi": {
|
||||||
|
"extensions": [
|
||||||
|
"./extensions"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=22"
|
||||||
|
}
|
||||||
|
}
|
||||||
+456
@@ -0,0 +1,456 @@
|
|||||||
|
/**
|
||||||
|
* NVIDIA Speech NIM realtime ASR client (Riva realtime WebSocket API).
|
||||||
|
* Zero dependencies, Node >= 22 (global WebSocket).
|
||||||
|
*
|
||||||
|
* Docs: https://docs.nvidia.com/nim/speech/latest/reference/api-references/asr/realtime-asr.html
|
||||||
|
*
|
||||||
|
* Verified against the live deployment (behind Traefik `replacePathRegex`
|
||||||
|
* ^/openai/streaming-asr/(.*) -> /v1/$1 plus the `qwen-audio-api-key` middleware):
|
||||||
|
*
|
||||||
|
* 1. GET {base}/health/live | /health/ready Bearer key -> {status:'live'|'ready'}
|
||||||
|
* 2. POST {base}/realtime/transcription_sessions Bearer key -> session + client_secret.value
|
||||||
|
* 3. WS {base}/realtime?intent=transcription
|
||||||
|
* Sec-WebSocket-Protocol: "realtime, realtime-token.<value>" (never a URL query param)
|
||||||
|
* 4. transcription_session.update -> transcription_session.updated
|
||||||
|
* 5. input_audio_buffer.append (base64 pcm16) * N
|
||||||
|
* 6. input_audio_buffer.commit -> input_audio_buffer.committed <-- REQUIRED, see below
|
||||||
|
* 7. server emits conversation.item.input_audio_transcription.delta|.completed|.failed
|
||||||
|
*
|
||||||
|
* Three behaviours differ from the published docs and are handled here:
|
||||||
|
* a) `input_audio_buffer.done` alone never flushes the decoder -> zero results. An explicit
|
||||||
|
* `input_audio_buffer.commit` is what triggers recognition.
|
||||||
|
* b) `delta` is CUMULATIVE (each frame repeats the whole utterance so far), not incremental.
|
||||||
|
* c) `is_last_result` never arrives as true, so completion is detected by event quiescence.
|
||||||
|
* And the server sends two `.completed` frames per segment: the first carries `words_info`,
|
||||||
|
* the second has empty `words_info` -> the word-bearing one must be kept.
|
||||||
|
* d) Auth failures arrive as an HTTP **403 during the upgrade handshake**, not as WebSocket
|
||||||
|
* close code 1008 like the docs claim. A tokenless handshake returns 101 with NO echoed
|
||||||
|
* subprotocol, which Node's global WebSocket (undici) treats as a fatal error
|
||||||
|
* ("Server did not respond with sent protocols") -> the minted token is mandatory in Node.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { spawn } from "node:child_process";
|
||||||
|
import { readFileSync } from "node:fs";
|
||||||
|
import { readFile } from "node:fs/promises";
|
||||||
|
import { homedir } from "node:os";
|
||||||
|
import { extname } from "node:path";
|
||||||
|
|
||||||
|
export const EV = {
|
||||||
|
conversationCreated: "conversation.created",
|
||||||
|
sessionUpdated: "transcription_session.updated",
|
||||||
|
bufferCommitted: "input_audio_buffer.committed",
|
||||||
|
bufferCleared: "input_audio_buffer.cleared",
|
||||||
|
delta: "conversation.item.input_audio_transcription.delta",
|
||||||
|
completed: "conversation.item.input_audio_transcription.completed",
|
||||||
|
failed: "conversation.item.input_audio_transcription.failed",
|
||||||
|
error: "error",
|
||||||
|
} as const;
|
||||||
|
|
||||||
|
export interface AsrConfig {
|
||||||
|
baseUrl: string;
|
||||||
|
apiKey: string;
|
||||||
|
language: string;
|
||||||
|
model: string | null;
|
||||||
|
sampleRate: number;
|
||||||
|
channels: number;
|
||||||
|
punctuation: boolean;
|
||||||
|
wordOffsets: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const DEFAULT_BASE_URL = "https://ai.iaminsane.online/openai/streaming-asr";
|
||||||
|
|
||||||
|
export interface AsrFileConfig { baseUrl?: string; apiKey?: string }
|
||||||
|
|
||||||
|
/** Optional on-disk config so the key never has to live in source: ~/.pi/agent/asr.json (or $ASR_CONFIG). */
|
||||||
|
export function configFilePath(): string {
|
||||||
|
return process.env.ASR_CONFIG ?? [homedir(), ".pi", "agent", "asr.json"].join("/");
|
||||||
|
}
|
||||||
|
|
||||||
|
let cachedFileConfig: AsrFileConfig | null = null;
|
||||||
|
export function readConfigFile(): AsrFileConfig {
|
||||||
|
if (cachedFileConfig) return cachedFileConfig;
|
||||||
|
try {
|
||||||
|
cachedFileConfig = JSON.parse(readFileSync(configFilePath(), "utf-8")) as AsrFileConfig;
|
||||||
|
} catch {
|
||||||
|
cachedFileConfig = {};
|
||||||
|
}
|
||||||
|
return cachedFileConfig;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Env wins, so the key can live outside the file: ASR_API_KEY / ASR_NIM_URL. */
|
||||||
|
export function defaultConfig(overrides: Partial<AsrConfig> = {}): AsrConfig {
|
||||||
|
return {
|
||||||
|
baseUrl: (overrides.baseUrl ?? process.env.ASR_NIM_URL ?? readConfigFile().baseUrl ?? DEFAULT_BASE_URL).replace(/\/+$/, ""),
|
||||||
|
apiKey: overrides.apiKey ?? process.env.ASR_API_KEY ?? readConfigFile().apiKey ?? "",
|
||||||
|
language: overrides.language ?? "en-US",
|
||||||
|
model: overrides.model ?? null, // null => whatever the NIM advertises on mint
|
||||||
|
sampleRate: overrides.sampleRate ?? 16000,
|
||||||
|
channels: overrides.channels ?? 1,
|
||||||
|
punctuation: overrides.punctuation ?? true,
|
||||||
|
wordOffsets: overrides.wordOffsets ?? false,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const uid = (p: string = "event"): string => `${p}_${crypto.randomUUID()}`;
|
||||||
|
const sleep = (ms: number): Promise<void> => new Promise(r => setTimeout(r, ms));
|
||||||
|
|
||||||
|
export interface WordInfo {
|
||||||
|
word: string;
|
||||||
|
start_time: number;
|
||||||
|
end_time: number;
|
||||||
|
confidence?: number;
|
||||||
|
speaker_tag?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface TranscriptResult {
|
||||||
|
text: string;
|
||||||
|
segments: { transcript: string; isLast: boolean }[];
|
||||||
|
words: WordInfo[];
|
||||||
|
failed: unknown[];
|
||||||
|
seconds: number;
|
||||||
|
model: string | null;
|
||||||
|
sessionId: string;
|
||||||
|
protocol: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
type Listener = (ev: any) => void;
|
||||||
|
|
||||||
|
export class RealtimeAsrClient {
|
||||||
|
cfg: AsrConfig;
|
||||||
|
verbose: boolean;
|
||||||
|
private listeners: Map<string, Set<Listener>> = new Map();
|
||||||
|
|
||||||
|
constructor(opts: Partial<AsrConfig> & { verbose?: boolean } = {}) {
|
||||||
|
this.cfg = defaultConfig(opts);
|
||||||
|
this.verbose = opts.verbose ?? false;
|
||||||
|
}
|
||||||
|
|
||||||
|
log(...a: unknown[]): void {
|
||||||
|
if (this.verbose) console.error("[asr]", ...a);
|
||||||
|
}
|
||||||
|
|
||||||
|
on(type: string, fn: Listener): () => void {
|
||||||
|
if (!this.listeners.has(type)) this.listeners.set(type, new Set());
|
||||||
|
this.listeners.get(type)!.add(fn);
|
||||||
|
return () => this.listeners.get(type)?.delete(fn);
|
||||||
|
}
|
||||||
|
|
||||||
|
private emit(ev: any): void {
|
||||||
|
for (const fn of this.listeners.get(ev?.type) ?? []) { try { fn(ev); } catch {} }
|
||||||
|
for (const fn of this.listeners.get("*") ?? []) { try { fn(ev); } catch {} }
|
||||||
|
}
|
||||||
|
|
||||||
|
private headers(extra: Record<string, string> = {}): Record<string, string> {
|
||||||
|
const h: Record<string, string> = { ...extra };
|
||||||
|
// Gateway middleware `qwen-audio-api-key` -> 403 {"Invalid or missing API Key"} without it.
|
||||||
|
if (this.cfg.apiKey) h.Authorization = `Bearer ${this.cfg.apiKey}`;
|
||||||
|
return h;
|
||||||
|
}
|
||||||
|
|
||||||
|
get wsUrl(): string {
|
||||||
|
return `${this.cfg.baseUrl.replace(/^http/, "ws")}/realtime?intent=transcription`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The gateway answers 403 {Invalid or missing API Key} without a key, which is cryptic; fail with instructions instead. */
|
||||||
|
private requireKey(): void {
|
||||||
|
if (this.cfg.apiKey) return;
|
||||||
|
throw new Error(
|
||||||
|
"No ASR API key configured. Set ASR_API_KEY, or write { \"apiKey\": \"sk-...\" } to " + configFilePath() +
|
||||||
|
" (or point ASR_CONFIG at your file). Base URL: " + this.cfg.baseUrl,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** GET {base}/health/live | /health/ready */
|
||||||
|
async health(which: "live" | "ready" = "live", timeout: number = 15000): Promise<any> {
|
||||||
|
this.requireKey();
|
||||||
|
const r = await fetch(`${this.cfg.baseUrl}/health/${which}`, {
|
||||||
|
headers: this.headers(),
|
||||||
|
signal: AbortSignal.timeout?.(timeout),
|
||||||
|
});
|
||||||
|
const t = await r.text();
|
||||||
|
if (!r.ok) throw new Error(`health/${which} -> HTTP ${r.status}: ${t.slice(0, 200)}`);
|
||||||
|
try { return JSON.parse(t); } catch { return { raw: t }; }
|
||||||
|
}
|
||||||
|
|
||||||
|
/** POST {base}/realtime/transcription_sessions — single-use token, 60s TTL, so mint right before connecting. */
|
||||||
|
async mint(timeout: number = 15000): Promise<any> {
|
||||||
|
this.requireKey();
|
||||||
|
const r = await fetch(`${this.cfg.baseUrl}/realtime/transcription_sessions`, {
|
||||||
|
method: "POST",
|
||||||
|
headers: this.headers({ "Content-Type": "application/json" }),
|
||||||
|
signal: AbortSignal.timeout?.(timeout),
|
||||||
|
});
|
||||||
|
const t = await r.text();
|
||||||
|
if (!r.ok) {
|
||||||
|
const why = r.status === 403 || r.status === 401 ? "gateway rejected the API key" : `HTTP ${r.status}`;
|
||||||
|
throw new Error(`mint session failed: ${why}: ${t.slice(0, 200)}`);
|
||||||
|
}
|
||||||
|
return JSON.parse(t);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Payload for transcription_session.update. The server substitutes *defaults* for anything
|
||||||
|
* omitted (model silently becomes "conformer", which is not loaded here and yields zero
|
||||||
|
* results), so echo back the minted session and apply overrides on top.
|
||||||
|
*/
|
||||||
|
sessionConfig(minted: any = {}, overrides: Record<string, any> = {}): Record<string, any> {
|
||||||
|
const base: Record<string, any> = {
|
||||||
|
modalities: minted.modalities ?? ["text"],
|
||||||
|
input_audio_format: "pcm16",
|
||||||
|
input_audio_transcription: {
|
||||||
|
language: this.cfg.language,
|
||||||
|
model: this.cfg.model ?? minted.input_audio_transcription?.model,
|
||||||
|
prompt: minted.input_audio_transcription?.prompt ?? "",
|
||||||
|
},
|
||||||
|
input_audio_params: {
|
||||||
|
sample_rate_hz: this.cfg.sampleRate,
|
||||||
|
num_channels: this.cfg.channels,
|
||||||
|
},
|
||||||
|
recognition_config: {
|
||||||
|
...(minted.recognition_config ?? {}),
|
||||||
|
max_alternatives: 1,
|
||||||
|
enable_automatic_punctuation: !!this.cfg.punctuation,
|
||||||
|
enable_word_time_offsets: !!this.cfg.wordOffsets,
|
||||||
|
enable_profanity_filter: false,
|
||||||
|
enable_verbatim_transcripts: false,
|
||||||
|
},
|
||||||
|
speaker_diarization: minted.speaker_diarization ?? { enable_speaker_diarization: false, max_speaker_count: 8 },
|
||||||
|
word_boosting: minted.word_boosting ?? { enable_word_boosting: false, word_boosting_list: [] },
|
||||||
|
endpointing_config: minted.endpointing_config ?? {
|
||||||
|
start_history: 0, start_threshold: 0, stop_history: 0, stop_threshold: 0,
|
||||||
|
stop_history_eou: 0, stop_threshold_eou: 0,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
return { ...base, ...overrides };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Open the socket. `onDelta` gets every cumulative partial, `onFinal` each completed segment. */
|
||||||
|
async open(opts: { token?: string; timeout?: number; onDelta?: (text: string) => void; onFinal?: (ev: any) => void } = {}): Promise<SocketHandle> {
|
||||||
|
const token = opts.token ?? null;
|
||||||
|
const protocols = token ? ["realtime", `realtime-token.${token}`] : ["realtime"];
|
||||||
|
const ws = new WebSocket(this.wsUrl, { protocols, headers: this.headers() });
|
||||||
|
const st: SocketHandle = { events: [], closed: null, protocol: null };
|
||||||
|
|
||||||
|
await new Promise<void>((res, rej) => {
|
||||||
|
const t = setTimeout(() => { try { ws.close(); } catch {} rej(new Error("WebSocket handshake timed out")); }, opts.timeout ?? 20000);
|
||||||
|
ws.onopen = () => { clearTimeout(t); st.protocol = ws.protocol; res(); };
|
||||||
|
ws.onerror = (e: any) => {
|
||||||
|
clearTimeout(t);
|
||||||
|
const msg = e?.message || e?.error?.message || e?.type || "unknown";
|
||||||
|
rej(new Error(msg.includes("did not respond with sent protocols")
|
||||||
|
? `${msg} — the server accepted a tokenless handshake but echoed no subprotocol; always mint a client_secret first`
|
||||||
|
: `WebSocket error: ${msg}`));
|
||||||
|
};
|
||||||
|
ws.onclose = (e: any) => { st.closed = { code: e?.code, reason: e?.reason || "" }; this.log("closed", st.closed.code, st.closed.reason); };
|
||||||
|
ws.onmessage = (m: any) => {
|
||||||
|
const raw = m?.data ?? m;
|
||||||
|
let ev: any;
|
||||||
|
try { ev = JSON.parse(typeof raw === "string" ? raw : Buffer.from(raw as any).toString("utf-8")); } catch { return; }
|
||||||
|
st.events.push(ev);
|
||||||
|
this.log("<-", ev.type);
|
||||||
|
if (ev.type === EV.delta && opts.onDelta) opts.onDelta(ev.delta ?? "");
|
||||||
|
if (ev.type === EV.completed && opts.onFinal) opts.onFinal(ev);
|
||||||
|
this.emit(ev);
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
st.send = (obj: Record<string, any>) => ws.send(JSON.stringify({ event_id: uid(), ...obj }));
|
||||||
|
st.expect = (type: string, ms: number = 20000) => {
|
||||||
|
const hit = st.events.find(e => e.type === type);
|
||||||
|
if (hit) return Promise.resolve(hit);
|
||||||
|
return new Promise<any>((res, rej) => {
|
||||||
|
const t = setTimeout(() => rej(new Error(`timeout waiting for "${type}" (saw: ${st.events.map(e => e.type).join(",") || "nothing"})`)), ms);
|
||||||
|
const off = this.on(type, ev => { clearTimeout(t); res(ev); });
|
||||||
|
if (st.closed) { clearTimeout(t); off(); rej(new Error(`socket closed (code ${st.closed.code}) while waiting for "${type}"`)); }
|
||||||
|
});
|
||||||
|
};
|
||||||
|
st.close = () => { try { ws.close(1000, "bye"); } catch {} };
|
||||||
|
return st;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Convenience wrapper: mint -> open -> configure (caller keeps the handle for live/mic streaming). */
|
||||||
|
async openSession(opts: { session?: Record<string, any>; onDelta?: (t: string) => void; onFinal?: (ev: any) => void } = {}): Promise<{ handle: SocketHandle; minted: any }> {
|
||||||
|
const minted = await this.mint();
|
||||||
|
const handle = await this.open({ token: minted.client_secret?.value ?? null, onDelta: opts.onDelta, onFinal: opts.onFinal });
|
||||||
|
handle.send({ type: "transcription_session.update", session: this.sessionConfig(minted, opts.session ?? {}) });
|
||||||
|
await handle.expect(EV.sessionUpdated, 20000);
|
||||||
|
return { handle, minted };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* One-shot transcribe of raw s16le pcm16 audio.
|
||||||
|
* `onDelta` receives cumulative partials (great for a live UI), return value has final text.
|
||||||
|
*/
|
||||||
|
async transcribe(pcm16: Buffer, opts: {
|
||||||
|
frameMs?: number; pace?: boolean; paceMs?: number; session?: Record<string, any>;
|
||||||
|
quietMs?: number; emptyGraceMs?: number; finalTimeout?: number; tailMs?: number;
|
||||||
|
onDelta?: (t: string) => void; signal?: AbortSignal;
|
||||||
|
} = {}): Promise<TranscriptResult> {
|
||||||
|
const buf = toBuffer(pcm16);
|
||||||
|
const frameMs = opts.frameMs ?? 100;
|
||||||
|
const frameBytes = Math.max(2, Math.round((this.cfg.sampleRate * frameMs) / 1000) * 2 * this.cfg.channels);
|
||||||
|
const secs = buf.length / (this.cfg.sampleRate * 2 * this.cfg.channels);
|
||||||
|
const finalTimeout = opts.finalTimeout ?? Math.max(30000, Math.round(secs * 1000) + 30000);
|
||||||
|
|
||||||
|
const minted = await this.mint();
|
||||||
|
if (opts.signal?.aborted) throw new Error("cancelled");
|
||||||
|
const handle = await this.open({ token: minted.client_secret?.value ?? null });
|
||||||
|
|
||||||
|
const deltas: string[] = [];
|
||||||
|
const finals: Map<string, any> = new Map();
|
||||||
|
const failures: unknown[] = [];
|
||||||
|
const offs = [
|
||||||
|
this.on(EV.delta, e => deltas.push(e.delta ?? "")),
|
||||||
|
this.on(EV.completed, (e: any) => {
|
||||||
|
// keep the word-bearing copy: the server sends a second, words-less duplicate per segment
|
||||||
|
const key = e.item_id ?? "_";
|
||||||
|
const prev = finals.get(key);
|
||||||
|
const nw = e.words_info?.words?.length ?? 0;
|
||||||
|
const pw = prev?.words_info?.words?.length ?? 0;
|
||||||
|
finals.set(key, prev && pw > nw && prev.transcript ? { ...e, words_info: prev.words_info } : e);
|
||||||
|
}),
|
||||||
|
this.on(EV.failed, (e: any) => failures.push(e)),
|
||||||
|
];
|
||||||
|
|
||||||
|
try {
|
||||||
|
handle.send({ type: "transcription_session.update", session: this.sessionConfig(minted, opts.session ?? {}) });
|
||||||
|
await handle.expect(EV.sessionUpdated, 20000);
|
||||||
|
|
||||||
|
for (let i = 0; i < buf.length; i += frameBytes) {
|
||||||
|
if (opts.signal?.aborted) throw new Error("cancelled");
|
||||||
|
handle.send({
|
||||||
|
type: "input_audio_buffer.append",
|
||||||
|
audio: buf.subarray(i, Math.min(i + frameBytes, buf.length)).toString("base64"),
|
||||||
|
});
|
||||||
|
if (opts.pace !== false) await sleep(opts.paceMs ?? frameMs);
|
||||||
|
}
|
||||||
|
|
||||||
|
handle.send({ type: "input_audio_buffer.commit" }); // <-- the actual flush
|
||||||
|
const t0 = Date.now(), deadline = t0 + finalTimeout;
|
||||||
|
const quietMs = opts.quietMs ?? 1500, emptyGraceMs = opts.emptyGraceMs ?? 8000;
|
||||||
|
let lastCount = 0, lastChange = t0;
|
||||||
|
while (Date.now() < deadline) {
|
||||||
|
if (opts.signal?.aborted) break;
|
||||||
|
const n = finals.size;
|
||||||
|
if (n !== lastCount) { lastCount = n; lastChange = Date.now(); }
|
||||||
|
if (failures.length) break;
|
||||||
|
if (n > 0 && Date.now() - lastChange >= quietMs) break;
|
||||||
|
if (n === 0 && Date.now() - t0 >= emptyGraceMs) break;
|
||||||
|
await sleep(120);
|
||||||
|
}
|
||||||
|
|
||||||
|
handle.send({ type: "input_audio_buffer.done" });
|
||||||
|
await sleep(opts.tailMs ?? 600);
|
||||||
|
|
||||||
|
const list = Array.from(finals.values());
|
||||||
|
const text = list.length
|
||||||
|
? list.map(f => f.transcript ?? "").filter(Boolean).join(" ").replace(/\s+/g, " ").trim()
|
||||||
|
: (deltas.at(-1) ?? "").replace(/\s+/g, " ").trim();
|
||||||
|
|
||||||
|
return {
|
||||||
|
text,
|
||||||
|
segments: list.map(f => ({ transcript: f.transcript ?? "", isLast: f.is_last_result ?? false })),
|
||||||
|
words: list.flatMap(f => f.words_info?.words ?? []),
|
||||||
|
failed: failures,
|
||||||
|
seconds: +secs.toFixed(3),
|
||||||
|
model: minted.input_audio_transcription?.model ?? null,
|
||||||
|
sessionId: minted.id ?? "",
|
||||||
|
protocol: handle.protocol,
|
||||||
|
};
|
||||||
|
} finally {
|
||||||
|
offs.forEach(f => f());
|
||||||
|
try { handle.send?.({ type: "input_audio_buffer.clear" }); } catch {}
|
||||||
|
handle.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SocketHandle {
|
||||||
|
events: any[];
|
||||||
|
closed: { code: number; reason: string } | null;
|
||||||
|
protocol: string | null;
|
||||||
|
send?: (obj: Record<string, any>) => void;
|
||||||
|
expect?: (type: string, ms?: number) => Promise<any>;
|
||||||
|
close?: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
function toBuffer(x: any): Buffer {
|
||||||
|
if (Buffer.isBuffer(x)) return x;
|
||||||
|
if (x instanceof Uint8Array) return Buffer.from(x);
|
||||||
|
if (x instanceof ArrayBuffer) return Buffer.from(new Uint8Array(x));
|
||||||
|
if (x?.buffer instanceof ArrayBuffer) return Buffer.from(new Uint8Array(x.buffer));
|
||||||
|
throw new Error("expected pcm16 audio as Buffer/Uint8Array/ArrayBuffer");
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ------------------------------------------------------------------ audio -- */
|
||||||
|
|
||||||
|
/** Decode wav/mp3/m4a/aac/ogg/flac/mp4/video... to interleaved s16le at the target rate. Needs ffmpeg. */
|
||||||
|
export async function decodeToPcm16(
|
||||||
|
path: string,
|
||||||
|
{ sampleRate = 16000, channels = 1, start = 0, speed = 1 }: { sampleRate?: number; channels?: number; start?: number; speed?: number } = {},
|
||||||
|
): Promise<Buffer> {
|
||||||
|
const args: string[] = ["-v", "error"];
|
||||||
|
if (start > 0) args.push("-ss", String(start));
|
||||||
|
args.push("-i", path);
|
||||||
|
if (speed !== 1 && speed > 0) {
|
||||||
|
// atempo only accepts 0.5..100; chain it for extremes
|
||||||
|
let rate = Math.max(0.25, Math.min(4, speed));
|
||||||
|
const filters: string[] = [];
|
||||||
|
while (rate < 0.5) { filters.push("atempo=0.5"); rate /= 0.5; }
|
||||||
|
while (rate > 2) { filters.push("atempo=2.0"); rate /= 2.0; }
|
||||||
|
if (rate !== 1) filters.push(`atempo=${rate.toFixed(6)}`);
|
||||||
|
if (filters.length) args.push("-af", filters.join(","));
|
||||||
|
}
|
||||||
|
args.push("-ac", String(channels), "-ar", String(sampleRate), "-f", "s16le", "-");
|
||||||
|
const chunks: Buffer[] = [];
|
||||||
|
const { code, stderr } = await run("ffmpeg", args, b => chunks.push(b));
|
||||||
|
const out = Buffer.concat(chunks);
|
||||||
|
if (!out.length) throw new Error(`ffmpeg produced no audio for ${path} (exit ${code}) ${stderr.slice(0, 300)}`.trim());
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Native RIFF/EXTENSIBLE parser so plain 16k-mono wav files work without ffmpeg on PATH. */
|
||||||
|
export async function readWavAsPcm16(path: string, sampleRate: number = 16000): Promise<Buffer> {
|
||||||
|
const buf = await readFile(path);
|
||||||
|
if (buf.toString("ascii", 0, 4) !== "RIFF" || buf.toString("ascii", 8, 12) !== "WAVE") throw new Error("not a wav file");
|
||||||
|
let pos = 12, fmt: any = null, data: Buffer | null = null;
|
||||||
|
while (pos + 8 <= buf.length) {
|
||||||
|
const id = buf.toString("ascii", pos, pos + 4);
|
||||||
|
const size = buf.readUInt32LE(pos + 4);
|
||||||
|
const body = buf.subarray(pos + 8, Math.min(pos + 8 + size, buf.length));
|
||||||
|
if (id === "fmt ") fmt = { format: body.readUInt16LE(0), channels: body.readUInt16LE(2), rate: body.readUInt32LE(4), bits: body.readUInt16LE(14) };
|
||||||
|
else if (id === "data") data = body;
|
||||||
|
else if (id === "LIST" || id === "INFO" || id === "fact") { /* skip */ }
|
||||||
|
pos += 8 + size + (size % 2);
|
||||||
|
}
|
||||||
|
if (!fmt || !data) throw new Error("wav is missing fmt /data chunks");
|
||||||
|
if (fmt.bits !== 16 || fmt.format !== 1) throw new Error(`wav is ${fmt.bits}bit/format${fmt.format}, need 16bit PCM (use ffmpeg) — falling back`);
|
||||||
|
if (fmt.channels !== 1) throw new Error(`wav has ${fmt.channels} channels, need mono (use ffmpeg)`);
|
||||||
|
if (fmt.rate !== sampleRate) throw new Error(`wav is ${fmt.rate}Hz, need ${sampleRate}Hz (use ffmpeg)`);
|
||||||
|
return data;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** PCM16 with ffmpeg-first strategy and a native wav fast path. */
|
||||||
|
export async function loadPcm16(path: string, opts: { sampleRate?: number; channels?: number; start?: number; speed?: number } = {}): Promise<Buffer> {
|
||||||
|
const sampleRate = opts.sampleRate ?? 16000;
|
||||||
|
const channels = opts.channels ?? 1;
|
||||||
|
if (!opts.start && !opts.speed && extname(path).toLowerCase() === ".wav") {
|
||||||
|
try { return await readWavAsPcm16(path, sampleRate); } catch { /* fall through to ffmpeg */ }
|
||||||
|
}
|
||||||
|
return decodeToPcm16(path, { sampleRate, channels, start: opts.start ?? 0, speed: opts.speed ?? 1 });
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function hasFfmpeg(): Promise<boolean> {
|
||||||
|
try { await run("ffmpeg", ["-version"], () => {}); return true; } catch { return false; }
|
||||||
|
}
|
||||||
|
|
||||||
|
function run(cmd: string, args: string[], onData: (b: Buffer) => void): Promise<{ code: number; stderr: string }> {
|
||||||
|
return new Promise((res, rej) => {
|
||||||
|
const child = spawn(cmd, args, { stdio: ["ignore", "pipe", "pipe"] });
|
||||||
|
let stderr = "";
|
||||||
|
child.stdout.on("data", onData);
|
||||||
|
child.stderr.on("data", b => { stderr += b.toString(); });
|
||||||
|
child.on("error", e => rej(new Error((e as any).code === "ENOENT" ? `${cmd} is not installed / not on PATH` : e.message)));
|
||||||
|
child.on("close", code => res({ code: code ?? 0, stderr }));
|
||||||
|
});
|
||||||
|
}
|
||||||
Reference in new issue
Block a user