first commit

2026-10-29 10:03:05 +01:00
commit 50ef195c35
@@ -0,0 +1,67 @@
+# LFM3-Audio-2B streaming voice demo
+
+Web UI + FastAPI WebSocket server for [LiquidAI/LFM3-Audio-2B](https://huggingface.co/LiquidAI/LFM3-Audio-2B):
+speak into the browser, get streamed text and speech back, with debug data for each turn.
+
+## Run
+
+```sh
+uv sync
+uv run python server.py        # http://127.0.0.1:8000
+```
+
+Hold the button (or **Space**) to talk; release to send, or type a message and press Enter. Modes: **voice chat**
+(interleaved text + speech, multi-turn history), **voice + tools**, **ASR**, and a **TTS** button.
+
+## Tool calling
+
+"Voice + tools" mode passes the schemas in `tools.py` to the chat template, with the system prompt
+`Respond with function calls, and interleaved text and audio.\n\nList of tools:` and no prefill. The model either
+emits `<tool_call><function=…><parameter=…>` blocks or answers with speech. The server parses the calls, runs them,
+appends the assistant `tool_calls` message and a `tool` result message, and generates again (up to 4 rounds).
+Test tools:
+
+- `get_current_time(timezone)`: real clock
+- `get_weather(city)`: **simulated**, stable per city
+- `calculate(expression)`: safe arithmetic evaluator
+- `set_theme_color(color)`: changes the page's accent color, so a successful call is visible right away
+
+Calls and results appear as cards in the chat and in the debug panel's "Tool calls" box; the raw markup shows
+in the token stream.
+
+**Force speech** (on by default): with greedy text decoding the model sometimes answers in plain text without
+speech. When on, rounds after a tool result are prefilled with `<|mixed_start|>`, and a first round whose first
+token isn't `<tool_call>` or a speech marker is stopped and restarted with that prefill (about 230 ms extra). The
+debug panel counts these as "speech restarts".
+
+## How it streams
+
+- **Mic → server:** AudioWorklet posts float32 PCM at the browser's native rate over the WebSocket while you hold;
+  the server resamples to 16 kHz on release.
+- **Server → browser:** `model.generate(streamer=…, audio_streamer=…)` runs in a worker thread. Text tokens are sent
+  as JSON events. Speech frames (12 codebooks, 80 ms each) go to a separate decoder thread that uses
+  `LiquidCodecStreamingCache` and sends 24 kHz float32 PCM as binary messages, every `LFM_CHUNK_FRAMES` frames
+  (default 3, i.e. 240 ms). The browser schedules them gaplessly.
+
+## Debug panel
+
+Per turn: input duration/RMS, prompt tokens, time to first text token / speech frame / audio chunk, generate time,
+RTF (gen time ÷ audio seconds, < 1 = faster than real time), frames/s (12.5 = real time), codec decode time and
+backlog, and playback underruns. There is also a client-side timeline, a raw token stream with special tokens
+highlighted, the exact prompt, and an event log. Each reply and recording gets a playable/downloadable WAV.
+
+## Config (env vars)
+
+| var | default | |
+|---|---|---|
+| `LFM_DEVICE` | cuda → mps → cpu | LM device |
+| `LFM_DTYPE` | bfloat16 (float32 on cpu) | LM dtype |
+| `LFM_CODEC_DEVICE` | cpu | codec device |
+| `LFM_CHUNK_FRAMES` | 3 | frames per audio chunk sent |
+| `HOST` / `PORT` | 127.0.0.1 / 8000 | |
+
+## Scripts
+
+- `scripts/smoke_test.py`: headless TTS → ASR → chat → ASR round trip; writes `smoke_out/*.wav`.
+- `scripts/ws_client_test.py <wav>`: drives the WebSocket like the browser does (2 turns).
+- `scripts/tools_test.py`: typed and spoken tool-calling requests against a running server.