68 lines
3.3 KiB
Markdown
68 lines
3.3 KiB
Markdown
# LFM3-Audio-2B streaming voice demo
|
|
|
|
Web UI + FastAPI WebSocket server for [LiquidAI/LFM3-Audio-2B](https://huggingface.co/LiquidAI/LFM3-Audio-2B):
|
|
speak into the browser, get streamed text and speech back, with debug data for each turn.
|
|
|
|
## Run
|
|
|
|
```sh
|
|
uv sync
|
|
uv run python server.py # http://127.0.0.1:8000
|
|
```
|
|
|
|
Hold the button (or **Space**) to talk; release to send, or type a message and press Enter. Modes: **voice chat**
|
|
(interleaved text + speech, multi-turn history), **voice + tools**, **ASR**, and a **TTS** button.
|
|
|
|
## Tool calling
|
|
|
|
"Voice + tools" mode passes the schemas in `tools.py` to the chat template, with the system prompt
|
|
`Respond with function calls, and interleaved text and audio.\n\nList of tools:` and no prefill. The model either
|
|
emits `<tool_call><function=…><parameter=…>` blocks or answers with speech. The server parses the calls, runs them,
|
|
appends the assistant `tool_calls` message and a `tool` result message, and generates again (up to 4 rounds).
|
|
Test tools:
|
|
|
|
- `get_current_time(timezone)`: real clock
|
|
- `get_weather(city)`: **simulated**, stable per city
|
|
- `calculate(expression)`: safe arithmetic evaluator
|
|
- `set_theme_color(color)`: changes the page's accent color, so a successful call is visible right away
|
|
|
|
Calls and results appear as cards in the chat and in the debug panel's "Tool calls" box; the raw markup shows
|
|
in the token stream.
|
|
|
|
**Force speech** (on by default): with greedy text decoding the model sometimes answers in plain text without
|
|
speech. When on, rounds after a tool result are prefilled with `<|mixed_start|>`, and a first round whose first
|
|
token isn't `<tool_call>` or a speech marker is stopped and restarted with that prefill (about 230 ms extra). The
|
|
debug panel counts these as "speech restarts".
|
|
|
|
## How it streams
|
|
|
|
- **Mic → server:** AudioWorklet posts float32 PCM at the browser's native rate over the WebSocket while you hold;
|
|
the server resamples to 16 kHz on release.
|
|
- **Server → browser:** `model.generate(streamer=…, audio_streamer=…)` runs in a worker thread. Text tokens are sent
|
|
as JSON events. Speech frames (12 codebooks, 80 ms each) go to a separate decoder thread that uses
|
|
`LiquidCodecStreamingCache` and sends 24 kHz float32 PCM as binary messages, every `LFM_CHUNK_FRAMES` frames
|
|
(default 3, i.e. 240 ms). The browser schedules them gaplessly.
|
|
|
|
## Debug panel
|
|
|
|
Per turn: input duration/RMS, prompt tokens, time to first text token / speech frame / audio chunk, generate time,
|
|
RTF (gen time ÷ audio seconds, < 1 = faster than real time), frames/s (12.5 = real time), codec decode time and
|
|
backlog, and playback underruns. There is also a client-side timeline, a raw token stream with special tokens
|
|
highlighted, the exact prompt, and an event log. Each reply and recording gets a playable/downloadable WAV.
|
|
|
|
## Config (env vars)
|
|
|
|
| var | default | |
|
|
|---|---|---|
|
|
| `LFM_DEVICE` | cuda → mps → cpu | LM device |
|
|
| `LFM_DTYPE` | bfloat16 (float32 on cpu) | LM dtype |
|
|
| `LFM_CODEC_DEVICE` | cpu | codec device |
|
|
| `LFM_CHUNK_FRAMES` | 3 | frames per audio chunk sent |
|
|
| `HOST` / `PORT` | 127.0.0.1 / 8000 | |
|
|
|
|
## Scripts
|
|
|
|
- `scripts/smoke_test.py`: headless TTS → ASR → chat → ASR round trip; writes `smoke_out/*.wav`.
|
|
- `scripts/ws_client_test.py <wav>`: drives the WebSocket like the browser does (2 turns).
|
|
- `scripts/tools_test.py`: typed and spoken tool-calling requests against a running server.
|