# LFM3-Audio-2B streaming voice demo

Web UI + FastAPI WebSocket server for [LiquidAI/LFM3-Audio-2B](https://huggingface.co/LiquidAI/LFM3-Audio-2B):
speak into the browser, get streamed text and speech back, with debug data for each turn.

## Run

```sh
uv sync
uv run python server.py        # http://127.0.0.1:8000
```

Hold the button (or **Space**) to talk; release to send, or type a message and press Enter. Modes: **voice chat**
(interleaved text + speech, multi-turn history), **voice + tools**, **ASR**, and a **TTS** button.

## Tool calling

"Voice + tools" mode passes the schemas in `tools.py` to the chat template, with the system prompt
`Respond with function calls, and interleaved text and audio.\n\nList of tools:` and no prefill. The model either
emits `<tool_call><function=…><parameter=…>` blocks or answers with speech. The server parses the calls, runs them,
appends the assistant `tool_calls` message and a `tool` result message, and generates again (up to 4 rounds).
Test tools:

- `get_current_time(timezone)`: real clock
- `get_weather(city)`: **simulated**, stable per city
- `calculate(expression)`: safe arithmetic evaluator
- `set_theme_color(color)`: changes the page's accent color, so a successful call is visible right away

Calls and results appear as cards in the chat and in the debug panel's "Tool calls" box; the raw markup shows
in the token stream.

**Force speech** (on by default): with greedy text decoding the model sometimes answers in plain text without
speech. When on, rounds after a tool result are prefilled with `<|mixed_start|>`, and a first round whose first
token isn't `<tool_call>` or a speech marker is stopped and restarted with that prefill (about 230 ms extra). The
debug panel counts these as "speech restarts".

## How it streams

- **Mic → server:** AudioWorklet posts float32 PCM at the browser's native rate over the WebSocket while you hold;
  the server resamples to 16 kHz on release.
- **Server → browser:** `model.generate(streamer=…, audio_streamer=…)` runs in a worker thread. Text tokens are sent
  as JSON events. Speech frames (12 codebooks, 80 ms each) go to a separate decoder thread that uses
  `LiquidCodecStreamingCache` and sends 24 kHz float32 PCM as binary messages, every `LFM_CHUNK_FRAMES` frames
  (default 3, i.e. 240 ms). The browser schedules them gaplessly.

## Debug panel

Per turn: input duration/RMS, prompt tokens, time to first text token / speech frame / audio chunk, generate time,
RTF (gen time ÷ audio seconds, < 1 = faster than real time), frames/s (12.5 = real time), codec decode time and
backlog, and playback underruns. There is also a client-side timeline, a raw token stream with special tokens
highlighted, the exact prompt, and an event log. Each reply and recording gets a playable/downloadable WAV.

## Config (env vars)

| var | default | |
|---|---|---|
| `LFM_DEVICE` | cuda → mps → cpu | LM device |
| `LFM_DTYPE` | bfloat16 (float32 on cpu) | LM dtype |
| `LFM_CODEC_DEVICE` | cpu | codec device |
| `LFM_CHUNK_FRAMES` | 3 | frames per audio chunk sent |
| `HOST` / `PORT` | 127.0.0.1 / 8000 | |

## Scripts

- `scripts/smoke_test.py`: headless TTS → ASR → chat → ASR round trip; writes `smoke_out/*.wav`.
- `scripts/ws_client_test.py <wav>`: drives the WebSocket like the browser does (2 turns).
- `scripts/tools_test.py`: typed and spoken tool-calling requests against a running server.