GitHub - BoumedineBillal/esp32-pie-asr: Live English speech recognition on the ESP32-P4: a 65.7M-parameter streaming transducer in 2/4-bit weights, faster than
**Live English speech recognition on the ESP32-P4.** Open vocabulary, streaming, faster than real time, on the microcontroller alone: the board's microphone in, the words on its screen as you speak. No cloud, no neural accelerator; the two RISC-V cores and their PIE SIMD instructions.
The model is Kroko (Banafo), a 65.7M-parameter streaming Zipformer2 transducer, with its weights brought from 71 MB to 26.65 MB in 2 and 4 bits. The weights are on Hugging Face: BoumedineBillal/esp32-pie-asr-kroko-en.
| | | | --- | --- | | Accuracy | LibriSpeech test-clean **4.34 %** WER, test-other **10.26 %** (beam search, 4 hypotheses) | | Speed | **0.70× real time**: each 1.28 s of audio in 0.90 s, both cores at 400 MHz | | Delay | a word is final about **2 s** after it is spoken (median 1.9 to 2.3 s, 95 % within 4.1 s) | | Weights | **26.65 MB**, read from a microSD card at boot in 2.9 s | | Memory | weights and state in the 32 MB PSRAM; 121 KB of internal RAM left free (104 KB with the demos) | | Bit-exact | the chip computes exactly what the engine's PC build computes: the PC's WERs are the chip's | | Phrases | words and sentences to listen for (hotwords), switched on the screen, no retraining | | Voice control | six demos driven by spoken orders through word rules, no language model |
The firmware, `firmwares/voice_control`, opens on live transcription, then the voice-control demos. The transcriber's numbers were measured with `firmwares/live_transcriber`, the transcriber alone on the same engine, where the engine's board tests are.
Compared
[](https://github.com/BoumedineBillal/esp32-pie-asr#compared) | | Parameters | Weights | Runs on | Streaming | test-clean | test-other | Speed | | --- | --- | --- | --- | --- | --- | --- | --- | | **esp32-pie-asr** | 65.7M | **2/4-bit, 26.65 MB** | **ESP32-P4** (2 × 400 MHz RISC-V) | yes | **4.34 %** | **10.26 %** | **0.70× real time** | | Kroko as released (sherpa-onnx) | 65.7M | int8, 71 MB | PC | yes | 3.94 % | 9.17 % | | | Oído, streaming, with LM | 13M | int8 | ESP32-S3 (240 MHz Xtensa) | yes | 4.86 % | 11.03 % | final text 4.1 to 11.1 s after clips of 3.6 to 10.6 s | | Oído, whole utterance | 13M | int8 | ESP32-S3 | no | 3.70 % | 8.23 % | 1.97× real time (one core) | | Moonshine Tiny | 27.1M | float | CPU | no | 4.52 % | 11.71 % | | | Whisper tiny.en | 37.8M | float | CPU | no | 5.66 % | 15.45 % | | | ESP-SR MultiNet | | | ESP32-S3 | | a fixed list of up to 200 commands, not open vocabulary | | |
esp32-pie-asr and Kroko as released: measured here on all 2620 / 2939 clips with Whisper's English text normalizer (Kroko with sherpa-onnx 1.13.8, the same beam width, 2 s of silence after each clip so its last chunk is decoded). The others: as their authors publish them.
Where the accuracy goes, test-clean:
| | greedy | beam 4 | | --- | --- | --- | | Kroko as released, int8 ONNX | 4.03 % | 3.94 % | | 2/4-bit weights, float activations (PyTorch simulation) | 4.53 % | | | 2/4-bit weights, integer activations: **the chip** | **4.52 %** | **4.34 %** |
The 2/4-bit weights cost about half a point; the integer arithmetic on the chip costs nothing.
Phrases
[](https://github.com/BoumedineBillal/esp32-pie-asr#phrases)
A text file on the SD card lists words and sentences to listen for; a word the model spells differently gets its spoken forms (`ESP32 = ESP thirty two`). Each one is switched on or off on the screen. On 140 text-to-speech clips (35 sentences, 4 voices) with the default list:
| | the list's words found | its sentences found | false alarms (40 clips without them) | WER | | --- | --- | --- | --- | --- | | phrases off | 0 of 60 | 62 of 72 | 0 | 19.5 % | | **phrases on** | **46 of 60** | **70 of 72** | **0** | **3.4 %** |
Voice control
[](https://github.com/BoumedineBillal/esp32-pie-asr#voice-control)
The firmware's second page has six demos in which spoken orders drive a device simulated on the screen: a robot vacuum, a smart home, a car cabin, a factory line, a kitchen, a presenter. No language model: each demo is a table of word rules (an order's verb, then its target; numbers, rooms, "not the bedroom"), and the screen shows which words fired each action. Each demo has its own phrase list.
On 188 text-to-speech clips per demo (25 orders and 22 near misses, ordinary sentences with the demo's words, 4 voices):
| | orders exact, no list | with its list | false actions (88 near misses) | | --- | --- | --- | --- | | robot vacuum | 92 % | 96 % | 0 | | smart home | 93 % | 97 % | 0 | | car cabin | 93 % | 97 % | 0 | | factory line | 86 % | 94 % | 0 | | kitchen | 97 % | 98 % | 0 | | presenter | 97 % | 99 % | 0 |
The lists were chosen on these clips; every miss is a word heard wrong. The board gives the PC's actions on every clip. From the end of an order to its action: median 1.84 s on the vacuum's page, 1.97 s on the factory's, at most 2.57 s. With a demo's page drawing, the engine stays under real time (at most 0.81×).
Run it
[](https://github.com/BoumedineBillal/esp32-pie-asr#run-it) Hardware: an ESP32-P4 of chip revision v3 (ESP32-P4X) with 32 MB PSRAM, a microphone, a screen, a microSD card. Made and measured on Espressif's ESP32-P4X-Function-EV-Board (ES8311 microphone, 1024 × 600 MIPI-DSI panel).
The weights (26.65 MB) do not fit the board's 16 MB flash, so they live on a microSD card: at boot the firmware reads them from the card into the 32 MB PSRAM (2.9 s) and runs from there.
1. Put the weights on a FAT-formatted microSD card, in its top folder: git clone https://github.com/BoumedineBillal/esp32-pie-asr cd esp32-pie-asr python models/download.py --card <the microSD card's folder> # downloads, checks, copies 2. Put the card in the board's microSD slot. 3. Flash the firmware, from the browser (Chrome or Edge, no tools to install): **boumedinebillal.github.io/esp32-pie-asr**, or with ESP-IDF v6.1: cd firmwares/voice_control idf.py set-target esp32p4 idf.py -p PORT flash
At the first boot the firmware also writes its phrase lists on the card: `phrases.txt` for the transcriber and one per demo (`vacuum_phrases.txt`, `smarthome_phrases.txt`, ...). To change a list, edit its file on a PC: one word or sentence per line, `shown = spoken | spoken` for a word the model spells differently, `#` for a note; the screen shows up to 10 words and 10 sentences.
`firmwares/live_transcriber` (the transcriber alone) builds and flashes the same way.
The debug build adds a link to a PC over the board's USB High-Speed port (commands, the log, the words, screenshots) for the board tests (`debug/tests/run_all.py`, Windows):
idf.py -B build_debug -D SDKCONFIG=build_debug/sdkconfig \ -D SDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.debug" -p PORT flash
Inside
[](https://github.com/BoumedineBillal/esp32-pie-asr#inside) How the model fits and runs, in docs/design.md:
- **2/4-bit weights** in groups of 128, with a Hadamard rotation and one norm per group; 2 or 4 bits chosen per matrix by its sensitivity, without training.
- **Integer arithmetic**: int8 into the products, int16 between them, int8 attention caches, activations by table.
- **PIE SIMD kernels** written in assembly, each with a C reference that the host build runs; the products split over both cores.
- **Bit-exact with the PC**: deterministic math on both sides, checked on every test clip (the encoder's output CRC and the text) and on every kernel.
- **Streaming** in 1.28 s chunks with fixed-size caches; a beam search with phrase biasing; the transcript's word and line rules.
- **Voice control** by word rules, tested on the PC; on its pages a pause commits the search's best hypothesis, so an order acts about 1.9 s after its end instead of 3.1 s.
| Folder | | | --- | --- | | `pieasr/` | the engine: an ESP-IDF component, its host build (`host/`), its tests (`tests/`) | | `components/` | the speech service, the voice rules, the microphone, the screen and touch, the SD card, the debug link | | `firmwares/voice_control/` | the firmware: live transcription and the six demos, their clip tests on the board and the PC | | `firmwares/live_transcriber/` | the transcriber alone, with the engine's board tests | | `models/` | the weights' download | | `tools/eval/`, `benchmarks/` | the WER measurements and every text they scored |
Reproduce
[](https://github.com/BoumedineBillal/esp32-pie-asr#reproduce)
cmake -S pieasr/host -B build/host -DPIEASR_WEIGHTS=$PWD/models/kroko-en-q24-v1.bin cmake --build build/host && ctest --test-dir build/host # unit tests and the 8 golden clips python tools/eval/get_librispeech.py --set test-clean # 2620 clips; --set test-other: 2939 bash tools/eval/run_wer.sh data/test-clean int+beam4 # the chip's configuration, on the PC bash pieasr/tests/target/run.sh # every PIE kernel against its C reference, on the board
Next
[](https://github.com/BoumedineBillal/esp32-pie-asr#next)
- Support for the ESP32-S31, which has the same PIE 2.2 SIMD instructions as the ESP32-P4X.
- Arabic: a model for Modern Standard Arabic.
Credits
[](https://github.com/BoumedineBillal/esp32-pie-asr#credits)
- Kroko by Banafo: the model (community license, CC-BY-SA).
- Cactus Quants by Cactus Compute: the weight format (Hadamard-rotated groups of 128, Lloyd-Max levels).
- LibriSpeech (CC BY 4.0): the 8 test clips in `pieasr/tests/data` and the WER sets.
- Piper and its voices: the text-to-speech test clips (made by the tests' scripts, not in the repo).
- Espressif: ESP-IDF, and the ES8311 and EK79007 register values (Apache-2.0).
- Inter (SIL OFL 1.1) and stb_truetype (public domain or MIT).
License
[](https://github.com/BoumedineBillal/esp32-pie-asr#license) The code: Apache-2.0 (LICENSE). The weights: CC-BY-SA 4.0, as Kroko's.
Author
[](https://github.com/BoumedineBillal/esp32-pie-asr#author)
**Boumedine Billal**: GitHub · Hugging Face · LinkedIn · boumedinebillal@gmail.com
@misc{boumedine2026esp32pieasr, author = {Boumedine Billal}, title = {esp32-pie-asr: streaming speech recognition on the ESP32-P4}, year = {2026}, url = {https://github.com/BoumedineBillal/esp32-pie-asr} }