--- description: Overview of automatic speech recognition in the ABR SDK, covering on-device transcription, streaming, and modes. --- # Overview The ABR SDK transcribes speech to text on-device, with no network round-trip. You push audio in as it arrives and receive text chunks back through a callback in real time. ::::{lang-set} :::{lang} python The ASR API is provided by the {api}`module.asr` module. The main class you interact with is {api}`Asr`. ::: :::{lang} c The ASR API is the {c:struct}`abr_lib` function table, loaded from an ASR package with {c:func}`abr_lib_open`. An {c:type}`abr_app_t` created from it is the application you stream audio through; the {doc}`session helper ` from the tarball's `examples/common` directory wraps the streaming in {c:func}`abr_session_push` and {c:func}`abr_session_wait_for_completion`. ::: :::: ## How a transcript is built ASR produces text incrementally as audio arrives. Each piece of output is an {api}`AsrChunk`; the {api}`AsrTranscript` helper collects chunks and applies all overwrites automatically if you only need the final text. The full mechanics (chunk types, model output modes, and replace ranges) are covered in {doc}`/sdk/asr/transcription-stages`. ## Streaming transcription Streaming transcription is the primary way to use ASR in the SDK. You push audio in chunks as it arrives, and a callback fires each time the model produces new text. This works for live microphone input, network audio streams, and long files you want to start transcribing before they finish loading, as well as for short clips and complete recordings. :::{attention} The function below shows the pattern. `audio_source` can be any iterable that yields raw PCM bytes (a microphone reader, file reader, or network stream). For copy-paste-ready code, see {doc}`/sdk/asr/examples`. ::: ::::{lang-set} :::{lang} python ```{example} streaming.py ``` The {api}`Asr.push` call returns as soon as all input bytes have been accepted; it can block briefly when the model's input buffer is full, and the `on_chunk` callback may fire during the call as chunks are produced. The callback passed to the first `push()` of a session is used for the whole session; the `on_chunk` argument on later calls is ignored (passing the same callback each time, as the examples do, is harmless). {api}`Asr.wait_for_completion` blocks until the model has finished processing all audio that has been pushed, and may also invoke `on_chunk`. ::: :::{lang} c ```{example} streaming.c ``` `abr_session_push` returns as soon as all input bytes have been accepted; it blocks while the model's input buffer is full, and `on_chunk` fires during the call as chunks are produced. `abr_session_wait_for_completion` pushes a flush marker and blocks until the model has finished processing all audio that has been pushed, invoking `on_chunk` for the last chunks. Both drive {c:member}`abr_lib.buf_push`, {c:member}`abr_lib.buf_pull` and an event set; `examples/asr.c` shows that loop in full. ::: :::: For benchmarking against a fixed dataset, see {doc}`/sdk/asr/benchmarking`. ## Transcription modes The SDK offers two transcription modes: - {api}`AsrMode.FAST` emits text as soon as possible, around 150 milliseconds after the audio arrives. It does not wait for additional context. - {api}`AsrMode.ACCURATE` uses a small amount of subsequent audio as context before emitting, roughly one second after the audio arrives, which improves accuracy at the cost of latency. If you do not specify a mode, the library defaults to ACCURATE. Both modes run the same network at the same compute cost, so FAST is not faster for batch processing. The mode only changes when the model commits to output. You set the mode when constructing the {api}`Asr` instance: ::::{lang-set} :::{lang} python ```python from abr_sdk.asr import Asr, AsrMode with Asr(LIBRARY_PATH, mode=AsrMode.FAST) as asr: ... ``` ::: :::{lang} c The mode is the `asr_mode` configuration option of {c:member}`abr_lib.app_create`, `"fast"` or `"accurate"`: ```c abr_config_t cfg; cfg.key = "asr_mode"; cfg.value.type = ABR_VAR_STR; cfg.value.value.str = "fast"; abr_session_t session; abr_session_open(&session, LIBRARY_PATH, &cfg, 1); ``` ::: :::: The right choice depends on what consumes the transcript. {api}`AsrMode.FAST` suits a live display where text must appear as soon as possible. {api}`AsrMode.ACCURATE` suits everything else, including subtitles and transcripts sent to a downstream language model. See {doc}`/sdk/asr/transcription-stages` for guidance on picking between them. ## The overwrite model **Text appears early and is revised in place.** The model output, FAST or ACCURATE depending on your mode setting, appears first. Later chunks overwrite it with punctuated, spell-corrected text. This is how the SDK delivers low-latency output while still refining it before it is considered final. How you handle the overwrites depends on where the text is rendered. A terminal can rewrite a line in place; a web UI tracks byte ranges and updates DOM spans; a downstream language model should receive the final transcript after {api}`Asr.wait_for_completion`. The full mechanics (model output modes, post-processing, and the replace-range fields) are covered in {doc}`/sdk/asr/transcription-stages`. ## What you need to use ASR To run ASR you need three things: ::::{lang-set} :::{lang} python - **The SDK Python package** (`pip install abr-sdk`). ::: :::{lang} c - **The SDK C binding**, compiled into your program (see {doc}`/sdk/getting-started/installation`). ::: :::: - **An application package** for your target platform. This archive contains the compiled model, the network weights, and supporting files. You download it from the [ABR developer portal](https://dev.appliedbrainresearch.com) and extract it on the device. See {doc}`/sdk/concepts/application-packages`. - **Audio in the expected format.** ASR consumes raw PCM bytes at a specific sample rate. The required format and sample rates are listed in {doc}`/sdk/asr/input-format`. ::::{lang-set} :::{lang} python The {api}`Asr` constructor takes the path to the package's shared library. ::: :::{lang} c {c:func}`abr_lib_open` takes the path to the package's shared library, and {c:member}`abr_lib.app_create` creates the ASR application from it. ::: :::: :::{admonition} Next steps :class: hint - New to the SDK? Start with {doc}`/sdk/getting-started/asr-quickstart`. - {doc}`/sdk/asr/transcription-stages`: how the model produces and revises output, and how to handle chunk overwrites. - {doc}`/sdk/asr/examples`: full, runnable examples for microphone, file, LLM feed, and more. :::