--- description: Using Asr.process() to benchmark ASR throughput and accuracy against a fixed audio dataset. --- # Benchmarking ::::{lang-set} :::{lang} python {api}`Asr.process` is a blocking call that takes a complete audio clip and returns the final transcript. It is suited for benchmarking throughput and accuracy against a fixed audio dataset. ::: :::{lang} c Pushing a complete audio clip and then waiting for completion gives the final transcript of that clip, which is what benchmarking throughput and accuracy against a fixed audio dataset needs. With the {doc}`session helper ` from the tarball's `examples/common` directory, that is one `abr_session_push` followed by one `abr_session_wait_for_completion`. ::: :::: :::{attention} Accuracy benchmarks should run in ACCURATE mode, the default. FAST mode trades accuracy for latency and inflates the measured WER/CER. FAST is not faster for batch processing either, because both modes run the same network at the same compute cost. ::: For real-time or streaming use, see {doc}`/sdk/asr/overview`. ## Processing a clip ::::{lang-set} :::{lang} python ```python Asr.process(data: bytes) -> AsrTranscript ``` Pushes all of `data` through the ASR pipeline, waits for the neural network to finish, and returns the transcript. The call blocks until the entire input has been processed; it does not return partial results. {api}`Asr.process` can be called multiple times on the same {api}`Asr` instance. Each call is self-contained: no audio state carries over between clips, so you can process a batch of clips back to back. `data` is PCM audio encoded as a little-endian 16-bit byte array at 16,000 Hz; see {doc}`/sdk/asr/input-format` for format requirements. The call returns an {api}`AsrTranscript`; call `.text` to get the final string. ::: :::{lang} c `abr_session_push` writes the whole clip into the input buffer, delivering chunks as they are produced, and `abr_session_wait_for_completion` pushes a flush marker and delivers the rest. The flush marker ends the segment: no audio state carries over to the next clip, so a batch of clips can go through the same application back to back. Underneath, that is {c:member}`abr_lib.buf_push` with `flush` set once the clip is in, and {c:member}`abr_lib.buf_pull` until it reports the marker; `examples/asr.c` in the tarball shows the loop in full. ::: :::: ## Example ::::{lang-set} :::{lang} python ```{example} benchmark.py ``` ::: :::{lang} c ```{example} benchmark.c ``` ::: :::: Each `.pcm` file must contain raw PCM audio in the format the application expects. See {doc}`/sdk/asr/input-format` for details on sample rate, bit depth, and channel layout. ## Scoring transcripts When computing WER or CER, score the final transcript: `transcript.text` on the returned {api}`AsrTranscript`. This text includes the post-processing pass (punctuation, capitalization, spell correction). Scoring intermediate chunks, or only the model output, measures text the pipeline would have revised and overstates the error rate. Apply the same text normalization to the reference and the hypothesis before scoring, and use the same normalizer the numbers you are comparing against were measured with. Niagara's published accuracy figures come from the [HuggingFace Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard), which applies its own text normalizer to both sides; scoring with that normalizer makes your numbers directly comparable. ## Errors ::::{lang-set} :::{lang} python | Exception | When raised | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `abr_sdk.exceptions.AbrSdkError` | Called on a closed `Asr` instance, for example after the `with` block has exited or after `asr.close()` was called explicitly. | | `abr_sdk.cabi.AbrSdkCAbiError` | The underlying C library returned a failure status. Check the log output for details. | ::: :::{lang} c {c:member}`abr_lib.buf_push` and {c:member}`abr_lib.buf_pull` return a negative {c:enum}`abr_status` when the application has failed, and keep returning it until the application is freed. The `session.h` helper records the first such status in `abr_session_t.status`, logs it, and turns later calls into no-ops. Check the log output for details. ::: ::::