Benchmarking#

process()buf_push is a blocking call that takes a complete audio clip and returns the final transcript. It is suited for benchmarking throughput and accuracy against a fixed audio dataset.

Pushing a complete audio clip and then waiting for completion gives the final transcript of that clip, which is what benchmarking throughput and accuracy against a fixed audio dataset needs. With the session helper from the tarball’s examples/common directory, that is one abr_session_push followed by one abr_session_wait_for_completion.

Attention

Accuracy benchmarks should run in ACCURATE mode, the default. FAST mode trades accuracy for latency and inflates the measured WER/CER. FAST is not faster for batch processing either, because both modes run the same network at the same compute cost.

For real-time or streaming use, see Overview.

Processing a clip#

Asr.process(data: bytes) -> AsrTranscript

Pushes all of data through the ASR pipeline, waits for the neural network to finish, and returns the transcript. The call blocks until the entire input has been processed; it does not return partial results.

process()buf_push can be called multiple times on the same AsrAsr instance. Each call is self-contained: no audio state carries over between clips, so you can process a batch of clips back to back.

data is PCM audio encoded as a little-endian 16-bit byte array at 16,000 Hz; see Input format for format requirements. The call returns an AsrTranscriptAsrTranscript; call .text to get the final string.

abr_session_push writes the whole clip into the input buffer, delivering chunks as they are produced, and abr_session_wait_for_completion pushes a flush marker and delivers the rest. The flush marker ends the segment: no audio state carries over to the next clip, so a batch of clips can go through the same application back to back.

Underneath, that is abr_lib.buf_push with flush set once the clip is in, and abr_lib.buf_pull until it reports the marker; examples/asr.c in the tarball shows the loop in full.

Example#

benchmark.py#
from pathlib import Path

from abr_sdk.asr import Asr, AsrMode

LIBRARY_PATH = "/path/to/niagara-38m-live.en-linux-x86_64/libniagara_38m_live.so"
CLIP_PATHS = ["clip1.pcm", "clip2.pcm"]

# ACCURATE mode (the default, set explicitly here) is the right mode for
# accuracy benchmarks; FAST trades accuracy for latency and inflates WER/CER.
with Asr(LIBRARY_PATH, mode=AsrMode.ACCURATE) as asr:
    for path in CLIP_PATHS:
        transcript = asr.process(Path(path).read_bytes())
        print(transcript.text)
benchmark.c#
#include <stdio.h>
#include <stdlib.h>

#include "common/asr_transcript.h"
#include "common/session.h"

#define LIBRARY_PATH \
    "/path/to/niagara-38m-live.en-linux-x86_64/libniagara_38m_live.so"

static const char *const CLIP_PATHS[] = {"clip1.pcm", "clip2.pcm"};

static char text[1 << 20];
static abr_asr_transcript_t transcript;

/**
 * Reads a whole file into a buffer the caller frees.
 */
static uint8_t *read_file(const char *path, size_t *n_bytes) {
    FILE *file = fopen(path, "rb");
    if (file == NULL) {
        return NULL;
    }
    fseek(file, 0, SEEK_END);
    *n_bytes = (size_t)ftell(file);
    fseek(file, 0, SEEK_SET);
    uint8_t *data = malloc(*n_bytes);
    if (data != NULL && fread(data, 1, *n_bytes, file) != *n_bytes) {
        free(data);
        data = NULL;
    }
    fclose(file);
    return data;
}

int main(void) {
    // Accurate mode (the default, set explicitly here) is the right mode for
    // accuracy benchmarks; fast trades accuracy for latency and inflates
    // WER/CER.
    abr_config_t cfg;
    cfg.key = "asr_mode";
    cfg.value.type = ABR_VAR_STR;
    cfg.value.value.str = "accurate";

    abr_session_t session;
    if (abr_session_open(&session, LIBRARY_PATH, &cfg, 1) != ABR_OK) {
        return 1;
    }

    for (size_t i = 0; i < sizeof(CLIP_PATHS) / sizeof(CLIP_PATHS[0]); i++) {
        size_t n_bytes;
        uint8_t *clip = read_file(CLIP_PATHS[i], &n_bytes);
        if (clip == NULL) {
            fprintf(stderr, "cannot read %s\n", CLIP_PATHS[i]);
            break;
        }
        abr_asr_transcript_init(&transcript, text, sizeof(text));
        abr_session_push(&session, clip, n_bytes, abr_asr_transcript_on_output,
                         &transcript);
        abr_session_wait_for_completion(&session, abr_asr_transcript_on_output,
                                        &transcript);
        free(clip);
        printf("%s\n", transcript.text);
    }

    abr_session_close(&session);
    return session.status == ABR_OK ? 0 : 1;
}

Each .pcm file must contain raw PCM audio in the format the application expects. See Input format for details on sample rate, bit depth, and channel layout.

Scoring transcripts#

When computing WER or CER, score the final transcript: transcript.text on the returned AsrTranscriptAsrTranscript. This text includes the post-processing pass (punctuation, capitalization, spell correction). Scoring intermediate chunks, or only the model output, measures text the pipeline would have revised and overstates the error rate.

Apply the same text normalization to the reference and the hypothesis before scoring, and use the same normalizer the numbers you are comparing against were measured with. Niagara’s published accuracy figures come from the HuggingFace Open ASR Leaderboard, which applies its own text normalizer to both sides; scoring with that normalizer makes your numbers directly comparable.

Errors#

Exception

When raised

abr_sdk.exceptions.AbrSdkError

Called on a closed Asr instance, for example after the with block has exited or after asr.close() was called explicitly.

abr_sdk.cabi.AbrSdkCAbiError

The underlying C library returned a failure status. Check the log output for details.

abr_lib.buf_push and abr_lib.buf_pull return a negative abr_status when the application has failed, and keep returning it until the application is freed. The session.h helper records the first such status in abr_session_t.status, logs it, and turns later calls into no-ops. Check the log output for details.