Benchmarking#
process()buf_push is a blocking call that takes a complete audio clip and returns
the final transcript. It is suited for benchmarking throughput and accuracy against a fixed audio
dataset.
Pushing a complete audio clip and then waiting for completion gives the final transcript of that
clip, which is what benchmarking throughput and accuracy against a fixed audio dataset needs. With
the session helper from the tarball’s examples/common
directory, that is one
abr_session_push followed by one abr_session_wait_for_completion.
Attention
Accuracy benchmarks should run in ACCURATE mode, the default. FAST mode trades accuracy for latency and inflates the measured WER/CER. FAST is not faster for batch processing either, because both modes run the same network at the same compute cost.
For real-time or streaming use, see Overview.
Processing a clip#
Asr.process(data: bytes) -> AsrTranscript
Pushes all of data through the ASR pipeline, waits for the neural network to finish, and returns
the transcript. The call blocks until the entire input has been processed; it does not return
partial results.
process()buf_push can be called multiple times on the same
AsrAsr instance. Each call is self-contained: no audio state carries over
between clips, so you can process a batch of clips back to back.
data is PCM audio encoded as a little-endian 16-bit byte array at 16,000 Hz; see
Input format for format requirements. The call returns an
AsrTranscriptAsrTranscript; call .text to get the final string.
abr_session_push writes the whole clip into the input buffer, delivering chunks as they are
produced, and abr_session_wait_for_completion pushes a flush marker and delivers the rest. The
flush marker ends the segment: no audio state carries over to the next clip, so a batch of clips can
go through the same application back to back.
Underneath, that is abr_lib.buf_push with flush set once the clip is in, and
abr_lib.buf_pull until it reports the marker; examples/asr.c in the tarball shows the
loop in full.
Example#
from pathlib import Path
from abr_sdk.asr import Asr, AsrMode
LIBRARY_PATH = "/path/to/niagara-38m-live.en-linux-x86_64/libniagara_38m_live.so"
CLIP_PATHS = ["clip1.pcm", "clip2.pcm"]
# ACCURATE mode (the default, set explicitly here) is the right mode for
# accuracy benchmarks; FAST trades accuracy for latency and inflates WER/CER.
with Asr(LIBRARY_PATH, mode=AsrMode.ACCURATE) as asr:
for path in CLIP_PATHS:
transcript = asr.process(Path(path).read_bytes())
print(transcript.text)
#include <stdio.h>
#include <stdlib.h>
#include "common/asr_transcript.h"
#include "common/session.h"
#define LIBRARY_PATH \
"/path/to/niagara-38m-live.en-linux-x86_64/libniagara_38m_live.so"
static const char *const CLIP_PATHS[] = {"clip1.pcm", "clip2.pcm"};
static char text[1 << 20];
static abr_asr_transcript_t transcript;
/**
* Reads a whole file into a buffer the caller frees.
*/
static uint8_t *read_file(const char *path, size_t *n_bytes) {
FILE *file = fopen(path, "rb");
if (file == NULL) {
return NULL;
}
fseek(file, 0, SEEK_END);
*n_bytes = (size_t)ftell(file);
fseek(file, 0, SEEK_SET);
uint8_t *data = malloc(*n_bytes);
if (data != NULL && fread(data, 1, *n_bytes, file) != *n_bytes) {
free(data);
data = NULL;
}
fclose(file);
return data;
}
int main(void) {
// Accurate mode (the default, set explicitly here) is the right mode for
// accuracy benchmarks; fast trades accuracy for latency and inflates
// WER/CER.
abr_config_t cfg;
cfg.key = "asr_mode";
cfg.value.type = ABR_VAR_STR;
cfg.value.value.str = "accurate";
abr_session_t session;
if (abr_session_open(&session, LIBRARY_PATH, &cfg, 1) != ABR_OK) {
return 1;
}
for (size_t i = 0; i < sizeof(CLIP_PATHS) / sizeof(CLIP_PATHS[0]); i++) {
size_t n_bytes;
uint8_t *clip = read_file(CLIP_PATHS[i], &n_bytes);
if (clip == NULL) {
fprintf(stderr, "cannot read %s\n", CLIP_PATHS[i]);
break;
}
abr_asr_transcript_init(&transcript, text, sizeof(text));
abr_session_push(&session, clip, n_bytes, abr_asr_transcript_on_output,
&transcript);
abr_session_wait_for_completion(&session, abr_asr_transcript_on_output,
&transcript);
free(clip);
printf("%s\n", transcript.text);
}
abr_session_close(&session);
return session.status == ABR_OK ? 0 : 1;
}
Each .pcm file must contain raw PCM audio in the format the application expects. See
Input format for details on sample rate, bit depth, and channel layout.
Scoring transcripts#
When computing WER or CER, score the final transcript: transcript.text on the returned
AsrTranscriptAsrTranscript. This text includes the post-processing pass (punctuation,
capitalization, spell correction). Scoring intermediate chunks, or only the model output, measures
text the pipeline would have revised and overstates the error rate.
Apply the same text normalization to the reference and the hypothesis before scoring, and use the same normalizer the numbers you are comparing against were measured with. Niagara’s published accuracy figures come from the HuggingFace Open ASR Leaderboard, which applies its own text normalizer to both sides; scoring with that normalizer makes your numbers directly comparable.
Errors#
Exception |
When raised |
|---|---|
|
Called on a closed |
|
The underlying C library returned a failure status. Check the log output for details. |
abr_lib.buf_push and abr_lib.buf_pull return a negative
abr_status when the application has failed, and keep returning it until the application is
freed. The session.h helper records the first such status in abr_session_t.status, logs it, and
turns later calls into no-ops. Check the log output for details.