Optimizing Contextual Speech Recognition with Vector Quantization for Efficient Retrieval

TL;DR

• Contextual biasing for speech recognition often relies on cross-attention between audio and a biasing catalog, which can be prohibitively costly for large catalogs.
• The paper proposes an approximation to cross-attention scoring based on vector quantization to enable compute- and memory-efficient retrieval of bias entries grounded on audio.
• The approach is combined with a retrieval-based biasing workflow and is agnostic to the underlying biasing method (full cross-attention, prompting with LLMs, or both).
• Empirical results show that retrieval-based shortlisting enables leveraging catalogs of thousands to a million entries, delivering up to 71% relative error rate reduction in personal entity recognition and reducing compute time by ~20% and memory usage by 85–95% for large catalogs.
• The work was accepted to the IEEE Spoken Language Technology Workshop (SLT) 2024. IEEE SLT 2024

Context and background

Neural contextual biasing is a growing approach to improve transcription accuracy by injecting contextually relevant information into speech recognition models. Traditional biasing mechanisms typically rely on a cross-attention module that attends to a catalogue of biasing entries in conjunction with spoken audio. While effective, this strategy scales poorly as the biasing catalogue grows, leading to significant computational and memory demands. This limits both the size of the biasing catalogue that can be practically used and the potential accuracy benefits obtainable from larger, more diverse catalogs. The challenge is to balance the desire for expansive, context-aware biasing with the realities of resource-constrained deployment.

What’s new

The authors address the scalability bottleneck by introducing a quantized retrieval approach that serves as an efficient pre-filter for biasing candidates. Specifically, they propose an approximation to cross-attention scoring based on vector quantization. The workflow proceeds as follows: first, a fast, quantized retrieval module grounds biasing entries on the audio signal and shortlists a subset of entries; then, the retrieved subset is used for biasing, in conjunction with the chosen biasing method. A key finding is that this retrieval-based shortlisting can be combined with various biasing strategies. The authors investigate using full cross-attention, prompting with large language models (LLMs), and a combination of the two. This flexibility is important because practitioners can choose the most effective integration depending on their resource constraints and task requirements. Crucially, the approach scales to large bias catalogs, enabling the use of thousands to up to one million bias entries. The study reports substantial improvements in recognition accuracy, particularly for personal entity recognition, while achieving meaningful reductions in compute and memory usage compared with standard dot-product cross-attention on large catalogs.

Why it matters (impact for developers/enterprises)

For developers and enterprises deploying speech recognition systems in real-world settings, the ability to leverage extensive contextual bias catalogs without incurring prohibitive compute and memory costs is transformative. This work demonstrates that you can harness large, diverse bias catalogs to improve transcription quality—especially for personal entities—without sacrificing latency or scalability. From an engineering perspective, the proposed vector-quantization based retrieval module reduces the burden of maintaining massive bias catalogs in memory and helps keep inference times predictable even as catalogs grow. For organizations building voice-enabled applications with strict performance or energy constraints, this approach offers a practical pathway to richer contextualization without backend or hardware overhauls.

Technical details or Implementation (how it works)

• Problem framing: Contextual biasing in ASR traditionally uses cross-attention between the audio representation and a biasing catalogue. The attention scoring step becomes a bottleneck as catalogue size increases.
• Vector quantization-based approximation: The authors introduce an efficient quantized retrieval module that grounds biasing entries on audio. This module shortlists candidates by quantizing representations and performing approximate matching, dramatically reducing the search space before biasing.
• Retrieval-first then bias: After shortlisting, the retrieved entries are fed into the biasing stage. Because the shortlisted set is small, downstream biasing computations—whether using full cross-attention, LLM prompting, or a combination—become tractable for much larger catalogs.
• Agnostic to biasing method: The retrieval step is designed to be agnostic to the choice of biasing mechanism. This enables experimentation with traditional cross-attention, LLM prompting, or both within the same retrieval framework.
• Scale and results: The method supports catalogs up to one million entries. In comparative terms, the approach reduces compute time by about 20% and memory usage by 85–95% relative to standard dot-product cross-attention on large catalogs, while delivering substantial accuracy gains (notably up to 71% relative ER reduction in personal entity recognition).

Key takeaways

• Vector quantization can effectively approximate cross-attention scoring for contextual biasing, enabling scalable retrieval from large catalogs.
• A retrieval-based shortlisting stage reduces computational and memory loads, making it feasible to use bias catalogs with thousands to a million entries.
• The approach is versatile: it can be paired with full cross-attention, LLM prompting, or both, without being tied to a single biasing method.
• Empirical gains include up to 71% relative improvement in personal entity recognition, with ~20% faster compute and up to 85–95% memory reduction for very large catalogs.

References

• https://machinelearning.apple.com/research/optimizing-contextual -IEEE SLT 2024 (as described in the paper linked above)