State-of-the-art audio retrieval, in production: A collaboration with Oxford VGG
The third entry in Epidemic Sound’s Applied AI & ML Research series, covering audio retrieval.
At Epidemic Sound, millions of monthly searches run across our catalog of music and sound effects. The queries are often deceptively simple: “footsteps on gravel,” “tense ambient drone,” “a cinematic crowd reaction,” and so on. But turning natural language into the right audio result, at the scale and editorial quality our users expect, is one of the harder open problems in multimodal retrieval today.
For the past year, we’ve partnered with the Visual Geometry Group (VGG) at the University of Oxford to push that frontier. The result is a new audio-text retrieval framework. Submitted to IEEE CBMI 2026, it re-purposes Multimodal Large Language Models as a unified backbone for both fast retrieval and precise re-ranking.
The framework outperforms the previous state of the art on every benchmark we tested while using roughly 1% of the training data of comparable systems. As a result, it now powers sound effects search inside Epidemic Sound, in production.
Alongside the research paper, we’re open-sourcing a 1,297-clip validation benchmark of carefully selected, human-annotated sound effects, under an academically permissible license. Open evaluation data is a chronic bottleneck for audio-text research; this is our contribution back.
This post covers how we went from a research paper to production, and why we think this is a template for applied AI teams engaging with the cutting edge.
We’ll discuss:
- The model: Methodology, the new Hybrid NCE training objective, and the MLLM-based bidirectional re-ranker.
- Deployment: How the model integrates into Epidemic Sound’s search, and what it takes to run a 7B multimodal model in a production retrieval system.
- The open benchmark: What’s in our 1,297-clip evaluation set, how it was annotated, and why we’re releasing it openly.
- Testing and evaluation: How we evaluate a new search system for classic and agentic search use cases.
The Model
Our multimodal model, AuroLA, is based on the open source Qwen2.5-Omni-7B model. Our model makes use of three parts: the large language model, the audio encoder, and the audio-to-text projector.
Turning a generative model into a retrieval model
Qwen2.5-Omni is a generative model. Given audio or text in, it produces text. To turn it into a retrieval model, we adopt the Explicit One-word Limitation trick: We prompt the model to summarize the input into a single word, then take the hidden state of that word as the embedding.
We use an additional special token, $<embed>$, as the “summary slot,” treating its final-layer representation as the embedding for both audio and text. Because the same MLLM is used on both sides, the two modalities land in the same space by construction.
Encoding text
For text, we concatenate the caption $t_i$ to be embedded with $Prompt_T$ and append $<embed>$ to the end of the sequence:
$$t_i = \ LLM([t_i,\ Prompt_T,\ <embed>)$$
$Prompt_T$ is defined as the string:
Summarize the above text in one word:
Encoding audio
To embed the audio, we must first encode it. We prepare audio by resampling it to 16 kHz, then transform the resulting waveform to a 128-channel mel spectrogram with a window size of 25ms and a hop size of 10ms.
The audio encoder then produces dense audio features for the spectrogram. Finally, we project the audio features to the language space with the projection model and encode the following sequence:
$$a_i = \ LLM([\ Projector(\ AudioEncoder(a_i)),\ Prompt_A,\ <embed>])$$
The $Prompt_A$ is defined as:
Summarize the above audio in one word:
The final token produced then acts as representation for $a_i$.
The three stages of training
Out of the box, Qwen2.5-Omni knows nothing about producing embeddings; it has only ever been trained to generate text. We bridge that gap in three stages.
Stage 1: Pre-training on text only
We first run text-only contrastive learning on the Natural Language Inference (NLI) dataset. The model sees pairs of related and unrelated sentences and learns to use the $<embed>$ token as a real summary representation. This is essentially a warm-up as it pushes the LLM’s latent space into a shape that’s usable for retrieval, before any audio shows up.
Stage 2: Audio & text training
Next, we plug in the audio side and train on AudioVerse, our 1.4M multi-granular audio-text dataset (described later). For each clip, we randomly sample one of its three captions — long, short, or tag — as the text input.
We adopt HybridNCE as the objective, a novel loss function. It pulls in audio clips that share semantic tags as additional “soft positives,” and also re-weights negatives. This means genuinely confusable hard negatives contribute more to the gradient than trivially easy ones.
Stage 3: Re-ranker
Finally, we train a separate re-ranking model on top. We use the Stage 2 retrieval model to mine hard negatives: texts that look superficially similar to the query but aren’t a true match. We then fine-tune another LoRA-adapted Qwen2.5-Omni-7B as a binary cross-modal judge: given an (audio, text) pair, output Yes or No. The probability of Yes after softmax is the re-ranking score.
AudioVerse training dataset
A model is only as good as the data it hears, and most public audio-text corpora are bottlenecked on a single source (typically AudioSet) and a single caption per clip. To train AuroLA, we built AudioVerse: a 1.4M-clip dataset that pools audio from 11 different public sources.
We use Qwen3-Omni-30B as an automated captioner to generate three captions at different granularities:
- A long caption: Detailed temporal narrative
- A short caption: One-line summary
- A tag caption: Handful of structured keywords
Soundly benchmark dataset
To give back to the community, we’re releasing a 1,297-clip sound effects benchmark based on the test dataset used to develop AuroLA. Not only are we releasing the audio together with its original metadata, but also with descriptive captions annotated by real people.
The dataset is available on Hugging Face and is released under the CC BY-NC-ND 4.0 license.
High-quality data for evaluating sound effects models is scarce. We hope this dataset can help people build more soundtracking features.
Deployment
AuroLA is a 7B-parameter multimodal model. Serving it on every sound effects query meant meeting a tight latency budget and a per-query cost that scales with traffic. Neither of these are guaranteed for a model this size.
This section describes the serving architecture we landed on, the optimizations that made it viable, and the improvements it delivered over our previous sound effects search stack.
Retrieval is split across an offline indexing path and an online query path:
- Offline: Each sound effect is encoded once into two AuroLA embeddings: aurola_tags_embedding (from metadata) and aurola_audio_embedding (from audio), which is indexed into Elasticsearch.
- Online: At query time, the search service encodes only the query text. Elasticsearch then executes a hybrid retrieval: a lexical branch over title, tags, and taxonomy, and a semantic KNN branch over the precomputed embeddings, with the two branches score-fused into a single ranking.
This keeps all audio encoding offline, so the online deployment has a single, well-bounded role: map a query string to a text embedding within the request’s latency budget. That made the encoder the only component we had to optimize for latency, and the focus of the work below.
We serve the encoder with SGLang. This provides the primitives that matter for high-throughput, low-latency embedding inference: continuous batching, CUDA graphs with Triton kernels, a RadixAttention KV cache, and a native embedding mode (is_embedding=True) that pools the final hidden state directly, so we never pay for token generation. Continuous batching is the key primitive that fuses concurrent requests into a single prefill step, rather than running each in isolation.
Relative to a vanilla single-request-per-forward-pass deployment on Transformers, SGLang configuration reduced the container count 5 times, cut p90 latency by 40%, and halved cold-start time.
Testing and evaluation
Beyond the model-level benchmarks in the paper, we wanted to know how much our production search improves with AuroLA added as a semantic branch, relative to the previously deployed sound effects search. For AuroLA-vs-prior-SOTA comparisons and ablations, see the paper.
We evaluate on two datasets that probe different failure modes:
The historical top 500: Our most popular real-traffic queries, which (lacking ground-truth labels) we use only to check whether retrieved clips sit closer to the query in AuroLA’s embedding space.
The Soundly test set: Sampled from Soundly’s catalog and labeled by sound effects category, which we score with ranking metrics (MRR, Precision@3/@10) and embedding-space cosine.
Across every query shape and sound effects category, each metric improves over the previous system with no regressions. Ranking quality on Soundly rises 7–9% (MRR +7.3%, P@3 +8.5%), and embedding-space proximity rises 16–24% on both datasets.
By category, Foley and Transition gain most. Ambiance is hardest, since its textures overlap heavily across nearby labels (“rain,” “outdoor,” “city background”), and hybrid retrieval narrows that gap without closing it.
Wrapping up
A year ago, the question was whether a frontier multimodal model could become the retrieval backbone of a real product without compromise on quality, cost, or the operational realities of serving every query through a 7B-parameter model. Our collaboration with Oxford VGG answers it.
AuroLA beats prior SOTA on every benchmark we ran while training on a small fraction of the data comparable systems consume. The production search it powers measurably improves ranking and semantic alignment over what ran before, at a cost that lets us put it on every query in the product.
The bigger payoff is downstream of search itself. Studio, Epidemic Sound’s AI-powered soundtracking workflow, analyzes an uploaded video and picks sound for it, which means it’s calling search and acting on what it gets back. When an agent is the consumer of retrieval rather than a person, semantic understanding matters even more. That’s where AuroLA shines.
The same technology is now in the hands of sound designers through Soundly, whose Natural Language Search lets users find sounds by describing them in plain language.
The full research paper and Soundly benchmark are linked below.
Thanks to the VGG team at the University of Oxford for a year of genuinely collaborative research, rather than the usual hand-off model; to the Soundly and Epidemic Sound annotators, who built the evaluation set we’re releasing; and to the engineering, ML, and search teams at Epidemic Sound and Soundly.