Why an Icelandic company built a new ASR benchmark with Hugging Face
The benchmark measures how models fare in real-world scenarios
Whenever voice AI companies release a model, they try to make a case for its model’s superiority. While latency and price are key factors in ASR (Automatic Speech Recognition) models, the most important point is accuracy. These companies usually present their own benchmarks, but those are hard to trust because, most of the time, they favor their own models.
There are independent benchmarks like the Open ASR leaderboard, but they don’t always capture real-world scenarios with messy audio and noise. Plus, these kinds of leaderboards deal with clean audio in a near-field situation where a user is speaking into a phone or a headset.
This is why Hugging Face and Iceland-based voice simulation and testing company Treble decided to create a benchmark specifically for Far Field ASR (FFASR). It evaluates environments where a user is one meter or farther from the mic, such as conference room meetings or someone talking to a smart speaker.
Treble has a simulation where AI companies or device providers can test their models in real-life scenarios, such as various restaurants or cafes. The company told me that over the years, it has developed simulation tech that can create complex situations and test models much faster than before. Using these techniques, Treble can tell AI companies what kind of physics understanding they can build in voice models for better accuracy.
The startup’s CEO, Finnur Pind, said that until now, companies measured their models’ capabilities through the Open ASR leaderboard, which is built on clean audio. He also noted that it could be possible to prepare models for better benchmark performance, as the data is known.
“For Hugging Face and us, it was important to measure how ASR performs out in the real world in complex conditions with overlapping talkers, reverberation, and movement. We wanted to try to create a benchmark that tests these conditions,” Pind said.
The FFASR leaderboard has different datasets for comparison. There near-field speech sample of 8 hours as a base. Then, through Treble’s SDK, there are high, mid, and low signal-to-noise ratio datasets that test models in scenarios like the bathroom, living room, meeting rooms, and hallways with noises like coughing and the HVAC system. Other measurement parameters are of high, mid, and low signal-to-noise ratios with movement. All these tests are based on the Treble10 dataset that the Icelandic startup and Hugging Face collaborated on last year.
However, Pind mentioned that the core data is hidden to avoid gaming of the benchmark. He also mentioned that performance measurement on the old leaderboard had plateaued. And the new FFASR leaderboard will make way for a new kind of measurement.
When model companies test on this benchmark, they would get an average word error rate (WER) across datasets, and an RTFx score, which measures the duration of audio and the time it took for a model to process it. There is also a plot to show the correlation between the two.
“When you test out models on real-world conditions, the performance fluctuation is large for different scenarios. This new benchmark will show companies where their models have been lacking,” Pind said.
Looking ahead, Hugging Face and Treble plan to expand the platform. In a recent webinar, Hugging Face mentioned that in the future, it wants the community to contribute to benchmarking with new open-source datasets for far-field evaluation, ways to test hardware, new denoising algorithms, and different acoustic conditions to test out models. You can check out the benchmark here.