A consortium of 40 tech companies, including the likes of Facebook and Google, have come together to release a set of evaluation benchmarks for AI.
The benchmarks which cover image recognition, object detection and voice translation are meant to help companies compare various AI tools to see which work best for them as they pursue their own AI initiatives, said Peter Mattson, general chairman of the consortium, MLPerf, which counts 40 companies as members.
“For CIOs, metrics make for better products and services they can then incorporate into their organization,” said Mr. Mattson, a Google engineer.
A major consortium of AI community stakeholders today introduced MLPerf Inference v0.5, the group’s first suite for measurement of AI system power efficiency and performance. Inference benchmarks are essential to understanding just how much time and power is required to deploy a neural network for common tasks like computer vision that predicts the contents of an image.
The suite consists of 5 benchmarks that include English-German machine translations with the WMT English-German data set, 2 object detection benchmarks with the COCO data set, and 2 image classification benchmarks with the ImageNet data set.
The inference standards were decided upon over the course of the past 11 months by partner organizations such as Arm, Facebook, Google, General Motors, Nvidia, and Toronto University, MLPerf said in a statement shared with VentureBeat.
As well as providing best practice guidance for companies in the AI field, it’s hoped the benchmarks will help kick-start further innovation as, despite its hype, organizations have been slow to pick up the technology. In a statement, MLPerf’s general chair Peter Mattson said, “By creating common and relevant metrics to assess new machine learning software frameworks, hardware accelerators, and cloud and edge computing platforms in real-life situations, these benchmarks will establish a level playing field that even the smallest companies can use.”
