tech
Harmonic says humans should set AI benchmarks
The Nvidia-backed startup bets mathematics can provide a better test of AI progress.
TL;DR
- Harmonic is collaborating with the American Institute of Mathematics (AIM) on a new AI evaluation process.
- The benchmark will use mathematical problems chosen by working mathematicians.
- It will measure both the correctness of AI answers and their ability to assist mathematicians in research.
- The benchmark materials and criteria will be publicly released for contributions from academia and industry.
- This move reflects a growing demand for domain experts to define AI progress metrics.
- Harmonic believes mathematics offers a clear test of AI reasoning due to formal verification of answers.
- The company was founded in 2023 and is backed by Robinhood CEO Vlad Tenev.