How to evaluate an AI music detector

Should you trust a “99% accurate” detector claim?

Not on its own. Accuracy is a single number that hides the trade-off that actually matters: how often human music gets flagged as AI. A detector that labels almost everything “AI” scores well on a corpus that is mostly AI and is useless in practice. Without the class balance of the test set, an accuracy percentage cannot be interpreted at all.

The seven disclosures a real accuracy claim includes

1. Corpus composition. How many human tracks, how many generated tracks, which genres, and how they were sourced.
2. False-positive rate on human music, reported separately from the detection rate. This is the number that decides whether a tool is safe to use on your own work.
3. Generators and versions tested. “Suno” is not a version. Model updates change acoustic behaviour, so a result from an old version may not transfer.
4. Encode conditions. Lossless, 320 kbps, 128 kbps, re-encoded, loudness normalised. Compression removes exactly the high-frequency detail many detectors depend on.
5. Threshold and scoring rule. At what probability is a track counted as detected, and is confidence reported separately from probability?
6. Engine version and date. A number without a build is not reproducible; see this site’s changelog.
7. Failure cases. A published list of the conditions under which the tool is wrong. Every honest evaluation produces one.

Which metrics actually matter?

• Detection rate (TPR) per generator, not pooled — a strong average can hide a generator the tool cannot see at all.
• False-positive rate (FPR) per genre. Highly quantised electronic and heavily limited pop are where human music most often looks synthetic.
• Calibration. When a tool says 70%, are roughly 70% of those tracks actually generated? Uncalibrated scores are ranks dressed up as probabilities.
• Inconclusive rate. A tool that never abstains is hiding uncertainty rather than measuring it.

Five questions to ask any vendor

1. What is your false-positive rate on human-made electronic music?
2. Which generator versions were in your test set, and when was it built?
3. Does your score survive a 128 kbps re-encode of the same file?
4. Is my audio uploaded, stored, or used for training?
5. Will you publish the raw aggregates behind your headline number?

How this tool scores on its own checklist

Honestly: partially. The engine parameters, combination weights, thresholds, metric definitions and corpus design are all published, processing is local so nothing is uploaded, and every failure mode we know about is listed. What is not published is a measured accuracy figure — the benchmark protocol exists with every result cell null, and null means unmeasured. We would rather show an empty cell than an estimate dressed as a measurement.

Related reading

• Detector comparison — how the public tools differ in access, privacy and disclosure.
• What your result means — reading a probability and a confidence band correctly.
• Limitations — where this detector fails.