The AI music detection benchmark
An open, versioned reference for evaluating AI music detectors: how the corpus is built, what each metric means, which parameters this engine actually runs with, and what a published figure can and cannot be used to claim.
Benchmark v1.0 · engine build ensemble-1 · updated 2026-08-15 · CC BY 4.0
Status
The protocol and the engine parameters below are published and citable today. Result cells read not measured until the corresponding evaluation has been run. We do not publish estimated, modelled or vendor-supplied accuracy figures, and any cell that later carries a number will also carry the date and engine build that produced it.
Published engine parameters
These are measured properties of the shipped build rather than claims about it: each one is read from the public ensemble configuration the browser downloads before analysing a file, so anyone can verify them independently.
| Parameter | Value |
|---|---|
| Models in the ensemble | 4Acoustic, spectral-artefact, extended-feature and embedding detectors, combined by weighted vote. |
| Vote weights | spectral-artefact 2.2 · embedding 1.6 · acoustic 1.0 · extended-features 0.9Published in public/config/ensemble.json and applied at runtime in the browser. |
| Reported probability range | 15–85%The engine never reports 0% or 100%. No file-only measurement justifies certainty. |
| Shrinkage toward 50% | 0.94Every combined score is pulled slightly toward the middle before it is shown. |
| Confidence bands | moderate ≥ 0.50 · high ≥ 0.75Below the moderate band the result is reported as inconclusive rather than as a verdict. |
| Disagreement penalty | threshold 0.20 · penalty 1.1×When the models disagree, confidence is reduced rather than the probability being moved. |
| Where analysis runs | In the browserAudio is not uploaded to a server, which also means no evaluation corpus is collected from users. |
Metric definitions
Most published AI-detection accuracy claims are unusable because the metric is undefined. These are the five figures this benchmark reports, stated precisely enough to be quoted.
- Detection rate (TPR)
- Share of known AI-generated tracks scored above 50% by the engine, counted per generator rather than pooled.
- False-positive rate (FPR)
- Share of verified human-produced tracks scored above 50%. This is the metric we treat as primary, because a false accusation costs a musician more than a missed detection.
- Inconclusive rate
- Share of tracks where combined confidence falls below the moderate band and no verdict is issued. Abstentions are reported, not hidden inside accuracy.
- Calibration error
- Mean absolute gap between the reported probability and the observed frequency of AI origin within that probability bucket. A detector that says 70% should be right about 70% of the time.
- Encode stability
- Mean absolute change in reported probability for the same master rendered at 320, 192, 128 and 64 kbps against its lossless reading.
Corpus design
690 tracks in total, split between generated arms and human control arms. Sampling is stratified by genre and by production style rather than drawn from whatever is convenient, because both classes contain easy and hard cases and a corpus made only of easy ones produces a flattering, meaningless number.
Generated arms
| Generator | Version | Tracks | Detection rate | Inconclusive |
|---|---|---|---|---|
| Suno | v4 / v4.5 | 60 | not measured | not measured |
| Udio | v1.5 | 60 | not measured | not measured |
| Stable Audio | 2.0 | 40 | not measured | not measured |
| ElevenLabs Music | current | 40 | not measured | not measured |
| Riffusion | FUZZ | 40 | not measured | not measured |
| Seed Music | current | 30 | not measured | not measured |
| MiniMax Music | current | 30 | not measured | not measured |
| Mureka | current | 30 | not measured | not measured |
| Held-out generator | undisclosed until publication | 40 | not measured | not measured |
Human control arms
The control side is deliberately weighted toward the cases where a detector is most likely to embarrass itself: heavily processed commercial masters and human-authored electronic music.
| Source | Tracks | False positives |
|---|---|---|
| Commercially released studio recordingsHeavily processed human music is the most common false-positive scenario. | 80 | not measured |
| Independent bedroom productionsIn-the-box production with stock plugins looks superficially synthetic. | 80 | not measured |
| Live acoustic and single-room recordingsThe easiest control case; a failure here would be disqualifying. | 60 | not measured |
| Fully synthetic human-authored electronic musicQuantised, synthesiser-only human work is the hardest control case. | 60 | not measured |
| Hybrid human/AI tracks (AI stems in a human arrangement)Reported separately; there is no correct binary answer for these. | 40 | not measured |
Evaluation procedure
- Every track is held as a lossless master and rendered into the fixed encode ladder described in the compression study, so results can be reported per source condition instead of pooled.
- All files are analysed by a single pinned engine build. A configuration change invalidates the run; it does not get merged into it.
- Component outputs are recorded alongside the combined score, so a failure can be attributed to a specific model rather than to the ensemble in general.
- One generator is held out entirely and disclosed only at publication, to measure behaviour on a model the thresholds were never tuned against.
- Abstentions are counted, never dropped. Headline figures always appear next to the inconclusive rate for the same arm.
Limitations
Anyone citing a number from this page should cite these alongside it. Each has a stable anchor for that purpose.
- Lossy compression is the dominant error source. Several features are computed in exactly the frequency region an encoder discards first. Results are reported per source condition, never pooled across bitrates.
- Generator versions move faster than benchmarks. A figure measured against one model release says little about the next. Every number is stamped with the generator version and the engine build that produced it.
- Hybrid tracks have no ground truth. A human arrangement built around a generated stem is neither class. These are reported as a separate arm and excluded from headline figures.
- The benchmark is not adversarial. Deliberate evasion — re-recording, heavy remastering, aggressive time and pitch manipulation — is a separate study. Do not read these figures as robustness against an attacker.
- No claim about individual tracks. A population-level rate does not transfer to a single file. No figure on this page should be used as evidence about a specific song or person.
How to cite
The benchmark is published under CC BY 4.0. Cite the version, not just the URL, because figures change when the engine or the generator versions change.
AI Music Detector. "The AI Music Detection Benchmark" (v1.0), 2026-08-15. https://aimusicdetector.co/benchmark
The same data is served as JSON at aimusicdetector.co/benchmark.json, including null fields for every figure that has not been measured yet. Corrections, replication attempts and corpus contributions are welcome via the contact page; see also accuracy and limitations and the methodology.
Frequently asked questions
Because a percentage without a documented corpus, metric definition and engine build is unciteable. Publishing the protocol first also stops us retrofitting a story to whatever the data turns out to say.