The AI music detection benchmark
An open, versioned reference for evaluating AI music detectors: how the corpus is built, what each metric means, which parameters this engine actually runs with, and what a published figure can and cannot be used to claim.
Benchmark v1.1 · engine build ensemble-1 · updated 2026-08-24 · CC BY 4.0
Status
The engine behaviour study below is measured as of 2026-08-24. The accuracy arms — detection rate and false-positive rate — read not measured until the labelled corpus has been evaluated. We do not publish estimated, modelled or vendor-supplied accuracy figures, and any cell that later carries a number will also carry the date and engine build that produced it.
Study 1 — engine behaviour (measured)
The first thing a detector can be held to account on does not need ground-truth labels at all: given the same music, does it give the same answer, and how much does an encode, a downmix or a shorter excerpt move it? 168 detections were run over 12 reference tracks in 7 conditions on build ensemble-1.
Corpus. 12 procedurally rendered reference tracks (20–23 s, 44.1 kHz stereo) produced by the site's own sample generator across 12 genre presets. No third-party or copyrighted audio. Ladder. lossless WAV · MP3 320 · 192 · 128 · 64 kbps · mono downmix · 10 s excerpt.
Swipe the table sideways to see all columns.
| Measurement | Result |
|---|---|
| Repeat determinism | 0.0 ppEach of the 84 files was analysed twice in the same session. All 84 pairs returned an identical probability, so the engine adds no run-to-run randomness. |
| Encode stability (mean |Δp| vs lossless) | 2.4 ppAcross the four MP3 bitrates: 1.8 pp at 320 kbps, 1.8 pp at 192 kbps, 2.3 pp at 128 kbps, 3.5 pp at 64 kbps. Largest single shift observed: 26 pp, at 64 kbps. |
| Mono downmix sensitivity | 1.2 ppCollapsing stereo to mono moved the score by 1.2 pp on average (worst case 11 pp), even though stereo-field features are then unavailable. |
| Short-excerpt sensitivity | 4.4 ppA 10 s excerpt of the same master moved the score by 4.4 pp on average and up to 31 pp. This is why the tool asks for at least 30 s of audio. |
| Abstention behaviour | 83 of 84 files inconclusiveOn single-render synthetic reference audio the engine declined to issue a verdict in almost every case. It prefers abstaining to guessing — which is the designed behaviour, and also a reminder that an inconclusive result is a normal outcome. |
| Synthesiser-only false-positive risk | 3 of 12 tracks read above 70%Three of the twelve renders — all synthesiser-only, quantised, single-take material — produced high AI probabilities. Purely electronic human-authored music is the hardest control case, and this study confirms the risk rather than hiding it. |
| Analysis time | 406 ms medianMedian 406 ms, 90th percentile 532 ms per track measured on the harness, excluding upload time. Real-world runs include the network round trip to the detection service, so they will be slower. |
What this study does not show
- This is a behaviour and robustness study, not an accuracy study. It contains no AI-generated tracks and no verified human commercial recordings, so it yields no detection rate and no false-positive rate.
- The corpus is procedurally generated, which makes it reproducible but not representative of released music.
- All figures are for engine build ensemble-1. A configuration change invalidates them.
Published engine parameters
These are measured properties of the shipped build rather than claims about it: each one is read from the public ensemble configuration the browser downloads before analysing a file, so anyone can verify them independently.
Swipe the table sideways to see all columns.
| Parameter | Value |
|---|---|
| Models in the ensemble | 4Acoustic, spectral-artefact, extended-feature and embedding detectors, combined by weighted vote. |
| Vote weights | spectral-artefact 2.2 · embedding 1.6 · acoustic 1.0 · extended-features 0.9Published in public/config/ensemble.json and applied at runtime in the browser. |
| Reported probability range | 15–85%The engine never reports 0% or 100%. No file-only measurement justifies certainty. |
| Shrinkage toward 50% | 0.94Every combined score is pulled slightly toward the middle before it is shown. |
| Confidence bands | moderate ≥ 0.50 · high ≥ 0.75Below the moderate band the result is reported as inconclusive rather than as a verdict. |
| Disagreement penalty | threshold 0.20 · penalty 1.1×When the models disagree, confidence is reduced rather than the probability being moved. |
| Where analysis runs | Server-side, third-party serviceAudio is uploaded over HTTPS and classified by a specialist third-party detection service. No audio is retained by us, so no evaluation corpus is collected from users. |
Objective
Measure how the detector behaves on audio whose origin is known: recordings verified as human-created and recordings verified as AI-generated, across several generators, genres, durations and production conditions. The output of a completed run is a confusion matrix per arm, not a single headline percentage.
A second, weaker objective is already met: the behaviour study above establishes how stable the engine is when the same music is presented in different encodes. Stability is a precondition for accuracy, not a substitute for it.
The holdout principle
Any audio used to choose thresholds, weights or abstention rules is development data and can never appear in the evaluation set. A detector tuned and then scored on the same material reports how well it memorised that material. In addition, output from one generator is withheld from all design work and disclosed only at publication, so at least one arm measures behaviour on a model nothing was fitted to.
Transformation matrix
Real uploads have been through encoders, loudness processing and sometimes a phone speaker and a microphone. Each transformation below alters, and in several cases destroys, the acoustic detail detection depends on, so every arm is evaluated per condition rather than pooled into one average.
Swipe the table sideways to see all columns.
| Condition | What it changes | Measured effect |
|---|---|---|
| Original lossless (WAV/FLAC) | Reference condition — nothing removed. | Baseline in the behaviour study (12 tracks) |
| MP3 320 kbps | Mild high-band loss; psychoacoustic masking artefacts. | 1.8 pp mean |Δp| vs lossless (behaviour study) |
| MP3 192 kbps | Visible spectral ceiling; stereo joint coding. | 1.8 pp mean |Δp| vs lossless (behaviour study) |
| MP3 128 kbps | Hard low-pass; smeared transients. | 2.3 pp mean |Δp| vs lossless (behaviour study) |
| MP3 64 kbps | Severe band loss; the condition most likely to mimic generated audio. | 3.5 pp mean |Δp| vs lossless, worst single case 26 pp |
| AAC (where the decoder supports it) | Different masking model to MP3; distinct artefact signature. | Not yet measured |
| Loudness normalisation | Level only; no spectral change. | Not yet measured |
| Broad EQ (±3 dB shelves) | Moves tonal balance and spectral centroid. | Not yet measured |
| Dynamic-range compression | Reduces crest factor and micro-dynamics. | Not yet measured |
| Brickwall limiting to −8 LUFS | Removes peak structure; mimics commercial mastering. | Not yet measured |
| Mild clipping (0.5 dB over) | Adds broadband harmonic distortion. | Not yet measured |
| Resample 44.1 → 22.05 → 44.1 kHz | Destroys the upper octave permanently. | Not yet measured |
| Mono downmix | Removes all stereo-field evidence. | 1.2 pp mean |Δp|, worst case 11 pp (behaviour study) |
| Added noise floor (−60 dBFS) | Masks low-level detail; imitates an analogue chain. | Not yet measured |
| 10-second excerpt | Reduces the number of independent analysis windows. | 4.4 pp mean |Δp|, up to 31 pp (behaviour study) |
| Acoustic re-recording (speaker → microphone) | Adds room, mic and re-encode artefacts to generated audio. | Not yet measured |
Metric definitions
Most published AI-detection accuracy claims are unusable because the metric is undefined. These are the five figures this benchmark reports, stated precisely enough to be quoted.
- Detection rate (TPR)
- Share of known AI-generated tracks scored above 50% by the engine, counted per generator rather than pooled.
- False-positive rate (FPR)
- Share of verified human-produced tracks scored above 50%. This is the metric we treat as primary, because a false accusation costs a musician more than a missed detection.
- Inconclusive rate
- Share of tracks where combined confidence falls below the moderate band and no verdict is issued. Abstentions are reported, not hidden inside accuracy.
- Calibration error
- Mean absolute gap between the reported probability and the observed frequency of AI origin within that probability bucket. A detector that says 70% should be right about 70% of the time.
- Encode stability
- Mean absolute change in reported probability for the same master rendered at 320, 192, 128 and 64 kbps against its lossless reading.
Corpus design
690 tracks in total, split between generated arms and human control arms. Sampling is stratified by genre and by production style rather than drawn from whatever is convenient, because both classes contain easy and hard cases and a corpus made only of easy ones produces a flattering, meaningless number.
Generated arms
Swipe the table sideways to see all columns.
| Generator | Version | Tracks | Detection rate | Inconclusive |
|---|---|---|---|---|
| Suno | v4 / v4.5 | 60 | not measured | not measured |
| Udio | v1.5 | 60 | not measured | not measured |
| Stable Audio | 2.0 | 40 | not measured | not measured |
| ElevenLabs Music | current | 40 | not measured | not measured |
| Riffusion | FUZZ | 40 | not measured | not measured |
| Seed Music | current | 30 | not measured | not measured |
| MiniMax Music | current | 30 | not measured | not measured |
| Mureka | current | 30 | not measured | not measured |
| Held-out generator | undisclosed until publication | 40 | not measured | not measured |
Human control arms
The control side is deliberately weighted toward the cases where a detector is most likely to embarrass itself: heavily processed commercial masters and human-authored electronic music.
Swipe the table sideways to see all columns.
| Source | Tracks | False positives |
|---|---|---|
| Commercially released studio recordingsHeavily processed human music is the most common false-positive scenario. | 80 | not measured |
| Independent bedroom productionsIn-the-box production with stock plugins looks superficially synthetic. | 80 | not measured |
| Live acoustic and single-room recordingsThe easiest control case; a failure here would be disqualifying. | 60 | not measured |
| Fully synthetic human-authored electronic musicQuantised, synthesiser-only human work is the hardest control case. | 60 | not measured |
| Hybrid human/AI tracks (AI stems in a human arrangement)Reported separately; there is no correct binary answer for these. | 40 | not measured |
Evaluation procedure
The behaviour figures above are written up in full in two studies: the electronic-music false-positive study, which reports the three synthesiser-only renders that read above 70%, and the compression study, which reports the per-bitrate shifts.
- Every track is held as a lossless master and rendered into the fixed encode ladder described in the compression study, so results can be reported per source condition instead of pooled.
- All files are analysed by a single pinned engine build. A configuration change invalidates the run; it does not get merged into it.
- Component outputs are recorded alongside the combined score, so a failure can be attributed to a specific model rather than to the ensemble in general.
- One generator is held out entirely and disclosed only at publication, to measure behaviour on a model the thresholds were never tuned against.
- Abstentions are counted, never dropped. Headline figures always appear next to the inconclusive rate for the same arm.
Results
Controlled benchmark results have not yet been published. The accuracy cells on this page read not measured because the labelled corpus described above has not been assembled and evaluated, and we do not fill them with estimates, vendor figures or values carried over from other detectors. When a run completes, each cell will carry the date, the engine build and the generator versions that produced it.
Two studies that could be run without a labelled corpus are complete and published: the electronic-music false-positive study and the compression study.
Limitations
Anyone citing a number from this page should cite these alongside it. Each has a stable anchor for that purpose.
- Lossy compression is the dominant error source. Several features are computed in exactly the frequency region an encoder discards first. Results are reported per source condition, never pooled across bitrates.
- Generator versions move faster than benchmarks. A figure measured against one model release says little about the next. Every number is stamped with the generator version and the engine build that produced it.
- Hybrid tracks have no ground truth. A human arrangement built around a generated stem is neither class. These are reported as a separate arm and excluded from headline figures.
- The benchmark is not adversarial. Deliberate evasion — re-recording, heavy remastering, aggressive time and pitch manipulation — is a separate study. Do not read these figures as robustness against an attacker.
- No claim about individual tracks. A population-level rate does not transfer to a single file. No figure on this page should be used as evidence about a specific song or person.
How to cite
The benchmark is published under CC BY 4.0. Cite the version, not just the URL, because figures change when the engine or the generator versions change.
Swipe the table sideways to see all columns.
AI Music Detector. "The AI Music Detection Benchmark" (v1.1), 2026-08-24. https://aimusicdetector.co/research/benchmark
The same data is served as JSON at aimusicdetector.co/benchmark.json, including null fields for every figure that has not been measured yet. Corrections, replication attempts and corpus contributions are welcome via the contact page; see also accuracy and limitations and the methodology.
Frequently asked questions
Because a percentage without a documented corpus, metric definition and engine build is unciteable. Publishing the protocol first also stops us retrofitting a story to whatever the data turns out to say.