How AI Songs Are Created, Start to Finish

Stage one: the prompt and its conditioning

Everything starts with conditioning information — usually a short text prompt describing genre, mood, instrumentation and vocal style, often alongside lyrics and sometimes a reference audio clip. The model does not 'understand' this text in any interesting sense; it encodes it into a vector that steers generation towards regions of its learned distribution associated with similar descriptions in training.

This is why prompts behave the way they do. Genre words work well because genre is heavily represented in training metadata. Requests for a specific named artist's voice work uncomfortably well for the same reason, which is precisely why most providers now block them. And highly unusual combinations produce weaker results, because the model has fewer examples to interpolate between.

Stage two: audio as tokens

Modern song generators do not predict waveforms sample by sample — at 44.1 kHz that would be hopeless. Instead they compress audio into a much shorter sequence of discrete tokens using a neural audio codec, typically a residual vector-quantised autoencoder. A few seconds of music becomes a few hundred tokens across several quantiser levels.

The model then works in that token space, predicting sequences the way a language model predicts text, or denoising them the way an image diffusion model denoises pixels. Coarse tokens carry structure and pitch content; finer levels carry timbre and detail.

The codec is the single most important source of measurable artefacts. It has a finite bandwidth and a finite bit budget, and it learned to spend both on what mattered in training. Where it spends nothing, energy disappears — sometimes as a hard ceiling in the top octave, sometimes as smearing of transients that no longer survive quantisation.

Stage three: generating the sequence

Two families dominate. Autoregressive transformers predict the next token given everything before it, which makes them good at long-range musical coherence and prone to drifting or repeating over long durations. Diffusion models start from noise and denoise the whole token grid at once under the conditioning, which gives strong global consistency and tends to produce very even tonal balance from start to finish.

Both are usually run in stages: a coarse pass sets structure and melody, later passes fill in detail. Vocals may be generated jointly with instrumentation or handled by a dedicated branch conditioned on the lyrics, with a phoneme-level alignment keeping words attached to notes.

The consequences are measurable. Whole-track denoising is part of why cross-segment tonal agreement in generated music is often improbably high: the verse and the chorus were produced by the same global process at the same time, rather than by musicians playing on different days.

• Autoregressive pipelines: strong local phrasing, weaker long-form novelty, occasional loop-like repetition
• Diffusion pipelines: very even spectral balance across the whole track, softer transients
• Extension and inpainting: sections stitched from separate generations, which can create seams where measurements disagree

Stage four: decoding back to audio

The token sequence is decoded to a waveform by the codec's decoder, sometimes followed by a separate neural vocoder or a learned upsampler that raises the output to full sample rate. This stage is where the recognisable texture of generated audio lives: slight metallic sheen on cymbals, a certain glassiness on sibilants, reverb tails that decay a little too cleanly.

Upsamplers exist specifically to hide the codec's bandwidth limit, and they work. That is why the spectral-ceiling test has weakened so much: the missing top end is now synthesised back in, plausibly enough to fool a measurement even if not always a trained ear.

Stage five: the automatic master

Almost every consumer generator applies loudness normalisation and limiting before you hear the result, because trained on commercial masters, output that arrives quiet feels wrong to users. The practical effect is that generated tracks land in the same narrow dynamics window as commercial releases.

For detection this is doubly awkward. It removes crest factor as a discriminator, and it means anything a user does afterwards — re-encoding, re-mastering, uploading to a platform that normalises again — pushes human and generated material even closer together.

What this means for detecting the result

Read the pipeline backwards and the detection problem becomes clear. The signals worth measuring are the ones tied to stages the pipeline cannot easily hide: how consistent the whole-track statistics are, how much brightness moves over time, whether transients survive, whether the stereo image was constructed rather than captured.

The signals that used to work — hard spectral ceilings, obvious codec buzz, crushed dynamics — were tied to stages that have since been engineered away. Any detector claiming stable accuracy across generator versions is either not being evaluated properly or not saying which version it was evaluated on.

This is also why an honest tool reports a probability with a confidence level and an explicit inconclusive band. The pipeline changes faster than any fixed set of acoustic measurements can track.