Introduction
Music has long been a powerful means of human expression, shaped by deliberate creative choices that influence how it is experienced. Over time, it has grown increasingly complex, with layered instruments, melodies, and harmonies forming the foundation of modern songs. Producing music used to be a feat that required human emotion, cultural context, and years of experience.
Yet today, computer algorithms are capable of imitating these intricate musical structures with surprising accuracy. Songs can now be rearranged, modified, and even entirely generated within seconds. But how is this possible? How can machines, which do not have human emotions or lived experience, produce pieces that sound human? Understanding how AI generates music requires knowledge about how computers process sound and how computer models interpret vast collections of existing songs.
Processing Sound
Before an AI model can start generating music, it must first understand what music is. However, this cannot be achieved simply by playing video files to a computer. Computers do not have eardrums like humans, so they cannot "listen" to sounds. Instead, they need to interpret information about those sounds. Therefore, before any music is input into an AI training model, it is first converted into a spectrogram.
A spectrogram is a complex graph that shows which frequencies are present in a sound at each moment, with color or brightness indicating their amplitude. Most people are more familiar with waveform graphs, the type seen when recording voice memos, but they only show overall loudness over time. Although they highlight volume changes, they contain less detailed frequency information.
Therefore, spectrograms are the ideal format for AI models to interpret. This is because it breaks down the different parts playing of the song and gives crucial information, such as the loudness and pitch. For example, a high-pitched violin note and a low-pitched bass note produce very different patterns on a spectrogram, which a computer can distinguish.
As a result, the computer gains a comprehensive understanding of each layer of the song, similar to how sheet music breaks down each component of a performance. With this representation, the computer can effectively "hear" the music and begin to detect patterns across pitches, rhythms, and textures.
Interpreting Sound
Once a computer can "hear" music through spectrograms, it needs to recognize what makes music coherent, expressive, and stylistically distinctive. To do this, AI models are trained on thousands of existing songs across genres, instruments, and styles. Just as a human musician would develop their skills be listening to hundreds of songs, AI models gain a statistical understanding of musical strcutures by recognizing recurring patterns in a large dataset of music.
During training, AI identifies patterns in rhythm, melody, harmony, and texture. It learns how notes are sequenced, which chord progressions are common, and how instruments interact to create rich textures. It can detect subtle stylistic signatures: the swing feel of jazz, the punchy backbeat of pop, or the layered harmonies of classical music. By abstracting these patterns, the AI captures the essence of a musical style without storing exact copies of songs.
Importantly, the AI learns probabilities rather than memmorizing the note-to-note strucuture of a song. It learns what note is most likely to come after a certain chord or note, based on the patterns it found in its training data set. This allows the AI model to generate an entirely new piece of music that is stylistically accurate but not an exact replica of an existing song.
Once the AI model has interpreted the dataset and identified the patterns, it needs a method to create the music itself. There are three main models used to accomplish this, all of which will be explained below.
Generative Models
One common approach to generating music is the autoregressive model. This model creates music step by step, producing one note at a time. Each new note is based on the previous note or sequence of notes, chosen according to the patterns the AI learned from its training dataset. In other words, the model predicts what is most likely to come next, much like how your phone suggests the next word as you type a sentence.
Autoregressive models are particularly good at creating short, coherent musical phrases because each note logically follows the one before it. However, they struggle to produce longer, structured compositions, such as entire songs with verses, choruses, and bridges. This limitation arises because the model only considers the immediate context, the “next step," without a broader understanding of the overall musical structure. As a result, while the output may sound plausible in the short term, it can lose coherence when extended over time.
A simple way to visualize this is to imagine a melody where each note is predicted one by one: The model chooses each new note based solely on what came immediately before, rather than planning the melody’s arc from beginning to end. You can try doing this yourself in the short simulation below:
First, the encoder takes a spectrogram of a song and compresses it, extracting the most important musical features from the input data. This compressed representation is then passed to the decoder, which attempts to reconstruct the original song based on this simplified information. The reconstructed output is compared to the original, and the model receives feedback, allowing it to adjust and improve its output over time. This process does not aim to recreate the song exactly. Instead, the VAE learns a general “style” of the music, reaching a level of similarity where the characteristics are recognizable, but the result is not a one-to-one copy.
After being trained on thousands of audio samples, VAEs can preserve stylistic features, such as the distorted guitar sounds common in rock music, while still generating entirely new pieces. Because small variations are introduced during each reconstruction, the music produced by VAEs remains unique, making them particularly well suited for creative music generation.
The final model commonly used for generating music is the Transformer. Like the VAE, the Transformer takes a holistic approach, analyzing an entire song rather than focusing on individual notes. When processing a piece of music, it identifies interdependencies within the song, recognizing how different elements, such as motifs, harmonies, and rhythms, relate to one another across time.
By understanding these long-range relationships, the Transformer can detect recurring motifs, see how various sections of a song connect, and anticipate patterns in the overall structure. This knowledge allows it to generate music with a more nuanced and cohesive form than autoregressive models or VAEs, producing pieces that feel structured, deliberate, and stylistically consistent.
It is important to note that AI can be trained using multiple models simultaneously, and this is often the case in practice. By combining approaches, the system can leverage the strengths of each model: the autoregressive model ensures short-term cohesiveness, the VAE preserves stylistic characteristics, and the Transformer provides long-range structure and consistency.
Conclusion
To summarize, AI generates music by analyzing large datasets of songs, represented as spectrograms. It then uses different models to extract important features, recognize patterns, and generate new compositions that capture the style of the original music.
The creation of AI-generated music demonstrates just how advanced technology has become. Producing songs that can mimic human emotion, style, and musical intention is a remarkable achievement. At the same time, AI is not without limitations. While it can imitate structure and style, it does not truly experience or understand music as humans do. These boundaries and the broader implications of AI creativity will be explored further in the next article, “Limits of AI Creativity.”