MIR Overview
MIR means Music Information Retrieval: the part of audio analysis that turns sound into musical answers — tempo, beat positions, key, chords, pitch, timbre, and structure. This page is a map. It groups the terms you will meet across the docs and shows how they build on one another, so you know which feature to reach for and where it is computed.
These terms are grouped on purpose. They are not isolated functions — almost every MIR task is built on the same time–frequency foundation. Understanding that shared foundation once means the individual features stop looking like a long, unrelated list.
New here? Read this as orientation, not reference
This page explains how the pieces relate. For call signatures, go to the JavaScript API or Python API; for how each one is computed, see DSP Implementation Notes.
The shared pipeline
Most MIR features are derived from a small set of intermediate representations. You rarely build these by hand — libsonare computes them internally — but seeing the flow explains why so many features share parameters like nFft and hopLength.
Sharing these intermediates only pays off inside a single whole-track analysis. analyze (MusicAnalyzer) computes the STFT and its derivatives once and reuses them across BPM, key, chord, and section results.
The per-feature helpers do not share anything
detectBpm, detectKey, detectChords, and the section helpers each start again from the raw samples: four back-to-back calls mean four STFTs, two independent chromagrams, and a second onset envelope. There is no cache keyed on the audio. If you want more than one answer about the same track, call the whole-track analysis once instead of assembling it from the individual helpers.
Which question, which feature
| You want to answer… | Reach for | Built on |
|---|---|---|
| How fast is it? Where are the beats? | BPM / beat tracking | onset strength |
| What key is it in? | key detection | chroma |
| What chord is playing? | chord recognition | chroma |
| Where does the chorus start? | section analysis | MFCC (timbre) + chroma (harmony) + energy |
| What note is the melody? | pitch / melody tracking | pitch tracking (see Separation and pitch below) |
| What does it sound like (timbre)? | MFCC | mel spectrogram |
| Can I separate drums from the rest? | HPSS | spectrogram structure |
| What is the raw frequency content over time? | STFT / spectrogram | the waveform |
| What does the recording space sound like? | room-acoustic analysis | impulse-response (IR) decay or blind free-decay estimates |
Timing: BPM, beat, onset, section
The timing family builds up in layers:
| Feature | What it answers |
|---|---|
| Onset detection | Where notes, drums, or consonants begin: the spikes in an onset-strength envelope. |
| BPM | How periodic those onsets are. |
| Beat tracking | Where pulses land on the timeline. |
| Section analysis | Where longer spans such as intros, verse-like sections, chorus-like sections, and breaks begin and end. Unlike the three rows above it, this one does not read the onset envelope: boundaries come from timbre (MFCC) and harmony (chroma). |
Onset is the root of the rhythm family
BPM, beats, and tempograms all start from the same onset-strength envelope. If you want the time × tempo picture behind a BPM estimate, see the tempogram family in Realtime and Streaming.
Harmony: key, chord, chroma
Chroma compresses frequency content into 12 pitch-class bins (C, C♯, … B), folding every octave of the same note together. That makes it the natural substrate for harmony: key detection estimates the tonal center from the overall chroma distribution, and chord recognition estimates local harmony frame by frame.
Chroma trades octave and timbre detail for harmonic clarity
Folding octaves together is exactly what makes chroma good for key/chord work — and exactly what makes it the wrong tool for melody or timbre, where octave and spectral shape matter. Match the representation to the question.
Spectrum: FFT, STFT, spectrogram
The FFT (Fast Fourier Transform) is an efficient algorithm for the DFT (Discrete Fourier Transform), which converts a block of samples into frequency content.
The STFT (short-time Fourier transform) repeats that over many short, overlapping windows so frequency content can be tracked over time.
A spectrogram is the visual result: time on one axis, frequency on another, intensity as brightness.
Two parameters recur everywhere: nFft (window size — bigger means finer frequency resolution but blurrier timing) and hopLength (step between windows — smaller means more frames and smoother motion). The trade-off between frequency and time resolution is fundamental, not a libsonare quirk.
Perceptual features: mel, MFCC, CQT, VQT
These perceptual features answer different questions:
| Feature | What it emphasizes |
|---|---|
| Mel spectrogram | Frequency resolution shaped toward human hearing: fine detail low, coarser detail high. |
| MFCCs (mel-frequency cepstral coefficients) | A compact "timbre fingerprint". It is computed by taking the mel spectrogram, compressing its loudness with a logarithm, then summarizing each frame into a handful of numbers that capture overall spectral shape rather than exact pitch. |
| CQT / VQT (constant-Q / variable-Q transform) | Musically spaced bins — geometric spacing with a configurable binsPerOctave (12 by default, so one bin per semitone; 24 or 36 are common for chroma and tuning work). Useful when pitch relationships matter more than equal-Hz spacing. |
You can also run these transforms backwards for previews and debugging — see Inverse Features.
Separation and pitch: HPSS and pitch
HPSS means Harmonic/Percussive Source Separation. It splits sustained pitched material from transient hits by using their spectrogram shapes.
| Component | Spectrogram shape |
|---|---|
| Harmonic content | Mostly horizontal lines. |
| Percussive content | Mostly vertical lines. |
Separating them first often improves downstream tasks because drums and pitched instruments otherwise confuse each other.
Pitch estimation tracks the fundamental frequency: the lowest partial of the harmonic series, equivalently the rate at which the waveform repeats. It is not necessarily the strongest frequency — an upper harmonic often dominates a bright brass note or a bass heard through a small speaker, and the pitch is still heard at F0 even when no energy is present there at all. That is why the estimators work from the waveform period rather than from the loudest spectral peak. Useful for melody, vocals, monophonic instruments, tuning checks, and transcription-style workflows.
Adjacent: room acoustics
Room-acoustic analysis is adjacent to MIR. It describes the space captured by the recording rather than the notes, rhythm, or form of the music.
Use direct IR analysis when you have a clean impulse response. That path measures RT60 (the time reverberation takes to decay by 60 dB), EDT, C50, C80, D50, and band decay.
Use blind acoustic estimation when you only have a normal recording. That path reports room-decay cues with a confidence value because the free-decay evidence may be weak or missing. See Room Acoustics.
Implementation notes
libsonare exposes MIR functions across browser/WASM, JavaScript, Python, native bindings, CLI, and C++ APIs.
Many features share intermediate representations such as STFT, chroma, and spectral energy curves. That sharing lives inside MusicAnalyzer: a whole-track analyze computes each intermediate once and reuses it across BPM, key, chord, and section results. The single-feature entry points hold no cross-call cache, so each one rebuilds what it needs from the samples.
The browser demos are built for interactive use, but each one emphasizes a different part of the library:
| Demo | Main role |
|---|---|
| Music Analysis Studio | Full-file MIR: BPM, key, chords, sections, and related analysis. |
| Realtime views | BPM, key, and chord estimates that update over time through StreamAnalyzer. |
| Mastering Studio | Measurement-style APIs such as loudness measurement, reference comparison, and report export. |
Seeing those demos side by side shows which pieces are reusable across analysis work and finishing work.
Related: Introduction, Audio Basics, JavaScript API, Room Acoustics, DSP Implementation Notes, librosa Compatibility