MidiBERT for Music Understanding

#transformer models #music analysis #midi data #nlp #deep learning #tokenization #pre-training #BERT #music understanding #neural networks

1. What is MidiBERT?

What is MidiBERT?

MidiBERT is a transformer-based model specifically designed for symbolic music understanding, leveraging the success of BERT (Bidirectional Encoder Representations from Transformers) in natural language processing. Unlike traditional approaches that process raw audio waveforms, MidiBERT operates on symbolic music representations, such as MIDI files, which encode musical events like note pitches, durations, velocities, and timing. This allows the model to capture high-level musical structures, including harmony, rhythm, and melody, in a computationally efficient manner.

Architecture and Tokenization

MidiBERT employs a transformer encoder architecture, similar to BERT, but adapts the tokenization process to musical data. Instead of word or subword tokens, MidiBERT tokenizes MIDI events into discrete units representing musical elements. Each token corresponds to a specific musical event, such as:

The tokenization process converts a MIDI sequence into a series of discrete tokens, which are then embedded into a continuous vector space. These embeddings are fed into the transformer encoder, which learns contextual relationships between musical events through self-attention mechanisms.

Pre-training Objectives

MidiBERT is pre-trained using two primary objectives adapted from BERT:

$$ \text{MLM Loss} = -\sum_{i \in \mathcal{M}} \log P(x_i | x_{\setminus \mathcal{M}}) $$

where M represents the set of masked tokens, and x denotes the input sequence.

Applications and Advantages

MidiBERT excels in tasks requiring deep musical understanding, such as:

Its symbolic approach offers advantages over waveform-based models, including reduced computational complexity and explicit modeling of musical semantics. However, it relies on the availability of high-quality symbolic data, which may limit its applicability in scenarios where only audio recordings are available.

What is MidiBERT? – MidiBERT for Music Understanding – Tutorial Diagram
Diagram Description: The diagram would show the tokenization process of MIDI events into discrete units and how they are fed into the transformer encoder, illustrating the flow from raw MIDI data to embedded tokens.

Key Applications of MidiBERT in Music Analysis

MidiBERT, a transformer-based model pre-trained on symbolic music data (MIDI), excels in capturing hierarchical and long-range dependencies in musical sequences. Its architecture, inspired by BERT, enables it to process polyphonic music with nuanced understanding. Below are its key applications in advanced music analysis.

Music Generation and Composition

MidiBERT's bidirectional attention mechanism allows it to generate coherent musical sequences by predicting masked tokens in a given context. The model learns latent representations of musical structure, enabling it to compose melodies, harmonies, and even full arrangements. For conditional generation, the probability distribution for the next token yt given previous tokens y<t is computed as:

$$ P(y_t | y_{

where W and b are learnable parameters, and ht is the hidden state at position t.

Automatic Music Transcription

MidiBERT can transcribe polyphonic audio into symbolic notation by first converting audio to a piano roll representation, then predicting note onsets, durations, and velocities. The model's self-attention layers help disambiguate overlapping notes, a common challenge in transcription. Performance is measured using the F1-score for note-level accuracy:

$$ F1 = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} $$

Music Style Transfer

By fine-tuning MidiBERT on genre-specific datasets, the model can transform a musical piece from one style to another (e.g., classical to jazz). The key lies in its ability to isolate style-agnostic features (e.g., melody) from style-specific ones (e.g., rhythm patterns). Cross-attention mechanisms facilitate this disentanglement.

Emotion and Structure Analysis

MidiBERT's embeddings correlate with perceptual features like emotion (valence/arousal) and formal structure (e.g., verse-chorus segmentation). A linear probe trained on its embeddings achieves state-of-the-art performance in emotion classification, leveraging the attention weights to identify emotionally salient segments.

Real-Time Performance Assistance

In live performances, MidiBERT can predict upcoming notes or harmonies based on the performer's input, acting as an AI collaborator. The model's low-latency inference is achieved through optimized attention caching, reducing the computational complexity of autoregressive generation from O(n2) to O(n) for sequential decoding.

1.3 Advantages of Using Transformer Models for Music

Long-Range Dependency Modeling

Traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs) struggle with capturing long-range dependencies in sequential data due to their inherent architectural limitations. RNNs suffer from vanishing gradients, while CNNs are constrained by their fixed receptive fields. Transformer models, however, leverage self-attention mechanisms to compute pairwise relationships between all tokens in a sequence, regardless of their positional distance. For musical sequences, this enables the model to recognize harmonic progressions, melodic motifs, and rhythmic patterns that span hundreds of time steps.

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

Here, Q, K, and V represent the query, key, and value matrices, respectively, while dk is the dimension of the key vectors. The softmax operation ensures that the attention weights sum to one, allowing the model to focus on the most relevant parts of the input sequence.

Parallelization and Scalability

Unlike RNNs, which process sequences sequentially, Transformers process all tokens in parallel during both training and inference. This parallelization drastically reduces training time and enables efficient scaling to large datasets. For music applications, where sequences can be extremely long (e.g., MIDI files with thousands of time steps), this property is particularly advantageous. The computational complexity of self-attention is O(n2) for sequence length n, but techniques like sparse attention or memory-efficient variants mitigate this cost.

Hierarchical Feature Learning

Transformers inherently learn hierarchical representations through their multi-head attention mechanism and deep stack of layers. In music, lower layers may capture local features like note durations and intervals, while higher layers model global structures such as chord progressions and song form. This hierarchical abstraction aligns well with the multi-scale nature of musical compositions, where local motifs contribute to larger-scale themes.

Transfer Learning and Pretraining

Transformer architectures like MidiBERT benefit from pretraining on large-scale musical corpora, learning generalizable representations that can be fine-tuned for specific downstream tasks (e.g., melody generation, style transfer, or music classification). The masked language modeling (MLM) objective, where random tokens are masked and predicted, forces the model to develop a robust understanding of musical context. This approach mirrors successful NLP models like BERT, demonstrating the Transformer's versatility across domains.

Handling Polyphonic Music

Music is inherently polyphonic, with multiple notes sounding simultaneously. Transformers excel at modeling such complex interactions due to their ability to attend to multiple input tokens simultaneously. By representing musical events as discrete tokens (e.g., note-on, note-off, velocity), the model can learn intricate dependencies between concurrent and sequential musical events, capturing the rich texture of polyphonic compositions.

Flexible Input Representations

Transformers can accommodate various musical representations, including MIDI events, piano rolls, or symbolic notation. The tokenization process can be adapted to preserve musical semantics, such as pitch, duration, and dynamics. This flexibility allows the same architecture to be applied across different musical tasks without significant structural modifications.

2. Transformer-Based Architecture Overview

Transformer-Based Architecture Overview

The foundation of MidiBERT lies in the transformer architecture, originally introduced by Vaswani et al. in 2017. Unlike recurrent or convolutional neural networks, transformers rely entirely on self-attention mechanisms to model sequential data, making them particularly effective for capturing long-range dependencies in musical sequences. The architecture comprises several key components: multi-head self-attention, positional encodings, layer normalization, and feed-forward neural networks.

Self-Attention Mechanism

The self-attention mechanism computes a weighted sum of input embeddings, where the weights are determined by the compatibility between pairs of tokens. Given an input sequence X of dimension dmodel, the mechanism first projects X into queries (Q), keys (K), and values (V) using learned linear transformations:

$$ Q = XW_Q, \quad K = XW_K, \quad V = XW_V $$

where WQ, WK, WV are weight matrices of dimension dmodel × dk. The attention scores are computed as:

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

The scaling factor √dk prevents the dot products from growing too large in magnitude, which would otherwise push the softmax into regions of extremely small gradients.

Multi-Head Attention

Multi-head attention extends self-attention by applying multiple attention mechanisms in parallel, allowing the model to jointly attend to information from different representation subspaces. For h heads, the output is computed as:

$$ \text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W_O $$

where each head is an independent self-attention operation:

$$ \text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V) $$

and WO is a learned output projection matrix. In MidiBERT, this mechanism enables the model to simultaneously focus on different aspects of musical structure, such as harmony, rhythm, and melody.

Positional Encoding

Since transformers lack inherent sequential processing, positional encodings are added to the input embeddings to inject information about the order of tokens. For a position pos and dimension i, the encoding is defined as:

$$ PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right) $$ $$ PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right) $$

These sinusoidal functions allow the model to generalize to sequence lengths longer than those encountered during training, a critical feature for handling variable-length musical compositions.

Layer Normalization and Residual Connections

Each sub-layer in the transformer (self-attention, feed-forward network) is followed by layer normalization and a residual connection, which stabilize training and mitigate vanishing gradients. The output of a sub-layer is computed as:

$$ \text{LayerNorm}(x + \text{Sublayer}(x)) $$

where Sublayer(x) is the function implemented by the sub-layer itself. This architecture ensures smooth gradient flow during backpropagation, even in deep networks.

Feed-Forward Networks

Each transformer layer includes a position-wise feed-forward network (FFN) applied identically to each token. The FFN consists of two linear transformations with a ReLU activation in between:

$$ \text{FFN}(x) = \text{ReLU}(xW_1 + b_1)W_2 + b_2 $$

where W1, W2 are weight matrices and b1, b2 are bias terms. In MidiBERT, the FFN enables non-linear transformations of the attention outputs, capturing complex musical patterns.

By stacking multiple transformer layers, MidiBERT builds hierarchical representations of musical sequences, with lower layers capturing local motifs and higher layers modeling global structure. This architecture has proven particularly effective for tasks such as music generation, classification, and symbolic music understanding.

Transformer-Based Architecture Overview – MidiBERT for Music Understanding – Tutorial Diagram
Diagram Description: The diagram would show the transformer architecture's multi-head attention mechanism, positional encoding, and layer normalization flow, which are spatial and hierarchical relationships.

Tokenization Strategies for MIDI Data

Tokenizing MIDI data for transformer-based models like MidiBERT requires careful consideration of the hierarchical and multi-modal nature of musical sequences. Unlike text, MIDI encodes pitch, velocity, duration, and timing with parallel tracks, necessitating specialized tokenization approaches that preserve musical semantics while enabling efficient sequence modeling.

Event-Based Tokenization

The most common strategy decomposes MIDI into discrete events, each representing a musical action (e.g., note-on, note-off, tempo change). Each event is mapped to a token ID through a vocabulary that typically includes:

$$ t_i = \text{argmin}_k |d - \mu_k| \quad \text{where} \quad \mu_k = \alpha \log(1 + k/\beta) $$

Duration quantization uses log-scaling to prioritize precision at shorter note values, with α and β controlling bin distribution. This matches human perceptual sensitivity to rhythmic variations.

Temporal Resolution Handling

MIDI's temporal precision (often 480 ticks per quarter note) requires downsampling to prevent sequence fragmentation. Two dominant approaches:

$$ t_{\text{quant}} = \left\lfloor \frac{t}{\Delta} + 0.5 \right\rfloor \Delta $$

Multi-Track Representation

Polyphonic music requires strategies to handle concurrent tracks. The flattened approach interleaves events from all tracks in temporal order, while hierarchical methods use:

Vocabulary Construction

The token vocabulary size critically impacts model performance. For a 88-key piano with 32 velocity bins and 100 duration bins, the naive combination would yield 281,600 tokens. Practical implementations use:

Recent variants like Compound Word Tokenization decompose events into attribute sequences (e.g., [NOTE] [C4] [VEL64] [DUR200ms]), reducing vocabulary size while maintaining expressivity.

MIDI Tokenization Hierarchy Hierarchical block diagram showing MIDI tokenization structure with parallel tracks, event types, and temporal relationships. Time [PIANO] [DRUMS] Note Events Note-on Note-off Velocity (32) Duration (log) Control Legend Track Prefix Note Events Control Tokens MIDI Tokenization Hierarchy
Diagram Description: The diagram would show the hierarchical structure of MIDI tokenization, including parallel tracks, event types, and their temporal relationships.

2.3 Pre-training Objectives and Tasks

Masked Language Modeling (MLM) for Symbolic Music

MidiBERT adapts the Masked Language Modeling (MLM) objective from BERT to symbolic music data. Given a sequence of MIDI events X = (x1, x2, ..., xn), a subset of tokens is randomly masked with probability pmask. The model learns to predict the masked tokens based on the surrounding context. For MIDI data, tokens represent musical elements such as note pitch, velocity, duration, and timing. The loss function for MLM is:

$$ \mathcal{L}_{\text{MLM}} = -\sum_{i \in \mathcal{M}} \log P(x_i | X_{\backslash \mathcal{M}}) $$

where M is the set of masked positions. Unlike text, MIDI tokens exhibit hierarchical dependencies (e.g., a note's duration depends on its onset timing), requiring the model to capture both local and global musical structure.

Next Sentence Prediction (NSP) Adaptation

While BERT uses NSP to learn relationships between text sentences, MidiBERT replaces this with Next Segment Prediction (NSP), where the model predicts whether two musical segments are contiguous. Given segments S1 and S2, the objective is:

$$ \mathcal{L}_{\text{NSP}} = -\mathbb{E}[\log P(y | S_1, S_2)] $$

where y = 1 if S2 follows S1 in the original composition, and y = 0 otherwise. This task encourages the model to learn musical coherence beyond local note patterns.

Joint Pitch-Duration Modeling

MIDI events decompose into discrete attributes: pitch p, duration d, and velocity v. MidiBERT extends MLM to predict these attributes jointly. For a masked note, the probability decomposes as:

$$ P(x_i) = P(p_i) \cdot P(d_i | p_i) \cdot P(v_i | p_i, d_i) $$

This factorization reflects musical dependencies—e.g., duration distributions vary by pitch due to instrument-specific idioms.

Contrastive Learning for Musical Similarity

MidiBERT incorporates a contrastive loss to distinguish between positive (musically similar) and negative (dissimilar) segments. Given an anchor segment S, positive sample S+, and negatives S-, the InfoNCE loss is:

$$ \mathcal{L}_{\text{contrast}} = -\log \frac{\exp(\text{sim}(f(S), f(S^+)) / \tau)}{\sum_{k=1}^K \exp(\text{sim}(f(S), f(S^-_k)) / \tau)} $$

where f(·) is the model's embedding, sim is cosine similarity, and τ is a temperature hyperparameter. This objective aligns embeddings for musically equivalent segments (e.g., transposed melodies).

Bar-Level Positional Encoding

To capture metrical structure, MidiBERT uses bar-relative positional encodings. For a note at position t in bar b, the encoding combines:

This dual encoding allows the model to attend to both phrase-level (multi-bar) and beat-synchronous patterns.

Pre-training Objectives and Tasks – MidiBERT for Music Understanding – Tutorial Diagram
Diagram Description: The diagram would show the hierarchical token masking process in MLM for MIDI events and the bar-level positional encoding structure.

3. Datasets for Training MidiBERT

Datasets for Training MidiBERT

Training MidiBERT effectively requires large-scale, high-quality MIDI datasets that capture diverse musical styles, structures, and expressive nuances. The choice of dataset directly impacts the model's ability to generalize across musical genres and tasks such as melody generation, harmonization, or style transfer.

Key MIDI Datasets for MidiBERT

The following datasets are commonly used for training MidiBERT and similar music language models:

Preprocessing MIDI Data for MidiBERT

MIDI files require careful preprocessing to be compatible with transformer-based architectures like MidiBERT. The standard pipeline includes:

  1. Quantization — Aligning note onsets and durations to a fixed temporal grid (e.g., 16th notes) to reduce sparsity in the input representation. Given a time resolution δt, each event time t is quantized as:
    $$ t_q = \left\lfloor \frac{t}{\delta t} + 0.5 \right\rfloor \cdot \delta t $$
  2. Tokenization — Converting MIDI events (note-on, note-off, velocity, tempo) into discrete tokens. MidiBERT typically uses a vocabulary of 3,000–5,000 tokens, covering pitch, duration, velocity, and control events.
  3. Sequence Chunking — Splitting long MIDI sequences into fixed-length segments (e.g., 512 tokens) to fit the transformer's context window. Overlapping segments or sliding windows may be used to preserve continuity.

Dataset Augmentation Strategies

To improve robustness, MIDI datasets are often augmented through:

Evaluation Benchmarks

Standard benchmarks for evaluating MidiBERT's performance include:

3.2 Fine-Tuning for Specific Music Tasks

Fine-tuning MidiBERT for specialized music tasks involves adapting its pre-trained representations to downstream applications such as melody generation, chord recognition, or expressive performance modeling. The process leverages transfer learning by initializing the model with weights from pre-training and then updating them on task-specific labeled data. Given MidiBERT's transformer architecture, fine-tuning typically involves modifying the output layer and optimizing the entire model end-to-end.

Task-Specific Architecture Modifications

For classification tasks like genre identification or emotion recognition, a linear projection layer is appended to the [CLS] token's hidden representation:

$$ \mathbf{y} = \text{softmax}(\mathbf{W}_c \mathbf{h}_{[\text{CLS}]} + \mathbf{b}_c) $$

where Wc and bc are learnable parameters. For sequence labeling tasks (e.g., note-level articulation detection), each token's hidden state is projected:

$$ \mathbf{y}_t = \text{softmax}(\mathbf{W}_s \mathbf{h}_t + \mathbf{b}_s) $$

Optimization Strategy

The loss function combines task-specific objectives with optional regularization. For multi-task learning, a weighted sum of losses is used:

$$ \mathcal{L} = \sum_{i=1}^N \lambda_i \mathcal{L}_i(\theta) + \beta \|\theta\|_2^2 $$

where λi balances task importance and β controls L2 regularization. The learning rate is typically reduced by 10× compared to pre-training, with values between 1e-5 and 5e-5 empirically performing well for music tasks.

Data Augmentation for Music

Effective fine-tuning requires strategies to handle limited labeled music data:

Case Study: Expressive Performance Prediction

When fine-tuning for predicting performance attributes (velocity, timing deviations), the model processes aligned score-performance pairs. The input sequence interleaves:

$$ \mathbf{X} = [\text{Score}_1, \text{Perf}_1, \text{Score}_2, \text{Perf}_2, ..., \text{Score}_n, \text{Perf}_n] $$

where Score tokens represent nominal note values and Perf tokens encode realized performance features. The model learns cross-modal attention between score and performance contexts.

Computational Considerations

Fine-tuning efficiency can be improved through:

Addressing Challenges in Music Sequence Modeling

Long-Range Dependencies in Polyphonic Music

Music sequences exhibit intricate long-range dependencies, particularly in polyphonic compositions where multiple voices interact simultaneously. Traditional recurrent architectures struggle with these dependencies due to vanishing gradients, while standard attention mechanisms in transformers face quadratic complexity with sequence length. MidiBERT mitigates this by employing relative position embeddings, allowing the model to efficiently capture relationships between distant musical events without explicit positional encoding.

$$ A_{ij} = \frac{(W_Q h_i)^T (W_K h_j + R_{i-j})}{\sqrt{d_k}} $$

Here, Ri-j represents learnable relative position biases, enabling the model to attend to note patterns regardless of absolute position. This is critical for recognizing motifs or chord progressions that recur across measures.

Hierarchical Structure Modeling

Musical structure operates at multiple timescales—from individual notes to phrases and sections. MidiBERT addresses this through:

Discrete Token Representation Challenges

MIDI events are inherently discrete, creating sparse input representations. MidiBERT improves upon this by:

$$ e_i = W_p p_i + W_d d_i + W_v v_i $$

Where pi, di, and vi represent pitch, duration, and velocity embeddings respectively. This disentangled representation allows the model to separately model different musical dimensions while maintaining their interactions through attention.

Temporal Irregularity in Expressive Performances

Human performances introduce microtiming deviations and rubato that challenge rigid sequence models. MidiBERT incorporates:

$$ \mathcal{L} = \alpha \mathcal{L}_{CE} + (1-\alpha) \mathcal{L}_{MSE} $$

The cross-entropy loss LCE handles discrete note events while mean squared error LMSE captures continuous timing nuances.

Computational Efficiency for Long Sequences

Full-length musical works can span thousands of events. MidiBERT implements:

This maintains O(n log n) complexity while preserving the ability to model complete musical forms.

Addressing Challenges in Music Sequence Modeling – MidiBERT for Music Understanding – Tutorial Diagram
Diagram Description: The diagram would show the relative position embeddings mechanism and how bar-level masking operates across musical sequences.

4. Music Generation and Composition

Music Generation and Composition

Architecture and Tokenization in MidiBERT

MidiBERT employs a transformer-based architecture, leveraging self-attention mechanisms to model sequential dependencies in symbolic music representations. The input is tokenized into discrete events, including:

Each event is embedded into a continuous vector space, enabling the model to capture hierarchical relationships. The tokenization process maps MIDI data into a sequence of integers, analogous to word tokenization in natural language processing.

$$ \mathbf{E} = \text{Embedding}(\mathbf{T}) \in \mathbb{R}^{n \times d} $$

where n is the sequence length, d is the embedding dimension, and T represents the tokenized input.

Autoregressive Generation with Masked Language Modeling

MidiBERT utilizes a masked language modeling (MLM) objective during pretraining, where random tokens are masked, and the model predicts the missing elements. For generation, the model operates autoregressively, sampling from the output distribution at each step:

$$ P(\mathbf{y}_t | \mathbf{y}_{<t}) = \text{softmax}(\mathbf{W} \mathbf{h}_t + \mathbf{b}) $$

where ht is the hidden state at position t, and W, b are learnable parameters. Temperature scaling and top-k sampling are commonly applied to control diversity.

Harmonic and Rhythmic Coherence

The self-attention mechanism enables MidiBERT to learn long-range dependencies critical for maintaining harmonic and rhythmic coherence. Attention heads specialize in different aspects:

Fine-Tuning for Compositional Tasks

Pretrained MidiBERT can be fine-tuned for specific compositional tasks through:

$$ \mathcal{L} = \lambda_1 \mathcal{L}_{\text{MLM}} + \lambda_2 \mathcal{L}_{\text{task}}} $$

where λ1 and λ2 balance pretraining and task-specific objectives.

Evaluation Metrics for Generated Music

Quantitative evaluation employs both objective metrics and human assessment:

$$ \text{Entropy} = -\sum_{i=1}^{12} p_i \log_2 p_i $$

where pi represents the probability of pitch class i in the generated piece.

Music Generation and Composition – MidiBERT for Music Understanding – Tutorial Diagram
Diagram Description: The diagram would show the transformer architecture with attention heads processing tokenized MIDI events, illustrating how local/global attention and position embeddings interact.

4.2 Music Classification and Tagging

MidiBERT's architecture, derived from BERT's transformer-based framework, is particularly effective for music classification and tagging tasks due to its ability to model sequential dependencies in symbolic music data. The model processes MIDI events as tokens, leveraging self-attention mechanisms to capture long-range harmonic, rhythmic, and structural patterns. For classification, a task-specific head is appended to the pretrained MidiBERT backbone, typically implemented as a multilayer perceptron (MLP) with softmax activation:

$$ P(y|x) = \text{softmax}(W \cdot \text{MidiBERT}(x) + b) $$

where x represents the input MIDI sequence, W and b are learnable parameters, and y denotes the predicted class probabilities. The model is fine-tuned end-to-end using categorical cross-entropy loss:

$$ \mathcal{L} = -\sum_{i=1}^N y_i \log(\hat{y}_i) $$

Feature Extraction for Tagging

For multi-label tagging tasks (e.g., genre, mood, or instrument identification), MidiBERT employs a sigmoid-activated output layer with binary cross-entropy loss. The model extracts hierarchical features at different temporal resolutions:

Attention Analysis

The self-attention weights reveal interpretable patterns in music understanding. For instance, high attention scores often occur between:

This attention behavior enables MidiBERT to outperform convolutional and recurrent architectures on tasks like genre classification, where global context is critical. On the GTZAN dataset, MidiBERT achieves 85.2% accuracy, surpassing CNN-based models by 6-8%.

Transfer Learning Applications

Pretrained MidiBERT embeddings can be transferred to downstream tasks with limited labeled data. The embedding space organizes compositions by:

For few-shot learning, a k-nearest neighbors classifier using MidiBERT embeddings achieves 72% accuracy with only 20 examples per class on the MusicNet dataset.

Music Classification and Tagging – MidiBERT for Music Understanding – Tutorial Diagram
Diagram Description: The diagram would show the hierarchical feature extraction process in MidiBERT, illustrating how local, phrase, and global-level features are captured by different transformer layers.

4.3 Music Transcription and Symbolic Representation

Music transcription involves converting audio signals into symbolic representations such as MIDI or sheet music. MidiBERT leverages transformer architectures to model polyphonic music sequences, capturing both local and global dependencies in symbolic music data. The model treats musical events—notes, chords, and rests—as discrete tokens, enabling it to learn hierarchical structures analogous to language modeling in NLP.

Symbolic Music Representation

MIDI encodes music as a sequence of discrete events, each characterized by attributes like pitch, velocity, and duration. A symbolic representation S of a musical piece can be formalized as a sequence of tokens:

$$ S = \{ (p_i, v_i, d_i) \}_{i=1}^N $$

where pi denotes pitch, vi represents velocity (dynamics), and di is duration. MidiBERT tokenizes these attributes into a vocabulary of discrete symbols, enabling the use of transformer-based attention mechanisms.

Event-Based Tokenization

MidiBERT employs an event-based tokenization scheme, where each musical event is decomposed into a series of subtokens. For example, a note-on event is represented as:

This fine-grained tokenization allows the model to capture nuanced musical features, such as articulation and phrasing.

Transformer Architecture for Music

MidiBERT adapts the transformer architecture to handle symbolic music sequences. The self-attention mechanism computes relationships between all pairs of tokens, enabling the model to learn long-range dependencies. The attention weights A for a sequence of length L are computed as:

$$ A = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} \right) V $$

where Q, K, and V are query, key, and value matrices, and dk is the dimension of the key vectors. This mechanism allows MidiBERT to model complex musical structures, such as motifs and harmonic progressions.

Training Objectives

MidiBERT is pre-trained using two primary objectives:

These objectives enable the model to learn both bidirectional and sequential dependencies in music.

Applications in Music Transcription

MidiBERT's symbolic understanding facilitates applications such as:

The model's ability to handle polyphony and complex rhythms makes it particularly suited for classical and jazz music, where precise symbolic representation is critical.

Music Transcription and Symbolic Representation – MidiBERT for Music Understanding – Tutorial Diagram
Diagram Description: The diagram would show the event-based tokenization process of a musical note into subtokens (Note-On, Pitch, Velocity, Duration) and how these tokens are structured in a sequence.

5. Bias in Music Data and Models

5.1 Bias in Music Data and Models

Bias in music data and models manifests in multiple forms, from dataset composition to algorithmic decision-making. MidiBERT, like other transformer-based models, inherits biases present in its training data, which can propagate through tokenization, attention mechanisms, and downstream tasks. Understanding these biases is critical for ensuring fair and representative music generation and analysis.

Sources of Bias in Music Datasets

Training datasets for music models often exhibit imbalances in genre, cultural origin, and composer demographics. For instance, classical MIDI datasets disproportionately represent Western composers, while underrepresented genres like traditional African or Indigenous music are scarce. This skew affects MidiBERT's ability to generalize across diverse musical styles. The tokenization process further compounds this issue by encoding musical elements in ways that favor certain structures—such as 4/4 time signatures or diatonic scales—over others.

$$ P(\text{genre} = g \mid \mathcal{D}) = \frac{N_g}{\sum_{i=1}^k N_i} $$

Here, \( P(\text{genre} = g \mid \mathcal{D}) \) represents the probability of encountering genre \( g \) in dataset \( \mathcal{D} \), where \( N_g \) is the count of samples for genre \( g \). A model trained on such data will inherently favor overrepresented genres during generation or classification.

Algorithmic Amplification of Bias

MidiBERT's self-attention mechanism can amplify existing biases. Attention weights \( \alpha_{ij} \) between tokens \( i \) and \( j \) are computed as:

$$ \alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^n \exp(e_{ik})}, \quad e_{ij} = \frac{Q_i K_j^T}{\sqrt{d_k}} $$

If the query (\( Q_i \)) and key (\( K_j \)) vectors are biased toward certain musical patterns, the model will disproportionately attend to those patterns. For example, a preference for chord progressions common in Western pop music might overshadow less frequent but culturally significant progressions in other traditions.

Evaluating and Mitigating Bias

Quantifying bias requires metrics such as genre parity or cultural representativeness. One approach is to measure the Kullback-Leibler (KL) divergence between the model's output distribution \( P_{\text{model}} \) and an ideal uniform distribution \( P_{\text{uniform}} \):

$$ D_{\text{KL}}(P_{\text{uniform}} \parallel P_{\text{model}}) = \sum_{g} P_{\text{uniform}}(g) \log \frac{P_{\text{uniform}}(g)}{P_{\text{model}}(g)} $$

Mitigation strategies include:

Case Study: Bias in Chord Prediction

When MidiBERT predicts chord progressions, it tends to favor major and minor triads over extended or non-tertian harmonies (e.g., quartal or cluster chords). This reflects the prevalence of triadic harmony in its training data. A histogram of predicted chords for a diverse test set often reveals this skew, with diminished or augmented chords appearing less than 5% of the time despite their theoretical importance.

Chord Type Distribution in MidiBERT Predictions Major Minor Diminished Augmented

5.2 Intellectual Property and Copyright Issues

The application of MidiBERT in music generation and analysis raises critical intellectual property (IP) and copyright concerns, particularly when the model is trained on existing MIDI datasets. Unlike raw audio, MIDI files encode musical compositions in a structured symbolic format, making them subject to copyright protection under most jurisdictions. The legal landscape becomes complex when considering derivative works generated by the model, as well as potential infringement risks during training.

Training Data and Copyright Infringement

MidiBERT's pretraining typically involves large-scale MIDI datasets, many of which contain copyrighted compositions. Under U.S. law (17 U.S.C. § 106), the reproduction of copyrighted works during training may constitute infringement unless covered by fair use (17 U.S.C. § 107) or licensed appropriately. The four-factor fair use analysis becomes particularly relevant:

The European Union's Copyright Directive (Article 4) provides a text and data mining exception for research organizations, but its applicability to AI music models remains untested in court.

Output Generation and Authorship

When MidiBERT generates new musical sequences, determining copyright ownership involves multiple legal questions:

$$ P(\text{Infringement}) = \int_{\Omega} \mathbb{I}(\text{similarity}(x, y) > \tau) \cdot f_Y(y) \, dy $$

where x is the generated output, y represents copyrighted training examples, and τ is a similarity threshold that may constitute substantial copying. Current U.S. Copyright Office policy states that purely machine-generated works lack human authorship and are therefore ineligible for copyright protection (Compendium § 313.2), though human-AI collaborations may qualify.

Mitigation Strategies

Several technical and legal approaches can reduce IP risks:

The 2023 U.S. Copyright Office guidance on AI-generated works emphasizes that copyright protection requires human creative input that goes beyond mere prompt engineering. This suggests that MidiBERT outputs may need significant human curation or modification to qualify for protection.

International Considerations

Jurisdictional differences create additional complexity. Japan's Copyright Act (Article 30-4) explicitly permits AI training on copyrighted material regardless of purpose, while the UK's Computer Generated Works provision (CDPA 1988 § 9(3)) assigns authorship to the person who made arrangements for the creation. The EU's proposed AI Act may introduce new transparency requirements for training data provenance.

5.3 Limitations of Current Approaches

Current approaches to symbolic music understanding, including transformer-based models like MidiBERT, face several fundamental limitations that hinder their performance and generalizability. These limitations stem from architectural constraints, data representation issues, and computational inefficiencies.

1. Tokenization Challenges

Most symbolic music models rely on tokenization schemes that discretize continuous musical features (e.g., velocity, tempo) into fixed bins. This quantization introduces information loss, particularly for expressive performances where subtle variations carry semantic meaning. The trade-off between vocabulary size and computational efficiency forces models to operate at reduced resolution.

$$ \mathcal{L}_{quant} = \sum_{i=1}^{N} (x_i - \hat{x}_i)^2 $$

where \(x_i\) represents the original continuous value and \(\hat{x}_i\) its quantized counterpart. This reconstruction error accumulates across sequential predictions.

2. Context Window Constraints

Transformers process music as fixed-length sequences due to the quadratic complexity of self-attention:

$$ \mathcal{O}(n^2d) $$

where \(n\) is sequence length and \(d\) is embedding dimension. This limits the model's ability to capture long-range musical structures (e.g., sonata form) that may span thousands of tokens. While relative position encoding and sparse attention variants help, they still struggle with hierarchical musical relationships.

3. Lack of Explicit Music Theory Knowledge

Current models learn musical grammar purely from data without incorporating:

This leads to syntactically valid but musically nonsensical outputs, particularly in few-shot generation scenarios.

4. Multimodal Integration Gaps

Symbolic representations alone cannot capture:

Hybrid architectures that jointly model symbolic and audio representations remain computationally prohibitive for most real-world applications.

5. Evaluation Metrics

Standard metrics like perplexity or BLEU scores fail to assess:

The lack of robust evaluation frameworks makes comparative analysis between models particularly challenging.

6. Data Scarcity and Bias

High-quality symbolic music datasets suffer from:

This cultural bias limits the models' ability to generalize across musical traditions.

6. Key Research Papers on MidiBERT

6.1 Key Research Papers on MidiBERT

6.2 Open-Source Implementations and Tools

6.3 Recommended Books and Articles on Music AI