Abstract
Understanding how large audio models represent music, and using that understanding to steer generation, is both challenging and underexplored. Inspired by mechanistic interpretability in language models, where direction vectors in transformer residual streams are key to model analysis and control, we investigate similar techniques in the audio domain. This paper presents the first study of latent direction vectors in large audio models and their use for continuous control of musical attributes in text-to-music generation. Focusing on binary concepts like tempo (fast vs. slow) and timbre (bright vs. dark), we compute steering vectors using the difference-in-means method on curated prompt sets. These vectors, scaled by a coefficient and injected into intermediate activations, allow fine-grained modulation of specific musical traits while preserving overall audio quality. We analyze the effect of steering strength, compare injection strategies, and identify layers with the greatest influence. Our findings highlight the promise of direction-based steering as a more mechanistic and interpretable approach to controllable music generation.
Giorgio Strano
PhD Student
Donato Crisostomi
PhD Student
PhD student @ Sapienza, University of Rome | former Applied Science intern @ Amazon Search, Luxembourg | former Research Science intern @ Amazon Alexa, Turin
Irene Tallini
PhD Student
I’m a Computer Science PhD with Math Bachelor and passion. Right now I’m working on AI for music and vector quantile regression. I like to sing, also.
