UC San Diego Students Push Boundaries of AI and Music Collaboration

Researchers at the University of California, San Diego are exploring ways to integrate artificial intelligence into music composition and performance. Instead of focusing solely on generating songs, they aim to create a more responsive and controllable partner for musicians.

Their work has led to innovative projects that bring together students and faculty from various departments, including music, computer science, and engineering. These collaborations have resulted in experimental compositions, live performances, and generative models designed with artists and listeners in mind.

Three graduating students – Tornike Karchkhadze, Mingyang Yao, and Zachary Novack – have made significant contributions to this field. Each has approached AI from a unique perspective, reflecting their diverse backgrounds and interests.

Tornike Karchkhadze, a Ph.D. student in music, comes from a songwriting background and views AI as a tool for musicians rather than just users. He's developed systems that allow for more control over generated music, enabling artists to tailor the output to their specific needs.

Karchkhadze's research has focused on accompaniment generation, multi-channel music creation, and interpreting graphic notation. One notable project involved using AI to interpret Cornelius Cardew's Treatise, a landmark experimental score that defies conventional musical notation. This work earned runner-up honors at the 2024 IEEE International Conference on Big Data.

Karchkhadze has also created a real-time human-AI co-performance system, which enables musicians to plug in an instrument and receive AI-generated accompaniment in response. However, he acknowledges that creating an AI system with responsiveness similar to human collaboration is still a challenge.

Mingyang Yao, who recently graduated with double majors in mathematics-computer science and cognitive science, has explored how AI models learn musical structure and style. He's developed methods for pre-training symbolic music models on broad collections of classical, folk, and popular music before fine-tuning them to specific styles using limited data.

Yao's research demonstrated strong style adaptation with fewer than 300 pieces from a given composer, outperforming larger models trained on more extensive datasets. His work also involved creating an AI model that interprets harmony in written music by making decisions step-by-step and detecting chord boundaries.

Zachary Novack, another Ph.D. student in computer science, has focused on making generative music systems more useful, playful, and responsive for artists. He argues that current models often fall short of their potential, serving as mere 'toys' rather than creative tools.

Novack's research aims to move beyond one-shot song generation by developing AI systems that can be played with like instruments. His work on Presto, a model for accelerating music generation, has explored how to make AI music systems faster and more interactive.

Their projects collectively point toward a broader vision for AI and music at UC San Diego: not replacing human creativity but expanding the ways musicians compose, perform, improvise, and experiment.