The Research Behind Songsmith

I want to know more about how Songsmith works. Is it super-top-secret?

Au contraire, we’d love to tell you all about the inner workings of Songsmith. In fact, for the tech-folks among you, we’ve published several research papers about the underlying technology, and you can read them at the MySong project page. The MySong project was a collaboration among Dan Morris and Sumit Basu at Microsoft Research and Ian Simon at the University of Washington. Songsmith grew directly out of this research project.


How should I cite Songsmith in a research paper?

If you’d like to cite Songsmith in a research publication, please cite our CHI 2008 paper:

Simon I, Morris D, and Basu S. MySong: Automatic Accompaniment Generation for Vocal Melodies. Proceedings of ACM CHI 2008 (the 26th SIGCHI Conference on Human Factors in Computing Systems), p725-724.

...or our AAAI 2009 paper:

Morris D, Simon I, and Basu S. Exposing parameters of a trained dynamic model for interactive music creation. Proceedings of AAAI 2008 (the 23rd National Conference on Artificial Intelligence), p.784-791.


What’s the short version of how Songsmith works?

Songsmith basically does three things when it’s making music to go with your melody:

  1. Songsmith figures out roughly what frequencies (and thus what musical notes) you sang using a technique called autocorrelation. We don’t figure this out quite so precisely that we could write down sheet music for your melody (that’s still something a computer can’t do as well as a trained musician), but it turns out we don’t need to figure out exactly the melody you sang in order to make good backing music.
  2. Songsmith chooses chords to accompany the notes you sang, using the algorithm we describe in our papers about MySong. Basically, we’ve used a database of about 300 popular songs to train a mathematical model with the basic statistics of what chords sound good with each other and what chords sound good with different types of melodies. Note this is a statistical model... we’re not taking your melody and peeking into a database to find similar bits of melody when we make new chords. Instead, Songsmith has learned the basic statistics of chord sequences, and uses those statistics to generate new chords when you sing a new melody.

    Of course, there’s no single correct sequence of chords for a given melody – in fact there are lots and lots and lots of chord sequences that might sound good for your song. So Songsmith incorporates the “happy” and “jazzy” sliders to let users – both musical novices and experienced songwriters – quickly explore all those different sequences. If you want to know how those work, you’ll have to go visit the MySong project page.
  3. Then we turn those chords into the music you hear using different musical styles (Songsmith comes with 30 styles, ranging from pop to rock to country, etc.). Songsmith tells the styles what chords it wants to play, and the styles help Songsmith turn those chords into a musical arrangement. These styles are provided through a partnership with PG Music, and Songsmith gives you a way to purchase additional styles from PG Music if you want an even broader range of styles. Then we turn that musical arrangement into the sound you hear using a musical synthesizer and a (very high-quality!) set of virtual instruments provided through a partnership with Garritan and Plogue. And yep, you guessed it, Songsmith also gives you a way to purchase additional styles and instruments from Garritan if you want an even broader range of sounds.

Are you working on other music-related projects at Microsoft Research?

Definitely! A list of some of our music-related projects is available here.

The best part of being at Microsoft Research is being able to work on what excites us. Songsmith happened because a couple of enthusiastic researchers at Microsoft Research happen to love making music. We’re still enthusiastic, we’re still at Microsoft Research, and we still love to make music, so keep an eye out for other music-related research projects in the future, and get in touch with us if you’re interested in research collaborations or in summer internships!



Microsoft Research