januari 13th, 2010

Adaptive music for the new decade

– The trick is to weave a seamless and musical soundtrack that follows you and enhances the experience no matter how you choose to experience it. Simple, huh?

Using music to enhance and create feelings is as old as music itself. Remove the soundtrack from any movie and you loose a lot of the emotional impact. This is also true for the web. But there’s a catch.

The Catch
When you create music for strictly linear media, for example a movie, you know exactly what happens and when it happens. That guy who slips on a sloppily discarded banana peel at 1.32 will always slip on it at exactly 1.32 no matter how many times you watch it. It’s easy to compose a soundtrack that builds up, peaks perfectly and gives your masterly directed slip that comical punch it needs.
    When you’re working with non-linear media, like the web, the user has the ability to interact with stuff and mess things up. Maybe the user jumps to a new section just before the 1.32 slip, or plays back the scenes in the ”wrong” order, or simply just refuses to look at any scenes involving a banana. You are not in control, the user is. And the music? Well, all those efforts to communicate emotions are quite inevitably chopped into pieces.

Traditional Use of Music on The Web
Music on the web today usually involves static loops that doesn’t really react to any user interaction. When someone has tried to make it responsive it’s often done with the tact of a woodsman swinging an axe rather than the loving touch of an artist and a musician. The result is music that stops and/or restarts abruptly when jumping between sections and that changes completely without respect for the beat and the tonality of the music.

In some computer/console games and web sites variation is created by crossfading between tracks with different intensity, e.g. when the enemy attacks in that spiffy game the soft track is fades down and the danger track fades up. This might have been edge and really cool on a multimedia CD in 1996, but 2010?!

Adaptive Music Is Sooo 2010!
So what do we need to do to get away from the Internet’s musical downfall? What we need is music that reponds to the user and changes in a musical way, that is transformed from one feeling to another seamlessly, that is changing in subtle ways to create an organic feeling. We need adaptive music.

Adaptive music is, simply put, music that adapts to something. When applying it to the web it means that when you interact with a web site, the music will adapt (e.g change) accordingly to what happens on the screen. It can be subtle changes that sets different moods for different pages/subpages or it can be bombastic symphonic fanfares that play when you score a goal in a soccer game – all made seamlessly and as a part of the soundtrack that is being put together in real time.

Real Life Examples
Here are some examples of productions we made last year. They all rely heavily on adaptive music. (Check out the launches section for more examples).

Adidas Teamgeist – A grand production that demanded some grand music. We wanted to create a Hollywood motion picture style soundtrack that never was interrupted, a soundtrack that just flows from one section to another as it would in a movie. Since the application is interactive the different games and sections would vary in length from one time to another, and all the videos could be skipped at any moment, so the musical transitions could happen at any time. Visit the site and try skipping videos, jump back and forth between sections and be as crude as you can – whatever you do the music makes smooth transitions between the different parts. Most people don’t even think about the transitions if you don’t point it out for them, it just feels natural. Just as a great soundtrack should :-)

SAAB Change Perspective – The song used here was originally written for the TV-commercial. The old school way would be to just add the two and a half minutes long track to the site and be happy with it. We didn’t go the old school way. We really wanted to make the most out of the music and make it adapt to what the user did. Visit the site and try navigating to different sections and back to the main menu, all sections have their own feeling and the music changes seamlessly between them, please be brutal, the music forgives all. Did you try to ”drive” the pen? Did you notice how the music changes from calmer to more intense car-driving-king-of-the-world music when you do? It really makes me want to buy a SAAB, no kidding :-)

E-On Action – The music is a solo nylon stringed guitar that follows your navigation, and it’s also affected by some realtime controls (like the amount of wind). The goal was to create the feeling of someone playing and following every action you take. The music is really naked and would easily be killed by rough editing, but with some love and care the music flows uninterrupted whatever the user does. Subtle, clean and alive.

So What Is The Secret?
Disassembly and reassembly. We chop up the music in small parts, just some seconds long, some containing a whole mix, others containing a single instrument. When the user navigates and interacts with the site these parts are reassembled in real-time depending on what is happening. Since we have the music in small parts we can put it together in any way and any combination we want.

To make it musical we set up rules how the music should behave. E.g. ”If the chorus is playing and you are on a specific subpage when you navigate back to the main page; play this transition, do a drumroll and change to the verse”, ”If the intro is skipped during the crescendo; do a quick build down and queue the strings”, ”If it is raining in Shanghai, take it to the bridge”. Plus a million rules for all other possible cases.

This is a quite abstract process, even for us, we are in a learning process, and everyday we reach new heights and new potentials.

The Source of Adaptation
The fundamental source is navigation – to let the music seamlessly follow the different sections of a website. This is used in almost all our projects where music is used. Letting the music adapt to different interactive elements is another source, it really enhances the user experience of the interaction (driving the pen in the SAAB site for example) . The music could be used for increasing usability, e.g. by changing to a ”waiting” tonality when the user is expected to input data. Another example is to make game states more understandable, e.g. the music changes depending on if it is my turn or not (as in the Teamgeist soccer game). Game score could also affect the music.

The Future Is Here, And This Is What It Holds
As you might have noticed by now there is no end to what we can do with adaptive music. The most obvious (and effective) is to affect the music by interacting with a web site (music changes according to navigation and other events like; completing a game, skipping a movie, etc), a more untouched field that we would like to keep exploring is to let the users mood, personality and current state of mind affect the music too.

How can we use data from things like Facebook to create a richer experience? What would happen if we tracked the users facial expressions and had that affect the music? How about tracking eye movements and change the music accordingly to what words you are currently reading?
And how can we use other realtime data like time, date, weather, number of users logged in, number of unread mails in you inbox to enhance the feeling of ”Wow, this is really happening right here and right now!”. And what real life events would be interesting to accentuate?

It is by walking down this path we can make the borders between inner and outer world blur.





This article is part of DinahMoe’s 2010 New Year’s resolution.
Click here to read about all of them!