There is a lot of hoopla out there with AI generated music and virtual singers. Most folks think these will be used in studios to produce soundtracks, and they are right. For now.

Dynamic Orchestration

Right now, every playthough of a specific game has the same music. Same opening theme, same battle music, same ‘pause screen’ music. If 500,000 people are playing Baulder’s Gate – each of them are getting the same soundtrack.

With Dynamic Orchestration, each playthrough will have a unique soundtrack. In an RPG, you can use music for setting and mood. There should be a music difference between “campsite music for a party of 4”, and “campsite music for a party of 4 who just picked up a vampire helper… and the night may not end well.”

Human composers are finite. We can’t cover all possibilities. A machine can. In real time.

History

We’ve had hints of dynamic orchestration for a while. Changing tempo to match the situation is common (like when everything speeds up for the last lap on Mario Kart). You can also mute or mix different layers of a piece of music to go from calm to exciting (like in an open world RPG where if you wander to where there is nothing, the music starts becoming thinner, and if you wander closer to interesting things, the music picks up). These are fairly difficult to implement now. They require working in MIDI-like sections, where instead of producing one track, you must produce thousands of individual notes and the instructions on how to put those notes together.

Why isn’t this happening now?

As of August, 2023, there are still a few hurdles.

The audio quality is pretty low right now. Eventually, we will get 48khz resolution on the audio, but now we have to settle for a fraction of that. Every company that does AI audio is working toward this, it’s a matter of time.

The cost of running these models is comparatively quite high. To do dynamic composition, you would need to stream a lot of audio data from servers OR find efficiencies in the models to allow them to run on smaller, home-scale computing devices.

What this means for Composers

There are still some opportunities for composers, though the workload per-game will be decreased significantly. A composer may still come up with the overall music design and provide some examples of themes for different parts of the story. We won’t be doing 12-hour soundtracks anymore.

Won’t AI also be able to do the main themes?

Yes. Yes, it will.

Ethical Considerations

Many people think the AI will be reigned in by governmental mandates and lawsuits from artists. These actions will slow adoption, but they can’t stop it. There is already enough music in the Public Domain to train some frighteningly good audio AIs. We have access to tons of folk music, music by dead composers like Mozart and Tchaikovski, and music be people who WANT to share what they’ve done like me.

Supercharged Storytelling

As Candy Crush weaponized engagement with their incredible design, Dynamic Orchestration will be a major aspect of weaponizing emotional storytelling… if you want it to. I hope every experience has a settings slider to control the poignancy of the soundtrack. Sometimes… I just don’t want that much.

Timeline

<Speculation> There are still a few challenges to get this to market. I expect someone will have a streaming service capable of doing this for under $10 per hour in under a year. Maybe another year to get an optimized model that can run on a graphics card or TPU. So… about 2027 you’ll be seeing the games show up on Steam. </Speculation>

Going Beyond

Dynamic Orchestration has a lot of other capabilities that can be realized before real-time inclusion in video games:

• Bar and Restaurant background music: You can get an endless playlist of jazz standards that perfectly read the room and adjust dynamically to the goings-on there.
• TV/Film Scoring: A composer will be able to guide the Dynamic Orchestration into an excellent soundtrack pretty quickly. Record it once, and incorporate it into the film!
• Personal Soundtrack: While going on a walk, working out, or just cleaning your house, you’ll be able to have a dedicated soundtrack perfectly suiting to inspiring you! (It may also include some musical warnings if your family is coming over soon).