More API for analyzing music playback

Feature Requests Engine Features
#1

This summer I want to work on making a game that revolves around music and colorful light shows. Ideally I’d like to automate the colors and intensity of the effects dynamically based on the tone and intensity of the music that is playing, but there’s a lot of information missing from the API.

PlaybackLoudness is great, but it doesn’t provide as much information about the mood and energy of the music. I think I could create better automated effects if I had:

  • Overall data should be split into stereo channels, rather than being a mono channel.
  • Effects from the left channel will influence the left half of the stage, and vice versa right.
  • Information about how much the bass is influencing the “loudness” of a channel, versus treble.
  • In my use case, bass would have a stronger influence on the intensity of effects.
  • Information about what key the bass and treble is in (in hz?)
  • The color palette I use for effects would act on the color associated with the mood of the music.
  • Information about the BPM
  • Would help me figure out the pacing of the effects.

cc @spotco

8 Likes
#2

Oh yes please BPM info. As well a way to get the loudest point in the audio file without having to play the sound and record the highest number.

5 Likes
#3

I’ve got a general idea of what your use case for these features would be, but from what I understand it seems like some of these things you would normally want to do offline.

The quick rundown:

  • Some of these features are not free, so they’d take a pretty significant amount of work.
  • BPM isn’t applicable for most sound effects, so at best this would be a rough estimation at any given point in time.
  • The same applies for Key and pitch.

Long story short, these seem like pretty complex features for a pretty specific small use case so realistically not happening soon.

2 Likes