New Audio API [Beta]: Elevate Sound and Voice in Your Experiences

Updates Announcements
#314

Hey! Yeah – GetConnectedWires will be available for public scripting in version 618

3 Likes
#315

Yo, epic! Thanks a lot for hearing me out, I appreciate it! :heart:

3 Likes
#316

How long is this going to stay disabled?

1 Like
#317

Could we also get an AudioPlayer.Stopped event? Sound has this and I use it to delete the sound when it’s stopped.

2 Likes
#318

My feedback on the new API

*I’m open to alternative solutions: I focus on issues here, not solutions.

So starting thoughts: It’s so powerful and it’s so damn awesome. What isn’t provided I can typically implement myself, and I have designed a powerful system around the new audio API in my game. I’ve spent the past 5 days or so working with the audio API and I would like to share my thoughts on it here, because there were some pain points.

Some beliefs I will be holding onto for this post. If you don’t want to read this preface, just skim over the bold parts.

  • Audio needs to be performant. This is because most interactions with audio happen very frequently. Either per frame, or every time a sound effect plays (very frequently), or every frame while a sound effect is playing. Go below “every frame” and you will hear issues (trust me, I tried). Performance is especially important when you’re targeting mobile users, or even desktop users. Because Bloxstrap is popular now, a very sizeable chunk of users expect your game to run above 60fps, sometimes even above 144fps. There have been multiple days in my games where I have seen analytics that show the average FPS to be higher than 60, which, for me, means that the demand for performance nowadays is higher than ever.
  • I don’t want to fight the audio API to perfect my sound design. By “fight” I mean implement features myself in hacky ways. For example, I disable rolloff while keeping directional audio by creating an attachment positioned exactly 5 studs away from the camera in the direction that the source part is in. This lets have directional audio with no rolloff. It’s obvious that having to do this to get the sound I want for my game isn’t great developer experience.

Issue 1: Disabling/enabling effects

Often times I will want to create “effect pipelines” where I might dynamically apply reverb or a muffling effect to someone’s voice. This is possible to do, but there’s a problem: I need to re-map my Wire instances every single time in order to enable/disable effects! This is pretty annoying to implement, and it really sucks for performance.

My proposed solution is to create an Enabled property on every single effect, and if the effect is not enabled, it will pass through the wires anyway.

Issue #2: Debugging wire connections

It is really hard to debug the new audio API.
I’ll just put it in a nice list:

  • No way to get connected wires from an effect. *Solved in version 618, props: New Audio API [Beta]: Elevate Sound and Voice in Your Experiences - #314 by ReallyLongArms
  • It appears that setting the connected instance on a wire doesn’t make that instance forget which wire is connected, sometimes leading to a “recursive” error? I am not sure. I don’t even know how to repro because I can’t see what’s happening
  • The above consistently makes instance caching with wires not possible. I will try to get a minimum repro, but it looks to be inconsistent and I am not even sure if it’s an issue on your end… it would be nice if I could get an answer from an engineer, as it is a big pain point.

Issue #3: Lack of rolloff options on AudioEmitters

There’s no way to customize anything related to rolloff on AudioEmitters. This is not good! My “wishlist”:

  • I want to be able to completely disable the default rolloff, effectively using AudioEmitters for directional audio.
  • The old API on Sounds would be nice, with options like RolloffStart, RolloffEnd, and RolloffMode. However please please please just have RolloffEnabled too! I understand that “it should be done for you” but if I want to disable rolloff I can already (and I do, at least in my proximity vc test), it’s just harder. This new audio API lets me “customize” rolloff as much as I want already.

Issue #4: AudioEmitters do not support volumetric audio

Self explanatory, I’m guessing this is being worked on already, but if it hasn’t been brought up already, it is an issue. Ideally we could toggle volumetric audio per emitter :eyes:

Issue #5: New effects / making existing effects more powerful

  • The ability to pan audio
  • Multiple sidechain pins on AudioCompressors. Probably a bad idea to just have like, 3, but my use case is having multiple ambient noises as the sidechain. Perhaps I’m misunderstanding how AudioCompressor is meant to be used…? I don’t know.
  • AudioFader.Volume is capped to 3. This isn’t enough, sometimes the sound assets provided to me are super quiet, and I need to yank up the volume by 5x. More on that in another section though

Issue #6: AudioPlayer cannot “stack” sounds

If I have a sound effect that lasts for 0.5 seconds, and I play it every .15 seconds, it will cut off the sound every time I use :Play(). Sure, that’s fine. That’s expected and good behavior.

The problem is if I want the old audio track to keep playing, I need to duplicate the AudioPlayer and I need to duplicate the wire(s). This is problematic if I have a lot of AudioEffects chained with the AudioPlayer, because I need to duplicate many wires, which is really bad for performance. Especially because I can’t use instance caching!

Proposed solution: Perhaps some :PlayNew() could be added? AudioPlayers have the potential to massively save on memory, this problem is the only thing holding that back. However, I have noticed memory savings for sound effects which only get played once. The granular control over if the asset is loaded is also really really nice, and I love that. Being able to tell a client to pre-load map-specific audio assets, dynamically loading/unloading audio assets based on the current needs of the game…

Issue #7: AudioPlayer has no .Volume

While you can use an AudioFader instance to yank up the volume, this is a headache to manage because it means double the wires. Especially when stacked (pun intended) with the previous issue, it means I have to clone and destroy a lot of instances to play my sound effects.

Issue #8: AudioAnalyzer is not “good” enough to use.

(keep in mind I am mainly talking using :GetSpectrum() which is disabled atm
So to preface, I am not saying it’s not powerful. I am not saying it’s not performant. It is very powerful, and the instance itself is fine. The problem is in order to do anything with this, you need to do massive number crunching on the Luau side, which kills your performance.

Proposed solution: This can be done on the C++ side! The Blazingly Fast™ side. I would like if the common use cases for AudioAnalyzer could be exposed as read-only properties: perhaps, CurrentVolume, LoudestFrequency.

Thanks for reading this. Hope these issues can be fixed!

edit: I am bad at remembering to proofread
edit: clarified emitters not supporting volumetric audio

3 Likes
#319

Hey @ffrostfall, great and detailed feedback – thank you!

I need to re-map my Wire instances every single time in order to enable/disable effects!

I can see how this is cumbersome; we’ll discuss

There’s no way to customize anything related to rolloff on AudioEmitters

This is being actively worked on!

  • Multiple sidechain pins on AudioCompressors

You should be able to wire multiple things to the same sidechain pin; it will mix them and use the total as the sidechain – does that work for your use-case?

AudioPlayer cannot “stack” sounds

We probably will not add this – in the past, we’ve implemented systems like Sound.PlayOnRemove and SoundService:PlayLocalSound that, internally, create a copy of the underlying sound, and play that to completion, without creating an instance.
These are a headache to maintain and debug: the playback state of the copy is nowhere to be found (since there is no instance reflecting it), which makes it difficult to control. We’ve also found that these APIs are not significantly more efficient than spawning a new local Sound and playing that; so it doesn’t seem worth the trouble.

I would like if the common use cases for AudioAnalyzer could be exposed as read-only properties: perhaps, CurrentVolume , LoudestFrequency .

I think you might be able to use PeakLevel or RmsLevel as a substitute for CurrentVolume, but we can discuss LoudestFrequency

2 Likes
#320

Hi, with version 618 released the GetConnectedWires() function works now. So that great, but it takes a parameter called “pin”. I was wondering if you could help explain that to me. Thanks a lot. :+1:

#321

I tried this myself. I think I may have to re-check my code now :sweat_smile:
When I tried to use AudioCompressors with multiple sidechain pins, it appeared like one audio track was overriding the others. I’ll double check

That’s unfortunate but understandable. Perhaps what could be done is allow the “grouping” of audio tracks? That way I can duplicate an AudioPlayer but not have to duplicate any other instance. This would still retain a lot of benefits like usability, and maybe even have other use cases, without creating the problem you described.

The documentation for these AudioAnalyzer properties specify “last audio buffer”. This indicates (to me at least) that it’s analyzing a set period of time, say .1 seconds of data before the current frame.

I’ll try it out? My underlying use case here is voice activity; but it felt like these properties had some level of “delay”. Maybe it was placebo, but if not, perhaps the buffer size could be adjustable?

1 Like
#322

@ReallyLongArms
Will we be seeing updates to the EQ eventually? One of my major gripes at the moment with the audio engine as a whole. An essential item in a sound designers kit. It would open up many opportunities for sound design in engine without having to call for 30+ voices to achieve a similar effect.

What are the dB/oct slopes right now for the EQ curves?
Will we ever get a parametric EQ for audio emitters or master fader channels?
Will we get more control over the 3-Band EQ? I notice the mid band has control over max and min, but why not add control for the low end and high end? Change slope types (High Shelf, Low Pass, High Pass, etc.)? Steepness (6dB/Oct, 12, 24, 32, maybe even 48?).

This would allow for MUCH better control and frankly, it would make it more fun to work with the engine. Currently, audio inside the engine is VERY custom orientated (and annoying sometimes) to manage a large amounts of variations/sound instances and voices.

Will there be audio debugging tools in the future? View voice count or CPU draw/memory usage from FMOD?

As a sound designer, this update was a massive improvement and I’m happy to see the updates as we go through the paces. I’d really like to see some more individual control in the future. Not restrictive due to simplicity, real control.

Cheers,
Panzerv1 (Max)

3 Likes
#323

This is all amazing but how does one use this system to have global sounds such as ambience? If i want a sound at a constant volume throughout the entire workspace do I just need to use the original sound instance?

#324

The pin argument selects which end you want to list connected wires from; e.x. "Input", "Output", or "Sidechain"

1 Like
#325

Hey @panzerv1,

I notice the mid band has control over max and min, but why not add control for the low end and high end?

the MidRange property controls crossover frequencies between the Low → Mid and Mid → High bands; since they’re shared crossovers, this indirectly controls the ranges of the low and high bands as well – does that provide enough control for your use case?

What are the dB/oct slopes right now for the EQ curves?

At the moment, the crossovers happen to be 24dB/Octave; we can discuss making this configurable, but we’d want to do so in a way that doesn’t necessarily crystalize the current choice of filters – in case we find something better or more efficient.

As a sound designer, this update was a massive improvement and I’m happy to see the updates as we go through the paces. I’d really like to see some more individual control in the future. Not restrictive due to simplicity, real control.

Glad to hear it! This API is not intended to be restrictive, we’re actively developing improvements and I really appreciate the feedback. I’m excited to see (hear?) what you build with this!

1 Like
#326

ngl, as an audiophile, this is probably the most exciting thing since true parallelization!

Real-time raytraced audio is actually possible now, and, … oh my god… it’s beautiful…

W

10 Likes
#327

If only this was an engine feature, games would feel much more immersive! Great job!

3 Likes
#328

You could use an AudioPlayer that is routed through a Wire to an AudioDeviceOutput. It would play the sound globally instead of being proximity-based.

Here’s the example codeblock that’s provided on the Roblox Creator Documentation page for the AudioPlayer:

local audioPlayer : AudioPlayer = Instance.new("AudioPlayer")
audioPlayer.Parent = workspace
audioPlayer.AssetId = "rbxassetid://9112854440"

local deviceOutput = Instance.new("AudioDeviceOutput")
deviceOutput.Parent = workspace

local wire = Instance.new("Wire")
wire.Parent = workspace

wire.SourceInstance = audioPlayer
wire.TargetInstance = deviceOutput

audioPlayer:Play() -- Fixed the typo here; the page has it as "player:Play()"
1 Like
#329

thank you for the help character limit

1 Like
#330

Really appreciate the response!
The one thing I’d like control in is the Band Shape and dB/Oct slope.
What band type are the High and Low configurations? Are they shelfs or high/low passes?

Knowing that the dB/Oct curves are 24 is super helpful, but having them that steep can introduce boosted frequencies at the edge of the curve. Having an option to select a super shallow 6dB or 12dB curve would be quite helpful in taming the low pass filters in my game. Having a locked 24dB curve was a good choice for general application but it’s starting to work against me when I’m designing.

BoostedFreq
BoostedFreq794×654 86.8 KB

Having control over the slope steepness will allow for smoother transitions and more possibilities in engine to design and direct audio.

Slopes
Slopes966×698 97.2 KB

Allowing the choice of different band types such as Shelfs or Low/High Pass filters can allow for more creative and specific EQ’ing that’s not too harsh/drastic in engine.

Finally, some sort of tool that allows me to see the voice counts of sound instances playing in game at one time either in the microprofiler or a seperate debug screen so I can manage and optimize my audio for the best listening experience.

Overall, love this update, can’t wait for improvements and more attention to the audio side of roblox as the platform grows.
AND PLEASE, sometime in the future. Let us put an audio instance in the world on its own, and be able to move JUST the audio instance like you would a part. You could select the shape you want and scale just like in other engines such as unity, or unreal.

Cheers,
Panzerv1

3 Likes
#331

I am not sure if anyone has addressed this issue, but it appears to me that muting and unmuting yourself causes a 2-3 second lag spike.

Reproduction:

#332

As TTV_Jabit
mentioned, I really wish raytraced audio like this is something that could be considered to be implemented into the roblox engine in a future update.

#333

I re-opened the Audio API Tutorial test place a few minutes ago and it has started working again :slight_smile:


@ReallyLongArms
Now that AudioAnalyzer:GetSpectrum() has been re-enabled, it would be nice to update the original post of this thread to clarify that it is functional again, as that would be useful to know for anyone like myself who still thought it was temporarily turned off.