Performant Auto-Exposure (Screen-Space)

First, I’d like to disclose that this is a fork of @atfdaj 's auto exposure, refined, fixed and optimized. The logic is entirely designed by them, using Editable Images and CaptureService.

api

EditableImage | Documentation - Roblox Creator Hub
CaptureService | Documentation - Roblox Creator Hub

How Does This Resource Work?

As stated previously, @atfdaj designed most of the inner workings of the script, and I refined, optimized, and fixed the code.

For every Heartbeat, a screenshot is taken via CaptureService and converted into an EditableImage. In a parallel thread, this image is processed and the HSV Value of every ‘pixel’ is analyzed and averaged into a value. From hereon, the appropriate ExposureCompensation is calculated. The new Exposure is then applied in a RenderStepped (intentionally separate from the calculation; this part has less overhead, and the rest of the script could be modified to work at a more comfortable frequency) via a clamped lerp, which works according to the adjustTime variable.

CODE/model

Please make sure that the localscript is parented to an actor.

SCRIPT
-- // Developed by @atfdaj, @AborayStudios, and @A_Mp5
--[[
https://devforum.roblox.com/t/free-performant-auto-exposure-adaptation-of-atfdajs/3000200
]]
print(script.Parent:IsA("Actor") and "" or "⚠️ Make sure Auto Exposure is parented to an actor!")

local captureService = game:FindFirstChildOfClass("CaptureService")
local assetService = game:FindFirstChildOfClass("AssetService")
local lighting = game:FindFirstChildOfClass("Lighting")
local runService = game:FindFirstChildOfClass("RunService")

local screen

local adjustTime = 3 -- Amount of time before camera is fully adjusted
local calibrationConstant = 6.5 -- A higher value makes exposure darker
local resolution = 50 -- Numbers too far from 50 may cause instability/errors;
local targetExposureValue = 10

local updatedExposure = 0

local function log2(x)
	return math.log(x) / math.log(2)
end

local function lerp(start, goal, alpha)
	alpha = math.clamp(alpha,0,1)
	return start + (goal - start) * alpha
end

local function calculateNewExposure(brightness)
	local luminance = brightness * 100
	local ev = log2(luminance) + calibrationConstant
	local compensation = targetExposureValue - ev
	return compensation
end

local function getAverageBrightness()
	local brightnessValues = {}
	local totalBrightness = 0
	for x = 1, screen.Size.X, resolution do
		for y = 1, screen.Size.Y, resolution do
			local pixel = screen:ReadPixels(Vector2.new(x, y), Vector2.one)
			if pixel then
				local color = Color3.new(pixel[1], pixel[2], pixel[3])
				local _, _, value = color:ToHSV()
				totalBrightness += value
				table.insert(brightnessValues, value)
			end
		end
	end
	return totalBrightness / #brightnessValues
end

local function updateScreen(deltaTime)
	captureService:CaptureScreenshot(function(capture)
		screen = assetService:CreateEditableImageAsync(capture)
		task.desynchronize()
		local averageBrightness = getAverageBrightness()
		updatedExposure = calculateNewExposure(averageBrightness)
		task.synchronize()
		screen:Destroy()
	end)
end

local function updateExposure(deltaTime)
	lighting.ExposureCompensation = lerp(lighting.ExposureCompensation,updatedExposure, adjustTime * deltaTime)
end
runService.Heartbeat:Connect(updateScreen)
runService.RenderStepped:Connect(updateExposure)

Demo place (COPYLOCKED, requires FFlagEditableImageEnabled = True)

Examples & Video

Uncompensated vs Compensated (Brightness: 20)


Known Bugs

  • Adjustment may appear to spike when closer to interpolation goal rather than linearly transition throughout the entire time
  • The usage of CaptureService unintentionally makes the exposure adjust according to any UI elements, including the pause menu. (Possibly an upside if you take a look at flashbang effects, for instance.)
  • Editable image API fails on weird aspect ratios.
  • Possible memory leaks from the nature of the services used and their recency.
  • As of 2024/06/01, the script may inexplicably stop working after a few playtests in Studio. This is a Roblox issue.
  • As of 2024/06/01, the script does not work in production-game and studio without the Beta or FFlag enabled.
    • This doesn’t work, how can I fix it?

  • In-Studio, make sure the EditableImage beta is enabled. & restart
  • Ingame, using a custom bootstrapper like BloxStrap or manually editing the FFlag json, set FFlagEditableImageEnabled to True. (This goes without saying: only players with the fflag enabled playing your game can see the effect, otherwise console is filled with warnings. Bound to change when it’s out of beta.)
  • Encountering one of the previously mentioned bugs (2024/06/01) in studio, reopen the place file.

Performance???

FPS remains relatively unaffected on the spectrum of devices. The script is easily modifiable to be infrequent instead of framely.

Extremely low-end device:

~2 - 9 frame reduction
(1 - 5ms on microprofiler)

Mid-High end device:

~0 - 3 frame reduction
(< 0.1ms on microprofiler)

Conclusion

Will you use this resource?

  • :white_check_mark:Yes
  • :x:No (respond why)
  • :watch:In the future
  • :thought_balloon: Maybe

0 voters

18 Likes

Hey, I appreciate the credit! I’ll try this out in the morning when I wake up. I’m pretty excited to see how much it’s been improved here.

2 Likes

@Homemade_Meal @Maximum_ADHD @Elttob
You’re all innovators in the Roblox graphics field, I’d like to hear some opinions :slightly_smiling_face:

  • there is no way to free images generated by CaptureScreenshot
  • this will stop working after some time, CaptureService has a limit
    (in this case, this module might interfere with games that uses CaptureService as well)
1 Like

Where are you getting this from…?
Garbage is collected:
image
(see the drop when it reaches 10k?)

and capture service works perfectly fine capturing 1000 images per second
From what I’ve tested, duplicate captures are taken care of appropriately

When I went about solving this problem several years ago, I went with a very conservative approach that simply evaluated a line of sight between the camera and the sun, taking into consideration the depth of the raycast. It was decent enough for my needs.

Trying to analyze the entire frame buffer on the CPU in real-time isn’t super practical.

5 Likes

Yes, my initial thoughts too, however, that’s the best that can be done on Luau. Except this only samples a handful of pixels, evenly spaced out (at a variable ‘resolution’), making the capture API actually have the largest overhead instead of the rest of the logic.

Is this a shader mod you use for the sunrays?

1 Like

Nope, just vanilla sunrays with more strength. Boosted exposure compensation makes them glow more.

1 Like

try loading a previous temporary asset and you’ll see what i mean, it isn’t freed

Did you make them yourself? or did you download it cuz i kinda need it for trying to make the most realistic picture i could make in roblox

Given his description of how it works, it’s clear he made it himself:

If you’re trying to capture a picture, there’s no need to have a script automating the exposure. You can change exposure yourself in Lighting.ExposureCompensation, this approach would give you more artistic freedom!


If you’re really looking for the script, MaximumADHD has open-sourced it on Pastebin:

image

edit: production version may have a mem leak, but that’s entirely on roblox to fix

The claims made by @pixe_ated are indeed true. CaptureService stopped providing me with screenshots after about 15 minutes of messing around with the system. After 10 minutes of being in game, my memory usage jumped up by ~2GB in an empty baseplate with no other scripts running.

1 Like

I do believe it’s rather backwards to label this as performant when there has been acknowledgement of possible, or current memory leaks. I believe that adds up a lot more than the raw performance of the actual module itself.

I’d also like to add that Roblox uses the CaptureService internally for the Roblox Mobile Screenshot button mechanic, which could likely also add to the total limit of screenshots you’re able to capture

(Especially considering previous replies have stated there are obvious limits to this which will result in it’s failure to continue working).

It’s likely better to continue with approaches much like MaximumADHDs, where you raycast and determine distance than factor the overall exposure of the scene off that. It may not be super accurate but it definitely gets the job done, and doesn’t seem like it looks all too bad either. Nor do you have to worry about very black box things such as the CaptureService itself.

I could be wrong on a few things here though, but I do feel going with a more tame approach would make this more useful, and probably easier on some devices given the possible memory leaks.

Update on memory leak, for those who were concerned: (or rather, findings)

@pixe_ated @TheBrainy06 @Vyntrick
I will be updating this resource ASAP if/when proper capture memory control is released.