Benchmarker Plugin - Compare function speeds with graphs, percentiles, and more!

Major Update!

V 5.0

End Condition Setting - Run for a set amount of time OR for a set amount of function calls.
(and other minor improvements)

This feature was requested by @zeuxcg and @pobammer quite a while ago. Sorry for the long delay!

image image

Up until now, the benchmark tests would call each function X amount of times. This allows you to gather precisely sized datasets. However, some users would rather run the benchmark for X seconds, and gather an arbitrary number of datapoints. This is generally more user-friendly since all tests will always take the set time, so slow functions won’t make your tests take longer. Slow function tests will just have fewer datapoints in order to stay within your time constraint.
Therefore, I’ve added a setting to allow you to pick which test behavior you prefer!

One behavioral note for Run Time: The test might take a few milliseconds above your actual set time, because I make sure each function is run the same amount of times (so the test might go overtime to call the next functions) in order to ensure a balanced and accurate result.

5 Likes

Update!

V 5.1

Improved protection behavior - Mark test modules with “.bench” suffix in their name, rather than a CollectionService tag.
This makes it much easier to use this plugin with Rojo and GitHub, while still protecting you from accidentally running an unintended module and causing issues.
Requested by @Kampfkarren

4 Likes

Major Update!

V 6.0


  • New UI

With icons from the incredible @Elttob, V 6.0 brings a much cleaner and more compact menu interface, replacing the ugly long text buttons.

image


  • Library

As you may have noticed, there’s a new section in the menu. The Library serves two purposes.
Firstly, it provides you with commonly used tests so you don’t have to write them yourself. Secondly, these files act as examples that demonstrate how to use Benchmarker and how to write your .bench files.

If you would like to contribute to the Library, open a pull request here.

image


  • Profiler Improvements

Using the Profiler for a more detailed breakdown is a very useful tool. However, if using the Profiler alters your test results then it wouldn’t be much good. I spent some time optimizing it down to the microsecond level.

As you can see in the image below, running the same function with 5 labels (2 nested) actually ran 0.3 microseconds faster. That’s pretty impressive, if I do say so myself. It means the the Profiler had no significant impact at all!

In addition to this performance squeeze, it now displays the “dark matter” of a function under a label called [UNTRACKED]. If you put a portion of your function into a Profiler.Begin - Profiler.End wrap, but leave the rest of the funciton with no profiling, the rest of the function will be put into the [UNTRACKED] label.
If your entire function has profiling, then the [UNTRACKED] label represents the overhead of the Profiler itself (which should usually be well below 2 microseconds).

image


  • Minor improvements

I made various little tweaks throughout, such as it displaying what .bench file is currently running. Nothing worth explicitly noting but just general improvements to UX and speed.

8 Likes

Running into a bit of a problem with running a test. The plugin is just stuck at “Running Tests.bench” without any sort of errors returned. Chances are I am doing something wrong, just not sure where I am going wrong here.

Here’s my Tests.bench:

--[[

|WARNING| THESE TESTS RUN IN YOUR REAL ENVIRONMENT. |WARNING|

If your tests alter a DataStore, it will actually alter your DataStore.

This is useful in allowing your tests to move Parts around in the workspace or something,
but with great power comes great responsibility. Don't mess up your stuff!

---------------------------------------------------------------------

Documentation and Change Log:
https://devforum.roblox.com/t/benchmarker-plugin-compare-function-speeds-with-graphs-percentiles-and-more/829912/1

--------------------------------------------------------------------]]

return {

	ParameterGenerator = function()
		-- This function is called before running your function (outside the timer)
		-- and the return(s) are passed into your function arguments (after the Profiler). This sample
		-- will pass the function a random number, but you can make it pass
		-- arrays, Vector3s, or anything else you want to test your function on.
		return require(game.ServerStorage.Await)
	end;

	Functions = {
		["Await"] = function(Profiler, Await) -- You can change "Sample A" to a descriptive name for your function
		
			-- The first argument passed is always our Profiler tool, so you can put
			-- Profiler.Begin("UNIQUE_LABEL_NAME") ... Profiler.End() around portions of your code
			-- to break your function into labels that are viewable under the results
			-- histogram graph to see what parts of your function take the most time.
		
		
			-- Your code here
			Await(0.05)
		end;

		["Sample B"] = function(Profiler, RandomNumber)
			wait(0.05)
		end;

		-- You can add as many functions as you like!
	};

}

Here’s ServerStorage.Await:

local RunService = game:GetService("RunService")


local BinaryHeap = {}


function BinaryHeap.insert(value, data)
	local insertPos = #BinaryHeap + 1
	BinaryHeap[insertPos] = {
		value = value,
		data = data
	}
	while insertPos > 1 and BinaryHeap[insertPos].value < BinaryHeap[math.floor(insertPos / 2)].value do
		BinaryHeap[insertPos], BinaryHeap[math.floor(insertPos / 2)] = BinaryHeap[math.floor(insertPos / 2)], BinaryHeap[insertPos]
		insertPos = math.floor(insertPos / 2)
	end
end


function BinaryHeap.extract()
	local insertPos = 1
	if #BinaryHeap < 2 then
		BinaryHeap[1] = nil
		return
	end
	BinaryHeap[1] = table.remove(BinaryHeap)

	while insertPos < #BinaryHeap do
		local smallerChild = 2*insertPos + (BinaryHeap[2*insertPos].value < BinaryHeap[2*insertPos + 1].value and 0 or 1)
		if BinaryHeap[insertPos].value > smallerChild.value then
			BinaryHeap[smallerChild], BinaryHeap[insertPos] = BinaryHeap[insertPos], BinaryHeap[smallerChild]
		end
		insertPos = smallerChild
	end
end


local CPUTime = os.clock()


RunService.Stepped:Connect(function()
	CPUTime = os.clock()
	local PrioritizedThread = BinaryHeap[1]

	while PrioritizedThread do
		PrioritizedThread = PrioritizedThread.data
		local YieldTime = CPUTime - PrioritizedThread[2]
		if PrioritizedThread[3] - YieldTime <= 0 then
			BinaryHeap.extract()
			coroutine.resume(PrioritizedThread[1], YieldTime)
			PrioritizedThread = BinaryHeap[1]
		else
			PrioritizedThread = nil
		end
	end
end)


return function(Time)
	BinaryHeap.insert(Time or 0, {coroutine.running(), os.clock(), Time or 0})
	return coroutine.yield(), CPUTime
end

Might be worth pointing out that I have the settings set to have the plugin run for two seconds instead of running it a certain amount of times as well. Again, I am almost certain I am doing something wrong here and it isn’t the plugins fault. Just not sure where I am going wrong here.

image image

Roblox decided to moderate the add button. :roll_eyes:
Uploaded again, hopefully they let this one through. I’ll be updating the plugin with the new asset id if it works.
Sorry for the inconvenience.

You don’t need to require your module every single time the functions are called, just have it defined at the top of your bench script and use it in whatever function you want.


The problem is that your Await function doesn’t work in Edit mode, and is therefore yielding indefinitely. You made it rely on Stepped, which doesn’t run during edit since there’s no physics simulation being stepped! I switched from RunService.Stepped to RunService.Heartbeat. This fixed the issue completely.

I also changed
if PrioritizedThread[3] - YieldTime <= 0 then
to
if YieldTime >= PrioritizedThread[3] then.
Checking if subtracting something is less than zero is a really strange and roundabout way to check if something is larger. Not sure why you did that. This wasn’t the issue but it was absurd so I changed it.

Updated with fixed icon image (I think) and some changes to Library files.

1 Like

In the future, you could appeal in Mod Review Requests instead of making a new asset!

Edit: Oops, replied to the wrong post.

Update!

V 6.1


  • Edge Case Fixes

Thank you to @AstroCode for bringing these issues to my attention.

If you ran a really slow function for a Run Time that would only have enough time to call it a couple times, it would cause errors and funky behavior. It would attempt to process data that simply wasn’t captured, and the graph would be trying to draw data increments that lead to weird valleys.
Regardless of errors/misbehavior, having so few data points is useless because it’s not enough info to have accurate and reliable benchmark results.

To solve this, I made it error if you have too few data points, and it estimates how much time you’d need to run the bench for in order to get enough data points. I also made the graph drawing handle small datasets more intelligently.

image

3 Likes

I have no clue if this is a problem on my end on studio, but whenever I enable the benchmarker plugin, my mouse seems to invert rather than being at a normal state, usually after I enable / disable it.

Repo:

  • Open a new place or file.
  • Open the Benchmarker plugin (click it once or twice if you have it at start-up).
  • ig check the mouse:

Restarting the application or studio didn’t do the trick.

Roblox has been having issues with plugin widgets and the cursor lately. They’ve been reported already, so we just have to sit tight.

1 Like

Update!

V 6.2


  • Fool Proofing

With a complicated tool like this, it’s easy to do things wrong if you don’t understand it. I’ve noticed people misusing the plugin lately, so this update is an attempt at catching common mistakes before they happen. I’ve added protections against common mistakes like benching yielding functions or running sub-microsecond tests.
If you’ve made a mistake that the plugin could have caught for you, DM me what it was and I’ll try to incorporate some checks against that.

image
image

Funny 'feature'

If you run a blank function, it runs so fast that it’ll suggest you run it inf times
image image

3 Likes

Lol sorry im dumb. It was accident. Good work though…

I’ve recently gathered quite the collection of benchmark files and with that, it’s become inconvenient to manually display each bench.

It would be cool to have a feature that gathers and executes all of your selected bench files. With that, there should be a way to easily select the result of a specific bench. Adopting a design similar to nexus unit testing, would be beautiful and offer a ton of possibility. On each tab in the list, it could display the file path, then it drops down to each benchmark and a gist of the results. Clicking on one of the tabs (or specific benchmark) would open the graph and show all that fancy stuff.

2 Likes

Every time I insert a benchmark with the “Create Bench” button, I remove all of the comments and extra code because it is overwhelming and distracting. Would you consider adding a barebones .bench to the Library? Something like

return {
	ParameterGenerator = function()
		return
	end,

	Functions = {
		["Sample A"] = function(Profiler, Parameter)

		end,

		["Sample B"] = function(Profiler, Parameter)

		end
	}
}
2 Likes

Sort of a bump, but ditto to this nevertheless. I end up just copying over a barebone modulescript across places.

1 Like

Hey there,

I have done what the plugin asks, i have added a loop onto the function and the error won’t go away, any tips?

Here is the code:

--[[

|WARNING| THESE TESTS RUN IN YOUR REAL ENVIRONMENT. |WARNING|

If your tests alter a DataStore, it will actually alter your DataStore.

This is useful in allowing your tests to move Parts around in the workspace or something,
but with great power comes great responsibility. Don't mess up your stuff!

---------------------------------------------------------------------

Documentation and Change Log:
https://devforum.roblox.com/t/benchmarker-plugin-compare-function-speeds-with-graphs-percentiles-and-more/829912/1

--------------------------------------------------------------------]]

return {

	ParameterGenerator = function()
		-- This function is called before running your function (outside the timer)
		-- and the return(s) are passed into your function arguments (after the Profiler). This sample
		-- will pass the function a random number, but you can make it pass
		-- arrays, Vector3s, or anything else you want to test your function on.
		return math.random(1000)/10
	end;

	Functions = {
		["table create()"] = function(Profiler, RandomNumber) -- You can change "Sample A" to a descriptive name for your function
			for _ = 1,1000 do
				wait(1)
				local t = table.create(4000)
				for i = 1,4000 do
					wait()
					t[i] = RandomNumber
				end
			end

		end;

		["Sample B"] = function(Profiler, RandomNumber)
			for _ = 1,1000 do
				wait(1)
				local t = {}
				for i = 1,4000 do
					wait()
					t[i] = RandomNumber
				end
			end
		end;

		-- You can add as many functions as you like!
	};

}

You cannot put a wait in a speed test, that ruins the entire purpose.

3 Likes

Update: Quality of Life Changes

  • Improved the coloring system to avoid conflicting colors and more evenly spread across the color space. This adapts to both light and dark theme.
    Before (Multiple purple hues) | After (Four distinct colors)


    (Thank you to @MrGreystone for the new color algorithm)

  • Added the ability to drag resize the profiler/gap bounds. It was previously annoying to read many stacked profiler labels since there wasn’t enough space, now you can make it larger while you read.

9 Likes

Is there any way that you could implement assert()? I was trying to benchmark assert() vs

local function MaybeQuickAssert(Condition, ErrorMessage)
   if not Condition then
   	print(ErrorMessage) -- Plugin thinks warn() is the code erroring so we use
   	return false -- print() and return which together should have a similar footprint
   end
end

and when passed a condition that caused it to alert the user then it gives an error and doesn’t benchmark. I’m guessing probably not without a custom function that would make the benchmark redundant since it doesn’t use the same code and therefore would take a different amount of time. Although, a custom function with an attached time that replaces the time it takes for the function to execute with the time it takes assert() to execute could work although it’s likely more trouble than it’s worth since the time it takes for a function to run depends on the specs of the user. Anyway, I was bored so I made this as a kind of proof of concept (minus the tricky bit lol).

-- Inside a ModuleScript
local TimerModule = {}

function TimerModule.new()
	local Timer = {
		__PauseStartTime = 0,
		StartTime = 0,
		PauseLength = 0,
		EndTime = 0,
		RunTime = 0
	}

	function Timer:Start()
		self.StartTime = os.clock() 
	end

	function Timer:Pause()
		self.__PauseStartTime = os.clock()
	end

	function Timer:Play()
		self.PauseLength += os.clock() - self.__PauseStartTime
		self.__PauseStartTime = 0
	end

	function Timer:AddTime(Time)
		self.RunTime += Time
	end

	function Timer:Stop()
		self.EndTime = os.clock()
		local RunTime = self.RunTime + (self.EndTime + self.PauseLength) - self.StartTime
		self.RunTime = RunTime
		return RunTime
	end

	function Timer:Destroy()
		self.__PauseStartTime = nil
		self.StartTime = nil
		self.PauseLength = nil
		self.EndTime = nil
		self.RunTime = nil
	end

	return Timer
end
return TimerModule


-- Inside a Script
-- Setup and other plugin stuff here
local Timer = TimerModule.new() 

-- Run button pressed here

local Code -- However you get the users code

if string.find(Code, " assert(") then  -- Space means it's not part of another function name and only one bracket since there would be arguments in between
	Timer:AddTime(1) -- Just a random number here for now, but it could possibly be based off the users framerate?
	Code = string.gsub(Code, " assert(", " FakeAssert(") -- And FakeAssert defined in the preset script (or maybe added by this bit of code?)
	Code = [[
local function FakeAssert(Condition, ErrorMessage)
	if not Condition then
		warn(ErrorMessage)
	end
end \n
]]..Code -- It's a bit of a mess but it *should* add the FakeAssert function to the top (also this bit and the gsub bit could be put in one for a performance enhancement)
end

Timer:Start()
-- Code is executed here