Thank you to @AstroCode for bringing these issues to my attention.
If you ran a really slow function for a Run Time that would only have enough time to call it a couple times, it would cause errors and funky behavior. It would attempt to process data that simply wasn’t captured, and the graph would be trying to draw data increments that lead to weird valleys.
Regardless of errors/misbehavior, having so few data points is useless because it’s not enough info to have accurate and reliable benchmark results.
To solve this, I made it error if you have too few data points, and it estimates how much time you’d need to run the bench for in order to get enough data points. I also made the graph drawing handle small datasets more intelligently.
I have no clue if this is a problem on my end on studio, but whenever I enable the benchmarker plugin, my mouse seems to invert rather than being at a normal state, usually after I enable / disable it.
Repo:
Open a new place or file.
Open the Benchmarker plugin (click it once or twice if you have it at start-up).
ig check the mouse:
Restarting the application or studio didn’t do the trick.
With a complicated tool like this, it’s easy to do things wrong if you don’t understand it. I’ve noticed people misusing the plugin lately, so this update is an attempt at catching common mistakes before they happen. I’ve added protections against common mistakes like benching yielding functions or running sub-microsecond tests.
If you’ve made a mistake that the plugin could have caught for you, DM me what it was and I’ll try to incorporate some checks against that.
Funny 'feature'
If you run a blank function, it runs so fast that it’ll suggest you run it inf times
I’ve recently gathered quite the collection of benchmark files and with that, it’s become inconvenient to manually display each bench.
It would be cool to have a feature that gathers and executes all of your selected bench files. With that, there should be a way to easily select the result of a specific bench. Adopting a design similar to nexus unit testing, would be beautiful and offer a ton of possibility. On each tab in the list, it could display the file path, then it drops down to each benchmark and a gist of the results. Clicking on one of the tabs (or specific benchmark) would open the graph and show all that fancy stuff.
Every time I insert a benchmark with the “Create Bench” button, I remove all of the comments and extra code because it is overwhelming and distracting. Would you consider adding a barebones .bench to the Library? Something like
I have done what the plugin asks, i have added a loop onto the function and the error won’t go away, any tips?
Here is the code:
--[[
|WARNING| THESE TESTS RUN IN YOUR REAL ENVIRONMENT. |WARNING|
If your tests alter a DataStore, it will actually alter your DataStore.
This is useful in allowing your tests to move Parts around in the workspace or something,
but with great power comes great responsibility. Don't mess up your stuff!
---------------------------------------------------------------------
Documentation and Change Log:
https://devforum.roblox.com/t/benchmarker-plugin-compare-function-speeds-with-graphs-percentiles-and-more/829912/1
--------------------------------------------------------------------]]
return {
ParameterGenerator = function()
-- This function is called before running your function (outside the timer)
-- and the return(s) are passed into your function arguments (after the Profiler). This sample
-- will pass the function a random number, but you can make it pass
-- arrays, Vector3s, or anything else you want to test your function on.
return math.random(1000)/10
end;
Functions = {
["table create()"] = function(Profiler, RandomNumber) -- You can change "Sample A" to a descriptive name for your function
for _ = 1,1000 do
wait(1)
local t = table.create(4000)
for i = 1,4000 do
wait()
t[i] = RandomNumber
end
end
end;
["Sample B"] = function(Profiler, RandomNumber)
for _ = 1,1000 do
wait(1)
local t = {}
for i = 1,4000 do
wait()
t[i] = RandomNumber
end
end
end;
-- You can add as many functions as you like!
};
}
Improved the coloring system to avoid conflicting colors and more evenly spread across the color space. This adapts to both light and dark theme.
Before (Multiple purple hues) | After (Four distinct colors)
(Thank you to @MrGreystone for the new color algorithm)
Added the ability to drag resize the profiler/gap bounds. It was previously annoying to read many stacked profiler labels since there wasn’t enough space, now you can make it larger while you read.
Is there any way that you could implement assert()? I was trying to benchmark assert() vs
local function MaybeQuickAssert(Condition, ErrorMessage)
if not Condition then
print(ErrorMessage) -- Plugin thinks warn() is the code erroring so we use
return false -- print() and return which together should have a similar footprint
end
end
and when passed a condition that caused it to alert the user then it gives an error and doesn’t benchmark. I’m guessing probably not without a custom function that would make the benchmark redundant since it doesn’t use the same code and therefore would take a different amount of time. Although, a custom function with an attached time that replaces the time it takes for the function to execute with the time it takes assert() to execute could work although it’s likely more trouble than it’s worth since the time it takes for a function to run depends on the specs of the user. Anyway, I was bored so I made this as a kind of proof of concept (minus the tricky bit lol).
-- Inside a ModuleScript
local TimerModule = {}
function TimerModule.new()
local Timer = {
__PauseStartTime = 0,
StartTime = 0,
PauseLength = 0,
EndTime = 0,
RunTime = 0
}
function Timer:Start()
self.StartTime = os.clock()
end
function Timer:Pause()
self.__PauseStartTime = os.clock()
end
function Timer:Play()
self.PauseLength += os.clock() - self.__PauseStartTime
self.__PauseStartTime = 0
end
function Timer:AddTime(Time)
self.RunTime += Time
end
function Timer:Stop()
self.EndTime = os.clock()
local RunTime = self.RunTime + (self.EndTime + self.PauseLength) - self.StartTime
self.RunTime = RunTime
return RunTime
end
function Timer:Destroy()
self.__PauseStartTime = nil
self.StartTime = nil
self.PauseLength = nil
self.EndTime = nil
self.RunTime = nil
end
return Timer
end
return TimerModule
-- Inside a Script
-- Setup and other plugin stuff here
local Timer = TimerModule.new()
-- Run button pressed here
local Code -- However you get the users code
if string.find(Code, " assert(") then -- Space means it's not part of another function name and only one bracket since there would be arguments in between
Timer:AddTime(1) -- Just a random number here for now, but it could possibly be based off the users framerate?
Code = string.gsub(Code, " assert(", " FakeAssert(") -- And FakeAssert defined in the preset script (or maybe added by this bit of code?)
Code = [[
local function FakeAssert(Condition, ErrorMessage)
if not Condition then
warn(ErrorMessage)
end
end \n
]]..Code -- It's a bit of a mess but it *should* add the FakeAssert function to the top (also this bit and the gsub bit could be put in one for a performance enhancement)
end
Timer:Start()
-- Code is executed here
I just am curious as to what you specifically mean by this. I am not very skilled when it comes to statistics as of now, but I am pretty sure if you took the weighted average of a data set that had a good bit of size to it the results would not be severely skewed from a couple of outliers. Clarification on this would be great.
Let’s say you have 10 data points of function timings. 9 of them are 100 milliseconds exactly, but 1 happened to be skewed by a GC step or something and ended up at 800ms.
Average = ((100*9)+(800))/10 = 170ms
Fiftieth percentile = 100ms
The average is 70% increased over the real world expected performance thanks to a single outlier, while the fiftieth percentile is a much more meaningful and accurate value.
Regardless, I provide both values in this plugin.
With the plugin’s histogram, you get to view how your code really behaves, rather than just a single metric. This plugin gives you as much information as possible so you can make informed decisions.
Any chance to use 2x versions instead so that they aren’t blurry on HiDPI screens? Roblox Studio is getting HiDPI support soon (FFlagStudioWindowsDpiScale), which makes it SO much easier to use it on 4K screens.
This… doesn’t… look… right? The points & lines are extremely thick, is that intentional? Plus the text is obscured?
It looks in your screenshots like you’re using the plugins on a HiDPI screen without the FFlag enabled, so things look much smaller to you. Have you tested the plugin on smaller screens (mine is effectively 1080p, just with sharper details)? Maybe enable that FFlag and see what it looks like at the correct scale?
P.S. Try using UIStroke instead of text stroke, it behaves properly in HiDPI.
Feature request: Add an option for Benchmarker to yield every X seconds, i.e. 0.1 or so, during benchmarking. This would allow me to run benchmarks for many minutes while I work on other things. Having it enabled could also add a Cancel button to the “Running benchmark…” screen.
Enable numbers of calls larger than 10k, but only when the yield option is enabled.
It would also give a bit more varied results when the functions being benchmarked include wildcards like Instance.new (which many of your library benchmarks do use, which is kind of a bad practice but whatever), although that’s not always desirable, hence the optionality.
I really want to buy this plugin, but I didn’t before because via robux it was like really expensive and just wasn’t worth it for the shares that Roblox would get, but, now I’ve noticed it moved to a real purchase thing, but now I can’t buy it, because, itch will not use the debit function (and might not be able to do so as I’m not on the USA), and I can’t use credit or use PayPal (as I’m -18), my bank does have an option for buying things with credit but working just like debit, but it won’t work for international purchases, so it doesn’t work either. Do you know like any other method that I could use to purchase Benchmarker? (apart from Robux which isn’t even a thing right now)
