Benchmarker Plugin

boatbomber · October 21, 2020, 12:33am

benchmarker_blue

by boatbomber

Get the plugin here!

What is this plugin?
This plugin helps you easily benchmark & visualize your function’s speed and compare its performance to other functions.

Why use this over free alternatives?

Before I get into the advantages of this plugin, I will say that it’s often best to use multiple tools to double check accuracy. Those other tools are great too, shoutout to those creators!

That said, this plugin does offer a variety of advantages. The most obvious one is the histogram. Being able to visualize the distribution is incredibly informative, and those others simply do not give you that information, visually or numerically.

Another key advantage is that this plugin uses the fiftieth percentile for its calculations and results, whereas others (as of writing this) only use the average value. Average values are a generally unhelpful metric, because one or two statistical outliers will completely skew the average value and it will no longer be representative of the function’s real behavior. This plugin provides you with percentiles, mins, maxes, averages, and totals so that you get a wide range of metrics for the most comprehensive results possible.

As you can clearly see in the image (of an older version) below, thanks to an outlier or two the average value is 384 times more than the function’s real behavior! (50th: 10.7 microseconds, Avg: 4116.9 microseconds)

Another advantage is that this plugin allows you to run your functions in an isolated environment with controlled input parameters- it doesn’t require you to run your whole game just to test one function and you can control exactly what the function is operating on.

The plugin is just a great all-in-one package- it does everything you need. Profiling, histograms, bar graphs, and detailed stats. It covers everything that the alternatives do, combined into a single plugin.

Get the plugin here!

Change Log

V1.0 - Initial release
V1.1 - Stats UI improvements
V1.2 - Graph UI improvements
V1.3 - More Graph UI improvements
V1.4 - Settings UI improvements
V2.0 - API changed to allow custom parameters + UI improvements
V3.0 - Graph changed to histogram + UI improvements
V3.1 - Added Graph Outliers setting + UI improvements
V3.2 - Graph UI improvements
V3.3 - Added visibility toggle + GUI restructured
V4.0 - Profiler + Bar Graph/Flame Chart
V4.1 - Bugfix + Color improvements
V4.2 - Dynamic measurements
V5.0 - End Condition setting + Minor improvements
V5.1 - Change protection mark behavior
V6.0 - New UI, Added Library, Profiler improvements
V6.1 - Edge Case Fixes
V6.2 - Fool Proofing
V7.0 - New UI, Improved Accuracy, Persistent Results

PoliceFighter761 · October 21, 2020, 2:19am

After trying out the plugin with some arbitrary test I noticed that sometimes it doesn’t correctly sort.

I don’t think this is intentional.
Also, a setting to toggle what graphs are shown would also be nice so you can compare 2 functions with each other directly in a visual form.

Other than that, this is another great plugin! Thank you for sharing it.

boatbomber · October 21, 2020, 2:20am

That looks correctly sorted to me! It sorts by the 50th % value since that’s the most valuable one.

PoliceFighter761 · October 21, 2020, 2:21am

Ah okay! I thought it sorts by average. Thank you for clearing it up.

boatbomber · October 21, 2020, 2:22am

I’ll redo that UI to display the values more intuitively, thank you for pointing out that ambiguous design!

Update!
V 1.1

New order is much more intuitive and useful:

R0bl0x10501050 · October 21, 2020, 2:23am

Ngl, my brain died looking at the graph/numbers.

Is it telling you how much ms (of what?) each code segment takes up? Sorry, I’m so confused since I’m extremely tired… school…

EDIT: This is super helpful tho! Nice work!

boatbomber · October 21, 2020, 2:24am

How many milliseconds it took to run the function.
(Get some rest, taking care of yourself is more important than my plugin!)

R0bl0x10501050 · October 21, 2020, 2:25am

1000 ms = 1s right?

That’s cool that it does this! I can never read the microprofiler so this is super helpful.

EDIT: Me reads the price and is super grateful he just did a commission lol. Jokes aside, this is pretty cheap for a good system like this I especially need this for a game I’m making literally as I type this since its using like 3-5 for loops nested in each other to work.

FrogmanExtract · October 21, 2020, 11:27am

This looks good, I’m gonna try this rn. But also @R0bl0x10501050 the plugin is free so what do you mean when you say

Do you mean that other plugins cost robux or something?

boatbomber · October 21, 2020, 3:18pm

Update!

V 1.2

The graph’s point values and axis values are now displayed in milliseconds rather than seconds.
This makes it match the stats window’s units, and its just a generally more useful number for these tiny values.
Space is no longer wasted by having everything start with 0.000, which is nice.

I’ve also made it specify “ms” on it because someone told me that it wasn’t entirely clear what the measurements were.

boatbomber · October 21, 2020, 8:51pm

Update!

V 1.3

Point labels now display the colored (RichText) function name above the value. This is helpful when the lines are too close together, as well as for colorblind people who need text since the colors might not be distinct enough.

R0bl0x10501050 · October 21, 2020, 9:35pm

It doesn’t appear free to me…

boatbomber · October 21, 2020, 10:03pm

Update!

V 1.4

Settings is now a dropdown menu inside of the main menu panel, reducing the number of widgets this plugin uses.

Settings also saves now!

Settings

boatbomber · October 22, 2020, 1:26am

Major update!

V 2.0

New API for handling custom function parameters, better env protection and error handling, and a few minor tweaks!

Thank you to @Coeptus for the feedback and feature suggestions!

I’ve redone the demo video using the latest plugin version, and I’ve made sure to be more clear in the video this time around.

SirMing · October 22, 2020, 2:37pm

With the new Luau, I wanted to check if it’s still necessary to localize globals like math.max and Vector3.new. However, the plugin always show Sample A as fastest even when I switched the code around:

boatbomber · October 22, 2020, 2:59pm

That’s not what is happening here, but I’m glad you brought this up.

Firstly, because dictionaries in Lua have no iteration order, Sample A being defined first has no impact.
Secondly, your images are showing a difference of roughly 20μs, which is so small that it is very difficult to measure accurately. (Also, means that “optimizing” for that is probably unnecessary.)

When dealing with times that tiny, it’s best to raise your loop count in the function so that the function takes longer in order to be more measurable.

When comparing those functions myself, I found that your localization “optimization” has no significant impact on run time. Whichever one comes first is nearly a roll of the dice. If you run multiple times, you’ll see differing results.

It would appear the answer is no, since this test has shown that there is no real run time difference.

TL;DR: The plugin is not broken, your functions are just not different in a significant way so results are basically luck.

bobbybob2131 · October 26, 2020, 9:28pm

Another great plugin from boatbomber! A couple of bugs though, it seems not to render the lines (only the points) when the Graph Points is over or equal to 135:

Also, the logo image at the top is hard to see in light mode:

boatbomber · October 26, 2020, 11:44pm

Super odd, haven’t seen this bug before. I set it to higher than 135 in the demos, it definitely works. What’s your resolution? Maybe it’s making the lines less than a pixel thick?

Solution: Join the superior team and use dark theme

bobbybob2131 · October 27, 2020, 9:41am

It appears that it only renders points when display widget is below a certain size and the points are above 135.

Repro:

Set Graph Points to 135+
Run any functions (the number of functions doesn’t matter)
Dock the widget and resize the X and Y, Y until it stops resizing and X so it takes up about 2/3 of the screen.

1366 x 768

Edit: It seems to depend on the size of the window, at minimum size it can only render below 77 points, and as you increase the size you can have more points. https://gyazo.com/a4d5c86608c76d4fa045959871ecbec5

Yep!

boatbomber · November 12, 2020, 5:41am

Big Update!

V 3.0

The graph is actually useful now.

The old graph system naively drew the dataset as if it were the output of a simple line function. Iteration count went along the X axis, and time taken was on the Y axis. This type of graphing is handy for things where you are measuring a change over time, like CPU temperature. However, it is unhelpful if you’re trying to visualize a set of distinct data points aren’t correlated, like our function calls.

Scrapping that system, the graph now calculates and draws the distribution (aka histogram) of our data set. This is far more useful information, as it demonstrates the functions performance, consistency, and stability in a clear and intuitive way.
The graph also draws a vertical line for the 50th percentile, so that you can quickly pinpoint which function performs better, even if their distributions are overlapping.
The Baseline Zero setting now applies to the X axis instead of the Y axis, but it’s the same concept and usage as before.

In terms of UI changes, the X axis is now labeled clearly, since those values can change and are now actually relevant in this new format. The points and lines are now a set size, so your resolution shouldn’t cause any issues like mentioned earlier in this thread. Also, the label that shows up when you hover on a point will no longer clip out of frame. I also made some minor changes to light theme.

(As always, please don’t hesitate to DM me if you find a bug or mathematical error! This was a massive change that I did in a single sitting and some sneaky mistake could have slipped past me.)