Proactive Testing: Profiling, Unit, and Fuzz testing
In the previous tutorial, we talked about using Roblox’s built in debugger. One major limitation of the debugger as a bug finding tool is that it is very difficult to find an error that you don’t already know about. Any issue that causes a red error in the output can easily be worked this way, but hidden bugs may still exist in your code unnoticed for a long time, until they start causing problems later. In this section we will talk about some proactive methods for finding bugs.
For this tutorial I will be providing this module with pre-made tools for the methods discussed. The tools include a basic unit test runner, a “wrapper” function for coverage and profiling, and some other helper functions.
Unit testing
In classical computer science, functions are used to organize code around specific tasks that take the same list of inputs and produce some output based on the value of those inputs. A Pure Function is especially easy to test, since its effect by definition only depends on its inputs, and it always gives the same result for the same input values (It has no Side Effects).
Unit testing is any testing of functions that ensures they give the expected result. If the functions are indeed pure, unit testing can rule out specific causes of bugs by “proving” that the tested functions are not the cause. Unit testing is often used continuously during development to catch bugs that may be introduced by code changes as soon as they occur, this type of unit test usage is known as regression testing.
Unit testing comes in two flavors, positive and negative. A positive test gives a function valid input values and checks to make sure the result is correct. The programmer provides these values and the expected result manually in a list called the test vectors. Negative testing gives a function invalid inputs and makes sure the invalid inputs result in an explicit error. That is, your code stops immediately or enters a special handler.
--Debug module included in this tutorial
local debugTools = require(game:GetService("ReplicatedStorage"):WaitForChild("DebugTools"))
--Function which adds 2 to a number, but errors if the number is negative or 10
local function MyFunctionUnderTest(input : number)
if input < 0 then
error("Input must be positive")
end
if input == 10 then
error("Oopsie woopsie")
end
return input + 2
end
--Test vectors for MyFunctionUnderTest
local testVectors = {
-- Test Name Test type (+/-) Input values
-- Function to be tested Expected value
{"MyTest1Name", MyFunctionUnderTest, "Return", 5, 3},
{"MyTest2Name", MyFunctionUnderTest, "Return", 3, 1},
{"MyTest3Name", MyFunctionUnderTest, "Error" , nil, -1},
{"MyTest4Name", MyFunctionUnderTest, "Error" , nil, 10},
}
--Run a unit test for each test vector
--Prints a "Pass" or "Fail" message for each test in the testVectors list
--Try breaking MyFunctionUnderTest and see what the test runner prints
--See the module code for how this runner works
debugTools.UnitTestRunner(testVectors)
Sometimes it is useful to temporarily “inject” a test into existing code, without modifying the actual source, thus reducing the chance of mistakes when editing said source. Since Lua treats functions like any other variable, we can do this with some sneaky code.
--Simple test injection example
MyFunctionUnderTest = debugTools.DebugWrapper(MyFunctionUnderTest,
--Debug wrapper returns a new function which will call this before calling the main function
function(input : number)
print("Calling function with input = " ..tostring(input))
end)
We can easily remove this test later if necessary to save performance. This kind of wrapper is incredibly useful for some other kinds of testing shown later. The important thing is that this wrapper code can be added to any existing code in your project without the risk of making mistakes while adding or removing test code directly from the target functions. It can also be turned on and off automatically:
--Optional toggle for debug wrappers
--This is useful if you have tests that are expensive to run
--You may want to not run them in a live game
local debugWrappersOn = true
if debugWrappersOn then
--Replace functions with DebugWrapped functions
end
Fuzz Testing
Sometimes unit testing is not practical due to the sheer number of test cases we would have to come up with. It can also be impractical if the function is too complex to verify the result by hand. This is a problem, because any case that we do not test may still contain a bug. Fuzz testing is similar to unit testing in that it compares inputs and outputs for correctness. However, instead of manually entering values, we randomly generate values in a range that we know will be correct or incorrect. This way, we can check many more combinations of inputs than would be possible if we had to enter them manually.
--Function which should always return a vector with length 1
local function RandomDirectionGenerator()
local xc = CFrame.fromAxisAngle(Vector3.xAxis, math.pi * math.random())
local yc = CFrame.fromAxisAngle(Vector3.yAxis, math.pi * math.random())
return (xc * yc).LookVector
end
--Basic fuzz test
for i = 1, 1000 do
local result = RandomDirectionGenerator()
if result.Magnitude > 1 then
error("Result vector too long!")
end
end
Sometimes fuzz testing can be purely visual. This is often the case for testing things like random generation. We can generate a bunch of parts in the workspace to visualize the possible outputs of RandomDirectionGenerator. This particular generator produces slight clumping towards one end of the hemisphere of possibilities.
--Visual fuzz test
for i = 1, 1000 do
local result = RandomDirectionGenerator()
local p = Instance.new("Part", workspace)
p.Anchored = true
p.BrickColor = BrickColor.new("Lime green")
p.Transparency = 0.8
p.Position = result
p.Size = Vector3.one / 10
end
Profiling
The wrapper technique used earlier is useful for some other testing tricks, the simplest of which is profiling. Profiling tests are anything which measures performance, which may refer to actual time taken to run, number of function calls, or any other indirect measurement of efficiency.
--Using DebugWrapper to set up profiling
local calls = 0
local totalTime = 0
local lastCallStartedTime = 0
ExpensiveFunction = debugTools.DebugWrapper(ExpensiveFunction,
--Called before with function args
function()
lastCallStartedTime = tick()
end,
--Called after with return value
function()
local callTime = tick() - lastCallStartedTime
--Safe to do anything here that takes extra time
print("Call time: " ..callTime)
calls += 1
totalTime += callTime
end)
--Test expensive function 100 times
--These could also be calls from existing code (injected test)
for i = 1, 100 do
ExpensiveFunction()
end
print("Average time to run ExpensiveFunction once: " ..(totalTime / calls))
Profiling should always be the first step before attempting to make code “faster”, since good testing will be able to tell you which functions will benefit the most from improvements; usually it is the functions that run the most or have the most other processes waiting on them.
Harder Test Setups
As a final example, here is a non pure function which must always add an item to a list. The list in question is global with respect to the function, meaning it is defined somewhere else in the code and is probably not easily simulated as part of a unit test. A test that makes sure an item is always added could be set up like this:
--More difficult unit test example (non-pure function)
--Below function must be called with a number NOT already an index in globalState
local globalState = {}
--Something bad happens if the same number is given twice,
--but this is hard to detect with a plain unit test
local function MustAddToGlobalState(a : number)
--Bug
if a == 3 then
return
end
table.insert(globalState, a)
end
--Debug wrapper and extra for checking valid input
local debugGlobalStateCount
local debugLastArgumet
MustAddToGlobalState = debugTools.DebugWrapper(MustAddToGlobalState,
--Called before with function args
function(a : number)
debugLastArgumet = a
debugGlobalStateCount = #globalState
end,
--Called after with return value
function(ret : number)
local newCount = #globalState
if newCount ~= (debugGlobalStateCount + 1) then
--tostring in case debugLastArgument is nil
error("Function didn't add to globalState!: " ..tostring(debugLastArgumet))
end
end)
--Test function for inputs 1 through 10
for i = 1, 10 do
MustAddToGlobalState(i)
end
While these examples are, by definition, contrived, they give the outline of tools you can use in your projects. The most important factor in any troubleshooting task is your creativity and imagination. Imagination is needed in order to imagine what could go wrong, and Creativity is needed in order to create a system which can adequately catch such a mistake. You’ll have to take my word for it that even a simple if then error style check can save you infinitely more time in future debugging than would be saved by not writing the check. Your code’s run time is relatively inexpensive compared to the time you spend working.