Add way to get lcov code coverage in studio, engine, and in open-cloud luau-execution-session-task

Add way to get lcov code coverage in studio, engine, and in open-cloud luau-execution-session-task.

As a Roblox developer, it is currently too hard to get code-coverage stats from our unit testing runs. This prevents us from tracking coverage and code quality through code-coverage. Code coverage is a metric that measures the extent to which a program’s source code has been executed during testing, helping identify areas that might need more attention. While luau and luau-cli has the ability to output lcov files - a standard code coverage format, the Roblox engine itself has no such mechanism.

While unit testing games is typically a bad idea, we have many core libraries which we depend upon. We want unit tests to ensure we don’t break these during refactors. And while code coverage is often a bad statistic to force in terms of velocity, just reporting code coverage itself can be a powerful mechanism to increase test quality.

Proposed workflows

  1. Write unit test
  2. Roblox studio reports lcov coverage and a plugin can report the actual coverage percentage
  3. Unit test runs in open cloud
  4. A script extracts the lcov files and sends them back to github action
  5. A standard lcov github action bot can parse the files and report coverage

This would make a big difference, and is a core requirement for higher code quality and more professional development on Roblox. This is also a feature we can’t do without Roblox’s help.

18 Likes

This is a great feature request, thanks! We agree we need a story for code coverage analysis.

We don’t provide access to the full debug library in the engine due to security concerns. This means the approach taken by LuaCov & similar to do code coverage won’t work in the Roblox runtime.

Bytecode instrumentation is probably the right approach here (and is what we do for internal Luau development). However - that means instrumentation needs to be done when the scripts are compiled. As such, there’s a path to us supporting in Open Cloud Luau Execution (through some sort of flag) but supporting in plugins / Studio would be much more complicated.

From your feature request - it sounds like you were expecting this already? Let me know though!

I will talk to the engineering team and get a sense of what this would take to support. Will update here in due course :slight_smile: