Expose low level general compression as a lua function

Feature Requests Engine Features
#1

As A Roblox Developer, it is currently too hard to compress data. It is obviously possible to compress data ourselves, but Implementing general compression can save us a lot of space in datastores and make it easier for developers to approach more open ended and creative gameplay.

Using a combination of my own data structure and an LZW compression module made by 1waffle1, I managed to fit HUGE builds in HomeBuilder 2 into single datastore keys. This is largely in thanks to the LZW compression module which cut JSON string lengths down by more than half. The average developer doesn’t have access to any general compression, and scripting it with Lua is not only difficult, but it performs slower than it would if it were built into the engine.

Huge HB2 Builds

The bottom screenshot is a city built with multiple tall buildings, built with a modular build set, and it takes up around half of the datastores 260,000 character limit.

image
image918×967 229 KB

image
image1366×746 417 KB

This strikes me as being not very difficult to implement as a function for developers to use, and extremely useful for anyone developing games that are highly personal or creative.

16 Likes
Release Notes for 431
#2

Compression is often fit to purpose - it depends on how the data needs to be read. I personally think it’s better to leave it up to the scripter than to clutter API with a generic compression algorithm.

[edit: In retrospect, I’d support exposing specific compression but I’m not too sure on the general.]

1 Like
#3

There are also tricks to avoid unnecessary data, and so you should only store what you need.

For instance, I have a building game where users can place buildings on the baseplate and rotate them. Now, most people might think I should just store the CFrame of the object to save its location in the game. But that’s 12 numbers! Instead, since I can only rotate on the Y axis and know what the Y position is always going to be, all I need to store is the X & Z positions, and the Y rotation. So now it’s just 3 numbers!

This is similar to database design. You never store data that can just be calculated from the other values, and usually you would not store data that is constant and defined elsewhere.

Think about that sort of stuff when trying to serialize large pieces of data in your game. Doing that, I don’t think you will often have to use compression methods.

6 Likes
#4

Yeah I already had my own compression but on top of that, adding the general compression still added huge benefit. There’s no reason not to run it through a general compression also. I really got the homebuilder2 save files very compressed myself with a clever data structure but either way at the end of the day, applying LZW on top of it still reduced my size by over 50%

@Arch_Mage I don’t see how adding one single function would be clutter to be honest with you

I mean I got my idea from zeuxcg so it couldn’t be that bad

The devforums don’t seem to want to link this quote correctly so here it is https://devforum.roblox.com/t/im-so-excited-about-my-compression/75042/19?u=coldsmoke

8 Likes
#5

I really think having compression exposed for developers to use would be amazing. We don’t have any ports that I can find of LZ4 for example and LZ4 is an extremely fast/efficient compression algorithm. It’s now 2020 and the quote mentioned above from @Zeuxcg has a lot of people in support of it. Even simply having a CompressionService of some kind with LZ4 on it would be absolutely great to have.

Here are some features I’d love to see included:

  1. Support for datastores (e.g. a bit width argument which would for sure be my preferred method as it allows for more control, or alternatively a dataStore boolean)
  2. Support for several commonly used compression algorithms (especially LZ4) for processing binary data such as PNGs (deflate) and RBXM (lz4) files.
5 Likes