Efficiently turn a table into a string?

Help and Feedback Code Review
#1

The code does what its name saids: turns a table into a string.

There has to be a more efficient way as I benchmarked it with printing a table with the built-in and mine. It detects cyclic table references (Which I can also do by passing a table of tables it has visited) and is still somehow faster.

Maybe this could be a bit faster without introducing state? (i.e. making it not pure)

local stringify

stringify = function(v, spaces, usesemicolon, depth)
	if type(v) ~= 'table' then
		return tostring(v)
	elseif not next(v) then
		return '{}'
	end

	spaces = spaces or 4
	depth = depth or 1

	local space = (" "):rep(depth * spaces)
	local sep = usesemicolon and ";" or ","
	local s = "{"

	for k, x in next, v do
		s = s..("\n%s[%s] = %s%s"):format(space,type(k)=='number'and tostring(k)or('"%s"'):format(tostring(k)), stringify(x, spaces, usesemicolon, depth+1), sep)
	end

	return ("%s\n%s}"):format(s:sub(1,-2), space:sub(1, -spaces-1))
end
2 Likes
#2

Doesn’t JSON turn tables into strings?

#3

Yes!
But I am turning tables into strings that are readable by lua (you know, for lookup tables)

#4

Ozzypig has a module called repr which is useful for printing tables. While it may be superseded by Roblox’s new behaviour for printing tables which is to create an interactive collapsible instead of showing the memory address, repr does return a string version of the table. You might be able to reference that, or even use that itself, for getting a string representation of a table.

1 Like
#5

While the module you linked me to is related, it doesn’t achieve the effect I need it to, runs slower due to the massive amounts of elseif checks and boilerplate code, useless checks that never occur, uses 150 lines of code when it can be done with merely 20 lines as I demonstrated, and is less feature-rich.

Sorry if this sounds like a rant, but
I am looking for improvements, not linked to an alternative that is considerably less sophisticated.

#6

I did some benchmarking of my own and it seems the built-in print is actually quicker to print tables than to print non-table values like numbers, strings, etc.

image
image1787×499 45.1 KB

My theory is that when you print a table it knows it may take a while and so it does it asynchronously. Regardless, this unfortunately means we cannot accurately benchmark the built-in table print and as such cannot fairly compare yours to it. Your method might be 10x faster than theirs and we’d never know, unless I’ve made a mistake.

Benchmark Test Code (using boatbomber's Benchmarker plugin) 
return {

	ParameterGenerator = function()
		return {RandomNumber = math.random(1000)/10, RandomTable = {
			math.random(1000)/10,
			math.random(1000)/10,
			math.random(1000)/10,
			math.random(1000)/10,
			math.random(1000)/10,
			math.random(1000)/10,
			math.random(1000)/10,
			math.random(1000)/10,
			math.random(1000)/10,
			math.random(1000)/10,
			math.random(1000)/10,
			math.random(1000)/10,
			math.random(1000)/10,
			{
				math.random(1000)/10,
				math.random(1000)/10,
				math.random(1000)/10,
				math.random(1000)/10,
				math.random(1000)/10,
				math.random(1000)/10,
				math.random(1000)/10,
			}
		}}
	end;

	Functions = {
		["table print"] = function(Profiler, Randoms)
			print(Randoms.RandomTable)
		end;

		["number print"] = function(Profiler, Randoms)
			print(Randoms.RandomNumber)
		end;
	};

}
3 Likes
#7

Very interesting find. This might mean that there is absolutely no way to compare mine with built-in fairly, while others that are out in the wild seem to be much slower and less sophisticated.

Thank you for the heads up!

EDIT: Actually on a second thought, it might actually be because roblox stack their output together.
What I mean is, the output’s previous state is concatenated with the next output, which might be slower (?)
I notice that when I am working with a very large table printed, printing a simple number will literally freeze half a second.
So the output has to be cleared for this to be fair.

Some very primitive testing of my own, however, shown that printing a large table is somehow faster than a number still. This further consolidated that the builtin print table is done asynchronously.

1 Like
#9

The real question to ask here is: why do you need something faster? What’s the goal? Is an existing tool just simply not fast enough to accomplish a task you’re actually trying to do? An effective code review can only be given if your intent is stated. I can think of a few reasons you’d want to turn a table into a string…

#1: Data serialization/deserialization, usually for saving game data

There’s existing tools for this that are already reasonably fast for any game. JSON is one such tool, although there’s plenty of other options of course. Probably not what you’re looking for, though.

It looks like from one of your previous posts that this is probably your intent. My primary comment on it is that you should not be using Lua for data-serialization because of how slow it will be to come up with the strings. It smells like a wrong-tool for the wrong-job type of situation, but we don’t know your full situation.

#2: Debugging, or more particularly inspecting the state of your code

For what it’s worth, repr isn’t written to be highly performant. It’s written to be performant enough, as should any software. It has two goals, and one of them is to be useful during debugging, which may/may-not include the use of print and Roblox’s output window. You could, potentially, send its output via HttpService for instance.

I keep seeing people say repr is superseded because the new Output window prints explorable tables. Don’t get me wrong, if it’s more useful, do use it! But, it still does one thing that print doesn’t and that’s give you a string to do whatever you please with, including-but-not-limited-to printing.

Furthermore on printing data for the sake of debugging: more does not imply better when it comes to print messages. We have a debugger which can evaluate expressions on-the-fly. Converting and printing tables for the sake of debugging should never even get close to the point where performance matters on this. You’re likely just getting a bad signal-to-noise ratio and there’s almost certainly a better way to debug.

#3: Code golfing

Not touching this with a 10-foot pole personally; got better things to do. But if that’s your goal, it should be stated. I don’t think this topic is really suited to it entirely, but I’m not an authority on that. -shrug-

2 Likes
#10

Furthermore, by using the HTTP service, the module becomes impure. I am not looking to create a new format that’s not readable by lua, and then translate it in run time.

There is a clear difference between golfing and concise code. Do not mix them up.

1 Like
#11

Hi, uhm, just to be clear, you are aware of HttpService | Roblox Creator Documentation and HttpService | Roblox Creator Documentation, and you think they are too slow?

In general within your server there is no reason to convert tables to strings… Only when you want to send it ‘outside’ your server, such as to a HTTP address. Actually I once tried to use strings as a manual ‘save file’ that the user could copy to a .txt, but found no way to pass long strings to the clipboard X) Anyway, between scripts, if it can be turned to a string safely, just send the table instead, even between client-server…

The only use case I could come up with within the server, is writing some config files in JSON for some reason, but efficiency/performance would not be an issue at all. And for all other use cases, sending your table to outside the server, your bandwidth is so limited that the stringification algorithm’s performance is hardly important. I may be wrong, or I may be missing the point, but I agree with what @Ozzypig wrote

#12

I feel like I have to state my use case or everyone is going to question it instead of giving ways to improve it. (efficiency-wise)

Is it just me, or does everyone generate their lookup table in run time?
printing the table with the studio default print something like this

{
    [1] =  ▼  { -- arrows that lua doesn't ignore
       [1] =  ▼  {
          [1] =  ▼  {
             [1] =  ▼  {
                [1] = {} -- weird spaces that lua can't parse
             }
          }
       }
    }
}

So at the end of the day, I either to have manually remove the arrows and change all these weird spaces manually, or use something that automates it.

Which I found annoying as I often have to do something like this

print(([[tablegeneratedontheoutput]]):gsub(weirdspacecharacater, asciispace):gsub(arrow, ''))

Then pasting it to Lua Beautifier so there will be space between stuff, i.e. a=2 vs a = 2

TL;DR, I am looking for ways to optimize it further if possible, while learning these techniques along the way.

#13

Concatenating in a loop is a red flag. Instead, it’s best to build the string, then concatenate when we’re done. When v had 100,000 key-value pairs, with integers for the keys and values, the old version exhausted the allowed execution time, and the new version (based on a completely unprofessional benchmark) took under 0.5 seconds on my computer.

local stringify
local insert = table.insert

stringify = function(v, spaces, usesemicolon, depth)
	if type(v) ~= 'table' then
		return tostring(v)
	elseif not next(v) then
		return '{}'
	end

	spaces = spaces or 4
	depth = depth or 1

	local space = (" "):rep(depth * spaces)
	local sep = usesemicolon and ";" or ","
	local concatenationBuilder = {"{"}
	
	for k, x in next, v do
		insert(concatenationBuilder, ("\n%s[%s] = %s%s"):format(space,type(k)=='number'and tostring(k)or('"%s"'):format(tostring(k)), stringify(x, spaces, usesemicolon, depth+1), sep))
	end

	local s = table.concat(concatenationBuilder)
	return ("%s\n%s}"):format(s:sub(1,-2), space:sub(1, -spaces-1))
end

If we knew how many keys were in v, we could use table.create() to create concatenationBuilder to avoid automatic table resizing as we insert into it.

I used table.insert() because it is faster than using t[#t + 1].

Further optimizations, if desired, could include caching of results during string building, or converting the function from a recursive approach to an iterative one.

5 Likes
#14

Theres already a built in lua functions

local yes = {"ok", "yes", 1}
-- make the table a string
local yesString = table.concat(yes, "%") -- the second param is a split, it can be ignored
print(yesString) -- ok%yes%1
-- make string back into a table
local newYes = string.split(yesString, "%")
#15

This depends on how much “complexity” you want - do you want your table-to-string to not support cyclic tables, because you are sure it wouldn’t occur in your code? do you only serialize primitive types (i.e no CFrame’s, Vector3’s etc)? The best solution would be to make a function best tailored for your own usecase. Currently though, according to benchmarks, my implementation is the fastest:

image
image1716×524 39.2 KB

You can find the module here: Table to String converter

#17

how do i make it so if there’s a string value it adds quote marks around the string? like for example i have

{
	[1] =  {
		[1] = 6,
		[2] = "D",
		[3] = 1732465503.091523,
		[4] = "L",
		[5] = "all"
	}
}

here i want the number 2, 4 and 5 to have the quote marks when i plug it into the function, how can this be done?

#18

Not sure if this is a good look in terms of performance, but it worth looking at if you’re interested in simplicity.

could json encode the table, then just do some regex with it and it should look like a table. im guessing the json encode method itself runs faster in c than we can replicate in luau. The question is just is that small speed worth the tradeoff for the ping.

from there you could just format how you like, replacing things as you need. the foundation is all there its just in json. or, if you’re normal, you would use a regex module ported into luau and just replace like that.