DataStoreService method to verify if a value is saveable & bring more attention to malicious strings

[EtSapientisMagna (Acedia)]

As a Roblox developer, it is currently too hard to verify if an object can be saved or not.

The engine currently lacks a function to verify if data can be saved and has bad documentation on how to verify the safety of strings, which was my fault for providing improper fixes to the problem. The lack of any proper way to do this causes various issues and bugs that are incredibly difficult to deal with.

Roblox’s documentation is very unclear about what types data stores can save or not. If you were to dig and try saving every type, you’d find that some of Roblox’s data types can be saved while others can’t be saved, and the userdata that does save is just nil when using GetAsync. The only places you’ll find mentions of this are on the GlobalDataStore page in SetAsync and UpdateAsync and the security tactics page. On the GlobalDataStore page the documentation is also wrong claiming “values greater than 127 are used exclusively for encoding multi-byte codepoints, so a single byte greater than 127 will not be valid UTF-8 and the GlobalDataStore:SetAsync() attempt will fail”, which is wrong since characters like “ï” (195) are over 127 and will save completely fine.

Why is this problematic?

Roblox datastores have limitations on the data that can be stored; many data types cannot be saved; tables with specific indices and strings with invalid unicode will error. If instances were allowed to be saved, that would also not solve the issues, since when a player loads in the data, it may error and instead corrupt the player’s data. However, in cases where an instance is passed, it can be reliably checked using typeof. Strings and tables containing strings are much more problematic because you have to manually figure out what in strings can be saved and what should be allowed since checking if it is ascii is not enough or using utf8.len is not enough and results in various bugs and bypasses.

Certain characters not saving is incredibly problematic when we don’t have a proper way to protect against this. Depending on the game the results an exploiter may get range from economy breaking to bypassing bans. If you were to have a loot box system in your game and an exploiter had a rare crate they could open they would be able to roll back their data constantly until they got the item they wanted, this could be repeated for other cases in games too. Alternatively if you had a system that banned players after a certain amount of infractions, and an exploiter decided to put a string that would prevent their data from being saved they’d be completely negating a ban they would receive.

Currently, as a developer, you might not even be aware of this, and the documentation Roblox provides explaining how to deal with the issue uses bad advice by only allowing ascii, which was based on my own bad advice I showcased in my github gist. While my github gist helped bring attention to the problem, it never provided proper fixes to verifying if a string is safe or not, utf8.len being bypassable, and my pattern only allowing ascii characters.

You’ll find that this check using string patterns based on my original [^\0-\127] pattern seems good and serves as a good check; however, it becomes problematic when certain characters that will save now are rejected, such as ï (naïve) and many non-English characters like Chinese characters and Japanese hiragana (pinyin isn’t safe either).

These checks exclude a lot of players that may be playing your game in countries that don’t use English, possibly leaving them confused when certain actions don’t work. The only proper way to verify if a string is safe as a developer is to manually try saving all characters, making a list of all the ones that work and all the ones that don’t, and developing a check based on that.

A list of all data types Roblox can save in data stores [January 19, 2023]

Axes -> errored
BrickColor -> errored
CFrame -> errored
CatalogSearchParams -> errored
Color3 -> errored
ColorSequence -> errored
ColorSequenceKeypoint -> errored
DateTime -> errored
DockWidgetPluginGuiInfo -> errored
Enum -> saved
EnumItem -> errored
Enums -> saved
Faces -> errored
FloatCurveKey -> errored
Font -> errored
Instance -> errored
NumberRange -> errored
NumberSequence -> errored
NumberSequenceKeypoint -> errored
OverlapParams -> saved
PathWaypoint -> errored
PhysicalProperties -> errored
Random -> errored
Ray -> errored
RaycastParams -> errored
Rect -> errored
Region3 -> errored
Region3int16 -> errored
TweenInfo -> errored
RotationCurveKey -> saved
UDim -> errored
UDim2 -> errored
Vector2 -> errored
Vector2int16 -> errored
Vector3 -> errored
Vector3int16 -> errored
userdata -> saved
number -> saved
string -> saved
function -> errored
table -> saved
boolean -> saved

How can this be remedied?

To summarize, Roblox should introduce a way to verify if a value can be saved. A function like DataStoreService:IsSaveable would work effectively. The function should be able to accept a variable amount of arguments, so multiple values can be checked at once instead of calling individually.

local d = game:GetService("DataStoreService")
local f = function() end
local h = "Hello World!"
local s = "平"
local n = "\237\190\140"

print(d:IsSaveable(f)) --> false
print(d:IsSaveable(h)) --> true
print(d:IsSaveable(s)) --> true
print(d:IsSaveable(n)) --> false
print(d:IsSaveable(h, s)) --> true
print(d:IsSaveable(h, n)) --> false
print(d:IsSaveable(f, h, s, n)) --> false
d:IsSaveable() --> error: missing argument

Roblox should bring more attention to this rather than having three paragraphs total explaining it that most developers will likely miss. A minor announcement warning developers of this would be useful to bring attention to this. I’ve talked with many developers of large games who told me they were completely unaware of this, and they themselves found it difficult to prevent these exploits other than just locking all the characters to ascii ones.

[funwolf7]

I do agree that it could be more clear about how to check for invalid inputs, and a dedicated method for it may be justified.

That being said, you can already check if something is valid for a DataStore relatively easily by trying to JSON encode it. This ends up working very well because DataStores actually JSON encode whatever you pass into it, meaning the restrictions on DataStores are due to the JSON standard.

local HttpService = game:GetService("HttpService")

local function isJSONAble(test: any): boolean
	return (pcall(HttpService.JSONEncode, HttpService, test))
end

print(isJSONAble("平")) --> true
print(isJSONAble("\237\190\140")) --> false

I will say that this doesn’t work for any of the Roblox datatypes (JSONEncode allows them and replaces them with either null or a string representing their state), but I feel like if Roblox datatypes are somehow being inserted into your data then you probably have bigger concerns.

EDIT: after further thought and discussion, this method is not preferable. If you need to check if a string is valid UTF-8, a reply further down gives a function that is based on the Unicode standard. Checking for if a table is encodable this way is probably also not a good idea, though I would recommend structuring your data and requests in a way where you don’t need to do this.

[stravant]

Here’s one of my modules to deal with the problem when writing Datastore-safe binary blobs:

https://gist.github.com/stravant/04312466d675c073a5e4052e95189808

[EtSapientisMagna (Acedia)]

Having to actual serialize data to json just to verify if it is safe is not a proper fix, its a work-around just like any other work-around that is provided. I was already aware of JSONEncode being a work-around you could use but it comes with its own problems, like it not being a performant way to check the safety of something. Aside from just performance JSONEncode does not do any verification for you so you have to verify inputs before you even check if they’re saveable.

local t = {}

for i = 1, 255 do 
	local newT = table.create(255, t)
	t = newT
end 

for i = 1, 50 do 
   task.spawn(SampleRemote.InvokeServer, SampleRemote, t)
end 

[EtSapientisMagna (Acedia)]

This is similar to how I deal with the issue in my own codebases, I work with some rather old existing codebases though and updating them to use any kind of packing is problematic for backwards compatibility and sometimes can’t be done in a good way.

Since Luau is introducing buffer doing this may be even easier soon.

[koob85]

On RoCitizens, we encountered issues where people traded items away and somehow corrupted their data file, essentially causing it to revert. It was incredibly difficult to discover the problem, but since we store user-inputted strings, we were susceptible to this issue completely unknowingly.

When it was discovered, we couldn’t think of a solution besides restricting text to known keyboard characters. Having a function like this would be extremely useful for us, especially since the solutions mentioned above tend to harm translation and foreign unicode characters.

[EtSapientisMagna (Acedia)]

Made a proper work-around that will actually check the validity of utf8 if anybody needs a proper way to do this. Thanks @funwolf7 for helping test this.

local function is_valid_utf8(input)
	local i = 1

	while i <= #input do
		local c = string.byte(input, i)

		if c >= 0x00 and c <= 0x7F then
			--// Fallthrough
		elseif c >= 0xC2 and c <= 0xDF then
			local next_byte = string.byte(input, i + 1)
			
			if not next_byte or next_byte < 0x80 or next_byte > 0xBF then
				return false
			end
			
			i = i + 1
		elseif c == 0xE0 then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			
			if not next_byte1 or not next_byte2 or
				next_byte1 < 0xA0 or next_byte1 > 0xBF or
				next_byte2 < 0x80 or next_byte2 > 0xBF then
				return false
			end
			
			i = i + 2
		elseif c >= 0xE1 and c <= 0xEC then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			
			if not next_byte1 or not next_byte2 or
				next_byte1 < 0x80 or next_byte1 > 0xBF or
				next_byte2 < 0x80 or next_byte2 > 0xBF then
				return false
			end
			
			i = i + 2
		elseif c == 0xED then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			
			if not next_byte1 or not next_byte2 or
				next_byte1 < 0x80 or next_byte1 > 0x9F or
				next_byte2 < 0x80 or next_byte2 > 0xBF then
				return false
			end
			
			i = i + 2
		elseif c >= 0xEE and c <= 0xEF then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			
			if not next_byte1 or not next_byte2 or
				next_byte1 < 0x80 or next_byte1 > 0xBF or
				next_byte2 < 0x80 or next_byte2 > 0xBF then
				return false
			end
			
			i = i + 2
		elseif c == 0xF0 then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			local next_byte3 = string.byte(input, i + 3)
			
			if not next_byte1 or not next_byte2 or not next_byte3 or
				next_byte1 < 0x90 or next_byte1 > 0xBF or
				next_byte2 < 0x80 or next_byte2 > 0xBF or
				next_byte3 < 0x80 or next_byte3 > 0xBF then
				return false
			end
			
			i = i + 3
		elseif c >= 0xF1 and c <= 0xF3 then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			local next_byte3 = string.byte(input, i + 3)
			
			if not next_byte1 or not next_byte2 or not next_byte3 or
				next_byte1 < 0x80 or next_byte1 > 0xBF or
				next_byte2 < 0x80 or next_byte2 > 0xBF or
				next_byte3 < 0x80 or next_byte3 > 0xBF then
				return false
			end
			
			i = i + 3
		elseif c == 0xF4 then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			local next_byte3 = string.byte(input, i + 3)
			
			if not next_byte1 or not next_byte2 or not next_byte3 or
				next_byte1 < 0x80 or next_byte1 > 0x8F or
				next_byte2 < 0x80 or next_byte2 > 0xBF or
				next_byte3 < 0x80 or next_byte3 > 0xBF then
				return false
			end
			
			i = i + 3
		else
			return false
		end

		i = i + 1
	end

	return true
end

Here’s my test sample.

local function is_valid_utf8(input)
	local i = 1

	while i <= #input do
		local c = string.byte(input, i)

		if c >= 0x00 and c <= 0x7F then
			--// Fallthrough
		elseif c >= 0xC2 and c <= 0xDF then
			local next_byte = string.byte(input, i + 1)
			
			if not next_byte or next_byte < 0x80 or next_byte > 0xBF then
				return false
			end
			
			i = i + 1
		elseif c == 0xE0 then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			
			if not next_byte1 or not next_byte2 or
				next_byte1 < 0xA0 or next_byte1 > 0xBF or
				next_byte2 < 0x80 or next_byte2 > 0xBF then
				return false
			end
			
			i = i + 2
		elseif c >= 0xE1 and c <= 0xEC then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			
			if not next_byte1 or not next_byte2 or
				next_byte1 < 0x80 or next_byte1 > 0xBF or
				next_byte2 < 0x80 or next_byte2 > 0xBF then
				return false
			end
			
			i = i + 2
		elseif c == 0xED then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			
			if not next_byte1 or not next_byte2 or
				next_byte1 < 0x80 or next_byte1 > 0x9F or
				next_byte2 < 0x80 or next_byte2 > 0xBF then
				return false
			end
			
			i = i + 2
		elseif c >= 0xEE and c <= 0xEF then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			
			if not next_byte1 or not next_byte2 or
				next_byte1 < 0x80 or next_byte1 > 0xBF or
				next_byte2 < 0x80 or next_byte2 > 0xBF then
				return false
			end
			
			i = i + 2
		elseif c == 0xF0 then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			local next_byte3 = string.byte(input, i + 3)
			
			if not next_byte1 or not next_byte2 or not next_byte3 or
				next_byte1 < 0x90 or next_byte1 > 0xBF or
				next_byte2 < 0x80 or next_byte2 > 0xBF or
				next_byte3 < 0x80 or next_byte3 > 0xBF then
				return false
			end
			
			i = i + 3
		elseif c >= 0xF1 and c <= 0xF3 then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			local next_byte3 = string.byte(input, i + 3)
			
			if not next_byte1 or not next_byte2 or not next_byte3 or
				next_byte1 < 0x80 or next_byte1 > 0xBF or
				next_byte2 < 0x80 or next_byte2 > 0xBF or
				next_byte3 < 0x80 or next_byte3 > 0xBF then
				return false
			end
			
			i = i + 3
		elseif c == 0xF4 then
			local next_byte1 = string.byte(input, i + 1)
			local next_byte2 = string.byte(input, i + 2)
			local next_byte3 = string.byte(input, i + 3)
			
			if not next_byte1 or not next_byte2 or not next_byte3 or
				next_byte1 < 0x80 or next_byte1 > 0x8F or
				next_byte2 < 0x80 or next_byte2 > 0xBF or
				next_byte3 < 0x80 or next_byte3 > 0xBF then
				return false
			end
			
			i = i + 3
		else
			return false
		end

		i = i + 1
	end

	return true
end

local function check_validity(input)
	local isValid = is_valid_utf8(input)
	local datastoreValidity = false 

	if isValid then 
		local s,e = pcall(function() game:GetService("DataStoreService"):GetDataStore("Sample"):SetAsync("Sample", input) end)

		if not s then 
			if e:find("maximum queue size") then
				datastoreValidity = "Data store limit reached"
			end
		else 
			datastoreValidity = true
		end
	end

	print(`Input: {input:gsub("%s", " ")} Validity: {isValid} Data store validity: {datastoreValidity}`)
end

-- Basic
check_validity("Hello World")
check_validity("\237\190\140")
check_validity("\ED\xA0\x80\xED\xB0\x80")
check_validity("Hello World\255")
check_validity("\237\190\140Hello World\237\190\140")
check_validity("\ED\xA0\x80\xED\xB0\x80Hello World\xED\xA0\x80\xED\xB0\x80")
-- Borrowed word sample
check_validity("ï")
check_validity("Hello Worldï")
-- Hiragana sample
check_validity("女")
check_validity("Hello World女")
-- Chinese sample
check_validity("我")
check_validity("Hello World我")
-- Surrogate
check_validity("\237\190\140Hello Worldï")
check_validity("\237\190\140Hello World女")
check_validity("\237\190\140Hello World我")
-- Surrogate
check_validity("\ED\xA0\x80\xED\xB0\x80Hello Worldï")
check_validity("\ED\xA0\x80\xED\xB0\x80Hello World女")
check_validity("\ED\xA0\x80\xED\xB0\x80Hello World我")
-- Emojis
check_validity("😀 😃 😄 😁 😆 😅 😂 🤣 🥲 🥹 ☺️ 😊 😇 🙂 🙃 😉 😌 😍 🥰 😘 😗 😙 😚 😋 😛 😝 😜 🤪 🤨 🧐 🤓 😎 🥸 🤩 🥳 😏 😒 😞 😔 😟 😕 🙁 ☹️ 😣 😖 😫 😩 🥺 😢 😭 😮‍💨 😤 😠 😡 🤬 🤯 😳 🥵 🥶 😱 😨 😰 😥 😓 🫣 🤗 🫡 🤔 🫢 🤭 🤫 🤥 😶 😶‍🌫️ 😐 😑 😬 🫨 🫠 🙄 😯 😦 😧 😮 😲 🥱 😴 🤤 😪 😵 😵‍💫 🫥 🤐 🥴 🤢 🤮 🤧 😷 🤒 🤕 🤑 🤠 😈 👿 👹 👺 🤡 💩 👻 💀 ☠️ 👽 👾 🤖 🎃 😺 😸 😹 😻 😼 😽 🙀 😿 😾 ")
-- Hiragana
check_validity([[あいうえおかきくけこさしすせそたちつてとなにぬねのはひふへほまみむめもやゆよらりるれろわをん]])
-- Katakana
check_validity([[アイウエオカキクケコサシスセソタチツテトナニヌネノハヒフヘホマミムメモヤユヨラリルレロワヲン]])
-- Russian
check_validity([[АБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯбвгдеёжзийклмнопрстуфхцчшщъыьэюя]])
-- Arabic 
-- abjadī
check_validity([[أبجدهوزحطيكلمنسعفصقرشتثخذضظغ]])
-- hijāʾī
check_validity([[ابتثجحخدذرزسشصضطظعغفقكلمنهوي]])
-- Greek
check_validity([[ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩαβγδεζηθικλμνξοπρστυφχψω]])
-- Egyptian hieroglyphics
check_validity([[𓀀 𓀁 𓀂 𓀃 𓀄 𓀅 𓀆 𓀇 𓀈 𓀉 𓀊 𓀋 𓀌 𓀍 𓀎 𓀏 𓀐 𓀑 𓀒 𓀓 𓀔 𓀕 𓀖 𓀗 𓀘 𓀙 𓀚 𓀛 𓀜 𓀝 𓀞 𓀟 𓀠 𓀡 𓀢 𓀣 𓀤 𓀥 𓀦 𓀧 𓀨 𓀩 𓀪 𓀫 𓀬 𓀭 𓀮 𓀯 𓀰 𓀱 𓀲 𓀳 𓀴 𓀵 𓀶 𓀷 𓀸 𓀹 𓀺 𓀻 𓀼 𓀽 𓀾 𓀿 𓁀 𓁁 𓁂 𓁃 𓁄 𓁅 𓁆 𓁇 𓁈 𓁉 𓁊 𓁋 𓁌 𓁍 𓁎 𓁏 𓁐 𓁑 𓁒 𓁓 𓁔 𓁕 𓁖 𓁗 𓁘 𓁙 𓁚 𓁛 𓁜 𓁝 𓁞 𓁟 𓁠 𓁡 𓁢 𓁣 𓁤 𓁥 𓁦 𓁧 𓁨 𓁩 𓁪 𓁫 𓁬 𓁭 𓁮 𓁯 𓁰 𓁱 𓁲 𓁳 𓁴 𓁵 𓁶 𓁷 𓁸 𓁹 𓁺 𓁻 𓁼 𓁽 𓁾 𓁿 𓂀 𓂁 𓂂 𓂃 𓂄 𓂅 𓂆 𓂇 𓂈 𓂉 𓂊 𓂋 𓂌 𓂍 𓂎 𓂏 𓂐 𓂑 𓂒 𓂓 𓂔 𓂕 𓂖 𓂗 𓂘 𓂙 𓂚 𓂛 𓂜 𓂝 𓂞 𓂟 𓂠 𓂡 𓂢 𓂣 𓂤 𓂥 𓂦 𓂧 𓂨 𓂩 𓂪 𓂫 𓂬 𓂭 𓂮 𓂯 𓂰 𓂱 𓂲 𓂳 𓂴 𓂵 𓂶 𓂷 𓂸 𓂹 𓂺 𓂻 𓂼 𓂽 𓂾 𓂿 𓃀 𓃁 𓃂 𓃃 𓃄 𓃅 𓃆 𓃇 𓃈 𓃉 𓃊 𓃋 𓃌 𓃍 𓃎 𓃏 𓃐 𓃑 𓃒 𓃓 𓃔 𓃕 𓃖 𓃗 𓃘 𓃙 𓃚 𓃛 𓃜 𓃝 𓃞 𓃟 𓃠 𓃡 𓃢 𓃣 𓃤 𓃥 𓃦 𓃧 𓃨 𓃩 𓃪 𓃫 𓃬 𓃭 𓃮 𓃯 𓃰 𓃱 𓃲 𓃳 𓃴 𓃵 𓃶 𓃷 𓃸 𓃹 𓃺 𓃻 𓃼 𓃽 𓃾 𓃿 𓄀 𓄁 𓄂 𓄃 𓄄 𓄅 𓄆 𓄇 𓄈 𓄉 𓄊 𓄋 𓄌 𓄍 𓄎 𓄏 𓄐 𓄑 𓄒 𓄓 𓄔 𓄕 𓄖 𓄗 𓄘 𓄙 𓄚 𓄛 𓄜 𓄝 𓄞 𓄟 𓄠 𓄡 𓄢 𓄣 𓄤 𓄥 𓄦 𓄧 𓄨 𓄩 𓄪 𓄫 𓄬 𓄭 𓄮 𓄯 𓄰 𓄱 𓄲 𓄳 𓄴 𓄵 𓄶 𓄷 𓄸 𓄹 𓄺 𓄻 𓄼 𓄽 𓄾 𓄿 𓅀 𓅁 𓅂 𓅃 𓅄 𓅅 𓅆 𓅇 𓅈 𓅉 𓅊 𓅋 𓅌 𓅍 𓅎 𓅏 𓅐 𓅑 𓅒 𓅓 𓅔 𓅕 𓅖 𓅗 𓅘 𓅙 𓅚 𓅛 𓅜 𓅝 𓅞 𓅟 𓅠 𓅡 𓅢 𓅣 𓅤 𓅥 𓅦 𓅧 𓅨 𓅩 𓅪 𓅫 𓅬 𓅭 𓅮 𓅯 𓅰 𓅱 𓅲 𓅳 𓅴 𓅵 𓅶 𓅷 𓅸 𓅹 𓅺 𓅻 𓅼 𓅽 𓅾 𓅿 𓆀 𓆁 𓆂 𓆃 𓆄 𓆅 𓆆 𓆇 𓆈 𓆉 𓆊 𓆋 𓆌 𓆍 𓆎 𓆏 𓆐 𓆑 𓆒 𓆓 𓆔 𓆕 𓆖 𓆗 𓆘 𓆙 𓆚 𓆛 𓆜 𓆝 𓆞 𓆟 𓆠 𓆡 𓆢 𓆣 𓆤 𓆥 𓆦 𓆧 𓆨 𓆩 𓆪 𓆫 𓆬 𓆭 𓆮 𓆯 𓆰 𓆱 𓆲 𓆳 𓆴 𓆵 𓆶 𓆷 𓆸 𓆹 𓆺 𓆻 𓆼 𓆽 𓆾 𓆿 𓇀 𓇁 𓇂 𓇃 𓇄 𓇅 𓇆 𓇇 𓇈 𓇉 𓇊 𓇋 𓇌 𓇍 𓇎 𓇏 𓇐 𓇑 𓇒 𓇓 𓇔 𓇕 𓇖 𓇗 𓇘 𓇙 𓇚 𓇛 𓇜 𓇝 𓇞 𓇟 𓇠 𓇡 𓇢 𓇣 𓇤 𓇥 𓇦 𓇧 𓇨 𓇩 𓇪 𓇫 𓇬 𓇭 𓇮 𓇯 𓇰 𓇱 𓇲 𓇳 𓇴 𓇵 𓇶 𓇷 𓇸 𓇹 𓇺 𓇻 𓇼 𓇽 𓇾 𓇿 𓈀 𓈁 𓈂 𓈃 𓈄 𓈅 𓈆 𓈇 𓈈 𓈉 𓈊 𓈋 𓈌 𓈍 𓈎 𓈏 𓈐 𓈑 𓈒 𓈓 𓈔 𓈕 𓈖 𓈗 𓈘 𓈙 𓈚 𓈛 𓈜 𓈝 𓈞 𓈟 𓈠 𓈡 𓈢 𓈣 𓈤 𓈥 𓈦 𓈧 𓈨 𓈩 𓈪 𓈫 𓈬 𓈭 𓈮 𓈯 𓈰 𓈱 𓈲 𓈳 𓈴 𓈵 𓈶 𓈷 𓈸 𓈹 𓈺 𓈻 𓈼 𓈽 𓈾 𓈿 𓉀 𓉁 𓉂 𓉃 𓉄 𓉅 𓉆 𓉇 𓉈 𓉉 𓉊 𓉋 𓉌 𓉍 𓉎 𓉏 𓉐 𓉑 𓉒 𓉓 𓉔 𓉕 𓉖 𓉗 𓉘 𓉙 𓉚 𓉛 𓉜 𓉝 𓉞 𓉟 𓉠 𓉡 𓉢 𓉣 𓉤 𓉥 𓉦 𓉧 𓉨 𓉩 𓉪 𓉫 𓉬 𓉭 𓉮 𓉯 𓉰 𓉱 𓉲 𓉳 𓉴 𓉵 𓉶 𓉷 𓉸 𓉹 𓉺 𓉻 𓉼 𓉽 𓉾 𓉿 𓊀 𓊁 𓊂 𓊃 𓊄 𓊅 𓊆 𓊇 𓊈 𓊉 𓊊 𓊋 𓊌 𓊍 𓊎 𓊏 𓊐 𓊑 𓊒 𓊓 𓊔 𓊕 𓊖 𓊗 𓊘 𓊙 𓊚 𓊛 𓊜 𓊝 𓊞 𓊟 𓊠 𓊡 𓊢 𓊣 𓊤 𓊥 𓊦 𓊧 𓊨 𓊩 𓊪 𓊫 𓊬 𓊭 𓊮 𓊯 𓊰 𓊱 𓊲 𓊳 𓊴 𓊵 𓊶 𓊷 𓊸 𓊹 𓊺 𓊻 𓊼 𓊽 𓊾 𓊿 𓋀 𓋁 𓋂 𓋃 𓋄 𓋅 𓋆 𓋇 𓋈 𓋉 𓋊 𓋋 𓋌 𓋍 𓋎 𓋏 𓋐 𓋑 𓋒 𓋓 𓋔 𓋕 𓋖 𓋗 𓋘 𓋙 𓋚 𓋛 𓋜 𓋝 𓋞 𓋟 𓋠 𓋡 𓋢 𓋣 𓋤 𓋥 𓋦 𓋧 𓋨 𓋩 𓋪 𓋫 𓋬 𓋭 𓋮 𓋯 𓋰 𓋱 𓋲 𓋳 𓋴 𓋵 𓋶 𓋷 𓋸 𓋹 𓋺 𓋻 𓋼 𓋽 𓋾 𓋿 𓌀 𓌁 𓌂 𓌃 𓌄 𓌅 𓌆 𓌇 𓌈 𓌉 𓌊 𓌋 𓌌 𓌍 𓌎 𓌏 𓌐 𓌑 𓌒 𓌓 𓌔 𓌕 𓌖 𓌗 𓌘 𓌙 𓌚 𓌛 𓌜 𓌝 𓌞 𓌟 𓌠 𓌡 𓌢 𓌣 𓌤 𓌥 𓌦 𓌧 𓌨 𓌩 𓌪 𓌫 𓌬 𓌭 𓌮 𓌯 𓌰 𓌱 𓌲 𓌳 𓌴 𓌵 𓌶 𓌷 𓌸 𓌹 𓌺 𓌻 𓌼 𓌽 𓌾 𓌿 𓍀 𓍁 𓍂 𓍃 𓍄 𓍅 𓍆 𓍇 𓍈 𓍉 𓍊 𓍋 𓍌 𓍍 𓍎 𓍏 𓍐 𓍑 𓍒 𓍓 𓍔 𓍕 𓍖 𓍗 𓍘 𓍙 𓍚 𓍛 𓍜 𓍝 𓍞 𓍟 𓍠 𓍡 𓍢 𓍣 𓍤 𓍥 𓍦 𓍧 𓍨 𓍩 𓍪 𓍫 𓍬 𓍭 𓍮 𓍯 𓍰 𓍱 𓍲 𓍳 𓍴 𓍵 𓍶 𓍷 𓍸 𓍹 𓍺 𓍻 𓍼 𓍽 𓍾 𓍿 𓎀 𓎁 𓎂 𓎃 𓎄 𓎅 𓎆 𓎇 𓎈 𓎉 𓎊 𓎋 𓎌 𓎍 𓎎 𓎏 𓎐 𓎑 𓎒 𓎓 𓎔 𓎕 𓎖 𓎗 𓎘 𓎙 𓎚 𓎛 𓎜 𓎝 𓎞 𓎟 𓎠 𓎡 𓎢 𓎣 𓎤 𓎥 𓎦 𓎧 𓎨 𓎩 𓎪 𓎫 𓎬 𓎭 𓎮 𓎯 𓎰 𓎱 𓎲 𓎳 𓎴 𓎵 𓎶 𓎷 𓎸 𓎹 𓎺 𓎻 𓎼 𓎽 𓎾 𓎿 𓏀 𓏁 𓏂 𓏃 𓏄 𓏅 𓏆 𓏇 𓏈 𓏉 𓏊 𓏋 𓏌 𓏍 𓏎 𓏏 𓏐 𓏑 𓏒 𓏓 𓏔 𓏕 𓏖 𓏗 𓏘 𓏙 𓏚 𓏛 𓏜 𓏝 𓏞 𓏟 𓏠 𓏡 𓏢 𓏣 𓏤 𓏥 𓏦 𓏧 𓏨 𓏩 𓏪 𓏫 𓏬 𓏭 𓏮 𓏯 𓏰 𓏱 𓏲 𓏳 𓏴 𓏵 𓏶 𓏷 𓏸 𓏹 𓏺 𓏻 𓏼 𓏽 𓏾 𓏿 𓐀 𓐁 𓐂 𓐃 𓐄 𓐅 𓐆 𓐇 𓐈 𓐉 𓐊 𓐋 𓐌 𓐍 𓐎 𓐏 𓐐 𓐑 𓐒 𓐓 𓐔 𓐕 𓐖 𓐗 𓐘 𓐙 𓐚 𓐛 𓐜 𓐝 𓐞 𓐟]])

-- All characters
for i = 1, 255 do 
	check_validity(string.char(i))
end

local function verify_json(str: string): boolean
	return (pcall(game:GetService("HttpService").JSONEncode, game:GetService("HttpService"), str))
end

local function test(CHAR_COUNT: number): ()
	local total: number = (2^(CHAR_COUNT * 8)) - 1
	for n: number = 0, total do
		if n % 100000 == 0 then
			print(n / total)
			task.wait()
		end

		local chars: {string} = {}
		for field: number = 0, (CHAR_COUNT * 8) - 1, 8 do
			table.insert(chars, string.char(bit32.extract(n, field, 8)))
		end

		local testString: string = table.concat(chars)
		if is_valid_utf8(testString) ~= verify_json(testString) then
			print("DISCREPENCY:", string.byte(testString, 1, #testString))
			check_validity(testString)
		end
	end
end

test(3)

print("Done")

[holderman_1206]

Did you know that you can check the validity of a utf8 string with utf8.len()?

Edit: Should’ve read the entire post before saying anything. Sorry for bothering you.

[EtSapientisMagna (Acedia)]

No you cannot and this has been thorougly discussed on discord with @zeuxcg. If you had read the feature request you’d have noticed that it mentions utf8.len is not a proper solution and I had updated my github gist (the place where you got utf8.len being a fix from!) with both the new fix and added notices that they’re not good fixes.

Try to pass \237\190\140, \255 and Hello World to utf8.len and you’ll notice very quickly it is not a proper solution.

[EtSapientisMagna (Acedia)]

Ping, would love to get some kind of update on this.

[lambarini]

Bump, would love to get some kind of update on this x2

[funwolf7]

It should be mentioned that the utf8 library was updated a while ago to fix the issue with surrogates; utf8.len can now be used to validate UTF-8 strings. The security tactics page has also been updated to show this.

There could still be merit in a function to validate that a table is acceptable in a DataStore entry, but at least the most complicated thing to validate (strings) now has a proper function for it.

[EtSapientisMagna (Acedia)]

Referencing the screenshot sent above data store service has other checks aside from just invalid strings. UTF-8 strings can be checked but that doesn’t mean there shouldn’t be a dedicated method to verify if something can be saved.

Getting any kind of update one this would be good.

[roalex2008 (Dottik)]

It is absurd Roblox hasn’t even tried addressing this, it can potentially ruin complete games’ internal economy, it is as if Roblox themselves left an endpoint to add Robux , at this point the only brute fix i can think of is to use buffers, from what I know, they can be saved (buffers themselves), and even if they contain gibberish, I always questioned myself why Roblox made the worst documentation for the topic, and even if it improved, it doesn’t provide developers with an easy method to know if something can be saved, which this feature request addresses perfectly fine, and I don’t understand how complicated this could be, being that it could be simple regex on the data, and they could (potentially) just Ctrl + C && Ctrl + V the code they use when they validate it to save on the server once it is sent.

[PysephDEV (Pyseph)]

Bump! very critical for beginner developers, and also experienced developers who may not be familiar with this issue and its consequences.

[metatablecatmaid (metatablecat)]

utf8 and its consequences on modern society, still surprised this hasn’t been adressed given its a major security issue.