Super Efficient DataStorage

[Magus_ArtStudios]

Before I figured this out I was creating individual values for a lot of things. But I wanted a solution to storing the data of my randomly generated NPCs very efficiently. All the data for these npc is stored in over 35 different value instances. It was based off my character customization system that uses all the native R15 character features for customization such as scaling ,clothing and interchangeable body parts.

I don’t know if you have ever thought about doing something similar for data storage but being able to store table data in a string is pretty useful. I just discovered it moments ago

-- Define a function to convert a folder of values into a table
local function FolderToTable(folder)
    local table = {}
    for _, value in pairs(folder:GetChildren()) do
        table[value.Name] = value.Value
    end
    return table
end

-- Define a function to convert a table into a string
local function TableToString(table)
    local string = ""
    for key, value in pairs(table) do
        string = string .. key .. ":" .. tostring(value) .. ";"
    end
    return string
end

-- Define a function to convert the string back into a table
local function StringToTable(string)
    local table = {}
    for key, value in string:gmatch("(.-):(.-);") do
        table[key] = tonumber(value) or value
    end
    return table
end

[FireStrykerAzul]

I don’t really get why it’s useful please can you explain ?

[Magus_ArtStudios]

That is a good question. For my application it stores the character data of the npc in one object by essentially compressing it from over 70 objects into 1 string then it interprets the string and reconstructs the data.
This encrypts your data with this algorithm as well. forcing any would be exploiter to have to print the table and reverse engineer it. You could take this further and even use a variation of this algorithm to encrypt the data with a algorithm that replaces letters based on a random seed. You can then store the seed in a seperate value and require that specific seed to decrypt the string making it totally impractical to ever try to exploit your game.

But generally I just think it is a good resource, it kind of blew my mind a bit and I thought it was pretty exciting because I have a random quest generator but the problem was that the randomly generated characters would be gone when the player left. Resulting in the quests not working with them very well. So this is a very efficient way of storing the data of many characters.Replicating thousands of string value objects can be resource intensive when scaled up. But with this I can store hundreds of different characters.

Also Datastores come in the form of tables.that means with this you could potentially store your entire games data into a single object, which is nothing new but something that should be known and considered

[KrimsonWoIf]

why not store the data in a dictionary on the server?

[Magus_ArtStudios]

I did not know about that! But this is for a recruitment system kind of similar to pokemon in a way. The characters are randomly generated so you do quests for them and then if you have done enough they will join your party. So their would be a lot of unused characters But what is it about storing dictionaries on the server?
For example Dragon Selene there was unique and will likely never be seen again

[KrimsonWoIf]

Storing it in a dictonary means that there is no actual object that contains it but its stored directly in the code. This method is guaranteed to never have data loss because the game doesnt have to retrieve the data when the player is leaving the game since it already has it in the code.

For example instead of getting the coins when they leave you game you store the coins value everytime it gets changed and then you save that stored value when they leave. A dictonary is a table that can contain multiple tables in it.

for example

local dictonary = {

	["Players"] = {
		
		["User1"] = {
			["Coins"] = 0,
			["Hats"] = {}
		},
		
		["User2"] = {
			["Coins"] = 100,
			["Hats"] = {"Hat1", "Hat2"}
		}
	},
	
}

dictonaries are also faster since they arent organized so like its quicker for data retrieval or some techniocal thing. Its not some advanced datastore concept its basic scripting.

I am not good at explaining things so here:

CHATGPT:
In the context of Roblox Lua, dictionaries are implemented using tables, which are a fundamental data structure in the language. Tables in Roblox Lua serve as associative arrays, meaning they can store key-value pairs. These key-value pairs are flexible and can contain any type of data, including other tables, allowing for nested and structured data.

Now, let’s compare using dictionaries (tables) directly in a data storing script with the alternative approach of storing all data in one string encrypted in an IntValue:

1. Using Dictionaries (Tables) in Data Storing Script:

In this approach, you can use dictionaries (tables) to organize and store player data, such as coins, hats, and any other relevant information. Dictionaries provide fast and efficient data retrieval based on the keys, and they are designed for quick access to specific data points.

Advantages:

• Data Organization: Dictionaries allow you to have structured data, making it easier to manage player information. You can store coins, hats, and other data associated with each player separately, which improves readability and maintainability.
• Efficient Data Retrieval: Accessing data using dictionary keys is generally faster than searching through a string or parsing encrypted data.
• No Data Loss: Since the data is stored directly in memory, there is no risk of data loss due to slow retrieval.

2. Storing Data as an Encrypted String in IntValue:

In this approach, you could potentially store all player data as a single encrypted string within an IntValue object. The string would need to be encrypted to ensure data security and integrity.

Challenges and Drawbacks:

• Data Parsing: Storing all data in a single encrypted string means that you’ll need to parse and extract specific information when needed. This could introduce additional complexity and slower data retrieval compared to using dictionaries directly.
• Data Updates: When modifying specific data points (e.g., updating a player’s coins), you would need to decrypt the entire string, make changes, and then re-encrypt the string, which could be resource-intensive and slow.
• Limited Structure: Storing data as a single string might lead to a less structured and organized approach. As the data grows or becomes more complex, maintaining and managing it becomes more challenging.

In conclusion, using dictionaries (tables) directly in a data storing script is generally more efficient and practical for managing player data in Roblox Lua. Dictionaries allow you to organize data in a structured manner and provide fast and direct access to specific data points. This approach is more maintainable, readable, and flexible compared to storing all data as a single encrypted string in an IntValue. Additionally, since the data is stored in memory, there is no need for slow data retrieval, reducing the risk of data loss or inefficiencies.

[corehimself (core)]

That’s actually a great point to address. I’ve yet to think of compressing it into one object and decoding it. It calls for more optimized performance.

Kudos for leaving that tip in there.

[Magus_ArtStudios]

I see what you’re getting at! you’re talking about storing the players data not within the player object . so in the case of the player object disappearing before saving has executed their would be potetial data loss. Also yes it would be fine to store the values in a dictionary. but you cannot write an array to a string value object . so if you do want to put the table on an object instance of a string value you would have to convert the table to a string.

local function FolderToTable(folder)
    local table = {}
    for _, value in pairs(folder:GetChildren()) do
        table[value.Name] = value.Value
    end
    return table
end

[KrimsonWoIf]

I appologize but why is nobody here even decent at scripting

[KrimsonWoIf]

I didnt say anything about that. my solution is not using 3rd part objects such as folders for data saving.

please read this

Intro to Dictionaries | Documentation - Roblox Creator Hub

it is a table in your game with multiple tables in it essentially

[corehimself (core)]

1. Refrain from using ChatGPT on DevForums as it may mislead information.
2. They both have their pros and cons within each other, at the end of the day it’s self-preferenced. Keep me out of it, I could care less.

I’ve seen a method I’ve yet to try and admired it, don’t target me for your idiocy.

[Mokiros]

Ever heard of JSON?

HttpService | Documentation - Roblox Creator Hub

DataStore uses it to save objects by the way.

[Magus_ArtStudios]

Yes those are points. But converting a string to a table is not a very resource intensive process. These set of functions convert a folder of values into a a table like the dictionary you wrote. then it converts the dictionary to a string to be saved as a string value.
I’m using this to compress a file. save it, then load it. because if I want to save a characters data based on an architecture of over 70 different values then scaling that up and storing all those individual values would be too many objects to replicate.

[KrimsonWoIf]

I do not see the purpose of converting strings to a table. It seems silly and noob like as well as unorganized and difficult to access.

In a situation where you are storing a lot of data you should do a method like this:

local PlayerData = {}
local ds = game:GetService("DataStoreService")
local coinService = ds:GetDataStore("CoinService")

game.Players.PlayerAdded:Connect(function(plr)
	
	local coins = coinService:GetAsync(plr.UserId)
	if not coins then
		coins = 0
	end
	
	plr:SetAttribute("Coins", coins)
	
	table.insert(PlayerData,
		{
			["UserId"] = plr.UserId,
			["Coins"] = coins
		}
	)
	
	plr:GetAttributeChangedSignal("Coins"):Connect(function()
		for i,v in pairs(PlayerData) do
			if v.UserId == plr.UserId then
				v.Coins = plr:GetAttribute("Coins")
			end
		end
	end)
	
end)

game.Players.PlayerRemoving:Connect(function(plr)
	
	for i,v in pairs(PlayerData) do
		if v.UserId == plr.UserId then
			coinService:SetAsync(plr.UserId, v.Coins)
		end
	end
	
end)

In my example I stored coins in the dictonary. This is an example and not an actual representation since the code is designed in cases where there is a lot of data since it wouldnt have to retrieve all the data.

[Magus_ArtStudios]

That’s quite okay. It’s not that deep.
I’m not creating thousands of new instances of an a string value for the purpose of storing player data into an array. I’m storing one value and desconstructing and interpreting the value for my algorithm so it can recreate the saved character

[KrimsonWoIf]

you should not be creating any objects to store data.

[Magus_ArtStudios]

You’re using attributes and loading the attribute data from the datastore array.
You do you buddy. I’m a very experienced programmer.

[KrimsonWoIf]

here is chatGpts response on it

Storing all of the data into one encrypted string on a StringValue object is not necessarily “silly” or “noobish,” but it may not be the most efficient or practical approach, especially when dealing with a lot of data or sensitive information. Here are some considerations to help you understand the trade-offs:

Advantages:

Simplicity: This approach might be easier to implement for developers who are new to more complex data storage mechanisms.

Single Retrieval: Storing all data in one string means you can retrieve all the player’s data with a single call, simplifying the retrieval process.

Disadvantages:

Data Parsing: Storing all data in a single encrypted string requires parsing the data whenever you need to access specific information. This can be more complex and slower compared to directly accessing a well-structured data format like a dictionary or table.

Data Updates: When updating specific data points, you need to decrypt the entire string, make changes, and then re-encrypt it. This can be resource-intensive and slow, especially if you’re dealing with large amounts of data.

Limited Structure: Storing all data in a single string might lead to a less organized and maintainable approach. As the data grows or becomes more complex, managing and maintaining it becomes more challenging.

Data Integrity: With all data in one string, there’s a higher risk of accidental data corruption or errors during encryption and decryption processes.

Security Considerations:

Storing sensitive data (e.g., player scores, in-game currency) in an encrypted format is essential for data security. However, the encryption and decryption process itself need to be secure to prevent unauthorized access to the data. Implementing strong encryption is a non-trivial task, and it’s best to rely on established encryption libraries or modules for this purpose.

Alternative Approaches:

If you have a significant amount of data or require a more scalable and efficient solution, consider the following:

Dictionaries or Tables: Use dictionaries or tables (as shown in previous examples) to organize and store player data. This provides a more structured approach and allows for direct access to specific data points.

In conclusion, the decision on how to store data ultimately depends on the complexity of your game, the amount of data to be managed, and the need for data integrity and security. While the StringValue approach may work for simple scenarios, more structured data storage solutions like dictionaries, DataStoreService, or databases are generally preferred for larger and more complex games.

[KrimsonWoIf]

you said that robotically like you used a code explainer or soemthing

[Magus_ArtStudios]

Yes now you are just being redundant and the AI has basically reiterated my points I have already made.
Also Yes we store our dictionary data points in the data store array.
Yes this method may add an extra layer of security to your game.
Yes this increases the complexity of accessing the data points. Yes their are a few applications for these functions. Are they useful yes.