Checking for additions/removals in a data store table

NinjoOnline · December 29, 2018, 2:09am

So here is the default table players start off with, which gets JSONEncoded, datastored, etc.

local data = {
	Classes = {
		['Knight'] = {
			['ID'] = 1,
			['Owned'] = true,
			['Weapons'] = {
				'Classic Sword',
			},
			['Armours'] = {
				'Knight Armour'
			},
			['Trails'] = {
				'None'
			},
		},
		['Archer'] = {
			['ID'] = 2,
			['Owned'] = true,
			['Weapons'] = {
				'Bow and Arrow'
			},
			['Armours'] = {
				'Archer Armour'
			},
			['Trails'] = {
				'None'
			},
		},
	},
	
	ClassEquips = {
		EquippedKnight = {
			Weapon = 'Classic Sword', 
			Armour = 'Knight Armour', 
			Trail = 'None'
		},
		EquippedArcher = {
			Weapon = 'Bow and Arrow', 
			Armour = 'Archer Armour', 
			Trail = 'None'
		},
	},
	
	EquippedClass = 'Knight',
	
	Level = 1,
	Exp = 0,
	Gold = 100,
	Gems = 0
}

Then loading it in:

function dataManager:GetData(player)
	local loadJSON = playerDataStore:GetAsync(player.UserId)
	local data = (loadJSON and httpService:JSONDecode(loadJSON)) or data
	
    playersData[player.UserId] = data
end

Now this is all fine for new players, but if a player loads in the for the first time, they’ll the data in the table above given to them, however, if I add more stuff to that data table, when they return they won’t have it, as it will just load in their old one. For example, adding another class:

local data = {
	Classes = {
		['Knight'] = {
			['ID'] = 1,
			['Owned'] = true,
			['Weapons'] = {
				'Classic Sword',
			},
			['Armours'] = {
				'Knight Armour'
			},
			['Trails'] = {
				'None'
			},
		},
		['Archer'] = {
			['ID'] = 2,
			['Owned'] = true,
			['Weapons'] = {
				'Bow and Arrow'
			},
			['Armours'] = {
				'Archer Armour'
			},
			['Trails'] = {
				'None'
			},
		},
		['Scout'] = {
			['ID'] = 3,
			['Owned'] = false,
			['Weapons'] = {
				'Dagger'
			},
			['Armours'] = {
				'Scout Armour'
			},
			['Trails'] = {
				'None'
			},
		},
	},
	
	ClassEquips = {
		EquippedKnight = {
			Weapon = 'Classic Sword', 
			Armour = 'Knight Armour', 
			Trail = 'None'
		},
		EquippedArcher = {
			Weapon = 'Bow and Arrow', 
			Armour = 'Archer Armour', 
			Trail = 'None'
		},
		EquippedScout = {
			Weapon = 'Dagger', 
			Armour = 'Scout Armour', 
			Trail = 'None'
		},
	},
	
	EquippedClass = 'Knight',
	
	Level = 1,
	Exp = 0,
	Gold = 100,
	Gems = 0
}

I know I could just change the datastore to be a different name, but that would wipe out the players original data. I want when a player who already has a table saved, the game to check this table, and if there are any additions to it, add it to the players table.

colbert2677 · December 29, 2018, 3:01am

You can’t, really. You can throw in a nil or default value to the player’s data table, but that’s as much as I know. It worked fine for me when I added or removed data values for an old project.

For additions: I establish a new variable and blank it out to a default value. If it needs to be based upon something, then I perform those functions and then add it to their data table.
For removals: I don’t create the variable for the value I want to remove, or blank it out in the data table. Upon saving, this value doesn’t get included. No part of any script uses that value either. It just naturally gets wiped.

Obviously an unclean method and leads to data having inconsistencies, but it’s far better than straight-wiping everyone’s data to make small changes or doing something that results in exhausting my request budgets. I don’t see OnUpdate working with this either; it doesn’t properly function, there’s a possibility that it may be deprecated (and removed - I can’t find the post stating this) and you still have to perform a way to apply the data structure updates to everyone.

Answer? Don’t check. Phase things in and/or out naturally.

NinjoOnline · December 29, 2018, 3:06am

A little confused…? Cause I mean chances are in the future I will come up with more classes over time, so I can’t really just create them all now and leave them set to nil or default stats, because I am 100% gonna be adding in more when more come to mind

colbert2677 · December 29, 2018, 3:20am

When you add new classes, does anything about those classes need to be based upon any data that a player might already have? If they don’t need to be based on anything, then you can create new tables with defaults in player data files on new servers. You’re better off tailoring your data store in such a way that allows you to phase in features, rather than trying to hard code them.

Normally if I need things added, what I do is I have a prefab of what a player’s data structure should look like. This can be a physical folder or a module or whatever you want. When a new player enters, this prefab is duplicated and assigned to the player, then their data is queried.

If there is an error, block all save attempts.
If there is data and the prefab has a key that matches a key within the fetched data table, fill it in. If the key wasn’t fetched, then it’ll use the default value from the prefab.
If you want to remove something, don’t include a variable for it in the prefab. When loading and there is no variable matching a fetched key, skip loading it into player session data. It disappears from this point.
If saving requests aren’t blocked, save this to a player’s storage data. It’ll automatically include any new value.

>> V1
DataPrefab: {Level = 1}
-> Fetched: {Level = 1}
-> For each key, fill it in on the table
-> SessionData: {Level = 1}
-> Save

>> V2
DataPrefab {Level = 1, ClassALevel = 2, ClassBLevel = 3}
-> Fetched: {Level = 1}
-> For each key, fill it in on the table
-> SessionData: {Level = 1, ClassALevel = 2, ClassBLevel = 3}

>> V3
DataPrefab {Level = 1, ClassALevel = 2}
-> Fetched: {Level = 1, ClassALevel = 2, ClassBLevel = 3}
-> For each key, fill it in on the table
-> SessionData: {Level = 1, ClassALevel = 2}

I might’ve, uh… made that a little more complicated to understand. Can you follow?

NinjoOnline · December 29, 2018, 3:23am

Umm kinda I think??

Guessing DataPrefab is what’s stored in the default table, that anybody without data saved is going to be given.

Now in V2, changes to the Prefab, it fetches the players old/saved data correct? Which is missing few values. But then not sure how to get it to fill.

Afraid filling it might fill in data they already have? Or overwrite their current data

colbert2677 · December 29, 2018, 3:38am

DataPrefab represents default data. It forms the player data structure appearance and what players without data are assigned. You get two birds with one stone: accounting for players who do not already possess any data for the game and those with.

Let’s use a container as an analogy so it’s easier to understand what I mean by phasing data in and out, rather than a technical explanation. I think it may be easier to follow along this way. A container will represent a prefab, while water represents player data.

Analogy:
Say you want to add a new class. You need a bigger container now so that your water has a place to stay. So you expand the container, but notice that you don’t add any water yet. That water doesn’t exist yet, it’s air. But you now have a bigger container, so you can fill that space with water. Now if you want to remove something, you need to make the container smaller. Now you have a smaller container, but more water than the container holds. What do you do? You clean up the overflowing water or ignore it. There’s no place to put the extra water.

Technical:
You have your data structure set up from the get go. You want to add a class. All you need to do is stick the class in with default values to the player tables as they enter new servers.

V1 Server Prefab: {Melee_LV1, Gunner_LV1}
When player enters, data = prefab, if they have data then reconfigure these values to what their data looks like

V2 Server Prefab: {Melee_LV1, Gunner_LV1, Archer_LV1}
Load data as normal. There’s now an archer class.Assign the prefab as the session data, fetch their data and then fill in the table. Everything gets loaded from V1 - their Melee and Gunner - but they have no data for Archer. So it stays level 1.

V3 Server Prefab: {Gunner_LV1, Archer_LVL1}
Melee class is now gone from the prefab. Load data as normal, they’ll still have the melee class in their fetched data. But, you’ve assigned the prefab as their session data. So when you’re filling in values, melee won’t have a variable, so your script skips over configuring any part of the table to what was saved as melee. Now when you save data and check it, it’ll be as if melee never existed in their data. It’s gone.

The keyword to my responses is “phase”. Phasing things and out. Instead of using saved player data for the session, give ALL players a prefab to use for the session (both new and returning players). The saved data should be configuring the prefab they were assigned to use saved values as opposed to being used in raw form. Basically, saved data acts as configuration to prefabs, rather than being used raw.

NinjoOnline · December 29, 2018, 3:44am

Ahhh ok I’m kinda understanding. So everyone always loads in with the prefab, but if they have presaved data, then that presaved data makes changes to the prefab which has been saved to the player?

No clue how to even being coding that though

colbert2677 · December 29, 2018, 3:57am

You got it exactly. Whether players have data or not, when they enter, assign them the prefab to use as the data for the session (server).

local sessiondata = {}
local lolprefab = {
	meleelv = 1,
	gunnerlv = 1,
}

game.Players.PlayerAdded:Connect(function (plr)
	-- i forgot how to copy tables
	sessiondata[plr.UserId] = {}
	for key, value in pairs(lolprefab) do
		sessiondata[plr.UserId][key] = value
	end
end)

After this, go ahead and get their data from a data store. Of course ignoring error handling for the purposes of this thread and assuming requests go through 100% of the time, if they have data, then go through it and use it as configuration. If they don’t have data, skip trying to set any kind of data and let them roll with this. They’ll use the prefab data, meaning they’re starting from the bottom as expected. If they have data, fill in keys only if those keys exist.

-- GetAsync returns {meleelv = 5, gunnerlv = 2} and is indexed as local variable "dat"

for pkey, pval in pairs(dat) do
	if sessiondata[plr.UserId][pkey] then -- check if key exists
		sessiondata[plr.UserId][pkey] = pval
	end
end

You see how I’m using save data as configuration as opposed to actually using the raw data that a player saves? Now it becomes really easy to add anything. If I add something:

local lolprefab = {
	meleelv = 1,
	gunnerlv = 1,
	archerlv = 1,
}

This becomes what every player starts with. Then we go back to the top and load their data with the same code. When a player’s data loads, they have the table {meleelv = 5, gunnerlv = 2} but Archer isn’t there. So the script will set their melee and gunner levels, but Archer will stay Level 1. So their session data now includes a defaulted Archer. When they save, Archer gets included.

Fast forward, scrap the melee class. Prefab becomes this:

local lolprefab = {
	gunnerlv = 1,
	archerlv = 1,
}

Again, every player that joins gets this prefab as their session data. Now a player that loads and has their data fetched will have a table that looks like this: {meleelv = 5, gunnerlv = 2, archerlv = 1}. See how melee is still in there? But again, we’re only using save data as configuration, not raw. So their table becomes:

sessiondata[userid] = {
	gunnerlv = 2,
	archerlv = 1,
}

There is no longer a meleelv key in the prefab. If you look at my setter code earlier, you’ll see that values are only set if a key exists. “meleelv” key exists in player data but not the prefab, so we do nothing with “meleelv”. Now when their data gets saved, they no longer have a melee level.

NinjoOnline · December 29, 2018, 4:00am

Would any of this change since I’m using JSON to encode all this? Since I’m saving a dictionary I was told to use JSON.

colbert2677 · December 29, 2018, 4:01am

Nope. The only thing that would change is that after fetching, you will need to decode that JSON and assign the decoded table to a variable (JSONDecode will return a table). I assume you know what to do for saving (just encode it and save the string returned).

For data, the only time you need JSON is when saving. When you actually need to use the data, it is far easier to work with a raw table.

NinjoOnline · December 29, 2018, 4:09am

Ok, not sure if I botched what I have, but idk, it looks like it’s working actually:

playersData[player.UserId] = {}
	for i, v in pairs(data) do
		playersData[player.UserId][i] = v
	end
	
	local loadJSON = playerDataStore:GetAsync(player.UserId)
	local setData = (loadJSON and httpService:JSONDecode(loadJSON)) or data
	
playersData[player.UserId] = setData

	for pkey, pval in pairs(data) do
		if playersData[player.UserId][pkey] then -- check if key exists
			playersData[player.UserId][pkey] = pval
		end
	end

This is what I got for when the player leaves (saving it)

local saveJSON = httpService:JSONEncode(playersData[player.UserId])
playerDataStore:SetAsync(player.UserId, saveJSON)

Idk, it looks to be working but people always manage to break it somehow

colbert2677 · December 29, 2018, 4:17am

local prefab = {--[[whatever you want]]}
playersData[player.UserId] = {}
for k, v in pairs(prefab) do
	playersData[player.UserId][k] = v
end

local loadJSON = playerDataStore:GetAsync(player.UserId)
local setData = (loadJSON and httpService:JSONDecode(loadJSON)) or nil

if setData then
	for key, value in pairs(setData) do
		if playersData[player.UserId][key] then
			playersData[player.UserId][key] = value
		end
	end
end

FTFY. Obviously don’t use raw, it would have to be configured in parts and fit into your current code.

Keep your saving code the same, that bit’s fine.

On a cautionary note, for tables, you might have to write a handler for that. Can be done by using type() against value and if it’s table, then iterating through it and setting values.

if type(value) == "table" then
	for k2, v2 in pairs(value) do
		playersData[player.UserId][key][k2] = v2
	end
end

Or something. I used value objects the last time I made a saving system like this, so I used FindFirstChild with the recursive argument as true. No key ever shared a name with another, so if I needed two keys with the same value, I’d specify it with a prefix (e.g. ALevel and BLevel, not Level and Level).

NinjoOnline · December 29, 2018, 4:19am

Actually think I just found a problem. It’s not saving their data, it’s just getting the data prefab. For example what I’ve got:

playersData[player.UserId] = {}
	for i, v in pairs(data) do
		playersData[player.UserId][i] = v
	end
	
	local loadJSON = playerDataStore:GetAsync(player.UserId)
	local setData = (loadJSON and httpService:JSONDecode(loadJSON)) or data
	
playersData[player.UserId] = setData

	for pkey, pval in pairs(data) do
		if playersData[player.UserId][pkey] then -- check if key exists
			playersData[player.UserId][pkey] = pval
		end
	end

	local user = playersData[player.UserId]

    print(user.Gold)

    if player.Name == 'NinjoOnline' then
        user.Gold = 500
    end
	
	print(user.Gold)

Prints 100 (base amount of gold given in the prefab) and then after prints 500, as it’s given me 500. But when I leave, remove the code about giving me 500 gold and just have it print my gold, it just prints 100.

colbert2677 · December 29, 2018, 4:20am

Read former reply and try again. LMK if that doesn’t work, I’ll create a repro for you.

NinjoOnline · December 29, 2018, 4:23am

Looks to be working saves the gold atleast

Will probably just do a more testing, making sure it’s saving everything, works with new and old players, etc

I’ll come back here if anything goes wrong Thank you!! Marked one of the responses as the solution

NinjoOnline · December 29, 2018, 4:36am

Far out, ok, normal data is saving, so like Gold, Gems, whatever. But now when I add in the new class to the list it ain’t showing up, returning nil when I print.

local data = {
	Classes = {
		['Knight'] = {
			['ID'] = 1,
			['Owned'] = true,
			['Weapons'] = {
				'Classic Sword',
			},
			['Armours'] = {
				'Knight Armour'
			},
			['Trails'] = {
				'None'
			},
		},
		['Archer'] = {
			['ID'] = 2,
			['Owned'] = true,
			['Weapons'] = {
				'Bow and Arrow'
			},
			['Armours'] = {
				'Archer Armour'
			},
			['Trails'] = {
				'None'
			},
		},
		['Scout'] = {
			['ID'] = 3,
			['Owned'] = false,
			['Weapons'] = {
				'Dagger'
			},
			['Armours'] = {
				'Scout Armour'
			},
			['Trails'] = {
				'None'
			},
		},
	},
},

(There is more data in this table, just kept the class section tho as that’s what’s not working)

So I’d test with just Knight and Archer in the list, reset DataStores so I join, gives me all that, works fine.

Leave, put the Scout class in, have it go

print(user.Classes.Scout)

expecting a table to be printed, got nil.

	playersData[player.UserId] = {}
	
	for i, v in pairs(data) do
		playersData[player.UserId][i] = v
	end
	
	local loadJSON = playerDataStore:GetAsync(player.UserId)
	local setData = (loadJSON and httpService:JSONDecode(loadJSON)) or nil
	
	if setData then
		for i, v in pairs(setData) do
			if playersData[player.UserId][i] then
				playersData[player.UserId][i] = v
			end
		end
	end

	local user = playersData[player.UserId]

	print(user.Gold)
	print(user.Gems)
	print(user.EquippedClass)
	print(user.Classes.Scout)

Obviously, resseting DataStore and keeping Scout inside the class menu works, but yeah, doesn’t work for returning players with previous data.

colbert2677 · December 29, 2018, 4:44am

This might be an issue regarding the line of code that checks key-value pairs. Are you experiencing this issue with only array values or do you experience them with other variables as well?

NinjoOnline · December 29, 2018, 4:48am

Well, here’s the full data table (prefab)

local data = {
	Classes = {
		['Knight'] = {
			['ID'] = 1,
			['Owned'] = true,
			['Weapons'] = {
				'Classic Sword',
			},
			['Armours'] = {
				'Knight Armour'
			},
			['Trails'] = {
				'None'
			},
		},
		['Archer'] = {
			['ID'] = 2,
			['Owned'] = true,
			['Weapons'] = {
				'Bow and Arrow'
			},
			['Armours'] = {
				'Archer Armour'
			},
			['Trails'] = {
				'None'
			},
		},
		['Scout'] = {
			['ID'] = 3,
			['Owned'] = false,
			['Weapons'] = {
				'Dagger'
			},
			['Armours'] = {
				'Scout Armour'
			},
			['Trails'] = {
				'None'
			},
		},
	},
	
	ClassEquips = {
		EquippedKnight = {
			Weapon = 'Classic Sword', 
			Armour = 'Knight Armour', 
			Trail = 'None'
		},
		EquippedArcher = {
			Weapon = 'Bow and Arrow', 
			Armour = 'Archer Armour', 
			Trail = 'None'
		},
		EquippedScout = {
			Weapon = 'Dagger', 
			Armour = 'Scout Armour', 
			Trail = 'None'
		},
	},
	
	EquippedClass = 'Knight',
	
	Level = 1,
	Exp = 0,
	Gold = 100,
	Gems = 0
}

When I make changes to just the variables (Gold, Gems, EquippedClass, etc.) they seem to change as well as be saved.

Just to test, I added a Test variable into there, printed it, and it printed that. So it seems to be adding in new variables from the list, but not stuff in arrays??

colbert2677 · December 29, 2018, 5:02am

Everything works but adding in new classes? The issue definitely lies within the part of the code that reads and writes data from prefabs into the session data then. The way I have the function setup is so that there isn’t any nested values, it only assumes that there is only top-level keys and values. Would explain why the arrays don’t work but the rest does. I’m at a loss here.

-- Prefab part
local prefab = {--[[whatever you want]]}
playersData[player.UserId] = {}

local function addressTable(receivedValue)
	local tableToReturn = {}
	
	for key, value in pairs(receivedValue) do
		tableToReturn[key] = (type(receivedValue) == "table" and addressTable(value) or value)
	end
	
	return tableToReturn
end

for k, v in pairs(prefab) do
	playersData[player.UserId][k] = (type(v) == "table" and addressTable(v)) or v
end

-- Data part
if setData then
	for key, value in pairs(setData) do
		if playersData[player.UserId][key] then
			playersData[player.UserId][key] = (type(value) == "table" and addressTable(value)) or value
		end
	end
end

This may work, try it out. It’s imperative that you only pass a table to addressTable, though that shouldn’t be a problem because I don’t see anyone other than you as the developer using this internal function. The way this change I made works is:

Created a new function called “addressTable” (will explain in a moment)
If the value is a table, pass tit to addressTable and use what the function returns as the value
If the value is not a table, then just use the raw value
- e.g. A = {"yes", {"no", "ok"}} gets passed to addressTable, but not B = true
addressTable is meant to return you a table value that handles the array
If there’s anything nested, the function calls itself to address a nested table before continuing
All of that is then given back to the code to function properly

ok i can’t explain it well but i trust it will somehow work.

NinjoOnline · December 29, 2018, 5:27am

Do I keep what I already have and just add this? Or remove certain parts?