Urgent! Memory Store Usage skyrocketed! How can I reduce this data?

Help and Feedback Scripting Support
, , , ,
#1

Hello! I just implemented a new feature into my game using MemoryStoreHashMaps.
I have a Daily Reward system where players earn rewards for spending time in-game, and since this data is not useful after a day, I figured hashmaps would be perfect for this implementation.

This is the simple data i’m attempting to save

type PlayerRewardData: {
	claimed: {number}, --[[rewards have a 1 digit int identifier
					   we are saving those already claimed]]
	savedTime: number --int seconds of total time played today
}

however, there was an instant surge in memory usage after having implemented this:

image
image891×425 25.7 KB

So I decided to search compression libraries to try to mitigate it to the best of my ability

First I tried string.pack() to get binary strings out of my ints, and they worked really well!
(i could just represent my entire array with just a single byte for each element!)
…And then I learned that JSON formatting has issues with non utf-8 characters. So that’s not an option.

Afterwards I went on to implement my current solution, in which I JSONEncode() the data and compress it with the lua port of LZW algorithm, and then create a buffer.fromstring() that is then saved on to the memorystore.

…but it didn’t seem to make much of a difference, if any at all

image
image982×425 25.3 KB

in fact, as of a few hours ago, i’ve officially went over 50% of my quota.

I’m at a loss over what I should do! and by extrapolating, I should probably be hitting my quota in the near future.

I don’t want to be forced to remove this feature, and if possible, I don’t want the solution to just be using datastores instead

I appreciate your time and your suggestions!

#2

I dont understand anything about this memorystorehashmaps, jsonencoding and buffer, but probably it “saves” the data really fast, so the memory usage is skyrocketing.

Could you try removing the old data and insert new data?

#3

Hello!

no! I’m not hitting any API request limits, memorystores have a size limit/quota (meaning i can only use up to 150k bytes across my entire experience)

#4

Try to do that and i think it will help
If not then i dont know how to help you
maybe we need to look in the script

In the quote i said to remove the old data and insert new, because i see you have an Daily reward system, so we should change the data of claimed state and date when player got the reward

And check if you can optimize the code inserting the savedTime into claimed table

Also, tables probably take up a lot of space, try to change claimed type from table to integer

#5

I don’t really use memorystores but isn’t there a way to transfer the data to a datastore I guess? I know you don’t wanna use it but you could try right?

#6

If you are using table for just one state, its not recommended because 1 and {1} are the same thing but using {} makes just a little bit longer code and more taken space

#7

I could just use datastores, but that would require more datastore API requests that I am already using for other player data and leaderboards, not to add that memorystores delete / expire the data automatically without any additional API requests (which is really handy and why I am using them)

#8

Show the actual code please, I need more context

#9

you are actually onto something here

dictionaries take a painful amount of bytes :sob:

EDIT:
The above information is irrelevant to this case, since tables are converted to a JSON string when saved to memorystores via SetAsync

#10

ma god i didnt even know that :moyai:
so that works i guess?

#11

There isn’t much more context, I’m just saving the before mentioned data when the player leaves

--within PlayerRemoving event
local json: JsonData = {
	claimed = data.claimed,
	savedTime = data.savedTime,
};

local compressed = LZW.Compress(HttpService:JSONEncode(json), false);
local dataBuffer = buffer.fromstring(compressed);

playerHash:SetAsync(
	player.UserId,
	dataBuffer,
	self.timeService.getTimeUntilTomorrow() --they all expire on 0:00 UTC
); --thats why theres those sudden drops at 7:00PM on the image
#12

yeah, I might have to just try saving everything as a single array

type PlayerRewardData: {savedTime, ...claimed}
--every additional element would be part of the claimed rewards

I still mostly wanna see if someone can come up with a proper way to serialize & compress this, and possibly leave my original table unaltered

EDIT:
might just ditch the array idea entirely, and just save a single string with my data and convert it back

PlayerRewardData = savedTime ..'|'..rewardID1..rewardID2..rewardID(n)
#13

so did it work or you tried another way?

#14

Whats the maximum amount of rewards a player can get per day? Unless it is a huge amount, I dont think you need compression here, which makes me think its something with how youre storing the rewards when its decided they should be given.

1 Like
#15

Only 12, so at most its a 12 element array, but players rarely get all twelve, usually get around ~5

an example of how it may look like is the following:

someonesRewards = {
	savedTime = 328 --seconds
	claimed = {3, 1, 5}
--claimed rewards identified by 3, 1 and 5
}
#16

Must note that I just disabled this feature, and the memory usage stopped increasing, so it is absolutely related to the total bytes each player is consuming when i’m saving this table

#17

yeah it worked, I substantially reduced size by just representing it as a string, and the int numbers converted to base 16 hex (sprinkled some base64 on the savedTime)

testData = {savedTime: 170, claimed: {1,2,3,4,5,6,7,8,9,10,11,12}}
--[[
new serialization:
'Bp|0123456789AB'

json:
{"claimed":[1,2,3,4,5,6,7,8,9,10,11,12],"savedTime":170}
]]

image
plain json was 56 bytes :wut:

this is really interesting, because this isn’t the only information i’m saving as a json in my game. I have a separate sorted map saving arrays of numbers too. I’ll have to implement this there aswell.

1 Like
#18

Figured i’d share the solutions i used if anyone comes across this (probably, as hashmaps come out of beta and become more widely adopted)

This one was insanely helpful to reduce savedTime:
(base64 allowed me to represent an entire minute (int 60) with just a single byte!)

and this I used to reduce the numbers ‘10’ ‘11’ and ‘12’ to one byte in the “claimed” array

1 Like
#19

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.