Memory usage reporting over capacity when its not

Either the graph is bugged or the limits are bugged here.

This is for our game yeet battles Yeet Battles - Roblox

Memory store graph is reporting that we are using 8125 bytes in our memory store.

Our memory store requests are failing with an out of memory error

Checking our game, the errors report a significantly higher memory usage than
A) The graph is showing
and
B) Should be possible with our current implementation of memory store

Luckily this is a feature in testing in case something like this happened, but this will prevent us from going live with the intended use case.

Our write implementation looks like

local FNV_PRIME_32 = 16777619
local FNV_OFFSET_BASIS_32 = 2166136261

local CommonUtils = {}

function CommonUtils.FNV1a16(str: string) : number
	local hash_val = FNV_OFFSET_BASIS_32

	for i = 1, #str do
		hash_val = bit32.bxor(hash_val, string.byte(str, i))
		hash_val = (hash_val * FNV_PRIME_32)
	end

	-- XOR fold to 16 bits
	local foldedValue = bit32.bxor(bit32.rshift(hash_val, 16), bit32.band(hash_val, 0xFFFF))
	return string.format("%04x", foldedValue)
end

function ActivePlayerMemoryMap.WriteAsync(userId: number, versionNumber: number)
    assert(versionNumber > 0 and versionNumber <= 127, "Version number must be between 1 and 127")

    local userIdHash = commonUtils.FNV1a16(tostring(userId))
    local versionChar = string.char(versionNumber)

    local success = false
    local attempts = 0
    while not success do
        success = pcall(function()
            activePlayerMemoryMap:SetAsync(userIdHash, versionChar, MAX_EXPIRATION_TIME)
        end)

        if not success then
            attempts += 1
            if attempts < RETRY_ATTEMPTS then
                task.wait(1)
            else
                break
            end
        end
    end

    if not success then
        warn("MemoryStore write failed after 5 attempts")
    end
end

Meaning we are writing one byte to a maximum of 65536 total keys throughout a memory store hashmap.

This SHOULD be impossible to hit limits with, as it is within the base limits. Regardless, the analytics graph is showing we are only using 8125 bytes of memory store storage.

Root cause may have something to do with https://create.roblox.com/docs/cloud/reference/features/storage#Cloud_FlushMemoryStore as this was used to clear out a previously flawed implementation somewhat recently.

Hey! Letting you know that I am taking a look into this - this is definitely strange.

First of all, I was wondering if you have a timestamp of when you last called the flush endpoint?

Hi,

Unfortunately I do not as the activity history has been lost in postman by now.

Sometime around 1/17 most likely.

Also of note is that within a day of posting this bug - our memory store usage has shot back to pre-fix levels and remained steady there.

Our team is still investigating this. Two questions:

  1. What is the value you set for MAX_EXPIRATION_TIME?
  2. Is the memory usage increase starting on 1/31 expected on your end (did you return to your previous implementation), or is it unexpected?
  1. Originally, 45 days. Currently - 8 days.
  2. The memory usage increase was NOT expected. Our new implementation uses the above hash to ensure our keyspace only has 65536 total keys, under which one byte is written. Memory usage under our current system should NEVER go above 65536 bytes.

Thanks for that, we’re still looking into this bug, a few more questions

  • Can you clarify if this timeline is correct?
    • Previous implementation had an expiration of 45d for each key
    • This was flushed around 1/17
    • After flush, was there a delay in between where you switched the implementation over to the new one?

I’m trying to determine if keys were written using the old implementation after the data was flushed, even if for a few hours. Unless they expire/are deleted, they’ll continue contributing to the memory quota error.

If it is still in testing, could you try flushing your universe again if it’s not too much trouble?

“After flush, was there a delay in between where you switched the implementation over to the new one?”
No.

“If it is still in testing, could you try flushing your universe again if it’s not too much trouble?”
It is still in testing. I have flushed the cache again. Flush was submitted 2/14 at 4:38 AM EST.

Experiencing a similar issue after using the flush endpoint.
About 2 minutes after flushing my memory “usage” jumped 3,000 bytes above the quota. (Nothing in production is currently writing to memory stores).

Flush occurred at ~5:45 PM EST today on experience 2200410891.

Not sure if this is related as we have not used the flush endpoint, but our game has suddenly started throwing TotalMemoryOverLimit as well. This is very strange because our usage before was always well under quota (1-2% of quota). Strangely as well a few hours before the issue started happening, the memory usage analytics for our experience go completely blank. I know our quota has not decreased severely either because the response being thrown says our quota is about the same as it has been before, a bit higher actually.

EDIT: The issue seems to have resolved itself but only after ~24 hours. Strangely enough as well for the entire period the issue was happening the memory usage analytics are completely blank


Update from my end:
We have made no changes, currently our memory usage has spiked down from ~500k total usage/is being reported as a negative number in the graph view when in the last one day date range.

Checking the last 7 days range, our total memory usage is being reported as something that is still higher than I expect - but is much closer to a believable value.

Hi, any updates on this issue? It’s relatively important to one of my game’s features and I’d really like to re-enable it soon. Thanks!

Hi folks,

We are still looking into the issue. In terms of the analytics you’ll are seeing - are they closer to what you’d expect now? There are a few bugs on our side that we have since fixed, but it may not have deterministically fixed this.

Please let me know if you are still seeing usage over capacity when that shouldn’t be the case. It’d also be helpful if you can briefly describe what data strucures and APIs you’ll are using, and what expiration you are setting on your keys.

Thank you for your patience as we figure this out, and sincere apologies for the delayed response.

1 Like

Hi,

Update from our end. We are still using much more memory store storage than expected. With our keyspace being “0000” to “FFFF” storing one character each, we would expect a maximum upper bound of 65536 bytes used.

We still see it fluctuate around 400k bytes used, and still have occasional anomalies where the storage used drops to 0 briefly.

Hello!

We shipped some fixes recently to address this issue. Can you confirm if the usage levels you are seeing are a more accurate representation of your data now?

Yes, this looks like an accurate representation of our data usage now.

I am no longer seeing any weird spikes. Full disclosure - we recently turned off most of our writes in favor of an API that Roblox recently released that makes our old system obsolete.

Thanks for the help!

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.