Introducing MemoryStore - High Throughput, Low Latency Data Service!

Updates Announcements
,
#112

Anything about the memory size limit that’s in bold still being wrong? This issue still seems to be ongoing and values still seem to be capped it seems as I am getting “The provided value is too long” for just over 1kb in a sorted map which makes this incredibly difficult to work with because 1kb is nothing.

gev8xp_109241
gev8xp_109241826×200 133 KB

Here’s an image, as you can see the error occurs when the length of the entire data is just over 1kb (1024). This is affecting a lot of people that use my module as the documentation is incorrect when it makes this claim:
rsz8jo_109251
rsz8jo_109251988×61 9.7 KB

I know that the entire memory isn’t over 65kb as well. There’s maybe 2kb in there at max.

Here’s also confirmation. I made a new game with 0 memory usage. And I made a test script and confirmed these suspicions:

2mgk4d_109261
2mgk4d_1092611178×144 13.3 KB

It’s fine on the first one, which is (just) under 1kb. But it errors on the second one which is over 1kb. And the total memory limit with 1 player is 65kb, which this is definitely not hitting.

And before you say it’s because I’m in studio, it’s not:

cpsyww_109271
cpsyww_1092711048×356 106 KB

Simple test code, literally just a string of characters 
local MemoryService = game:GetService("MemoryStoreService")
print("Memory limit size test:")
local test = ""
for i = 1, 1020 do
	test = test .. "1"
end
print("Length of data being added: " .. tostring(#test))

local map = MemoryService:GetSortedMap("Map")

map:RemoveAsync("Test")
map:RemoveAsync("Test1")

map:SetAsync("Test", test, 86400)

test = ""
for i = 1, 1100 do
	test = test .. "1"
end

print("Length of data being added: " .. tostring(#test))

map:SetAsync("Test1", test, 86400)

Please either fix the memory size limit on sorted maps or document it correctly.

8 Likes
[v2.1.0] MatchmakingService - ON HOLD
#113

I could be wrong but I think the error message for exceeding the expiration time is wrong.

image
image1585×94 32.3 KB

I tried to use that exact time and it wouldn’t let me. That’s 30,000 Days rather than 30?

#114

Nice, This is going to be so useful. Question: Does MemoryStore Data save accross places? And how can I use AddAsync? Do you type in the name of the value, The value itself then its priority then its experition? Because I need this to temporarily store player data for a game.

#115

so don’t use Queues but SortedMaps

#116

This service is useless for matchmaking. You’d expect to implement this using Queues, but this currently has no functionality to remove a specific item. You can’t account for players exiting the queue.

You could probably implement a hacky solution using SortedMaps, but this seems counterintuitive and seemingly makes Queues futile for matchmaking systems, which is what I was personally excited for.

Not sure if this was an oversight or some sort of limitation but seeing such a basic crucial feature missing is really disappointing for such a seemingly powerful service. :frowning_face:

4 Likes
#117

[v1.0.0] MatchmakingService - Resources / Community Resources - DevForum | Roblox look this guy

#118

To be fair, I do use a hacky method using sorted maps, so he isn’t wrong. It would be so much better if we got more control over memory queues.

#119

Perhaps my frustration was a bit too over-emphasized in my original post. It just sucks that this service went through a closed beta period and I’m failing to see a proper way to utilize Queues without all the previously mentioned drawbacks. It just doesn’t make life easier :frowning:

There are still games that are using MessagingService for matchmaking systems, which is honestly quite embarrassing. If it wasn’t for your effort of writing almost 2,000 lines of code, I’m not sure how many people would’ve wanted to put in this kind of effort to make their own system using SortedMaps.

I’m not sure how stress-tested this module is, but I’d assume it’s a great alternative.
Thank you for your contribution :slight_smile:

#120

I’m right there with you, I would much more prefer to use Memory Queues, and as I’ve mentioned in my matchmaking service post, they’re just not in a suitable position to be useful, and it’s rather unfortunate as well. Especially when they added the limit on sorted maps to 1kb from any amount that was within quota… That made my service that much more difficult to maintain. The 1kb limit in sorted maps is completely arbitrary as well and I don’t understand why it was added. I do hope that they add more features to memory queues because right now I would say that they’re pretty useless.

1 Like
#121

Could anyone help with this?

#122

i didn’t said that he was wrong just a link to your works but yes there are too much things to fix with MemoryStore Service like the size limit etc…

#123

Hey @V_ChampionSSR, thanks for the feedback. Can you explain more details about the challenge here? Why removing a specific item is critical to your use case? For matchmaking, we think you can validate if the player has dropped after reading from the head of the queue.

#124

Are you talking about the key length limit? Can you elaborate your challenge and what you want to achieve?

#125

When this service was initially announced, singular SortedMaps did not have a size limit as long as it did not go over the quota. Since then it was lowered to 1kb. I made a bug report about this before the docs were updated:

This just made it very challenging to use SortedMaps because 1kb is not a lot to work with.

As for memory queues, I would just like the ability to control them a bit more, such as getting specific values from the queue, or just added a Skip parameter to ReadAsync that will skip the number you give it and only return the values after. There also needs to be a way to remove specific items from memory queues.

The example for memory queues you guys give is a matchmaker. However, if a player leaves after they queue, there’s no way to remove them unless you read them first, which is rather inefficient because you also can’t just remove a single value while reading, it’s either you remove everything you read or nothing.

3 Likes
#126

I will also share my opinion on this topic.
SortedMaps are fine (except the 1kB limit), but Queues need to be reworked from scratch.

Here are the API members they should have:

:AddAsync(value: any, expiration: number, priority: number?)

This function stays the same.

:RemoveAsync(value: any)

Removes all items that are equal to value.

:UpdateAsync(transformFunction: (oldValue: {any}) → {any}, expiration: number)

Retrieves the queue and lets you update it via a callback function. The queue is a sorted array.

:GetRangeAsync(count: number?, direction: Enum.SortDirection?): {any}

Returns a sorted array of queue items. count specifies the number of items (if nil, all are returned). direction specifies whether it starts at the end of the queue (descending) or at the beginning (ascending). The default is Enum.SortDirection.Descending.
It would be nice if this function didn’t have a limit of how many items it can return (or at least bigger than 300).

.Added / .Removed / .Changed

We need some sort of event when the queue got updated.


Please make ReadAsync deprecated and remove it after some time. It is extremely inefficient.

I hope that you will realize at least a few of these functions. If you need a more detailed explanation, feel free to ask me. :slightly_smiling_face:

1 Like
#127

Hello. Thank you for the response. As was mentioned by @Extuls and @steven4547466 there is no efficient method including the one you have described. A thread that has more detail can be found here.

As for your question:

Let’s use ReadAsync for example, 20 results. 10 of those results are players that have left the matchmaking queue. That leaves us with 10 valid results. How can I remove those 10 invalid results, so that I only get 20 valid results?

There is no way to tell which results are valid or not. Even if there was a way to tell if a player was “dropped”, RemoveAsync would only let you remove all 20 results from the queue.

The reason most of us would prefer to have a specific amount of valid results from a Queue is so that we can make sure that there are enough players to start a match. Not to mention there is also that chance that TeleportService may fail, and some matched players aren’t able to make it. Being able to account for the margin of error would be incredibly useful.

All of this is not only to make life easier for us developers trying to make a matchmaking system but, will greatly improve user experience. We want players to be matched as efficiently and swiftly as possible.

What would be a potential solution is that instead of AddAsync() returning void, it could return an id that you can use in conjunction with RemoveAsync(). That way, if a player leaves the game, the queue, or whatever else we could handle this logic ourselves.

1 Like
#128

Thanks for the details. Can you share more about your use case? What type of data you’d like to store in sorted map? What’s max size do you expect?

#129

Some of the functions/events proposed seem too complicated and some may not be that practical in the long run. Returning all items in the queue for example, what if there are thousands of items? The same goes for the events, I think it would be more efficient to make a simple loop to fetch the queue instead of listening to events being fired hundreds of times a minute.

Also, why would you want the option to fetch the very back of the queue? The point of a queue is so that the elements in the front of the queue get processed first.

#130

My main use case is for my module MatchmakingService. Before the documentation was updated, it stated that, as long as you don’t go above the quota, SortedMaps had no limit. I was just wondering why that was scrapped for a 1kb limit. For example, I hold running games in memory to support adding players to games that have already been made. This allows players to start games with the minimum number of players required so they can just get in and play, and more players can join if spots are open. These games typically don’t last long so I hold them in a sorted map.

The problem with the 1kb limit comes with keeping track of running games, which are divided two ways: map and rating type. My service supports skill-based matchmaking (sbmm) which allows developers to enable it if they want a rating system for their game, they can have as many ratingTypes as they want so they can have multiple queue types (think “ranked” and “unranked” that both use sbmm, but for a different reason and aren’t connected). It also supports an arbitrary number of maps so that developers can have more than one map if they choose. All this boils down to is a lot of usage in memory, that doesn’t go over the size quota, but will go over the 1kb limit. I don’t save every game in the same sorted map, each game gets their own unique code.

In order to support adding players to running games I keep a list of every game code and put it into a group based on the rating type and map. It would be inefficient to use GetRangeAsync on every running game if none of the running games apply to the player’s queued game type and map, so (see bottom) I need to keep a list of every running game’s unique code for every map and rating type (I don’t store the entire game, it’s just a list of strings that represent the games), and at about 20 games that list exceeds the 1kb limit so I need to break that list up every 20 or so games, so I just have a counter that keeps track of every new sorted map so I can retrieve the list of running games based on rating type and map. I could shove more into the sorted map by reducing the size of the unique codes for every game, but eventually I’d still run into this problem, it’s unavoidable.

It just gets very complex very quickly with such a small limit and I’m really upset that the original unlimited size for sorted maps, as long as it didn’t exceed the quota, was scrapped for such a small limit.

I may switch to GetRangeAsync just to make it easier on myself to maintain, but, at least in my opinion, it shouldn’t be necessary. After re-reading the documentation on GetRangeAsync it wouldn’t be a suitable replacement as it only works on a single sorted map, which doesn’t help me in this case as I would need to get a list of all games running which are all different sorted maps. I could put all the running games into a single sorted map as, after re-reading the docs each value in the sorted map has the 1kb limit and maybe that could work, I’ll have to experiment with it though. However, the limit being so small is still not ideal.

#131

I doubt that there would be thousands of items in a matchmaking queue. If you maintain it in a first-in-first-out (FIFO) sequence, the maximum player count in the queue is approximately the player count in the matches. I have never seen matches with more than 100 players on Roblox.
If you don’t use FIFO, there still won’t be thousands of items. As I said in the post, I would be satisfied with a limit (of how many items it can return) that is 300 or bigger.

Polling is better when there are many queue changes and events are better when there are not many changes. The best would be probably a polling loop that adapts the amount of requests to the player count.

My bad, I mixed up the start and end of a queue. The default for direction should be Enum.SortDirection.Ascending.
I thought that it would be nice to have, but I wouldn’t know a use case for Descending right know either.

Do you have any other concerns?