Introducing MemoryStore - High Throughput, Low Latency Data Service!

That’s a valid comparison, but the official documentation fails to make any mention of that. When we want to access a tutorial, we’d rather know in what applications this is useful in than how much memory we’re allowed to store on it at a time. I’ve been developing for well over 7 years, so you could only imagine how someone with one or two years will approach this. The first paragraph mentions general use-cases, but is nowhere as in-depth as with the DataStoreService page.

Woah :exploding_head:


Do you have any plans to create an API for measuring Data size?

How are we supposed to know that our data is over 1KB?

It’s extremely difficult for us to remain under the limit without any tools that can help.

  • Memory Size Quota: 64KB + 1KB ⨉ [number of users]

  • Requests Quota: 1000 + 100 ⨉ [number of users]

while we can take notes, it’s almost impossible to get that information in a live game


There's a lot of documentation and links missing on the DevHub.

MemoryStoreService

MemoryStoreService:GetSortedMap

MemoryStoreService:GetQueue

Missing from MemoryStoreService

1 Like

Oh my gosh, thank you! This is definitely going to improve Ro-Chat with global server data!

Could I suggest that setting shorter expiration causes larger size limits? In my use case, I’d need the data for a max of a couple of hours. Keeping it for a month is pretty useless for me. But size may be an issue. Since the data is temporary, not sure why the size limit is smaller than DataStore’s is.

1 Like

SortedMap seems to just return a sorted array by division key instead of value. As such it doesn’t get any better when applied to the Global Leaderboard. SortedMap should add a way to get an array sorted by value.

4 Likes

Looks promising! I hope there will be some code examples in the future. Because I am unsure how I am supposed to use this for Global Leaderboard, which has been listed a use-case in the tutorial. Scores/Ranks are usually persistent so I would need to rely on DataStoreService anyhow. I don’t see how I would use MemoryStorage for a Global Leaderboard.

1 Like

I’ve used this method to measure the size of my datastores previously. It returns the length or how many letters is in the json string and each letter is about 1 byte.

print(#HTTPService:JSONEncode(table))
print(string.len(HTTPService:JSONEncode(table)))
1 Like

I’m assuming the example use case was if someone doesn’t use OrderedDataStores for global leaderboards. For example, a game can make a global leaderboard/ranking system based on Letters (A, B, C, etc) instead of numbers.

Id like to thank all the staff members who worked on this. Huge respect! :pray:

This has a lot of use cases and it’s great to switch to MemoryStore instead of using MessagingService for cross server communication.

Yay, now no need to worry about datastore cooldown.

Lovely dubbley, nice work! :heart_eyes:

New entries are usually missing full documentation on the Developer hub. I’m sure if you check back at some point the article and any other missing documentation will be there

1 Like

This is perfect for what I’m making right now!

I currently am making a system in where players input (filtered) text into a TextBox and this text would appear on a sign (across every server). I only wanted this text to be able to appear for a minute max and the MessangingService limits are just to small for me, so I was using datastores every 5 seconds but was quickly hitting my limits. So this service will really help me!

‘An error’ repeats in the tutorial, got scared I was reading it wrong for a bit!

Screenshot_20210921_172813

1 Like

Can an API be added to see what the rate limit is similar to how datastore has an API to view the rate limits on its API

3 Likes
  1. Will characters 0 and 128-255 (non-utf8) be supported properly without using JSON internally? string.pack is extremely useful, but DataStoreService and MessagingService currently fail when sending binary data. It’s important that we can use compact fix-sized data strings because of performance, storage quotas, and other limitations.

  2. Is MemoryStoreService:UpdateAsync preferable to DataStoreService:UpdateAsync if I’m implementing a save locking system for player data? The goal is to prevent a new server from overwriting a player’s data by waiting for their previous server to remove a “lock” appended to it (or by waiting for the previous lock to expire if their previous server crashed.) It would be possible to implement this lock using MemoryStoreService, and use DataStoreService:GetAsync/SetAsync afterwards. It seems like MemoryStoreService:UpdateAsync would significantly reduce bandwidth because the player’s entire save doesn’t need to be sent back with the lock appended to it, but it seems more complex to use multiple APIs to achieve this.

11 Likes

Sure thing!

My overall use case is that I have a bunch of AWS-based services that integrate with various external services we use to improve user experience - for example, we use AWS Lambda to handle log in/log out and fetch data from PlayFab, or other stores.

We use Amazon SQS to queue up messages that we want to send to servers; for example, sometimes we might want to tell a user they’ve been awarded a prize, received a communication from our support team, or something else that requires bridging from outside of Roblox to inside via some kind of Http request.

For point one, having a script connection that can listen for changes would mean we don’t have to poll for changes - we can do a system with MessagingService to indicate there is traffic that needs to be processed by the other servers (since generally, I’m not a fan of polling on repeat for changes) but this adds boilerplate to our setup which otherwise isn’t needed. So, I’m not too attached to this point since it is technically possible already but it would be nice :slight_smile:

For point two, having external API access (i.e. through a WebSocket gateway or a REST API) would allow us to introduce a backbone for communication and data storage between our Roblox servers backend APIs in AWS. We currently have a HA-oriented Redis setup which we use in conjunction with Simple Queue Service to figure out when there is information for a user that we don’t need/want to save in a database.

As an example use case, we have an in-game & discord based reporting system. People can use the report command via our Discord, or whilst they’re in game, to report behaviour that may be acceptable to Roblox (i.e. trading in game items or glitching across the map) but isn’t acceptable to us so we moderate it on top of that. Being able to unify the memory store between our in game and in-discord solutions would mean that we could respond instantly to users, or even set up a live chat setup for our support team to chat to users.

The current solution that most people use for this kind of communication is long polling, for example:
Reselim/rbx-engine.io: Engine.IO client for Roblox (github.com) but this is very heavy on requests / bandwidth and is prone to frequent issues. So, being able to send data over a queue via a REST API would be super useful.

6 Likes

I’ve noticed small misinformation on DevHub while looking on this page there are stated that the count parameter has maximum of 200 but when I’ve tried it in studio it gave me following error:

4 Likes

Also, why should this be used over messaging service?

I agree, an example of how to do a Global Leaderboard would be very helpful, I’m a little confused as well.

The way I am thinking you could do it would be to make a leaderboard SortedMap and have each of the player’s IDs as the keys with the “score” as the value, then call GetRangeAsync() with no args to get the entire SortedMap and then sort it by score after the fact - but that seems really expensive to do especially if you want to have it display updated values often and had a lot of players. I guess if you only needed to update the leaderboard when a server starts and every ~10 minutes after the fact it could work, but I feel like there is probably a more efficient way of going about it.

If anyone has some insight on the best way to do this sort of thing I’d love to hear it.

7 Likes

There should be ways of reading how many items are currently in a given queue to communicate information to players about how long a particular thing will take. For example, if a player has queued up to join a different game server, I need to be able to communicate to that player how many other players are ahead of them in the queue. If there are 100 other players ahead of them, they might not want to sit in queue for that long.

Thank you so much for all of these great updates roblox! I’m so happy to be in the developer community at this time!!