Previously, I did not compress the data before saving it, which meant it had to be split across more keys. Although Roblox performs its own internal compression, developers do not benefit from that compression when working within the per-key size limit, since the limit is based on the raw string value being written.
By compressing the data ourselves before saving it, we can fit more data into each key and reduce the total number of keys required.
I can share a snippet in a DM, but I’m no longer able to initiate DMs it seems! Could you start one with me, and I can follow up there?
Great change, I hope there are still more upgrades to come with more compression on the datastore side and hopefully a raise on the amount of MB we can get per lifetime player as I enter deeper into my game’s development I’m starting to get worried of going over 1 MB per player and having to break the bank to pay for extended services and probably ending up shutting down the game because of it.
I completely agree with this. I think that instead of 1MB per unique person, it should be 4MB, this is because the maximum size per any key is 4MB, so it only makes sense.
Are there even any legitimate use cases where most of a game’s player data is actually commonly using up those amounts of data storage to where the limit would be concerning to begin with though? 1MB already seems incredibly generous for pretty much any ordinary use case.
Yeah, there are numerous games that were on the brink of reaching their quota because they are data-heavy games. There’s only so much compression you can do.
We’re developing a large RPG using most of the platform’s new features, and we’re facing many challenges. And, unfortunately, getting support from Roblox isn’t always easy.
While this is good, Games would benefit better from faster and higher API limit than a datastore upgrade since I don’t think many games are reaching those size limit yet.
With a group API Key scoped for experience 383310974 making requests to /cloud/v2/universes/383310974, I am seeing rate limit headers that don’t seem to match the new documented limits:
Based on the quota charts in the Creator Hub, this is far below what would be expected for this experience. Additionally, this limit seems unchanged no matter what experience I actually target, including experiences that should have limits much lower than the headers suggest.
I second this, it would be really nice to have a filesystem API (either part of DataStores or its own thing) that would allow us to stream file content as it is needed rather than just yanking a 4MB file immediately.
This way we could have content larger than the limit but it has to be streamed in incrementally rather than needing to split data up across multiple keys, which maxes out read limits very quickly.
Buffers have a max size of 1GiB, (who would be saving a whole gigabyte of data on Roblox?), so its feasible to push the limit further than 4MiB.
We have launched the new custom alerts that should allow you to define alerting thresholds with webhook triggers. You can alert on your shared request budget, allowing you to action against rogue Open Cloud processes.
Thank you for bringing this up - this is the expected behavior. On your Open Cloud requests, we still apply per-API key rate limits and other undocumented limits intended for service protection. The headers you are observing refer to those limits - see here for more information.
To utilize the full unified limit in large experience, you may need to distribute your Open Cloud process across API keys and/or machines. To see Open Cloud usage against your unified limit, you will need to utilize the Data Stores Observability Dashboard.
Unfortunately, API Keys are already challenging to manage and audit at scale, much less rotating them. Made worse by user API Keys essentially being invisible. I’ll also add that the scales our game operates at, the sheer number of VMs with individual API Keys we would have to spin up is cost-prohibitive.
