Document Database System?

[Galicate (Gal)]

I’m working on a research game and I want users to be able to upload documents, however I also want HRs to be able to create things like folders so that documents have categories, how can I achieve this? I’ll likely be using a GUI system, where i’ll fetch all the data and use loops to create buttons to view them.

I’ll have two types of containers, “Folders” and “Documents”. Here is what my document format look like so far.

FileData.Document = {
	Title = "",
	Text = "",
	Clearance = 0,
	CreatedByUserId = 0,
	CreatedAtTimestamp = 0
}

I’m unsure where to start, like with how I would assign each document a parent, along with making folders with multiple sub folders. Sub folders aren’t necessary but they would be nice as I could organize even further.

If you are interested in how it’ll all work:

Summary

Researchers will write a document on a clipboard, and can approach a computer to upload a document from their clipboard. HRs will then have to review the document to make sure it isn’t a troll, and can then accept it or trash it. Documents won’t show up for others until it is accepted. If it is accepted it gets a clearance level and gets put into a folder. Researchers need certain clearances to access specific documents, so that they can’t open things they shouldn’t.

[manddilu]

Here’s AN approach, which is simple but assumes you won’t have thousands of documents and/or you’re fine with waiting a bit for them to load:

Use indices (1, 2, 3…) as unique keys in the documents DataStore. Make sure to set a limit for the length of the text of each document. Use another key (e.g. “counter”) to keep track of the number of documents.

When the server starts, get the counter value, and then start getting the documents one by one up until the counter value. Once that is reached, keep checking the counter value for a change, repeating the previous step when it changes. You have a 60 requests/minute budget for GET requests regardless of the number of players, so I’d personally add a 1 second wait between all requests that relate to the documents (including counter checks, and getting the documents that need to be reviewed), that would use even less than 60/min.

There are different ways of communicating changes to the clients, like RemoteEvents or adding Folders with different ValueBases in ReplicatedStorage, it’s up to you. Though whatever you do I’d say: to keep memory free, don’t transfer the text to the clients until they ask for a specific document, and make sure to free that up too once they’re done with it.

Keep the submitted documents in a separate DataStore, make sure to limit how often clients can submit (e.g. a 1 minute cooldown). Use :ListKeysAsync() on that DataStore for moderators, when they want to review documents, and then fetch the one they want to review through the delayed fetching system, but push these requests to the front of the queue, so they don’t have to wait for all the accepted documents to be fetched. Make sure to remove such requests from the queue if the moderator isn’t trying to view a submitted document anymore (they clicked away or left). It would also be good to test if :ListKeysAsync() uses up the budget, and then incorporate it into the queued system as well if it does.

Both accepted and rejected documents should be removed from the submissions DataStore, and if they are accepted they should be moved to the documents DataStore: :IncrementAsync() the counter key, and then SetAsync() the new document with the key that IncrementAsync returned. That way there should be no issues with servers writing to the same key, and you won’t need a complex tracking system. Though you should still take into account the possibility of two moderators making a decision about the same document, there are also multiple ways to solve that, but the easiest would be to try to get the same key (from submissions) once a moderator makes a decision, and then if it doesn’t exist (because it gets removed in both cases) don’t do anything, maybe inform the moderator that a decision was already made.

I haven’t talked about keys for the submissions DataStore, but tostring(os.clock()) would be an easy option.

That’s pretty much it, unless I’m forgetting something.

As for folders and sub-folders (alternatively “tags” or “categories”), the moderators can just add those as part of the document data (some multiple-option selection that you set). Clearance could be either manually set by moderators, or based off of folders and/or categories. I’d say do server checks for everything moderators do as well, you never know.

Let me know if you want me to address something else, or if I should expand on something. Some things do depend on what you’re planning to do in the future, but as for your current idea I think this is about as simple (but stable) as it can get, provided you’re fine with waiting for the documents a bit, if there’s ever many of them. If you want them fetched quicker, you’d have to do a more complex system where you’re storing many of them per key, and the complexity lies in making sure the servers don’t overwrite “each-other”, because the simple counter system wouldn’t be suitable there.

[Galicate (Gal)]

I’ve got a really basic saving and loading system, when the server starts I get all the documents, and each document is indexed using GUIDs database[Http:GenerateGUID()] = {}, and if I want to find a specific document I just search through all of them and check for a name value, just so documents can have the same names and documents won’t need super long names once there’s more than a few.

I’m just not sure how to go about the parent system, where a document would be inside of a “folder”. I could probably just give each document a variable like so document.Parent = "Research" and just create visual folders on the client when I detect a document has a parent. But then I can’t have things like sub folders or folders with clearances.

I’m probably gonna make it so that you can only upload a document every hour, to minimize the bandwidth and to prevent trolls from dossing the server. How long should the max character limit be, I figure 4000 characters shouldn’t be too bad right?

[manddilu]

@Galicate

and just create visual folders on the client

What else could you do? That’s all there is to it, and you can add more keys to the table to indicate “sub-folders” and so on, that’s the least complicated thing in the entire system, as long as you know what you want. I don’t see why you wouldn’t be able to have sub-folders and clearances. Just allow moderators to add these options to each document, and then represent it on the client however you want.

For example the players approach a shelf or drawers with documents, and a proximity prompt pops up, when they activate it they get to choose a “folder” then a “sub-folder” then a “sub-sub-folder” if necessary, and then they see the documents in there, if they have clearance for them. Or take a different visual approach, you could do it many ways.

@Galicate

I’m probably gonna make it so that you can only upload a document every hour, to minimize the bandwidth and to prevent trolls from dossing the server. How long should the max character limit be, I figure 4000 characters shouldn’t be too bad right?

An hour sounds too long, a player might want to submit multiple valid documents per hour, but I might be wrong, you have a better idea about how long it would take to write up a document. Bandwidth won’t be a problem unless they’re sending them every frame, and I doubt a troll will wait more than 10 minutes to submit another one, maybe some rare dedicated ones, to prevent this you could add a minimum play-time limit after which they can start submitting documents, in combination with a 10-15 minute cooldown.

The character limit is up to you, you have “4,194,304 per key”, characters that is, including commas and brackets and document data that’s not the text itself. Your previous reply is 987 characters long, so ask yourself “does 4 times that seem like enough”.

[Galicate (Gal)]

Oh you know what, I could store document data and document text separately, document data could all be stored in the same key for quick loading, and each documents text would have it’s own key that would only be loaded when it needs to be viewed. And then the same thing for un-reviewed documents, except since there’s less of them, I might be okay putting all of their info into one key.

local publicDatastore = DataStoreService:GetDatastore("PublicDocuments")
local privateDatastore = DataStoreService:GetDatastore("PrivateDocuments")

local publicDocuments = publicDatastore:GetAsync("PublicDocumentData")
local privateDocuments = privateDatastore:GetAsync("PrivateDocumentData")

local function getDocumentText(documentId)
    return publicDatastore:GetAsync(documentId)
end

local function getPrivateDocument(documentId)
    return privateDatastore:GetAsync(documentId)
end