QueryService: A Unified Querying Solution

[jonbyte (Jon)]

Preface

As a Roblox developer, it is currently too difficult to consistently find and react to instance existence based on a query in a scalable and unified manner. Right now we have QueryDescendants but we still need to combine other separate behaviors (ex: ChildAdded, TagAdded, AttributeChanged, custom scans, manual string matching) for common use case, which can be error-prone and hard to maintain in large projects.

Problem

The QueryDescendants method is a good foundation, but it falls short in situations where a more expressive querying model becomes necessary. It only evaluates based on current DataModel state. In experiences where content is replicated/streamed to the client it becomes essential to implement systems reactively. Currently, if I want to respond to any descendant matching some query when it exists, the approach is scattered and may not be so clear depending on the implementation:

1. connect ChildAdded/DescendantAdded
2. call QueryDescendants inside the callback
3. search (table.find) for the new instance

The way to check if one instance matches a query, as shown above, is to call QueryDescendants and then table.find the instance in the resulting array. Additionally, there is no current behavior that allows simple yielding until an instance exists matching some query. The workaround for this would involve a ChildAdded/DescendantAdded connection, polling, and thread management. Aside from unneeded complexity, a speculation on these workarounds would be increased memory allocation and potential performance degradation compared to a one unified, built-in approach.

In addition to the the problems listed above, the current query syntax lacks an essential feature: partial pattern matching. Many Roblox projects of mine and in general rely on naming conventions or classifications of objects. For example:

• GUI elements: Entry_1, Entry_2, … Entry_*
• prefixes: Right_*, Left_*, …
• other numbered/indexable children
• tag hierarchies: Interactable_Door, Interactable_Button

Without pattern matching support, we must resort to using QueryDescendants but also scanning the result with our string matching logic. No native support for this is poor for entity-component-like systems and generally any structured hierarchy in the DataModel.

I also mentioned tags because the concept of being able to query tags has been requested over the years. Currently, I would find it useful for a system where I have a specific tag on an instance (ex: entity-123), but I also must have an Entity tag on the instance so I can get all instances that are considered entities. Being able to query tags would have it so I only need the first tag and less error-prone since I no longer rely on both tags existing.

QueryService

Based on the limitations, I propose the addition of a QueryService service. The existence and API of this service promotes simplicity, clarity, and a one location for commonly needed behaviors related to queries. Below are listed the current ideas for the initial behaviors.

Note: All APIs listed are suggestions, not demands. The goal is to provide the idea of how these behaviors may be implemented.

local QueryService = game:GetService('QueryService')
-- the possible methods below are methods of this service

Syntax

QueryService uses a global syntax that supports pattern matching, logical combinators, and the existing , combinator. Selectors and some combinators depend on the domain in which the query is evaluated. The , combinator can be used for any domain method that returns a list of results. The backslash (\) can be used to escape any special character in this syntax.

All domains support string pattern matching which uses standard string.match rules, but patterns must be wrapped in curly brackets ({}) to avoid ambiguity with literal values or special characters.

All domains support logical operators which include !, &, and |. Operate precendence follows conventional rules: highest for !, then &, and finally |. Parentheses (()) may be used to group expressions for clarity or correctness.

For the DataModel domain, the existing > (child) and >> (descendant) combinators remain supported. This can also use familiar selector types such as ClassName, #Name, .Tag, [prop=value], [$attr=value]. Effectively, DataModel queries have at least the same syntax as current QueryDescendants does. However, the overall syntax supports what is listed above as well.

The Tag domain does not use any extra combinators and only selects plaintext (with pattern matching support).

Note: I have not verified that my suggested syntax avoids all ambiguity. More consideration may be needed.

Domain: DataModel

The current DataModel querying workflow is fragmented. QueryDescendants only returns the present state and doesn’t help on its own when content is streamed in or when attributes/tags change afterward. When considering how I might use QueryDescendants in my own projects I immediately ran into these issues as I often code reactively for streamed content.

Examples

• Needing to detect when an instance with a naming convention appears (ex: Item_*)
• Waiting for a descendant with some classification to be streamed in under some parent
• Attaching behavior to instances matching a query as and when they exist
• Lifecycle management for filtered instances when they no longer match the query, are destroyed, or streamed out
• Handling dynamic hierarchies where properties, attributes, or tags change after the instance already exists

Possible API

-- the string is the query string
-- the observer callback returns a function which is called when the instance no longer matches the query or it was destroyed or streamed out (common pattern in development for cleanup)

:Compare(Instance, string) : boolean

:HasDescendant(Instance, string) : boolean
:GetDescendant(Instance, string) : Instance?
:GetDescendants(Instance, string) : array<Instance>
:WaitDescendant(Instance, string, timeout: number?) : Instance?

:OnDescendantAdded(Instance, string, (descendant: Instance) -> ()) : RBXScriptConnection
:OnDescendantRemoved(Instance, string, (descendant: Instance) -> ()) : RBXScriptConnection
:ObserveDescendants(Instance, string, (descendant: Instance) -> (() -> ()) : RBXScriptConnection

:HasAncestor(Instance, string) : boolean
:GetAncestor(Instance, string) : Instance?
:GetAncestors(Instance, string) : array<Instance>
:WaitAncestor(Instance, string, timeout: number?) Instance?

:OnAncestorAdded(Instance, string, (ancestor: Instance) -> ()) : RBXScriptConnection
:OnAncestorRemoved(Instance, string, (ancestor: Instance) -> ()) : RBXScriptConnection
:ObserveAncestors(Instance, string, (ancestor: Instance) -> (() -> ()) : RBXScriptConnection

Domain: Tags

CollectionService only supports exact tag matches. In my own projects (and in many community patterns), tags are often organized into classifications. For example, there might be generic behavior we want to apply to enemies, but then specific behavior for kinds of entities. I described a similar issue before where I would have something like enemy_skeleton tag, but also each of those (and other classifications) would need a generic enemy tag. With QueryService we would be able to do something like :OnTagAdded('{enemy_%a+}', function(tag, inst) ... end) for reactive behavior.

Possible API

-- the string is the query string
-- the observer callback returns a function which is called when the instance no longer matches the query or it was destroyed or streamed out (common pattern in development for cleanup)

:HasTag(Instance, string) : boolean
:GetTags(string) : array<string>
:GetTagged(string) : array<Instance>
:WaitTag(string, timeout: number?) : string?
:WaitTagged(string, timeout: number?) : Instance?

:OnTagAdded(string, (tag: string, Instance) -> ()) : RBXScriptConnection
:OnTagRemoved(string, (tag: string, Instance) -> ()) : RBXScriptConnection
:ObserveTags(string, (tag: string, Instance) -> (() -> ())) : RBXScriptConnection

Conclusion

The best part about a QueryService is we can scale it to other domains without conflict. For example, there is potential for this API to be extended to attributes where the query would operate directly on attributes across the DataModel.

Querying and reacting to instances across hierarchy, tags, and naming conventions is currently fragmented across several APIs. The workarounds are not ideal and can lead to hard to manage and error-prone code. A QueryService with a simple API, unified syntax, reactive observers and listeners, pattern matching, and more would make workflows around queries much simpler and more maintainable. It consolidates common behaviors into a single provider that scales cleanly as a projects grows, improves code clarity, and reduces the need for extra custom infrastructure.

If these problems and suggestions were to be addressed it would solve 99% of the logical fragmentation and boilerplate around querying.

[avodey (avocado)]

Support! Querying and detecting changes to hierarchies definitely needs improvements. In my game, I use this code pattern a bunch:

local function onUser(user: Player)
    -- code
end

Players.PlayerAdded:Connect(onUser)

for _, user in Players:GetPlayers() do
    task.spawn(onUser, user)
end

But it would be shorter if it was like this:

-- Idk the format for descendant queries ;w;
Players:BindToDescendants("[class:Player]", function(user: Player)
    -- code
end)

You need to detect descendants AND iterate existing descendants. And if you need to apply a lot of conditions like detecting the class, name, etc, it starts to become less readable and optimized.

[woot3 (Woot)]

I’ve been thinking about this for a while and even built my own library to wrap CollectionService which lets you create ‘Collections’ and perform operations on them really easily.

For example:

local Collections = require(ReplicatedStorage.Collections)
local lavaParts = Collections:GetCollection("Lava")

lavaParts.Touched:Connect(function (hit)
  local humanoid = hit.Parent:FindFirstChildWhichIsA("Humanoid")
  if humanoid then
    humanoid.Health = 0
  end
end)

lavaParts.Material = "Neon"

It would be awesome if there were similar APIs to the ones we have for tags on CollectionService that take queries instead. With that I would extend the library to support them:

local Collections = require(ReplicatedStorage.Collections)
local lavaParts = Collections:GetCollectionFromQuery("BasePart[BrickColor='Bright orange'][Material='Neon']")

lavaParts.Touched:Connect(function (hit)
  local humanoid = hit.Parent:FindFirstChildWhichIsA("Humanoid")
  if humanoid then
    humanoid.Health = 0
  end
end)

[index_self]

While I’m sympathetic to your problems with replication and streaming, and I have nothing against what you’re proposing, I think the need for any :WaitForChild or similar yielding/reactive feature is actually symptomatic of a larger engine issue.

With streaming disabled, all instances are replicated to the client reliably. That means if you create a new model in the workspace and fire a remote to a client, that model is guaranteed to have been replicated on the client before the remote is received. Streaming breaks this behavior by making instance replication unreliable, forcing you to have to resort to these hacky pick-up systems for the replicated objects.

Now, this may seem a bit disconnected with that you’re talking about, but if “waiting for a descendant with some classification to be streamed in” is such a strong motivation for you, then I believe that’s really a deeper problem with streaming rather than DataModel querying. That isn’t to say your other motivations aren’t good–things like instance comparisons and dynamic hierarchies are all perfectly valid reasons to have this. I just wanted to point out the other side of this specific use case.

[jonbyte (Jon)]

@index_self

With streaming disabled, all instances are replicated to the client reliably. That means if you create a new model in the workspace and fire a remote to a client, that model is guaranteed to have been replicated on the client before the remote is received. Streaming breaks this behavior by making instance replication unreliable, forcing you to have to resort to these hacky pick-up systems for the replicated objects.

I always use StreamingEnabled for all my projects and Roblox also tends to promote it. Not using StreamingEnabled would not be a solution. The fact that is unreliable warrants features existing to code reactively without hacky custom solutions. DataModel query behavior is a powerful concept that should be expanded on.

[index_self]

Sorry, I wasn’t insinuating that you should stop using streaming to solve your problems. I was just mentioning that there may be a deeper-rooted issue here that is masking your need for more reactive solutions. I agree that DataModel querying is a powerful feature and should be expanded upon.

However, I disagree that the unreliable nature of streaming should be solved with more client reactivity. If anything, reactivity is the last thing you would want because it puts the whole burden of keeping everything in sync on the client all upon you. A better solution, I believe, would be to somehow mark certain critical models to always replicate reliably so you wouldn’t have to worry about them showing up asynchronously with streaming.

[jonbyte (Jon)]

@index_self

However, I disagree that the unreliable nature of streaming should be solved with more client reactivity. If anything, reactivity is the last thing you would want because it puts the whole burden of keeping everything in sync on the client all upon you.

I don’t really understand what you mean by this. There is pretty much no other solution than for the client to react or wait until streamed content exists. Also by “reactivity” I am referring to the commonly implemented observer pattern. It operates on what does exist and also listens for changes or additions.

I don’t see this as a burden and the client listening on change or add is also not an antipattern. It needs to be more well supported with query capability.

@index_self

A better solution, I believe, would be to somehow mark certain critical models to always replicate reliably so you wouldn’t have to worry about them showing up asynchronously with streaming.

I am pretty sure this is already supported with Persistent streaming for models. Not something you want for every instance with attached behavior (it defeats the purpose of streaming). I am unsure because I have rarely had a use for it yet. It seems great for very specific instances that should always exist, but not sets of instances with same behavior needing to be attached.

[index_self]

What I’m arguing is that you don’t need these patterns to solve the problem; they’re suggestive of asynchronous timing issues and only seem appropriate in these cases because they’re your only current option.

An alternative is making the engine guarantee certain models replicate synchronously, so you don’t need observer patterns in the first place. When the server creates a new model in the workspace, you could just send over a reference of it to the client, and the client can immediately start working with it.

---

@jonbyte

I am pretty sure this is already supported with Persistent streaming for models.

They’re close, but they only guarantee static models replicate before the client loads. If a persistent model is replicated dynamically, there’s still no guarantee it replicates before any remote events are received.

I’m actually surprised that you don’t use persistent models for any critical gameplay purposes. It’s extraordinarily frustrating for me to keep track of important stuff when they constantly stream in and out even with observers, and I can’t imagine how you can stand doing that for everything. It’s a breeding ground for race conditions.

[jonbyte (Jon)]

@index_self

An alternative is making the engine guarantee certain models replicate synchronously, so you don’t need observer patterns in the first place. When the server creates a new model in the workspace, you could just send over a reference of it to the client, and the client can immediately start working with it.

This could be a valid solution but this isn’t the objectively best solution possible. The client would have to listen for the replication event anyway. Reactive coding is definitely not an antipattern, but some may not prefer it. This is also deviating a bit from the topic of this feature request.

[TheGrimDeathZombie (Reapimus)]

@jonbyte

but this isn’t the objectively best solution possible

If you’re running code where there’s an expectation that the instance the server is referencing in a RemoteEvent/Function call to the client is replicated, then objectively the best solution possible is that the client doesn’t receive this event until it is confirmed the client has received the instance.

[jonbyte (Jon)]

Hopefully this will clarify what I am trying to say:

The goal is not to assume the instance should exist, but “react” to when it does exist. Forcing existence defeats the purpose of streaming. For example, say I had street lamps spread across the entire map and I want to attach behavior when they exist. Forcing them to always exist is not desired behavior. This is especially critical on low memory devices where we would want things to stream out when possible. A street lamp is not an important asset that should always exist if it doesn’t need to.

(With querying the example may not be as simple as a street lamp. For example, if the classification of an asset is hierarchical.)

tldr: You shouldn’t have an expectation that streamed assets exist when they are large quantity, same behavior; not “critical” for gameplay. If they are critical for gameplay just use persistent streaming or the solution that person proposed.

---

Bringing it back to the topic of this feature request: the current QueryDescendants reads whatever the current DataModel state is. Streamed content is an example of where that function is not useful with. And to reiterate the query would not be an exact name match; it would either use tags or follow a naming convention (or any query). The asset may not be gameplay critical. Generally, an instance can fail to match a query after it already exists. But, there’s no way to catch that without hierarchy/tag events or polling properties. The goal of this feature request is to explain these issues and only suggest a one service with a more expressive syntax for querying with compatibility with current syntax (for DataModel). This one service has a clear API and is scalable to multiple domains.

[crusherfire]

This would be nice, and I think having more selector capabilities (like checking for greater than, less than, etc.) would be good.

Additionally, being able to search for all instances with a given property, such as LocalTransparencyModifier, would make querying much easier.

And I agree with @index_self that working with instance streaming is annoying. The only reliable solution I’ve found that works best is to use observers and tags to detect when things stream in, stream out, and mark stuff that needs to stay as persistent. I haven’t run into any race conditions with an observer setup.

Yet streaming performance sucks when stuff is loading in and out because of frame spikes, bandwidth spikes, etc.

If my projects are small enough, I always disable instance streaming, because being able to directly index for instances is much nicer than having to set up an observer nightmare.

[TenebrisNoctua (Noctua)]

Coming soon under Server Authority!

[jonbyte (Jon)]

We have something even better now than what I suggested.

[Studio Beta] Collections

Hey Creators, Collections are now available as a Studio Beta. Collections build on CollectionService, the same service you already use for tags, and add a new way to group instances and run code on them. Collections are designed to: Make it easier to write code for streaming Extend the behavior of CollectionService to support more ways of grouping instances Reduce the amount of boilerplate needed with CollectionService today What are Collections? A Collection is a group of instances defi…