Collections: How to group instances with query patterns

Hi Creators, we are releasing an extension to the CollectionService called Collections. A Collection is like a set of tagged instances except that the tag is a query that defines the set.

Collections

We think of collections as a souped-up version of tags. Collections use a query instead of a tag; the collection automatically includes any instance that matches the query. Collections also have an interface for bulk-binding callbacks to events on instances and RunService.

Benefits

  • Powerful extension to tags: Simple declarative interface for managing groups of instances, fulfilling feature requests like this.
  • Reduces Boilerplate: Eliminates the need to track per-Instance signal connections, and lets you avoid manual Instance set management.
  • Easy Streaming: Collections only include streamed-in instances – they work like the Tags + Streaming workflow but are far more expressive.
  • Reduces Leaks (maids): Collections manage signals on instances automatically as instances enter and leave a collection.
  • Efficient DataModel Searching: Collections reduce the need to FindFirstChild and WaitForChild, because the query searches entirely in-engine.

Example


-- Creates a Collection of all BaseParts named KillOnTouch
local collection = CollectionService:CreateCollection("BasePart#KillOnTouch")

--OnAdded will fire whenever a DataModel change causes an instance to join a collection
function collection.OnAdded(killBrick: BasePart)
    killBrick.BrickColor = BrickColor.new("Neon orange")
    killBrick.Material = Enum.Material.Neon
end

-- Any signal on any type of instance can be bound to by name, with the “On” prefix.

function collection.OnTouched(killBrick: BasePart, otherPart: BasePart)
    local parentPart = otherPart.Parent
    local humanoid = parentPart:FindFirstChild("Humanoid")
    if humanoid then
        humanoid.Health = 0
    end
end

Query Syntax Reference [Instance:QueryDescendants]

-- Parts Named Wall
"Part#Wall"
 -- Children of any Model with a CollectionService tag “SwordPart”
"Model > .SwordPart"
-- Descendants of any Meshpart with an attribute "OnFire" set to true
"MeshPart >> [$OnFire = true]" 
-- MeshParts with no SurfaceAppearance or Texture children
"MeshPart:has(> :not(SurfaceAppearance, Texture))" 
 -- Players within 30 studs of TargetBuilding
"Model:has(> #HumanoidRootPart:in-radius(30, .targetBuilding))"

Why Collections

At a high level Collections solve the problem of binding code to groups of instances. If you want every enemy NPC with 1 health point to enter rage mode, you write a query that matches only those instances (“.NPC.Enemy[$health = 1]”). Code attached to this collection runs only on instances while they match that criterion. The alternative is to track two tags and wait on attribute changed events to check health.

We think Collections can help with multiple error-prone patterns on the platform today. Memory leaks are a common problem when connecting callbacks to instance events. Collections automatically manage connections so their callbacks are leak-free. DataModel searching with FindFirstChild and WaitForChild requires boilerplate for error checking as well as a working knowledge of Streaming. Collections populate automatically when instances match their query, so you don’t need to check whether an instance exists.

Details

Collections use engine queries which are also supported in our Style Engine and QueryDescendents. You can set a root Instance for the Collection, and it will only query for Instances descending from that root.

Collections can bulk bind callbacks to engine and instance events. In the KillBrick example every BasePart in the collection has the OnTouched function connected to the BaseParts Touched event. The first argument to every callback is the Instance the event fired on in the Collection. Connections are managed automatically. When an instance is added to a Collection, callbacks are connected to its events (and disconnected when it is removed).

Collections have permissive typing. A Collection with an OnTouched function containing both Folder and BasePart instances will connect the OnTouched function to the Touched events of BaseParts and ignore Folders. Generic Events like Added/Removed or RunService Events are always bound to all Instances.

Full API

CollectionService

  • CreateCollection(query: string, rootInstance : Instance) : Collection
    • query uses QueryDescendents selector syntax
    • query only applies to the subtree under rootInstance (default workspace)

Collection

  • Collection:ForEach(function(instance) end) – Calls the function on each Instance in the Collection, passing that Instance as the first and only argument
  • Collection:Destroy() – Removes every instance from the collection and stops tracking the collections query.

Valid Callbacks

Events on any Instance type can be hooked into a collection by defining a function with the event’s name prefixed by On: function OnTouched(collectionPart :Part, otherPart :Part) … end. The first argument is the Instance the event is firing on. This instance is always in the collection. The signature of every event callback is modified to include this instance.

Collections bind Instance Events to callback but do not use calls to Connect or return an ScriptSignal. When Instances are added and removed from the collection they are automatically connected to the callbacks defined by the Collection. Property and Attribute changed events are all available under the OnAttributeChanged and OnPropertyChanged namespaces. You cannot assign to a callback more than once (but we’d love some feedback on this point)

Callback API

  • function Collection.OnAdded(instance)
    • Fires when an Instance starts matching the Collection’s query
    • When assigned, this also fires on all Instances currently in the Collection (we’ll take feedback on this)
  • function Collection.OnRemoved(instance)
    • Fires when an Instance stops matching the Collection’s query
  • function Collection.OnHeartbeat(instance, dt: number)
    • Connects to RunService.Heartbeat
    • All non-deprecated RunService signals are supported (such as OnPostSimulation, OnPreRender, etc.)
  • function Collection.OnSimulate(instance, dt: number)
    • Works like BindToSimulation()
  • function Collection.OnPropertyChanged.{PropertyName}(instance)
    • Works like GetPropertyChangedSignal(“X”)
  • function Collection.OnAttributeChanged.{AttributeName}(instance)
    • Works like GetAttributeChangedSignal(“X”)
  • function Collection.On{EventName}(instance, …Args)
    • Connects to “EventName” on every Instance with “EventName” in their API (i.e., Touched, Pressed, etc.)

Added/Removed Semantics

Added and Removed are the backbone of Collections. These callbacks fire when an Instance matches or unmatched a Collection’s query. A Collection can exist prior to Added being set. When Added is first set it fires for all instances in the collection (see below). Remove fires for each instance when Destroy() is called.

-- Creates a Collection; All parts under folder KillBricks are killbricks.
local collection = CollectionService:CreateCollection("Folder#KillBricks > Part")

--collection is populated, but added is not assigned
function collection.OnAdded(killBrick)
    killBrick.BrickColor = BrickColor.new("Neon orange")
    killBrick.Material = Enum.Material.Neon
end

--added fires In immediate mode or is placed on the deferred queue here.

Query Performance

Tracking instances for Collections is not free. Each Collection adds a bit of runtime overhead when a DataModel change happens that could cause an Instance to be Added or Removed. The cost of each Query is related to what the query tracks. Below is an approximate chart of Query selectable data and its performance cost for a Collection.

:green_circle: Improves performance

  • RootContext
  • Class

:yellow_circle: Small cost

  • Tag
  • Property
  • Name
  • Attribute
  • not(…)

:red_circle: High cost

  • > (child)
  • >> (descendant)
  • has(…)
  • in-radius(suds, …)

Using a rootContext with a small subtree of your place is the best way to maximize Collection query performance. If the context is workspace or a large part of the DataModel the Collection must watch a large number of Instances to determine what belongs in the collection.

Adding a class specifier makes queries both faster and enables better typing. For example, Part#Killbrick is preferable to #Killbrick. Including the class allows the Collection to ignore all non-Part Instances, reducing the amount of work required. It also improves the developer experience by enabling more precise autocomplete and type checking, since the query is guaranteed to return Part Instances.

Internally, a Collection tracks changes to the DataModel to determine which Instances belong to it. The specificity of the query determines which changes must be observed. For a query like #Killbrick, the Collection must listen for name changes on every Instance because any Instance could potentially match. With Part#Killbrick, only Part Instances need to be monitored for name changes, significantly reducing the amount of bookkeeping.

A similar optimization applies to some property filters. For example, [Anchored = true] implicitly limits the query to classes that inherit from BasePart, allowing the Collection to ignore all other Instance types.

In contrast, tag and attribute specifiers do not make queries faster. They cannot be used to narrow which Instances must be monitored. In fact, they make queries slightly slower because the Collection must register additional listeners to detect tag and attribute changes.

Combinators are very expensive. Each combinator adds a layer of complexity to our tracking logic. Child queries are relatively easy to evaluate while Descendent combinators require extensive tracking of an entire subtree.

The Has pseudoclass is an expensive part of a query to evaluate. Has requires multiple evaluation steps up and down the DataModel hierarchy on each DataModel modification.

The in-radius is currently expensive to evaluate. In-radius must calculate an intersecting set of instances added to, and removed from the radius to generated added and removed signals.

In-radius

In-radius is a new, experimental addition to the query syntax. It is currently supported only by collection queries. in-radius selects BaseParts that are within n studs of a selector group For example: Part#Killbrick:in-radius(10, Part#HumanoidRootPart) selects all Parts named Killbrick that are within 10 studs of any Part named HumanoidRootPart. The radius is measured from the center of the selected Part to the nearest edge of the selector Part.

Lifetime

Collections are active until Collection:Destroy is called or the Collections root is Destroyed or garbage collected. Collections are intended to be long lived. They are kept alive by the CollectionService even if their reference returned by CreateCollection goes out of scope. Unparenting a Collections root will not cause the Collection to be garbage collected or stop running.

Query Typing

The first argument of a Collection callback gets its type from the query and from the callback. If a Query has no type information the return type will be Instance. For example if the query only checks name #SomeInstance, attribute [$SomeAttribute] or tag .SomeTag no information about the instance can be learned.

Typing information can be extracted from Class filters. The query ”Part” will type all instances in the collections as Parts, ”Part, Folder” will type each instance as Part or Folder.

The has() pseudoclass can give type information to Instances children. The Query Part:has(> Part#SomePart) lets us know that the selected part will always have a child Part named SomePart. Callbacks on this typed instance can safely access the instance’s child, including on the Client. There is no need to WaitForChild() on a child specified in the query.

local collection = CollectionService:CreateCollection("Part:has(>Part#SomePart")
function collection.OnAdded(part)
    part.SomePart.Position = Vector3.new(0,0,0)
end

Typing information is also extracted from Instance specific callbacks. If a collection may select many types of instances, but a callback can only be bound to a specific class that callback will get the correct typing information. For example:

local collection = CollectionService:CreateCollection(".FunnyTag")
function collection.OnAdded(instance)
    --the argument here will be typed as an instance
end

function collection.OnTouched(basePart)
    --the argument here is typed as a BasePart because this callback
    --will only ever fire on baseParts.
end

Explorer Search

We have added query: as a bespoke filter to the studio explorer search. Query accepts our query syntax and will display every instance that matches the query as a result.

Demo Place

We have created a demo obby filled with Collection examples called CollectionObby.rbxl (227.1 KB) The collection obby has a 5 level obstacle course and a few aesthetic elements implemented with Collections. It also includes a small museum with many of the collections.

What’s Next

Performance optimizations

Collections introduce tracking overhead, we’ve designed collection tracking very close to the CSS selectors in Blink. There are multiple avenues for performance improvement taken by Blink that we can implement in collections as well. RunService calls like OnHeartbeat can also be improved to reduce overhead invoking a large number of heartbeat invocations.

Extended Query API

Our current Query API only selects individual Instances. We have plans to extend the query syntax to capture multiple Instances per query, and to only data on Instances like properties and attributes. The goal of Query extension is to give an ECS-like frontend that enables fine-grained data parallelism like flecs.

Discoverability

A common problem on Roblox is identifying what code caused something to happen. We plan to extend the Collection API to check what collections are running on an Instance example (Instance:GetCollections).

FAQ

How does this relate to server authority?

Server authority requires the coordination of predicted behavior on the Client along with authoritative behavior on the Server. We believe that collections can simplify this coordination by providing a declarative way to specify Instance behavior. We believe that writing a single collection and instantiating it on both Client and Server is a simple way to align predicted and authoritative code.

How does this work with streaming?

Collection membership is automatically managed. Instances in a collection are only present when they are streamed in. Collection code written per-Instance thus avoids wait-for-child as the code can safely assume that the Instance is streamed in. We recommend the use of Atomic Modes in conjunction with Collections to simplify writing streaming safe code.

Why not use a library to build a string query?

Go for it! Collection query syntax is heavily related to CSS selectors and there is a rich ecosystem of query wrappers for css (JQuery, React, Cheerio). The main drawback is that using a library reduces what typing information we can provide for Collection callbacks. Using a string literal directly enables strict type checking and better auto complete.

Why ForEach and not __iter?

Collections are a forward thinking API, we hope to in the future support parallel execution across collections and within them. Ideally a safely written ForEach could be dispatched in parallel to increase performance. __iter on the other hand does not allow us to make use of this future parallel scheme. We may still release an __iter in the future as it can improve serial performance and may increase the usability of Collections overall.

62 Likes

I’ve been out of the scene for 3 months

1 hour ago I started a conversation about this precise and exact flaw with collection service, the second Roblox conversation I’ve had in 3 months

It took 1 hour for this post to come out following the conversation

Roblox I am impressed.

13 Likes

This CollectionObby experience cannot be viewed at the moment.

3 Likes

Thanks for the heads up, I’ve replaced that link with a place file that can be directly downloaded.

4 Likes

LOVE THIS

I am an avid user of QueryDescendants, so it’s seriously great to see this used in more places, ESPECIALLY Search bar.

Couple things:

  1. Why did you choose tags to be . and name to be #? Seems to me that it’d be significantly more fitting for those to be the other way around
  2. Can you introduce a conversion which converts GetFullName() to a compatible path using your new query system? You can recreate the path with a series of [Name=…] > [Name=…] but it’s a little janky. Would love if you had a way to convert that for us automatically, it’s the one area that makes GetFullName() pretty annoying to use

Absolutely amazing nonetheless!

5 Likes

Is this live?

local CollectionService = game:GetService("CollectionService")

local collection = CollectionService:CreateCollection("Lava.KillBrick")

function collection.OnAdded(brick: Part)
	print("added", brick)
end

3 Likes

Ah after updating studio, the error now says Collections are not enabled

4 Likes

I believe it’s one more restart with the beta feature enabled to get the flag working.

5 Likes

The allocation is slow, unfortunately as it is a lot of extra compute. Is this because you need a lock for the results and there is no guarantee for whether the data is stable to be passed to the returned array due to the native->luau barrier? If I may make an educated guess onto one of the concern is that the data can be moved around during iteration? Speaking out loud for a second, I am sure several ideas have been tried out including treating membership into collections as deferred commands to avoid this problem but I am sure this was discussed already and that would be both a huge undertaking, additional cognitive overhead, etc. But as it is, you can’t query every frame and that is a dealbreaker. But I think there is actually alot of iteration optimizations you can do to look up a batch of entities at once during iteration to quickly see which ones get prefiltered.

From what I understand this is quite the understatement. They would have significant overhead for testing membership to join a collection but also during invalidations that can get invoked by property, attribute, tag, scene graph changes that you have outlined. Especially for games with streaming enabled. If I may be crude, it is unusable at the moment in terms of performance. But I have lots of ideas on how you can speed this particular problem up for streaming enabled.

There are two different parallel mechanics in flecs, one is inter system and the other is assigning workers for some batches of entities, I am assuming you mean the latter with regards to stuff like world.set_threads that assigns workers to each batch of entities. In my experience that only starts scaling for rather big N.

I have studied Blink a lot, and I think that will at best help you with low hanging fruit. You mentioned flecs before, I highly urge you guys to explore optimisation techniques employed in query iterators that I think are far better that can actually look up multiple entities in batches instead. By default always testing membership per-instance won’t be very optimal. You need to care more about avoiding redundant invalidations and reevaluations rather than per instance testing optimisations.

P.S.
I am the author of jecs that have implemented a fast archetypal ECS and newly added a DontFragment (EnTT-like Sparse Sets) storage inspired by flecs. I have studied this particular subject a lot and implemented the exact feature that you are working towards with regards to observer/groups-like collections so as you can tell I have the enthuasism to discuss the implementation details.

Also been chatting with Sander Mertens the author of flecs and he seems to appreciate the mentions! Good people he says :smile:. So good on ya for the mention! Anyways would love a chat about the implementation, and see what I can lend help with in consulting this.

5 Likes

Thanks for taking an interest.

Did you notice a lot of overhead while using a Collection with streaming enabled in studio? We have some optimizations in place at the moment and a long list of ones we would like to make in the future. If there is a specific case or query you are working with that performed quite poorly we can try to address it directly.

Yes I 100% agree that’s the crux of building a fast membership engine. At the moment we only have a few indexes built that let us speed up or avoid invalidations. Our current cost for re-evaluating a tag added or a property change for simple queries around a microsecond and we plan to improve that.

3 Likes

This is extremely powerful! Super excited to see all the ways this will be used in the future! I was testing it out but it seems to be erroring every time I run the code.
“OnAdded is not a valid memeber of CollectionHandle”. So I am unable to actually test this out :sad:

Here is the code, please let me know if I am doing anything incorrectly here!

--//Services
local collectionService = game:GetService("CollectionService")
local players = game:GetService("Players")

--//Variables
local killBrickCollection = collectionService:CreateCollection("BasePart#KillBrick")

--//Functions
local function onKillBrickAdded(killBrick: Part)
	killBrick.Color = Color3.fromRGB(255, 0, 4)
	killBrick.Size = Vector3.new(2, 2, 6)
	killBrick.Material = Enum.Material.Neon
end


--//Signals
killBrickCollection.OnAdded(onKillBrickAdded)

Thanks!!!

3 Likes

You are suppose to assign a function, not call OnAdded.

killBrickCollection.OnAdded = onKillBrickAdded
1 Like

Oh haha my fault was looking at it through the lens of an event. Let me fix this!

1 Like

Is SelectorSyntax meant to link to https://roblox.atlassian.net/browse/API-126?

1 Like

No. Thank you for the catch this was an oversight.

3 Likes

My guess is that they were following along CSS, which uses . for class and # for ID. In CSS, the ID # is intended to be unique and class . is intended to be reused, which seems to match the case for names and tags (names # are intended to be unique while tags . are intended to be reused).

In Roblox’s case, it would make more sense to associate # with tags, but I think that was the general principle that they followed when deciding which symbols to use.

5 Likes