Great but when is it going to be client beta and released in the roblox player?
1 Like
We’re looking at ways to support both game development and new gameplay capabilities over time. For now, we want to make sure we iterate on these playtest agents while they are in beta.
4 Likes
Can I create my own tests similar Tests written for Test driven development (TestEZ, Jest) and have the Agent do the same test every time and not generated each session for consistency?
I have always wanted to have automated testing for this but having to manually type to the AI every time is not ideal
Let me know if I’m misunderstanding
1 Like
Pretty cool use of AI! Automated functional tests based on just text input would be awesome, not just to the vibe coders but also for eg pre-publish checks.
Biggest pain point for me when it comes to testing are multiplayer tests, it would save me a ton of time if I could just task:
“Start a multiplayer test with 2 clients. Client 1 clicks on client 2, select “high five” and client 2 accepts. Observe whether the animation starts successfully and is synced”
Going one step further would be an autonomous QA tester who would not just test the happy path, but also possible issues like incorrect input & glitches. A lot of development time goes into playtesting, so it’s super interesting to see how this tech will advance!
4 Likes
I think the Asset manager shows you the decals and stuff inside studio without going to the website. I don’t know if AI can see can read that data
1 Like
Yes, you can batch run multiple test cases consecutively that your AI agent reads from a file and assigns to the playtest subagent. We’re working to reduce this, but the subagent can report a test as “Pass” when a human playtester would spot an issue. We advise against relying solely on the playtest subagent to verify gameplay.
1 Like
Question:
Can the playtest agent interact with UI objects? (ie, Clicking/Dragging)
My game-in-construction relies solely on UI, requiring players to click, drag and hold on different elements on screen.
1 Like
The playtest agent is able to read UI and click using the Studio MCP mouse_input tool. However, the current form will not be reliable for dragging and holding in this early beta stage.
2 Likes
Ah, alright. Thanks for the info!
1 Like
@Mirrattar So I gave this feature a try with the beta enabled using the exact quote “using the playtest subagent” that you described.
As you can see, it worked like a charm! Completely functional:
It even understands Luau syntax!
2 Likes
Ah, thanks for sharing this. We’ve observed an issue in which sometimes creators are required by Assistant to approve the use of the start_stop_play MCP tool from inside of Studio. Do you know if that appeared for you? I see the playtest agent’s report notes a lack of access to that tool as a blocker in its reporting of the test as Inconclusive.
1 Like
No such prompt appeared anywhere.
1 Like
Thank you again for sharing this. We’ve been digging into it and caught a bug that has temporarily affected the ability for the subagent in Assistant to call MCP tools. We’re working on a fix now that should resolve this issue shortly and make it possible for the test session to start.
1 Like
The AI was also hallucinating and straight up using wrong luau syntax.
1 Like
playtest subagent is not able to simulate keyboard or mouse input, or clicking on a Gui button
1 Like
We believe this is related to the bug in which the playtest subagent lost access to several MCP tools including those to simulate input (and caused some of the erratic behavior). We released a fix for that tool access today. Please let us know if you continue to see the issue persist, and any additional details, and we’ll work to resolve. Thank you!
1 Like
If you ask the agent to accomplish three different subagent test tasks, it will fire off its subagents doing the exploration of the workplace model (codebase/workspace) and start streaming its chain of thought for all three tasks simultaneously. It seemingly is only actually controlling Roblox studio for the first subagent. So, it’s just burning tokens on two sub-agents that are doing information gathering and faking their act behaviors.
It says in it’s initialize response before firing the subagent it will test them one at time, butttt that’s a lie.
This also means that when the first task is complete, the second and third tasks don’t properly start because they think they’re already running, so they never re-activate.
I assume for now this is a rare edge case that doesn’t require addressing, but it does seem to show that there’s no stack or gating mechanism for subagent testing tasks.
This also results in what may be a timeout issue edge case, where the second or third task sits at an ACT cycle, and because it was running without feedback from a real studio-controlled playtest, it stalls out and gets you stuck at a “Generation is taking longer than expected” state. This requires hitting the “Stop” button to get back to a clean state.
I recommend adding a queue/stack for the agent to be able to delegate to itself a series of subagent executions that wait until the previous sub-agent is done.
On the upside, it’s almost as if the multiple subagents per Roblox Studio use case is ready to be supported! The queue/stack/gate can still be useful there as you’ll need to assign the relationship between sub-agent and playtest session/feedback stream.
I did test the same prompt again a few times, and it did seem to actually be capable of doing sequential usage of the sub-agent, only moving on after the first sub-agent completed, but the fact it’s variable performance trending towards the bad behavior indicates it may need either system prompt adjustments, or to give the agent a task sequencing tool to add/remove from it’s subagent queue, and add direct logic to the sub-agent tool that requires to to pull from the front of the subagent queue with the gating mechanism when starting/stopping subagents that can block starting a new one if there’s already one ongoing so it can’t just spam pop the queue. (Successful sequencing example below).
This will be more valuable/relevant as you get to delegating out multiple sub-agents in the same Roblox studio session/machine, but still slightly relevant currently!
It would be cool to be able to just drop a massive prompt in, asking the agent to test the 10 latest things built in the game, and walking away from the computer for an hour while the agent sequentially goes and tests them.
Thanks if you actually read all that! =)
Here’s a specific case as well where I confirmed I had focus, and it was unable to start the playtest.
Once it has it’s playtest plan, it says it’s starting playtest and beginning investigation, but I see no evidence in the Chain of thought logs that indicate it ever actually started a playtest or similar tool usage. It then has no confirmation mechanism to realize it’s still in edit mode and not in a playtest.
For example in the logs, it’s trying to find the “LocalPlayer” but saying it doesn’t exist yet, and it thinks because play mode isn’t properly running or because the player hasn’t spawned yet. It then takes 4 different additional observe cycles to try and guess if it’s in a playtest or not. Clear opportunity for a tool use or similar functionality feedback mechanism here. (I stopped the agent at this point, it didn’t break itself out of the loop, was tired of watching it struggle.)
I recommend improving the sub-agent prompt, clarifying the tool description for whatever is starting the playtests, and adding an additional tool or describing a method to the subagent to do an audit of if it’s actually in an active playtest scenario, or if it’s just investigating the workspace thinking its a live playtest instance.
Slightly unrelated question but maybe relevant: Does the sub-agent require the Roblox studio instance, or the game editor window to be in-focus?
I ran into a sub-agent use case where it failed to ever start a playtest, even though it attempted to across three sequential playtest scenarios.
(Summary screenshot from the agent)
Vague prompting leading to poor results would be massively improved if the subagents used visual feedback/screenshots to get more game context. For example if I ask it to use a subagent to go chop a tree in my tree chopping game, it has no idea what to do and just runs up to the tree, does some observe loops, and doesn’t figure it out.
If I tell the agent “Go chop a tree down, equip your woodcutting axe first, then use the proximity prompt modal to swing the axe.”, it goes and figures it out. It does this by doing these actions since it’s already been told there’s an axe to equip, and you swing with a proximity prompt.
In the first scenario that was too vague, if it had a screenshot tool available with clear instructions on when to use it, it could have realized when it was next to the tree that there’s a proximity prompt available and it has an inventory with an Axe in it.
Screenshots in the loop will reduce the cognitive load required to achieve user’s goals relative to the amount of detail provided in prompts.
TLDR: Interested in this being available to external/cloud-based tools too, ideally via a plugin accessible API or OpenCloud.
The cloud-based tools already have thousands of developers using them who can’t reach the subagent functionality through the 3rd party tools they use. Letting external MCPs forward to the subagent during beta would meaningfully expand the feedback surface you’re iterating against, on a population that’s specifically self-selected for AI-assisted Studio workflows.
External MCP servers often have more capabilities than the native Studio Assistant. They could use those tools to gather the context needed to write a richer subagent prompt, raising its likelihood of success. (Let us also customize the tools available to subagents?)
Two shapes that could unlock this, each serving a different use case:
-
An OpenCloud endpoint that lets an authenticated external service kick off a playtest run against a place on Roblox’s infrastructure, fully headless, no local Studio in the loop, with a structured result returned when it finishes.
- You already shipped LuauExecutionSessionTask which lets external tools execute Luau headlessly against a specific place version on your infrastructure (10 concurrent tasks per place may be worth revisiting for the regression testing case), and the docs cite “automated testing as part of a CI/CD pipeline” as a use case for it. The gap between that and a playtest agent endpoint is well-defined: a sibling endpoint or extension that brings up an actual playtest session with clients and physics enabled, runs the playtest subagent inside it, and returns the structured Pass/Fail/Inconclusive report. The “test work-in-progress changes without touching the live game” piece is already reachable today via a staging-place pattern (create a dedicated Staging sub-place in your universe once, OpenCloud-publish new versions to it from CI, run tests against those versions independent of the live slot), so I think the place-management plumbing is in place, the missing piece is the playtest-agent execution context of a spun up fully headless Roblox Studio.
- This enables parallelizable, headless regression and feature testing. A developer’s tool could analyze the game, generate a set of regression or new use cases (Can players buy from the new shop? Can they still complete the tutorial?), then fan those out as N concurrent runs against the staging version(s), and get results aggregated/rendered by the tool. The equivalent of running a true integration suite on every commit. Cost stays bounded because each run is metered through whatever billing model OpenCloud uses and developers self-rate-limit on their own budget. Maybe scale the budgets based on whatever game performance metrics you have so big games that need more usage and have the income to justify the costs can be appropriately served?
- The orchestration (test generation, fan-out, fan-in, reporting) lives in the developer’s tool, Roblox just exposes the primitive. Developers could spin up the rough equivalent of a QA team at scale, on demand.
-
A plugin-callable service (something like PlaytestService:RunSubAgent(prompt)) that exposes the same capability from plugin Luau. This would be the path that makes the playtest agent reachable to a developer who’s actively working in Studio with a cloud-based AI tool open in their browser. Their tool’s plugin can trigger the sub agent in their live Studio session and they see it run in their viewport, just like the official integration does today. No dependency on StudioMCP.exe, no extra binaries on the user’s machine.
I appreciate the cool stuff you all have been making available!
Exciting signals I’ve seen from Roblox team members in media lately:
-
“44% of the top 1,000 creators on Roblox use Roblox assistant or third-party AI tools via MCP…” - Shared by @ayyar on X (“Kartik”). (“Opinions not my employers.” disclaimer as well on his X).
-
“We have the philosophy of we want to enable creators to use whatever tools they want. So we want to meet creators where they are in the age of AI at Studio” - Lynn Yu Gong on the latest Roblox Tech Talks
Two additional questions:
- Is user_keyboard_input / user_mouse_input bound to Studio’s editor UI layer or to the Roblox engine’s input system? The fact the tool accepts instance_path targeting makes me think it’s engine-layer. If it’s bound to the editor UI layer the Opencloud approach would be more difficult to implement I would think.
- Is there documentation on the playtest types (playtest vs. search vs …?), and the “structured report” that is returned at the end?
Thanks =)