[Client Beta] In-experience Mesh & Image APIs now available in published experiences

Hello Creators,

We are excited to announce that you can now publish experiences with the EditableMesh and EditableImage APIs. With this update, these APIs are now available in Client Beta. This means we do not foresee any more breaking API changes and you can confidently publish experiences with these APIs. We are still considering this a beta release as we are going to be tweaking the memory budget and re-evaluating the permissions restrictions over the next few months, and this might impact what you can do with these APIs. Please see the “Best Practices” section for more information. This update also addresses a number of the known issues called out in our previous Developer Forum post.

With that, let’s dive into the details!

image2

Enabling the APIs

Since the EditableImage and EditableMesh APIs are powerful new APIs that require extra care to ensure compliance with our Terms of Use, you must be 13+, ID-verified, and explicitly opted-in to using the APIs in published experiences. ID verification allows us to increase creator accountability for APIs that could result in policy violating content on the platform.

  • You can visit Roblox.com > Settings to check if you are already ID-verified. If you’re not, complete the ID verification workflow.
  • Once you are ID-verified, go to Studio > Game Settings > Security and enable the “Allow Mesh / Image APIs” toggle (see screenshot below). Remember to review the Terms of Use before enabling the toggle.
  • If the experience is owned by a group, the group owner will need to be ID-verified and perform the above steps for the experience.
  • Please review the Best Practices section below for more details and tips to keep your experience safe

Note: You can use these APIs within Studio plugins (edit mode) without enabling this toggle.

Summary of APIs now available for published experiences

EditableMesh APIs

EditableMesh gives you direct access to the vertices, faces and attributes of a given Mesh. They can be procedurally created from scratch or created from an existing asset or MeshPart instance.

Lifecycle APIs

Click to read more:
  • AssetService:CreateEditableMesh() creates a new empty EditableMesh object for procedural creation.
  • AssetService:CreateEditableMeshAsync() creates an EditableMesh object from an existing mesh asset or an existing MeshPart instance’s mesh content
  • [New] AssetService:CreateMeshPartAsync() converts an EditableMesh into a new MeshPart instance
  • EditableMesh:Destroy() marks the object for garbage collection to free up resources
  • :warning: EditableMesh objects can only be created from mesh assets that are owned by you (or your group if the experience is owned by a group). Attempting to load an asset into an EditableMesh that you do not have permissions to will result in a permissions error.
  • :warning:The above creation APIs are subject to memory usage limits and might return nil if the device (or server) is out of memory reserved for Editable* objects. Always remember to check if your EditableMesh object was successfully created before using them in your scripts.

Direct access APIs

Click to read more:

EditableMesh Documentation

Vertices and Faces:

  • AddVertex adds a new vertex and AddTriangle adds a new triangle face to the EditableMesh object. RemoveFace removes a specific face, and RemoveUnused will remove unused vertices and attributes.
  • GetVertices,GetFaces and GetFaceVertices retrieves all the vertices or faces of the mesh or the vertices of a specific face that you can then iterate through. Note: Vertex and Face IDs are stable even as you add or remove new vertices or faces.
  • Get/SetPosition allows you to retrieve or update the position of a specific vertex ID.

Attributes:

  • AddColor, AddNormal and AddUV add a new attribute and assign it an id which can then be used to assign the attribute onto a face / vertices.
  • Get/SetColor, Get/SetColorAlpha, Get/SetNormal and Get/SetUV allow you to get or update the respective attribute for a specific Color, Alpha, Normal or UV ID.
  • ResetNormal will reset the normal to be automatically calculated based on the shape of the mesh.
  • GetColors, GetNormals and GetUVs retrieves all the instances of the respective attribute so you can iterate through them. Note: Color, Normal and UV attribute IDs are stable even as you add new ones or remove existing ones.
  • Get/SetFaceColors / Get/SetFaceNormals / Get/SetFaceUVs are used to get or assign Color, Normals or UV attribute IDs to the vertices of a face.
  • GetVerticesWithAttribute and GetFacesWithAttribute retrieves a list of all the vertices or faces that use a specific attribute ID.

Helper APIs:

  • FindClosestVertex returns the closest vertex relative to a point in 3D space and is great for scenarios where you might want to allow your users to do direct manipulation tasks.
  • FindClosestPointOnSurface returns the closest point on a mesh surface.
  • FindVerticesWithinSphere returns all vertices within a sphere and is also great for direct manipulation tasks.
  • GetSize and GetCenter return the size and extents of the mesh which can be useful as inputs to CreateMeshPartAsync.
  • GetAdjacentFaces and GetAdjacentVertices retrieves a list of faces or vertices that are adjacent to a specific face or vertex.
  • IdDebugString retrieves a string describing a stable ID which can be very useful for debugging purposes.
  • MergeVertices will merge all vertices in the mesh that are within a certain tolerance of each other.
  • RemoveUnused will remove all unused vertices, normals, colors and UVs.
  • RaycastLocal returns the first intersection of a ray in mesh local space.
  • Triangulate will split all faces on the mesh to be triangles.

EditableImage APIs

EditableImage objects allow for the runtime creation and manipulation of images by giving you direct access to the underlying pixels of the image.

Lifecycle APIs

Click to read more:
  • AssetService:CreateEditableImage() creates a new empty EditableImage object for procedural creation.
  • AssetService:CreateEditableImageAsync() creates an EditableImage object from an existing image asset or an existing MeshPart instance’s texture content and now supports an optional options table to customize the created EditableImage object.
  • [New] EditableImage:Destroy() marks the object for garbage collection to free up resources.
  • :warning: EditableMesh objects can only be created from mesh assets that are owned by you (or your group if the experience is owned by a group). Attempting to load an asset into an EditableMesh that you do not have permissions to will result in a permissions error.
  • :warning: The above creation APIs are subject to memory usage limits and might return nil if the device (or server) is out of memory reserved for Editable* objects. Always remember to check if your EditableMesh object was successfully created before using them in your scripts.

Direct access APIs

Click to read more:

EditableImage Documentation

Pixel APIs:

  • ReadPixelsBuffer and WritePixelsBuffer allow you to read and write pixel buffers directly from/to an image.

Drawing APIs:

  • DrawCircle, DrawRectangle and DrawLine allow you to easily draw primitive shapes / lines on an image. DrawRectangle is especially useful when you are trying to set or clear the entire image at once.
  • DrawImage allows you to draw another EditableImage into an EditableImage at a given position with a variety of blending options.
  • New: DrawImageTransformed also lets you draw one EditableImage onto another but also allows you to specify the position, rotation and scale. In addition to spatial transformations, you can also specify additional options like CombineType, SamplingMode and PivotPoint to further customize how the image is drawn.

Object and Content

As described in detail in the previous dev forum post, EditableMesh and EditableImage inherit from the new Object class. Along with the new Content data type, this enables multi-reference workflows where you can drive multiple instances like MeshPart from the same EditableMesh or EditableImage object.

You can read through the following explanations, code snippets and documentation to understand how common workflows like the following can be accomplished:

Best Practices

With these APIs, our goal is to set the foundation for our vision of collaborative in-experience co-creation, where one or more users come together in an experience and create a piece of artwork or sculpt together.

Thus, these APIs have been designed and optimized specifically for the following:

  • A limited number of Editable* objects created and being edited at the same time.
  • Async creation and conversion (e.g., CreateEditableMeshAsync, CreateMeshPartAsync).
  • Immediate mode edits (e.g., SetPosition, WritePixelsBuffer) once you have access to the Editable* object.
  • No guarantees that edits will be applied to rendering in the same frame.

The EditableImage and EditableMesh are incredibly powerful APIs due to how much flexibility they give you as creators. We recognize that a number of creators will choose to use these APIs for a variety of other use cases like procedural terrain generation, destructible environments, or even special effects. Since the APIs were designed for the above constraints, we recommend following these best practices to get the best results.

Memory Management

As introduced in the previous Studio Beta update post, Editable* objects provide random read and write access to vertices and pixels, and keeping that data available can be extremely memory-intensive. Roblox supports many platforms, including devices with limited memory. To allow your experiences to scale from low-end devices up to devices with more resources, creation of Editable* objects will be governed by a dynamic memory budgeting system within experiences and when running in Play Mode within Studio. If you are using Editable* objects for plugins in Studio Edit Mode, these budgets will not be applied.

:warning: Now that these APIs are available for published experiences, the engine will be enforcing a memory budget when creating new Editable* objects. Always remember to check if the Editable* creation APIs returned the object or nil before continuing to use the APIs.

:bulb:Using the Destroy APIs to free up memory

Click to read more:

You can use the EditableMesh:Destroy and EditableImage:Destroy APIs to explicitly free up memory after you are done using these objects. The follow code snippet shows a simple example:

local myEditableMesh = AssetService:CreateEditableMeshAsync(Content.fromUri("rbxassetid://ASSET_ID"))

If myEditableMesh == nil then

-- Not enough memory to create the EditableMesh. Show fallback UI or skip editing the mesh

return

end

-- myEditableMesh is valid, so safely continue editing the Mesh

-- When all edits are complete you can explicitly call Destroy to free up the reserved memory

myEditableMesh:Destroy()

Instances that reference Editable* objects when Destroy is called will fallback to default / empty data. (e.g. empty mesh for EditableMesh and black image for EditableImage)

:bulb:Setting the FixedSize option

Click to read more:

When using AssetService:CreateEditable* you can pass in an option that sets FixedSize. An EditableMesh object with FixedSize = true will not allow you to add any more vertices or attributes but will allow you to modify existing ones.

The benefit of setting FixedSize = true is that the engine knows the maximum memory the underlying object can ever use, so it does not need to reserve any extra memory for the case where more vertices, attributes or pixels are added.

Note:

  • When using AssetService:CreateEditableMeshAsync to create an EditableMesh object from an existing asset, FixedSize defaults to true.
  • EditableImage objects are not resizable and maintain the size they are originally created with by setting the Size option. Remember, EditableImages can use up a lot of memory quickly: one 1k x 1k image uses the same budget as four 512 x 512 images.

The following example shows how to explicitly set the FixedSize option:


local AssetService = game:GetService("AssetService")

-- This EditableMesh can have vertices / attributes added to it
local myEditableMesh = AssetService:CreateEditableMesh({ FixedSize = false})

-- This EditableMesh defaults to `FixedSize = True` because it is created from an asset.
-- The value can be overridden by directly passing in {FixedSize = False} to allow modifications
local myEditableMesh2 = AssetService:CreateEditableMeshAsync(Content.fromUri("rbxassetid://ASSET_ID"), { FixedSize = false})

-- This `EditableImage` will always have its `Size` be 32x32 pixels
local myEditableImage = AssetService:CreateEditableImage({ Size = Vector2.new(32, 32) })

Permissions

As introduced in the previous Studio Beta update post, to prevent misuse of assets using the EditableMesh and EditableImage APIs, these APIs will only be allowed to load assets:

  • that are owned by the creator of the experience if the experience is owned by an individual.

  • that are owned by a group, if the experience is owned by the group.

  • that are owned by the logged in Studio user if the place file has not yet been saved or published to Roblox.

The APIs will throw an error if they are used to load an asset ID that does not meet the criteria above.

:bulb: View assets that you own

Click to read more:

You can see all assets that you own by visiting the Creations page on Creator Hub or by going to the specific page for the asset by visiting: https://www.roblox.com/library/<Insert Asset ID Here>

:bulb: Use pcall to catch permissions errors

Click to read more:

You can use the following code snippet to account for cases where you might not have permissions to load an asset into an Editable* object:

local myEditableMesh

local result, errorMsg = pcall(function()

myEditableMesh = AssetService:CreateEditableMeshAsync(Content.fromUri("rbxassetid://ASSET_ID"))

end)

if result then

-- EditableMesh was created successfully and can now be edited

else

-- EditableMesh was not created successfully due to a permissions issue

print(errorMsg)

end

:bulb: Ensure you have the correct owner set when importing new assets

Click to read more:

When importing new mesh or texture assets into Studio, remember to verify that you have the correct owner set to prevent having to re-upload all the assets again with the correct owner.

  • :white_check_mark: When using the Import 3D workflow to import meshes, you can set the appropriate owner by selecting an option from this dropdown:

  • :white_check_mark: If you are using the new 3D importer queue Studio Beta you can select individual assets to select the appropriate owner

    image1

  • :warning: When using the existing bulk import tool to import a number of mesh or image assets at once, the owner is always set to the logged in Studio user and you cannot upload assets on behalf of your group from this flow.

Trust and Safety

Safety and civility is foundational to everything we do and these APIs are no exception. EditableMesh and EditableImage are powerful APIs that give you direct access to the faces and vertices of meshes and the pixels within images. With this power comes responsibility.

Our long-term vision is for the platform to support “real-time safety for in-experience creation,” so that users feel safe to create and collaborate freely using any tools creators provide. As we work towards that vision, we are taking the following steps to keep the platform safe in the interim:

  • Creators have to be ID-verified to publish experiences with these APIs.

  • Creators have to explicitly opt-in to using the APIs after reviewing the Terms of Use.

  • EditableMesh and EditableImage do not automatically replicate (i.e., edits are not automatically shared between clients or the server like other instances and properties on the platform).

As part of our recent Content Labels update (previously Experience Guidelines), certain free-form user creation experiences will be considered Moderate and restricted to 13+ users. If you enable Mesh & Image APIs using 3D assets that have individually gone through Roblox moderation (e.g. building a house with blocks, creating an avatar, customizing a car), then your experience can remain Minimal or Mild (previously All Ages and 9+ respectively).

However, if you allow users to draw or write in 2D and replicate those creations to other users without the completed creation going through Roblox moderation (e.g., writing or drawing on a chalkboard, whiteboard, or with spray paint), then you must update your Content Labels questionnaire (Creator Hub > Creations > Select an experience > Audience > Questionnaire), and your experience will be Moderate and restricted to 13+ users.

Tips for Safer Experiences

Remember, all experiences published on Roblox may be moderated for egregious user behavior within the experience as per the Terms of Use. In general, it is a good idea to:

  • Be careful when creating in-experience tools for free-form creation that could result in policy-violating content.

  • Be careful when enabling custom replication for user creations since you could be sharing policy-violating content.

  • Consider limiting users to UI like sliders or control values that can never result in policy-violating content. It’s much safer than free-form creation.

  • Do not load asset-like content from off-platform sites or unmoderated sources e.g. DataStores, source code, or StringValues. Storing or retrieving settings, slider values, or seed values for procedural generation is generally acceptable.

  • Try to make sure every free-form user action (decal, brush stroke, etc.) can be attributed back to a single user and potentially removed if it is policy-violating. Avoid mixing the work of multiple users into a single asset.

Updated example place file

We’ve updated the Hubcap customizer experience from the original Studio Beta announcement post to reflect the new Object and Content workflow as well as the permissions and memory management best practices described above.

Download the file here: HubCap_Image_v24.rbxl (437.4 KB)

Notes and Instructions:
  • In the RBXL file, all the scripts that make use of EditableMesh / EditableImage can be found under: StarterPlayer/StarterPlayerScripts/HubcapScript

  • In HubcapScript, take a look at the Init() function all the way at the bottom of the script for the main entry point. Here you will see how CreateEditableMeshAsync is used to create an EditableMesh object from an existing mesh asset. Note: This mesh asset has been whitelisted for all clients on Roblox so you do not need to have ownership permissions to load it into an EditableMesh object

  • The single hubcap spoke is then duplicated around the hubcap and the sliders are used to perform some straightforward manipulation of the mesh vertices

  • Also in HubcapScript, take a look at the updateCurrentDeform(meshInfos, emesh, params) function to see how the EditableMesh object is used to get the vertex positions and modify their positions based on the values from the UI sliders.

  • Take a look at the HubcapScript/MeshPainting ModuleScript to see an example of how EditableImage can be used. The function MeshPainting.Init() sets up a new EditableImage object and sets it as the SurfaceGui ImageLabelContent so you can preview edits.

  • Take a look at the castRayFromCamera(position) function in the HubcapScript/MeshPainting ModuleScript to see how the EditableMesh:RaycastLocal API is used along with the EditableImage object to allow the user to paint directly onto the hubcap

  • The doDrawAtPosition() function uses the DrawCircle API to mimic a circular “paint brush”

  • This place file also contains a number of additional helper ModuleScripts for a variety of common tasks like

    • Setting up UI sliders (HubcapScript/SetupSliders),
    • Calculating the mesh deformations necessary (HubcapScript/MeshMath)
    • Showing a color selection palette (HubcapScript/SetupPalette)

Bug fixes and Improvements

  • Improved overall stability

  • Introduced Content.none, the Content equivalent of an empty asset id

  • Introduced Destroy() for both EditableImage and EditableMesh

  • Introduced EditableMesh:GetCenter & EditableMesh:GetSize.

  • Fixed LocalRaycast missing edge hits

  • Fixed rendering culling meshes improperly

  • Fixed Luau autocomplete types for Content APIs

Known Issues

Here are a list of known issues that are actively working on fixing

  • Meshes with skinning information cannot currently be converted to EditableMesh.
  • A perfectly flat axis-aligned plane will not render properly.
  • Since EditableMesh and EditableImage objects do not currently replicate, we replicate an unusable Object of the same type in its place. Any attempt to read or write the contents of these “replicated” objects will throw. We’ll replace this with standard replication behavior in the future.
  • Setting vertices as transparent currently only works when the MeshPart is at least partially transparent.
  • There is a rare chance of the experience crashing during shutdown. The fix for this rare crash is rolling out to clients this week.
  • On some Android devices that fall back to OpenGL ES 2.0, EditableMesh objects might not be rendered correctly or rendered at all. We estimate that less than 2% of devices should be affected by this.
  • There are known issues with frustum culling for EditableMesh objects and how their bounding box is calculated. This could result in the meshes disappearing at some camera positions and angles. You can work around this by building your EditableMesh around the bounding box center and then moving it with the CFrame instead of moving the vertices themselves.
  • Plugins that use the Editable* APIs in places with Team Create enabled currently require the user to flip on the game Settings > Enable Mesh / Image APIs toggle. This will be fixed soon so any Studio plugin (edit mode) use of Editable* APIs will not require flipping on the toggle in Game Settings

What’s Next

  • We’ll keep iterating on the memory budget system to make it more dynamic, and work with other engine systems to scale down to create more headroom as needed.

  • EditableImage / EditableMesh and their contents are intended to replicate from server to client whenever an Instance referencing them would replicate. We’re still working on Object replication support and efficient image/mesh data replication.

  • We plan to provide an efficient option to use runtime-generated images and meshes once you are done reading or editing them, without needing to publish them, similar to in-experience Solid Modeling (CSG).

  • We plan to provide a plugin-security publishing API to publish Mesh and Image assets directly from Editable* objects.

  • We also eventually plan to provide in-experience publishing APIs to publish Meshes and Images to the Inventory. We are working out details on UX and scalability.

Getting these APIs out to the community has been a long road for us and all the feedback, bug reports, scenarios and workarounds you have shared with us over the months have been invaluable in getting to this state. This is just the start of the road for us though and we still have a number of APIs we would like to add and workflows we would like to enable.

Thanks,

@TheGamer101, @L3Norm, @monsterjunjun, @ContextLost, @LowDiscrepancy, @Penpen0u0, @LowDiscrepancy, @neutrinostack, @c0de517e, @FarazTheGreat, @syntezoid, @portenio, @FGmm_r2 and @gelkros on behalf of the Geometry, Rendering and Avatar teams.

175 Likes

This topic was automatically opened after 10 minutes.

Good feature, i dont like that its not guaranteed to be rendered immediately, i understand for moderation purposes but for drawing games it might take some time just to verify each pixel you want to draw

16 Likes

why do we still not have resize rotate and alike back :melting_face:

how to shatter the usability of this in one paragraph :frowning2:

congrats idk how this managed to get messed up but oh well i’ll continue coping…

39 Likes

WHOA.

It’s pretty remarkable just how BIG this is for Roblox.

I completely understand (and respect) the fine tightrope of safety concerns and am honestly happy it’s being treaded so carefully. I am massively excited to see the future and where these systems will be used. I’ve been playing around with these APIs for months, and developers and players alike will surely be looking forward to the crazy systems this will bring for 3D & UI designers. So many possibilities. Too many possibilities!

8 Likes

First, I just want to say that this is awesome. I have one issue though…

Can we please, for the love of all things holy, find a way to get out of this permissions disaster? It feels like each AssetType has its own permissions system at this point. Permissions need to be unified and aligned so that people working across multiple groups don’t have to re-upload.

Also there are cases where the mesh gets automatically uploaded to your profile rather than the group. Tears.

If any of you fix this problem of permissions, you will go onto the Roblox Hall of Fame and I will personally pay for a vacation of your choosing for you and your family.

Thank you in advance. Praise the sun.

50 Likes

A few questions.

  • With the memory limits, is there a method to seal a mesh in-place, by exporting it to a MeshPart, where it cant be edited any further?
    • My usecase here is allowing users to save their own meshes for further use, and if they import a lot of these meshes through the EditableMesh system, it might hit the memory limit.
  • Is this the final form of the API, or can we expect further changes to it
  • Will we get any form of automatic moderation from Roblox, whether done by AI or forwarding the UGC to Roblox’s moderation team
9 Likes

Let’s goooo!! What an update!

Great job.

1 Like

DrawImageTransformed supports this use case now! Let us know if there is some case that Resize or Rotate supported better.

12 Likes

idk who thought this was REMOTELY a good idea but oh man i am really NOT happy right now… all my time and excitement wasted for something i can’t use because i haven’t coughed up an id… great work :frowning2:

image

52 Likes

Very exciting additions to the Engine. I look forward to working with these features. Well done

2 Likes

:pray::pray: canva draw gonna be so much speedier now

3 Likes

Once i first saw the beta for EditableImages and EditableMeshes the first thing i thought of was something like a 2d clothes layering system (which people have been asking for years >_>), but the only thing these permissions do is get in the way.

Since the EditableImage and EditableMesh APIs are powerful new APIs that require extra care to ensure compliance with our Terms of Use, you must be 13+, ID-verified, and explicitly opted-in to using the APIs in published experiences.

Does it really need to be restricted to ID verification either? This feels like a good feature blocked by these unneeded restrictions. I know that they will be lowered as to not break things already made at launch as said in the previous post, but this is just unnecessary.

13 Likes

I hope I can also just use editable images completely client-sided for procedural effects.

I wouldn’t want a client-sided effect to be moderated because then things like paint gun splatters or other real-time-y effects would become horribly inefficient.

I don’t plan to use these features for drawing games but rather for more technical things or fun effects, it is very important that I can view the results of a generated image immediately.

8 Likes

awesome stuff!! this came out the day after I finished my fun little gif decoding project :smiley:
it decodes from URLs so im trying to be careful

8 Likes

:frowning:

I wanted to use the image APIs to pre-make some simple button backgrounds and release them as open-source. With this limitation, I need to handle both using the APIs and a fallback just because of the permission. I really wish it didn’t come to this, because it makes open-source projects and free models nearly pointless. The default is off, so most people will never turn it on.

14 Likes

The strict permissions regarding what assets can be used really kills off a lot of potential with this feature. I wanted to make a system that would allow users to import their own decals and then directly edit them in-experience, or for me to automatically resize/crop user-imported images if necessary, but that doesn’t seem possible anymore.

15 Likes

Great Work! :smiley: Thanks For The Update!

1 Like

Why is this restricted to having an ID? The reason given was for “safety”, but… what’s stopping me from opening blender (which doesn’t require an ID) and ruining my “safety” there? This feels like a bandaid for something else.

17 Likes

could you maybe chill a bit down with the “you need a 13+ ID verified account to use the API” thingy, please?

i know this doesn’t affect me, but releasing a super powerful API, and then limiting it to a pretty small number of people (because almost nobody wants to verify for privacy reasons) is a bit harsh. of course, i know that you are trying to add the ID restriction because you want to stay as clear as possible from inappropriate material, but im pretty sure that this won’t really be a big thing when your “no don’t let them kids join games w/out the questionnaire” rule takes place since most of these games will just be unverified games with little to no players. i know that you’re not able to verify these assets in real-time but still, it’s a bit rather silly how some legitimate people are being cut off from this just because they don’t want to share important documents with anyone.

i don’t find it realistic for a hit experience to suddenly just start using these APIs for EVIL and MISCHEVIOUS purposes. to my mind, you should lower the requirements to verifying your phone OR having an ID. pls ok thammk u ok bye :3

29 Likes