Introducing in-experience Mesh & Image APIs [Studio Beta]

Update 9/25/2024

TL;DR

  • EditableMesh and EditableImage are now available in Studio-beta and give you direct Luau access to Mesh and Image data at runtime.

  • Important Caveats:

    • Publishing edited Meshes / Images from within your experience is not available in this beta but is coming soon.

    • Real-time replication, moderation, and collaborative editing support for meshes / images is not available in this beta but is on the roadmap.

    • During the Studio beta, you can load & edit any Mesh or Image asset using these APIs however, for the final release, there will be checks to ensure your experience has permissions to load & edit the asset.

    • Right now, these APIs are only available on your local Studio client and you cannot publish live experiences with these APIs. They will be released sometime next year once known issues are resolved.

We wanted to get these APIs into your hands as soon as possible so you could give us direct feedback on the API surface, the workflows and the use-cases you have in mind.


Hello Creators!

We are very excited to finally get Mesh and Image APIs in your hands as a Studio beta. This has been a highly requested feature and really opens up what you can allow your users to do in your experiences.

For example, this experience illustrates how you can allow your users to customize their car’s hubcaps by modifying the geometry and painting different colors on them.

As we shared at RDC this year (Note: Some APIs have been renamed since this RDC talk), we have a long-term vision for Mesh & Image APIs on the platform. Our end goal here is to enable you to create real-time collaborative experiences that unleash your users’ creativity by giving you direct Luau access to Meshes and Images. We hope to eventually provide APIs that enable publishing Mesh and Image assets directly from within your experience as well as support real-time replication (with moderation) so everyone in your experience can edit meshes and Images collaboratively, in real-time.

As a first step down that path, this Studio Beta aims to get an initial set of Mesh and Image API features in your hands so we can 1) learn how you use them and what use-cases resonate the most with you and 2) get feedback from you on the actual API surface and the overall workflow.

There’s a lot to cover and we can’t wait to see what you build with these new powerful capabilities. So, let’s dive right in!

Usage

As with any Studio beta, your first step will be to enable the beta by going into the File > Beta features window and enabling the EditableImage and EditableMesh checkbox:

Once you have enabled the beta, you should now have access to the new APIs below from any Luau script. Remember, Studio Beta features only work in your own studio client session and will not work in any published places.

API Details

EditableMesh

Bulge_loop

The core promise of the Mesh API is to give you direct Luau access to any Mesh on the platform that you are allowed to edit. That means direct access to read / write vertices, faces, normals, and attributes on them. This is achieved is through a new instance called EditableMesh.

Creating an EditableMesh instance

An EditableMesh can be created from scratch for procedurally created meshes, by converting an existing MeshPart or directly from a Mesh Asset ID.

To create a new EditableMesh from scratch, you simply need to create a new EditableMesh instance like this:


local newMesh = Instance.new(“EditableMesh”)

To create an EditableMesh from an existing MeshPart, you can use the new CreateEditableMeshFromPartAsync() API on AssetService like this:


local AssetService = game:GetService("AssetService")

local existingMeshPart = workspace:FindFirstChildWhichIsA("MeshPart")

local newMesh = AssetService:CreateEditableMeshFromPartAsync(existingMeshPart)

Finally, if you would like to create a new EditableMesh directly from an Asset Id, you can use the new CreateEditableMeshAsync() API on AssetService and pass it a mesh asset Id directly:

(Remember to replace ASSET_ID below with the actual asset Id that you would like to create an EditableMesh from)


local AssetService = game:GetService("AssetService")

local newMesh = AssetService:CreateEditableMeshAsync("rbxassetid://ASSET_ID")

Keep in mind that creating an EditableMesh or EditableImage from an existing mesh \ image is an asynchronous call since the engine might need to download the actual Mesh \ Image asset and verify that you have permissions to edit it before it is available to your script.

Working with an EditableMesh

Once you have an EditableMesh, you can now get access to a whole slew of APIs that allow you to directly read, manipulate and write vertices, faces, and other attributes of the mesh as well as some helper APIs for common workflows

Direct access APIs:

  • GetTriangles() gets all triangles in a list of stable triangle IDs

  • GetVertices() gets all vertices in a list of stable vertex IDs

  • Get/SetPosition() allows you to retrieve or set the position of a specific vertex ID respectively

  • Get/SetVertexColor() and Get/SetVertexColorAlpha() allows you to retrieve or set the vertex color attribute and Alpha values for individual vertices

  • Get/SetUV() lets you retrieve and set the texture UV coordinates at individual vertices

  • Add/RemoveVertex() and Add/RemoveTriangle() allows you to change the topology of the underlying mesh by adding or removing vertices or triangles to it.

    Note: EditableMesh maintains stable vertex and face IDs even if you remove or add vertices / faces. This means you can be confident that IDs will not change underneath you

Helper APIs:

  • FindClosestVertex() returns the closest vertex relative to a point in 3D space and is great for scenarios where you might want to allow your users to do direct manipulation tasks.

  • FindClosestPointOnSurface() returns the closest point on a mesh surface.

  • FindVerticesWithinSphere() returns all vertices within a sphere and is also great for direct manipulation tasks

  • RaycastLocal() returns the first intersection of a ray in mesh local space

For the full list of APIs and further explanations, make sure to explore the docs here: EditableMesh | Documentation - Roblox Creator Hub

Previewing Edits

EditableMesh instances are not rendered in the scene by default. In many creation scenarios, however, it is likely you will want to allow your users to see a live preview of the edits they are making to the underlying mesh.

To preview an EditableMesh, simply place it as a child of a MeshPart instance to override the mesh that is rendered. You can use the following code snippet to re-parent an EditableMesh.


newMesh.Parent = existingMeshPart

Remember, you are simply rendering a visual preview of the EditableMesh and any edits to the Mesh will not reflect in the simulation yet (collisions, mass properties etc.)

Publishing a New Mesh Asset

The publish workflow is not yet available in this Studio beta but it is likely to follow the same pattern as the recently released PromptCreateAssetAsync() API that allows you to publish models from within experiences.

Once they are available, CreateMeshPartAsync and ApplyMesh, will allow you to create or update MeshPart meshes with updated collision geometry without requiring publishing the mesh as an asset immediately, similar to runtime created PartOperations returned from CSG APIs like UnionAsync. However, if you were to shut down your experience, all of these unpublished meshes would be lost! In order to persist the changes, you would need to publish the EditableMesh or the newly created MeshPart instance and get back an asset ID.

EditableImage

Paint_Loop

Editing images through the new Image API are very similar to the mesh editing workflow described above. Instead of using an EditableMesh, you would simply use an EditableImage instance instead.

Creating an EditableImage instance

Similar to creating an EditableMesh, an EditableImage can be created from scratch for procedurally created images or directly from an image Asset ID.

To create a new EditableImage from scratch, you simply need to create a new EditableImage instance:


local newImage = Instance.new(“EditableImage”)

To create a new EditableImage directly from an asset Id, you can use the new CreateEditableImageAsync() API on AssetService and pass it an image asset Id directly:

(Remember to replace ASSET_ID below with the actual asset Id that you would like to create an EditableImage from)


local AssetService = game:GetService("AssetService")

local newMesh = AssetService:CreateEditableImageAsync("rbxassetid://ASSET_ID")

Working with an EditableImage

Once you have an EditableImage, you can now get access to a whole slew of APIs that allow you to directly read, manipulate and write pixels and other attributes from/to the image as well as a set of higher-level APIs that make it much easier to draw common shapes and accomplish common workflows.

Direct access APIs:

  • EditableImage.Size retrieves the width and height of the image in pixels

  • Read/WritePixel() allows you to directly read and write pixel values to specific texture coordinates of the image relative to the top-left corner of the image.

Higher-level APIs:

  • Resize(), Rotate() and Crop() allows you to resize, rotate or crop the image using bilinear interpolation where necessary.

  • DrawRectangle(), DrawCircle() and DrawLine() can be used to draw a rectangle, circle or line directly onto the image instead of dealing with individual pixels

  • DrawImage() allows you to draw another EditableImage into this one at a given position

For the full list of APIs and further explanations, make sure to explore the docs here: EditableImage | Documentation - Roblox Creator Hub

Previewing and Publishing edits

Similar to EditableMesh, EditableImage allows you to preview changes to image if it is a child of the following instances:

  • As a child of a MeshPart, it will override the TextureId property

  • As a child of an ImageLabel or ImageButton, it will override the Image property

  • As a child of a Decal, it will override the Texture property

Just like EditableMesh we hope to introduce publishing new image assets from EditableImage instances as well using something similar to the PromptCreateAssetAsync API

Combining EditableMesh and EditableImage together

While the new EditableMesh/Image instances are powerful in their own individual rights, combining them can unlock even more powerful scenarios. A common creation scenario might be allowing your users to directly paint onto an object in your experience to customize it. This can now be achieved by combining these APIs together!

Using the EditableMesh:RaycastLocal() API, you can determine the exact point on the EditableMesh that a user is touching, get the UV texture coordinates for that location and then use an EditableImage instance to change the color / draw something on the underlying Image at those coordinates.

For an example, search for the MeshPainting module script in the example place below!

Example place file

Since there are a lot of new APIs we are introducing here, we thought it would be helpful to create a sample place file to show how everything could work together.

The following place file shows a simple experience where you can customize the shape of a wheel hub cap using a set of sliders and the EditableMesh API and then directly paint on it using the EditableImage API

Hubcap_RBXL_example

Hubcap_Image_v12.rbxl (426.3 KB)

Instructions:

  • Make sure you have the Studio beta enabled
  • Open up the RBXL file attached above in Studio and hit Play
  • Walk up to the car and hit F to switch to “Hubcap editing mode”
  • Use the sliders on the left to deform the car’s hub cap, add/remove spokes to it or change the taper
  • Use the slider on the right to change the brush size and after picking a color, you should be able to directly paint on the hub cap with your mouse.
  • Hit F again to exit “Hubcap editing mode” and you can now enter the car and drive it around with the new hubcaps

Notes:

  • In the RBXL file, all the scripts that make use of EditableMesh / EditableImage can be found under: StarterPlayer/StarterPlayerScripts/HubcapScript

  • In HubcapScript, take a look at the Init() function all the way at the bottom of the script for the main entry point. Here you will see how CreateEditableMeshAsync is used to create an EditableMesh instance from an existing mesh asset.

    • The single hubcap spoke is then duplicated around the hubcap and the sliders are used to perform some straightforward manipulation of the mesh vertices

    • Also In HubcapScript, take a look at the updateCurrentDeform(meshInfos, emesh, params) function to see how the EditableMesh instance is used to get the vertex positions and modify their positions based on the values from the UI sliders.

  • Take a look at the HubcapScript/MeshPainting ModuleScript to see an example of how EditableImage can be used. The function MeshPainting.Init() sets up a new EditableImage and sets it as a child of a SurfaceGui ImageLabel so you can preview edits.

    • Take a look at the castRayFromCamera(position) function in the HubcapScript/MeshPainting ModuleScript to see how the EditableMesh:RaycastLocal API is used along with the EditableImage to allow the user to paint directly onto the hubcap

    • The doDrawAtPosition() function uses the DrawCircle API to mimic a circular “paint brush”

  • This place file also contains a number of additional helper ModuleScripts for a variety of common tasks like

    • setting up UI sliders (HubcapScript/SetupSliders),
    • calculating the mesh deformations necessary (HubcapScript/MeshMath)
    • showing a color selection palette (HubcapScript/SetupPalette)

Important caveats

These new APIs come with incredible utility, but also some unusual limitations that are worth highlighting.

Never Saved (By Design)

EditableMesh and EditableImage instances are never saved in places, packages, or model files, even if Archivable is true. This is by design.

These instances represent an editing session and are intended to be eventually moderated and published as assets, or discarded. They’re meant to facilitate asset creation in-engine, not act as an alternative to the asset system.

Never Replicated (Temporary Safety Limitation)

EditableMesh and EditableImages instances currently do not replicate from servers to clients, unlike most instances. This is not by design. The technology to support safe replication isn’t finished yet. Accepting this temporary limitation allows us to get these APIs into your hands and get your feedback sooner! We felt these APIs were already incredibly useful, even without replication, that it justified sharing it, even in this “incomplete” form.

Our plan is for these to eventually replicate from servers to clients like a standard instance by default, with some additional safety filtering. Our goal is to support developers like yourselves building collaboration tools on top of these APIs, and standard replication is important for making that accessible. We’ll share more on the timeline and rollout plans later.

We don’t expect this safety work to be completed before we will first enable use of these editing APIs in experiences next year. This limitation on replication will exist for a while. It might take a year, or more.

EditableMesh:CreateMeshPartAsync is sadly also unavailable at this time. This API was designed to allow you to convert an EditableMesh back into a fully-simulated MeshPart. Without safe mesh replication, any MeshParts created by this method on the server would be invisible to all clients. Also, without special publishing support in Studio, these parts would be saved without meshes and remain invisible forever. We will be working to make this available to Studio plugins soon, but supporting safe runtime use of this method will take longer.

For now, avoid placing EditableMesh or EditableImage instances under Workspace or other replicated services on the server to avoid future compatibility headaches (for both of us!).

Asset Edit Permissions (Incoming Restrictions on Loading)

In this Studio Beta, you will be able to create an EditableMesh\Image from any existing mesh\image asset on Roblox.

However, we will start enforcing asset permissions sometime before these APIs are finally released and publishing support is enabled. You will only be able to load an EditableMesh\Image from an asset that your experience has permission to edit. (e.g. meshes and textures used by items in the Avatar Marketplace will not be able to be loaded into an Editable* unless the experience owner is the original creator of that item.) Stay tuned for more details in the coming months on how asset permissions will work for EditableMesh\Image instances.

For now, we recommend that you only use Mesh / Image assets you currently own even during this Studio beta to make your life easier when permissions are enforced by the API.

Known Issues

Issues that will be addressed before these APIs are released for use in experiences sometime next year:

  • [DONE see update] The APIs for Vertex Normals and other attributes will be improved significantly.
    • Normals, Colors, UVs, and other vertex attributes will get their own per-face attribute IDs to allow independent normal/color/uv seams.
  • Adding or removing vertices and faces will not be supported for skinned meshes until APIs for rigging and skinning are available.
  • [DONE see update]The EditableImage API will be updated to have a more unified approach to blend types rather than the current ImageCombineType enum that is used in the DrawImage API.
    • This will allow using alternative blend modes in other APIs like DrawCircle which will make creating things like erasers much easier.
  • [FIXED] EditableMesh preview does not work under Humanoid models. “FastCluster” rendering support is incomplete.
  • [FIXED] EditableMesh is currently disabled on devices running Mac lower than MacOs 11.0, subject to change.
  • [FIXED] Cloning an EditableMesh with 0 triangles can sometimes cause a crash. The fix for this issue will be released in a few weeks.
  • Meshes with skinning data cannot be loaded (i.e. most avatar meshes). This limitation will be removed once EditableMesh can store skinning bind data internally.
  • EditableMesh doesn’t have batch add/remove/set APIs yet. We’re getting to it!
  • In Studio copy & paste does not work because it currently relies on the class being marked as savable. Use Duplicate instead.
  • Live previewing of EditableMesh / EditableImage will likely change to use a new multi-owner reference mechanism, instead of supporting EditableMesh/EditableImage as a child.

Feedback

Again, we wanted to get these APIs in your hands as quick as possible to get your feedback. Let us know if you have any thoughts or suggestions regarding the API surface, the workflows, and the use-cases you have in mind.

Made with :heart: by, @L3Norm, @TheGamer101, @monsterjunjun, @igHunterKiller, @FarazTheGreat, @syntezoid, @theRealLogicalerror, @FGmm_r2, @TigerRabbit2, @NeoBuilder101 and @c0de517e

652 Likes

This topic was automatically opened after 10 minutes.

I am genuinely immeasurably excited to see these new classes coming. It’s like a dream come true in terms of allowing developers to do whatever they want with the Roblox engine.

Any word on when these might be enabled for use in games? I have a game that could use these new classes immediately, as I’ve already implemented another code path that uses EditableImage (it’s disabled right now obviously.) I don’t even need them to replicate, I actually assumed they wouldn’t when implementing it.

74 Likes

This is really cool! I can’t wait to start messing with all of these new features and seeing what they’re all capable of.

29 Likes

I am literally going to EXPLODE in joy and happiness, my goodness.
We got like 5 or 6 updates in nearly a single day now?

It almost feels too good to be true, you have NO IDEA how much and how long I’ve wanted this.
Editing meshes and decals has an unimaginable amount of usecases.

I can think of SO MANY ways to use this it’s almost scary.
Maybe I’ll even end up using the absolute heck out of these features for cool effects maybe even gameplay features, or whatever I can come up with.

58 Likes

Oh my god I have been waiting so long for this

28 Likes

what is going on today?! y’all are releasing amazing updates back to back to back :face_with_spiral_eyes:
i’ve been anticipating like all of these and i know all of my developer friends were too
whatever has possessed the team to churn these out, please keep possessing :pray: :pray: :pray:

33 Likes

I was going to talk about how amazing I believe this feature would be yet this paragraph has bogged me down quite a bit. Yes, I do respect the privacy of asset creators however avatar assets are quite literally forced into my experiences, I don’t have an option on whether these assets are added to my experience (as they are attached to avatars) so it feels severely unfair that you would limit my ability to edit meshes that are forced to be in my experience? I really hope this is reconsidered before launch, I understand why allowing publishing would be detrimental for asset-privacy however most use cases for EditableImages/EditableMeshes are not publishing related, for example; I just want to mess up the vertexes on my meshes for a cool effect, not allow players to publish such effects. I truly hope this change does not go through.

In terms of EditableMeshes/EditableImages, they are a good feature, but for the reason mentioned above I’ll never be able to use them for the use-cases I want.

60 Likes

you don’t have to push so many updates in a day but my god i am not gonna complain, this is amazing

21 Likes

It’s probably possible to create Blender inside of the Roblox engine now. The amount of plugins which would be able to be made with this is insane…
W

32 Likes

You guys are going crazy with the updates. Super excited to use this

20 Likes

This is just AMAZING. Almost feels like a dream come true. Can’t wait for some of the limitations get removed to implement this into my projects. Props to Studio’s engineers.

15 Likes

Are we celebrating Roblox birthday or something? These updates today are so good literally made my whole day especially this one!! Very excited to see what is possible with this single features

18 Likes

Christmas came early, love to see all these releases!

19 Likes

Bro the updates today were all AMAZING! I didn’t wanna seem sarcastic or repetitive so I didn’t really reply to the other topics but wow!

Drawing / painting is def what I’m planning to do with the EditableImages

14 Likes

Wow I was thinking about trying to make a modeling system a couple months ago. Wow yeah

14 Likes

I have several questions I need answers to.

  1. Can editable meshes be clustered for more optimal rendering and reducing draw calls?

  2. can multiple existing meshes be manually combined together into a single big mesh?

  3. Can we use editable images to paint onto meshes with very complex shapes, curves and with many UV islands and still get good-looking results?

    For a game where a user could paint their entire car or for a paintball game you’d ideally want the images to be able to wrap around surfaces and completely encapsulate a mesh and not have paint jobs appear cut-off or cause pixels bleeding onto other parts of the mesh.

  4. Can we or will we please also be able to read triangle, normal, vertex color information, etc of regular non-editable meshes?
    I’d love to have this feature, there are cases where I would not want to edit the mesh at all and just get surface information or perform a raycast.

  5. Will we be able to read pixel colors of existing non-editable images?

    Would love being able to raycast onto a part and just read it’s material or decal pixel colors.
    This can also be useful for games where you have toxic sludge or lava that spills on the floor and you want to raycast for green or orange pixels to deal damage to players that walk on it.

17 Likes

This is insane! I didn’t even know this was possible with the current roblox engine…

7 Likes

its time to make a video renderer so i dont have to pay to upload vids ezzzz

18 Likes

This is awesome that roblox has support for editing meshes, but it can even be done in-game

7 Likes