[Studio Beta] Major updates to in-experience Mesh & Image APIs

Updates Announcements
#1

Hello Creators,

As we have been promising since the initial Studio beta announcement and then again in the recent update, we have been working hard on a new workflow to enable shared references for EditableImage and EditableMesh.

Enabling strong references with shared ownership required us to make major changes to the core DataModel API design paradigms around the EditableMesh and EditableImage APIs. We’re finally ready to share these changes with you in this Studio beta feature update.

:warning: Note: If you already have scripts based on the previous Studio Beta APIs, you will need to update your scripts to match the new APIs and workflow described below. :warning:

Since a lot of these changes are equivalent between the EditableImage and EditableMesh APIs, we will just refer to them collectively as the Editable* APIs in this post for brevity.

Let’s dive into the details!

Object Instead Of Instance

When we first introduced the Editable* APIs, the only way for instances like MeshPart to reference Editable* data was by adding the Editable* instance as a child of the MeshPart with code like myEditableMesh.Parent = existingMeshPart.

A tradeoff to that design is that instances can only have one parent at a time. This doesn’t provide a way to have a single EditableImage drive multiple MeshPart instances since an EditableImage Instance could only be a child of one Instance at a time.

With this update, EditableMesh and EditableImage are now an Object instead of an Instance.

Object is a new base class of Instance that is being introduced to the Roblox Engine. The key difference is that Object does not have a Parent property and cannot be a member of the DataModel tree. This allows multiple instances (like MeshPart) to hold shared ownership of the same Editable* object without breaking API contracts for existing systems like streaming, replication and so on.

To learn more about Object, please take a look at the documentation: Object | Documentation - Roblox Creator Hub

New Content Data Type

Today, instances like MeshPart reference content through properties like MeshPart.MeshId and MeshPart.TextureID, which contain a Uniform Resource Identifier (URI) string that identifies content stored on Roblox servers, outside of the DataModel.

With EditableImage and EditableMesh now becoming Object classes, we are introducing a new Content data type that can wrap either a URI string or an Object reference and use an Editable* reference interchangeably with an asset URI identifying an asset of the same type.

Unlike typical Instance reference properties, which are weak references, Content references to objects are strong references that hold shared ownership of the Object. Just like strong references to variables from Lua, any Content.Object reference will extend the lifetime of that Object and prevent it from being garbage collected.

You can create Content data type values from an Object, URI string, or asset id number using one of the following APIs:

  • Content.fromObject(object: Object)
  • Content.fromUri(uri: string)

We are just starting on the path to deprecate and replace all string URI “…Id” properties with new “…Content” properties that use the new Content data type. In this release, some existing instances that reference content have been upgraded to support the new Content data type:

  • MeshPart.TextureID → MeshPart.TextureContent
  • MeshPart.MeshId → MeshPart.MeshContent
  • ImageButton.Image → ImageButton.ImageContent
  • ImageLabel.Image → ImageLabel.ImageContent

(These properties are not supported in published experiences yet, but will be enabled in a few weeks before the public release of these APIs)

You can expect all APIs and properties that support asset content ids or URI strings to be upgraded over time (eg. SurfaceAppearance, Decals, Texture, etc.).

If the “…Content” property contains an Object then the corresponding “…Id” property will return an empty string.

Any assignment to the legacy “…Id” property will do the equivalent of:

instance.XContent = Content.fromUri(instance.XId)

To learn more about Content, please take a look at the documentation: Content | Documentation - Roblox Creator Hub

Common Workflows

The following code snippets show how you would accomplish common workflows using the new Object and Content paradigm:

Creating A New Empty Editable* Object

Since Editable* objects are no longer instances, creating a new one from scratch can be done using the AssetService APIs. You will now get back an Object instead of Instance.

local AssetService = game:GetService("AssetService")
local myEditableMesh = AssetService:CreateEditableMesh()
local myEditableImage = AssetService:CreateEditableImage({ Size = Vector2.new(32, 32) })

Note that these APIs may sometimes return nil if the Object could not be created due to device memory constraints (see Memory Management below for more details). These APIs also have an optional options table you can include for more settings when creating new Editable* objects. Please refer to the AssetService documentation for more details.

Creating An EditableMesh From An Existing MeshPart

In this workflow, AssetService:CreateEditableMeshAsync() now requires some Content as a parameter so you can just pass in the MeshPart’s MeshContent property directly. The returned EditableMesh will now be an Object instead of an Instance.

local AssetService = game:GetService("AssetService")
local existingMeshPart = workspace:FindFirstChildWhichIsA("MeshPart")
local myEditableMesh = AssetService:CreateEditableMeshAsync(existingMeshPart.MeshContent)

Creating An Editable* From An Asset ID

When creating an EditableMesh or EditableImage from an asset, you will now need to pass in the URI as a Content object using the Content.fromUri API. You will get back an Object instead of an Instance.

local AssetService = game:GetService("AssetService")
local myEditableMesh = AssetService:CreateEditableMeshAsync(Content.fromUri("rbxassetid://ASSET_ID"))
local myEditableImage = AssetService:CreateEditableImageAsync(Content.fromUri("rbxassetid://ASSET_ID"))

Note that these APIs may throw an error if there is a failure to fetch the asset itself due to network issues or due to permissions. Remember to use pcall() to account for these cases. For more details and a code snippet, see the section on Permissions below. These APIs may also return nil if an Object could not be created due to device memory constraints.

Live-Rendering An EditableMesh

EditableMesh objects are not rendered in the scene unless they are linked to a MeshPart. In many creation scenarios it is likely you will want to allow your users to see a live preview of the edits they are making to the underlying mesh.

Previously, to render an EditableMesh, you would place it as a child of a MeshPart instance to override the mesh that is rendered. This prevents you from using the same EditableMesh data to drive content for multiple instances.

With the new Object and Content paradigm, this is no longer a limitation and the following code snippet shows how you would achieve live preview:

local AssetService = game:GetService("AssetService")
local myMeshPart = workspace:FindFirstChildWhichIsA("MeshPart")
local myEditableMesh = AssetService:CreateEditableMeshAsync(existingMeshPart.MeshContent)

-- Helper method to compute extents of a Mesh. This will eventually
-- be replaced with a direct getter method on EditableMesh
local function computeExtents(em: EditableMesh)
    local verts = em:GetVertices()
    if #verts == 0 then
        return Vector3.zero
    end
    local inf = math.huge
    local min = Vector3.new(inf, inf, inf)
    local max = Vector3.new(-inf, -inf, -inf)
    for _, id in verts do
        local v = em:GetPosition(id)
        min = min:Min(v)
        max = max:Max(v)
    end
    return max - min
end

-- Create a new MeshPart instance linked to this EditableMesh Content
-- Note: EditableMesh:CreateMeshPartAsync will be replaced with
-- AssetService:CreateMeshPartAsync(Content, …) before public release
local newMeshPart = myEditableMesh:CreateMeshPartAsync(computeExtents(myEditableMesh))

-- Apply newMeshPart which is linked to the EditableMesh onto myMeshPart
myMeshPart:ApplyMesh(newMeshPart)

-- Any changes to myEditableMesh will now live update on myMeshPart
local vertexId = myEditableMesh:GetVertices()[1]
myEditableMesh:SetPosition(vertexId, Vector3.one)

-- You can create more MeshPart instances that reference the same
-- EditableMesh content
local newMeshPart2 = myEditableMesh:CreateMeshPartAsync(computeExtents(myEditableMesh))
-- Drop newMeshPart2 under workspace to render it
newMeshPart2.parent = workspace

-- Any changes to myEditableMesh will now live update on both
-- myMeshPart and newMeshPart2
myEditableMesh:SetPosition(vertexId, Vector3.one*2)

-- Calling these two lines again will recalculate collision and fluid geometry
-- with a snapshot of the current edits and update myMeshPart.
-- It is generally recommended to do this at the end of a conceptual edit operation.
-- Note: EditableMesh:CreateMeshPartAsync will be replaced with
-- AssetService:CreateMeshPartAsync(Content, …) before public release
local newMeshPart = myEditableMesh:CreateMeshPartAsync(computeExtents(myEditableMesh))
myMeshPart:ApplyMesh(newMeshPart)

The diagram below corresponds to the code snippet above and illustrates the following flow:

  1. AssetService:CreateEditableMeshAsync() is used to create myEditableMesh

  2. newMeshPart is created from myEditableMesh so it is linked to the EditableMesh data

  3. newMeshPart is applied onto the existing myMeshPart to swap out the geometry while maintaining the reference to myEditableMesh

  4. Any edits made to myEditableMesh will live update on myMeshPart

  5. AssetService:CreateEditableMeshAsync() is used to create newMeshPart2, another MeshPart instance from myEditableMesh which is then parented under workspace to render it

  6. At the end of a conceptual edit, myEditableMesh:CreateMeshPartAsync() is called again to recalculate physics data with a snapshot of the current edits. Note: This is an Async function since it could take a while to generate the physics data.

  7. newMeshPart is once again applied onto the existing myMeshPart to swap out the rendered and physics geometry

927×643 80.3 KB

Multiple MeshParts Referencing The Same EditableImage

Previously, if you had multiple MeshParts that you wanted to drive from an EditableImage texture, you had to make individual copies of the EditableImage for each MeshPart.

With the new Object and Content paradigm, you can re-use the same EditableImage Content to drive multiple MeshPart instances like this:

local options = { Size = Vector2.new(32,32) }
local myEditableImage = game.AssetService:CreateEditableImage(options)
local content = Content.fromObject(myEditableImage)
myMeshPart.TextureContent = content
myMeshPart2.TextureContent = content

-- Any changes to newEditableImage will now update all the MeshParts 
-- linked to editableImageContent
myEditableImage:DrawRectangle(Vector2.zero, newEditableImage.Size, 
Color3.new(0, 255, 0), 0, Enum.ImageCombineType.Overwrite)

The diagram below corresponds to the code snippet above and illustrates how both myMeshPart and myMeshPart2 reference the same myEditableImage for their TextureContent properties.

Memory Management

Editable* objects provide random read and write access to vertices and pixels, and keeping that data available can be extremely memory-intensive. Roblox supports many platforms, including devices with very limited memory. To allow your experiences to scale from low-end devices up to devices with more resources, creation of Editable* objects will be governed by a dynamic memory budgeting system.

APIs that return Editable* object might sometimes return nil if you have reached the memory budget. The budgeting system will be conservative to start but will become dynamic in the future to enable more creative power and richer experiences, depending on the device.

As a general rule of thumb, you are less likely to hit these limits if you use/create EditableImage objects with fewer pixels (smaller images) or use fixed-size EditableMesh objects.

EditableMesh objects created from assets using AssetService:CreateEditableMeshAsync will have FixedSize true by default. When FixedSize is true you are not able to add or remove geometry so they use less of the budget, proportional to the complexity of the source asset, instead of budgeting for the maximum possible complexity supported by a non-fixed EditableMesh.

When using the Editable* APIs, you must check and account for cases where the Create* APIs might return nil. You can use code like the following snippet to do this:

local myEditableMesh = AssetService:CreateEditableMeshAsync(Content.fromUri("rbxassetid://ASSET_ID"))
If myEditableMesh == nil then
    -- Not enough memory to create the EditableMesh. Show fallback UI or skip editing the mesh
    return
end
-- myEditableMesh is valid, so safely continue editing the Mesh

Permissions

The EditableMesh and EditableImage APIs allow you to load in mesh and image assets and then start editing them with just the asset ID.

To prevent misuse of assets using these APIs, these APIs will only be allowed to load assets:

  • that are owned by the creator of the experience if the experience is owned by an individual.
  • that are owned by a group, if the experience is owned by the group.
  • that are owned by the logged in Studio user if the place file has not yet been saved or published to Roblox.

You can see all assets that you own by visiting the Creations page on Creator Hub or by going to the specific page for the asset by visiting: https://www.roblox.com/library/<Insert Asset ID Here>

The APIs will throw an error if they are used to load an asset ID that does not meet the criteria above.

local myEditableMesh
local result, errorMsg = pcall(function()
	myEditableMesh = AssetService:CreateEditableMeshAsync(Content.fromUri("rbxassetid://ASSET_ID"))
end)
if result then
	-- EditableMesh was created successfully and can now be edited
else
	-- EditableMesh was not created successfully due to a permissions issue
	print(errorMsg)
end

These permission rules will be revisited over time and with feedback from the community. Starting with a more restrictive rule set leaves room to loosen restrictions over time without inadvertently breaking experiences.

Updated example place file

Hubcap_Image_v21.rbxl (435.8 KB)

The Hubcap customizer experience from the original Studio Beta announcement post has been updated to reflect the new Object and Content workflow as well as the permissions and memory management best practices described above.

Notes:

  • In the RBXL file, all the scripts that make use of EditableMesh / EditableImage can be found under: StarterPlayer/StarterPlayerScripts/HubcapScript

  • In HubcapScript, take a look at the Init() function all the way at the bottom of the script for the main entry point. Here you will see how CreateEditableMeshAsync is used to create an EditableMesh object from an existing mesh asset.

  • The single hubcap spoke is then duplicated around the hubcap and the sliders are used to perform some straightforward manipulation of the mesh vertices

  • Also In HubcapScript, take a look at the updateCurrentDeform(meshInfos, emesh, params) function to see how the EditableMesh object is used to get the vertex positions and modify their positions based on the values from the UI sliders.

  • Take a look at the HubcapScript/MeshPainting ModuleScript to see an example of how EditableImage can be used. The function MeshPainting.Init() sets up a new EditableImage object and sets it as a child of a SurfaceGui ImageLabel so you can preview edits.

  • Take a look at the castRayFromCamera(position) function in the HubcapScript/MeshPainting ModuleScript to see how the EditableMesh:RaycastLocal API is used along with the EditableImage to allow the user to paint directly onto the hubcap

  • The doDrawAtPosition() function uses the DrawCircle API to mimic a circular “paint brush”

This place file also contains a number of additional helper ModuleScripts for a variety of common tasks like

  • Setting up UI sliders (HubcapScript/SetupSliders),
  • Calculating the mesh deformations necessary (HubcapScript/MeshMath)
  • Showing a color selection palette (HubcapScript/SetupPalette)

Bug Fixes / Improvements

  • Normals issues

    Normals were always being recomputed when converting a MeshPart to an EditableMesh. Now normals are taken from the original MeshPart.

    Normal computation on sliver triangles has also been made more robust. Notice the improvements to the lips and neck in the comparison image below.

    1416×707 51.9 KB

  • Normal maps
    EditableMesh objects now work with SurfaceAppearance normal maps, for meshes with valid UVs. Vertex tangents are automatically computed based on the UV coordinates.

  • Converting EditableMesh objects into MeshParts with Physics data

    With the new Object / Content workflow described above, this update also allows you to convert an EditableMesh object into fully simulatable MeshPart using EditableMesh:CreateMeshPartAsync.

    This means you can procedurally create a mesh or allow your users to edit a mesh and then fully simulate that mesh with collisions, aerodynamics and other foundational systems just by converting it into a MeshPart

    Note: EditableMesh:CreateMeshPartAsync will be removed before public release in favor of AssetService:CreateMeshPartAsync, which will be changed to take Content as its first argument, optimized, and made publicly available in the public release.

  • EditableMesh data can now be rendered in ViewportFrames
    With this update, you should now be able to use EditableMesh objects with Viewport frames

Known Issues

  • The Resize(),Rotate() and Crop() EditableImage APIs have been removed from this Studio Beta and will be consolidated into a new EditableImage API to transform Images.

  • The new Content properties are not yet shown in the Studio property inspector view.

  • Content.fromObject() does not yet check supported types. Only EditableImage and EditableMesh will be supported in the public release, passing anything else will throw an error.

  • After the place has been reloaded again in Studio right after saving or publishing to Roblox, permission checks on loading assets to ‘EditableMesh’ and ‘EditableImage’ may fail. We are fixing the issue, and the current workaround is to simply close the place and reload it to Studio again.

  • You currently need to set MeshPart.TextureContent before adding the part to workspace or the texture might not render correctly.

  • Automatically assigned UV ids sometimes behave unpredictably after removing and re-adding faces

FAQs

  • I tried this out but things are not working for me
    These changes are rolling out in Studio Release 648. You can go to File > About Roblox Studio and verify that you are on version 0.648.X.XXXX. Also, ensure that you have the Studio Beta enabled by going to File > Beta features and enabling EditableImage and EditableMesh

  • Can Editable* APIs be used on the client and the server? What about replication?
    Yes they eventually will be! Today, these APIs are only available in Studio Beta so you cannot publish an experience that can run this on a server yet. When these APIs are fully released, that should be possible. Remember that the server never renders anything though but you should be able to do everything else on either the client or the server.

    Due to safety reasons, Editable* does not replicate automatically. So any edits made on the client will stay on that client and any edits made on the server will not be shared to other clients. Once our platform-provided moderation tooling evolve to support more in-experience creation cases, we do plan on enabling this.

  • Can I publish Editable* data out of my experience as an asset?
    Today, all Editable* data is runtime-only and cannot be serialized or persisted. We eventually want to support this use-case with something similar to a PromptCreateAssetAsync API but we are still working on this. There are a number of issues that need to be resolved before we can get to this but we think this will open up a number of really cool use-cases for experiences!

  • EditableMesh:CreateMeshPartAsync() requires a Mesh Size parameter. How do I compute extents of my EditableMesh object?
    You can use the following code snippet to compute extents of your EditableMesh object for now. Since this is a common use case, we will add a getter function directly on EditableMesh to make this easier.

    Code snippet to compute Mesh extents 
    local function computeExtents(em: EditableMesh)
    local verts = em:GetVertices()
    if #verts == 0 then
        return Vector3.zero
    end
    local inf = math.huge
    local min = Vector3.new(inf, inf, inf)
    local max = Vector3.new(-inf, -inf, -inf)
    for _, id in verts do
        local v = em:GetPosition(id)
        min = min:Min(v)
        max = max:Max(v)
    end
    return max - min
end

em:CreateMeshPartAsync(computeExtents(em))

What’s Next

We are pushing hard to fully release these Mesh and Image APIs by the end of the year so you can publish experiences with them.You should expect to see a flurry of updates as we get closer to the full release.

Here are some upcoming changes you can expect:

  • Breaking API Change: EditableMesh:CreateMeshPartAsync will be removed before public release in favor of AssetService:CreateMeshPartAsync, which will be changed to take Content as its first argument, optimized, and made publicly available in the public release.

  • Breaking API Change: EditableMesh:GetContent() and EditableImage:GetContent() will be removed in favor of Object.fromObject().

  • Breaking API Change: AssetService:CreateEditableMeshFromPartAsync() will be removed since it is redundant with the new Object / Content workflow changes.

  • AssetService:CreateEditableImageAsync() ignores the options table right now. At public release, it will support a similar options table to CreateEditableImage and will allow you to set options like Size and others.

  • EditableMesh:Destroy() and EditableImage:Destroy() will be added in an upcoming release to allow you to explicitly release the resources and memory budget used by these objects. The memory budget will not be set until you have access to this Destroy() API so you have some control over how you use the budget.

  • A getter function will be added to compute the mesh extents on EditableMesh. As a workaround for now, you can use this code snippet in the FAQs section.

As always, we really appreciate all the feedback and bug reports you provide. These are vital to ensure we have a quality product that addresses your needs.

Thanks,
@TheGamer101, @ContextLost, @L3Norm, @Penpen0u0, @LowDiscrepancy, @monsterjunjun, @c0de517e, @FarazTheGreat, @syntezoid, @portenio, @FGmm_r2, @gelkros

181 Likes
[FIXED!] TerrainPaths V1.1 - Create smooth paths for terrain
[Studio Beta] Updates to In-experience Mesh & Image APIs
CanvasDraw - A powerful pixel-based graphics library (Draw pixels, lines, triangles, read/modify image data, and much more!)
Weekly Recap: October 21 - 25, 2024
EditableImage Baked Lighting: (Raytraced Global Illumination + Soft Shadows)
OSGL - EditableImage graphics library
Captures APIs are now available!
Ray Tracing Module #2
Editable instances uncreatable
Get Image Actual Resolution
#2

This topic was automatically opened after 10 minutes.

#3

Awesome changes!

Big thing I know a lot of people have been talking about though - is there an ETA on when we will be able to set individual surfaceappearance maps with texture contents?

21 Likes
#5

Can we have a Studio plugin API to save EditableImage and EditableMesh to Roblox, it would be amazing to build tools within Studio (especially a mesh editor) that I can then upload to Roblox.

This API partially exists with plugin.SaveSelectedToRoblox, but this wont work with the Object instance, since they do not exist in the DataModel tree, and also, all selections through this API get uploaded as models.

24 Likes
#6

This is awesome! One big limitation currently is that you can’t render an EditableMesh inside of a ViewportFrame. Is this something we can expect to change soon? Being able to draw arbitrary meshes inside of UI is very powerful (think drop shadows, vectors, etc.)

14 Likes
#7

Hey, we’re really excited for these changes. I’ve been wanting multi-owner Editables for a while now.

Regarding permissions… is there any chance we could see permissions allowing using arbitrary 2D clothing? A lot of the “cool factor” with Editable* comes from modifying the player, and personally I had some ideas that involved repacking the player’s clothing below another suit / making a persons clothing black & white / etc.

I’m not sure how different it’d be allowing just 2d clothing images compared to any arbitrary image on the catalog, but personally I’d love to see this possibility opened up

6 Likes
#8

Wow, this update looks really useful! I can definitely see it helping with dynamic meshes and bunch more cool stuff, Awesome Update ROBLOX! loving these editable stuff

6 Likes
#9

+1 to this, it would unlock a whole bunch of avenues to experiment with :'D

7 Likes
#10

i love this.

question, the last time i used editable mesh was a couple months ago - I got an error with something like “you cant use editable mesh on the server”.

I changed to client and it worked.

Still confused if its client or server?

6 Likes
#11

i am so lazy to update scripts

10 Likes
#12

please let us toggle aliasing for decals and meshes :pray: :pray: :pray: :pray: we need hd baldi textures

14 Likes
#13

I really like this because it will fix the super inconsistent property names like (Name), (Name)Id, (Name)ID and changes them into (Name)Content

Also how come ImageButton had its Image property changed into ImageContent, but not HoverImage and PressedImage? I know other instances have not had their properties changed yet, but only 1 of 3 was changed here.

10 Likes
#14

This should be working now, let us know if you have any issues with it

15 Likes
#15

Is there anything I need to do to get access to the new beta functionality?

I have the beta toggled on, and Roblox Studio doesn’t have any pending updates for me.
4d1e5cf65839c2c387cea40ac2bf2d87

5 Likes
#16

Apologies if I missed this and if this is already answered but.

Can editable images and meshes be used completely client-sided as well, with no replication to the server?
I think this is very important for client-sided effects that do not need to be visible to the server or other players.

Also, does generating meshes and images save them to Roblox servers or are they fully cleaned up when a game gets closed?

5 Likes
#17

Will MeshParts that are based upon an EditableMesh still use the traditional collision fidelity options, or will the collision be precise with the mesh?

3 Likes
#18

Yes please, I have been asking for this feature.

3 Likes
#19

Press Alt+S, if it says 647 as the version, there’s your problem.

Seems like Roblox hasn’t shipped 648 to everyone yet, this usually takes a day though so lets hope :pray:

5 Likes
#20

Yeah, that’s it. I’m a version behind.

3 Likes
#21

It will still use the traditional collision fidelity options, and run the EditableMesh through the same code to create collision geometry

4 Likes