Introducing in-experience Mesh & Image APIs [Studio Beta]

They are bunch of textured meshes being stacked resulting to this amazing view

Am I the only one thinking about generating a noise map? weā€™re getting close to doing shaders in Roblox the same unity did it in the past :+1:

1 Like

Will we ever be able to use editable images to make a skybox? If so this would come in really handy for certain things!

4 Likes

if these ever get collisions just like regular meshes then so many things can be done
currently im working on planet generation and i got it working perfectly with editable meshes but the only problem is that collisions dont work, if these get collisions that would help my project a lot

1 Like

Yes, I am genuinely terrified of the unnecessary moderation aspect. The amount of data, probably terabytes of data generated if this is used across the whole platform. It will be impossible to do it with a human, and AI will make many mistakes.

I need to make it clear that there already is an option to report a place, so if there is a developer abusing it, itā€™s as simple as reporting the place.

3 Likes

cylinders
I made a project which converts parts into meshes. All the mesh shapes lined up perfectly with the real deal except my cylinder, where I have made a horrifying discovery.

They were never perfectly round. The only thing they have going for them is cleaner topology than mine with the center vertex.

This really doesnā€™t matter but will annoy me forever.

3 Likes

Would it be possible to set the UV to not repeat? Like some sort of UV mode enum?
I ask this because Iā€™m making a module that uses UV to place projections/stickers onto mesh surfaces. Though, this comes with the issue that the image repeats as shown here.

(Only the red box should contain the image)

The only reasonable solution I can implement is to clone a chunk out of the mesh using a custom CSG implementation, though this takes multiple frames of time and isnā€™t really a good approach.
I ask because I know many systems allow you to customize the out-of-bounds behavior of UV coords.

Iā€™m not sure if this is possible to add in any reasonable amount of time but I thought I might as well ask.

2 Likes

Does anyone know if it is possible to get EditableImages to work as an Overlay on SurfaceAppearances?

Iā€™ve been messing around and unless Iā€™m just dumb I havenā€™t figured it out yet

Will the EditableImage resolution limit be increased?

1 Like

@TheGamer101 Iā€™m not sure if you are the correct person to bring this to, but, could we or will we potentially see a way to load/save images pixel data as buffers? It would result in some awesome performance gains. I would have loved if :ReadPixels() and :WritePixels() used buffers but itā€™s also understandable why they didnā€™t given the time frame. I guess itā€™s still technically possible to change the API or allow :WritePixels() to accept buffers, but Iā€™m not sure about for :ReadPixels() since it would break anything already written, and generally speaking itā€™d probably make more sense if they were separate APIs anyways.

We have considered adding a Read and Write pixels API that uses buffers in addition to the current APIs.

The performance difference isnā€™t actually substantial for most cases - ReadPixels and WritePixels are substantially faster but reading and writing from the buffer is slower than a table.

If you could provide more information on your specific use case, that would be interesting!

We have no current plans to increase the EditableImage resolution. Doing so in the near term would raise memory and performance scalability concerns. However, if you have a specific use case that requires higher resolution, please tell us about it, as it will help inform our future plans!

1 Like

This makes sense, my usecase was mainly just performance.

Tl;dr I realized the big performance gain is actually being able to pack colors into 32 bit values, not using buffers, which would be cool to get for tables, but Iā€™m not sure how it could be implemented or if it even makes sense. The buffer vs table details are also actually very interesting.


I investigated more (benchmarks in the final part of my post) and the reason why I believed buffers were overall faster here is actually because I encode single packed RGBA8888 colors. Buffers arenā€™t inherently faster, itā€™s actually because Iā€™m packing the colors into one writeu32 call.

The same savings can be made for tables by packing the RGBA values into one 32 bit value, which results in an approximately 2x speedup. Tables are indeed faster to write to, however, only by a little bit, which might actually be important enough to justify buffer but Iā€™m not exactly sure.


Reasons why buffer may still be beneficial

Most significantly, depending on how big the cost of tables are for EditableImageā€™s APIs under the hood I think that with --!native mode the performance difference between the two is so small (worst case I got was 4% slower) that the faster API implementation will overall win performance.

Worse reasons I debated including at all

I am also assuming that better JIT optimizations will eventually allow buffer writes to overtake the performance of table writes in native mode. I assume this because I know that buffers are basically just a little block of raw memory, the only concern is whether or not you are writing inside that block of memory, and those kinds of bounds checks are compiler optimizable (with a smart enough compiler, which, I would hope luau will grow into having even if itā€™s a long time from now)

Another weak reason that buffers still appeal to me in particular is because they compress when sent over the network, over teleports, and when stored in DataStores (a solution in search of a problem). I like the idea of that having essentially no implementation cost and not having the cost of translating from a buffer to a table to make a call to :WritePixels(), or doing the opposite with :ReadPixels() but I canā€™t say that this would ever really justify buffers


Performance of buffers vs tables, and single RGB8888 values

If the pixel colors are encoded in a single RGBA8888 value you can get some savings by packing into one write (2nd benchmark vs current :WritePixels() and :ReadPixels() format) which works with or without --!native and --!optimize 2.

If you use the RGB8888 format for the table, packing 32 bit color values, you can get almost equivalent performance between both (3rd benchmark) with buffer writes occasionally but not always outperforming table writes.

Benchmarks

All benchmarks use --!native and --!optimize 2 (I spent like 1 and a half hours working on these and generally just benchmarking/testing)

My CPU is a Ryzen 5 5600G, and I have a single 3200 MHz 16 GB stick of RAM.

Buffers vs tables no pixels 1:1 reads/writes

BuffersVsTables.bench.lua (724 Bytes)

Buffers (1 write) vs tables (4 writes)

BuffersVsTablesPixel.bench.lua (2.5 KB)

Buffers vs tables for packed RGB8888

BuffersVsTablesRGB888.bench.lua (2.3 KB)

Strangeness in the 3rd benchmark

Iā€™ve also noticed some general weirdness when I decode packed 32 bit RGBA8888 values. I think it must have something to do with the CPU cache or maybe it is some forbidden knowledge hardware optimization, but, it results in buffer reads being faster than table reads but only when youā€™re unpacking the value. Unpacking compared 1:1 with raw writes is still slower than if you donā€™t unpack a value of course, but 4x the writes is still ~2x slower in practice, so per pixel its faster.

It is kind of as if either unpacking the RGBA value made it slower when you read the next value from a table in the iteration or something, or it somehow is faster when you read the value from a buffer, and I have no idea which, only that this behaviour is 100% consistent no matter how hard I try to make it go away. I think the most likely explanation is that the CPU cache misses more on some internal table data during unpacking, but only for the table or something like that, but Iā€™m not really sure exactly, itā€™s certainly strange.

4 Likes

Hi, Iā€™m wondering if it is EditableImage or texture or decals that you are referring here? And is it a MeshPart or CSG or EditableMesh? Thanks!

Iā€™m not sure buffer would be better, but weā€™ll do some benchmarks to be sure when we get to batch APIs.

Batch APIs have always been planned, but theyā€™re going to be one of the last things we do after the rest of the API is stable. Thereā€™s still some big incoming changes to how normals and all other vertex attributes are handled on the way.

For an array of positions (vector values) buffer would only be ~25% more memory efficient. Now that Vector3 is a special VM-level vector type thereā€™s only the added overhead of a single int for the TValue type field. buffer might even be worse in terms of perf, because the individual buffer writes probably wonā€™t benefit from the same SIMD treatment that vector gets. Pre-allocated tables (table.create) might end up being the best bet for position arrays.

buffer might be more efficient for raw arrays of scalar attribute values, but even then, if table won for positions Iā€™m not sure if the confusion of some using tables and others using buffers would be worth it.

And for every batch operation that isnā€™t just appending new verts youā€™ll need a table of IDs anyway, partially because of our choice to use sparse IDs to help make it more friendly to replication and collaboration in the future. We probably would not use buffer for ID lists.

3 Likes

In this example Iā€™m using a SurfaceAppearance with the mode set to Transparency.
And to fully clarify, Iā€™m using :SetUV to move the image around the mesh. It is a EditableMesh

1 Like

Erm? Post about editable meshes not working on humanoid limbs

I think this is a really big limitation if it is intended behavior.

EditableMesh preview does not work under Humanoid models. ā€œFastClusterā€ rendering support is incomplete.

This is something we will fix in the future

1 Like

Will this feature be available before the complete release of EditableMeshes?

1 Like

How close are we to be allowed to load skinned meshes as editable meshes?