[Open Source] Painting Implementation Using Pen Tablet Pressure

[PhantomShiftGuh]

RBX Pen Showcase

This past week after seeing a post about the game Draw Me and trying it out, I got hit by the intrusive thought, “What if I could forward pen pressure to Roblox?” And as far as I’m aware? I think I’m the first to ever do it.

This project serves a couple of purposes besides being fun to work on:

1. A showcase of how currently pen pressure data can be forwarded to Roblox
2. An example of what Roblox could unlock for developers if pen pressure data was available
3. A case for why the current EditableImage limit should be increased
4. To be a free resource for learning about graphics and/or be referenced or used by others

This project consists of two parts: First is the OpenTabletDriver plugin written in C# which takes on the role of forwarding pen pressure data as a virtual/emulated controller (created and managed via ViGemBus), mapping the pressure data to trigger actuation. Note that this requires additional drivers to be installed, which means there is high friction on the player’s side to take advantage of this, along with the risk of faulty driver installation. This project serves only as an example of what could be done with pen pressure.

The second part is the painting software that actually makes use of this pressure data, which uses a combination of OSGL and some of my own graphics manipulation to present a relatively usable painting solution on Roblox.

The game can be tried here, whereas the code is freely available at GitHub - PhantomShift/rbx-pen-showcase: A showcase of tablet pen pressure in Roblox through an emulated controller. Hopefully this can serve as a relatively simple reference/example of how other developers could potentially take advantage of pen pressure data, assuming Roblox ever chooses to expose it. Secondarily, I hope to expand the scope of the painting software itself to serve as an interesting technical demonstration of what you can build on Roblox.

One major caveat as of writing, only 4 layers are allowed on the game client (999 layers in studio) due to the limits imposed on EditableImage. I do have a plan for working around this detailed in the README of the project, but currently this heavily limits potential workflows in-game.

Shortcuts

As of writing I do not currently have shortcuts written visibly anywhere in the place, so for now they are listed here.

Action	Shortcut	Description
Toggle Eraser	E	Switches between brush and eraser (eraser is just a brush that puts full transparency).
Zoom Canvas	LeftCtrl + Space + Drag	Zooms into the canvas when dragging right, zooms out when dragging left.
Pan Canvas	Space + Drag	Pans canvas based on cursor movement.
Change Brush Size	LeftShift + Drag	Increases brush size when dragging right, decreases when dragging left.
Eyedropper	LeftCtrl + Drag	Selects the color of the pixel currently under the cursor. Evaluates all layers.
Undo	LeftCtrl + Z	Pops the most recent action pushed to the undo stack. Pushes to redo stack.
Redo	LeftCtrl + Y	Pops the most recent action pushed to the redo stack. Pushes to the undo stack. The redo stack is cleared if another action occurs.

Currently the undo stack has a strict limit of 128 actions and the overall history a limit of 512MB, which is consumed depending on the nature of the action (i.e. low usage for moving layers, high usage for a stroke with a large bounding box). Actions that cause the stack to go beyond these limits cause the bottom of the stack to be freed and forgotten.

Media

Below are a couple of example illustrations that I’ve made using this project so far.

Very open to feedback and suggestions, especially for some future more computationally expensive/complex operations like layer compositing in order to overcome EditableImage memory limits while maintaining usable performance. I’d also love to learn about other ways you could potentially forward pen pressure data, especially if it’s simpler/less intrusive than what I’ve currently shown! Of course, I’m also open to direct contributions.

[kdn_ae (Kaden)]

This looks very cool and promising. If I remember I’ll test it out on my tablet tomorrow!

[A_Mp5 (amp)]

Phenomenal work. Using the gamepad inputs to deliver device inputs from other devices is frankly genius. I’ve known about the method for some time, but I see it rarely used

[MaddyRing]

OMG PEN PRESSURE IN ROBLOX BEFORE GTA 6

this works with all pens that support pressure sensitivity right? like including the logitech ones and maybe even apple ones

does this support 3d touch

[PhantomShiftGuh]

Love the enthusiasm, but this implementation essentially relies entirely on the device being supported by OpenTabletDriver (basically only traditional drawing tablets, i.e. Wacom). Realistically for an actual Roblox game, the engine needs to expose pen input (as us developers can’t just tell people to go download random drivers), which doesn’t sound like it would be particularly hard, but there’s probably very little incentive for them to do this. As I mentioned briefly, one of my hopes with this project is that it shows Roblox what developers could do if they did, as creating and sharing art is definitely another fun method of social interaction and providing artists with the tools to express themselves as they please would be a nice show of will from Roblox’s end. For now though it’s a very fun side project to work on.

@MaddyRing

does this support 3d touch

Unfortunately I’m not sure what this is referring to, so likely not.

[MaddyRing]

its basically like an old iphone feature
its a pressure senstivive screen
(3d touch)

[PhantomShiftGuh]

Ah I see, then unfortunately no. I’ve put in a little bit of thought about how this could potentially be done on mobile but ultimately mobile devices are so much more infinitely locked down compared to desktops that it would be a hard requirement for Roblox to actually interact with pens/pressure input themselves. It would be very cool if they did expose “touch pressure” for touch actions, they already use a Vector3 for input objects so using the Z axis for that would be nice.

[saaawdust (sawdust)]

This is amazing bro

Great use of the library too

[PhantomShiftGuh]

It’s a great library

Can’t wait for the next release, I’m currently a bit busy with uni but I’m in the middle of switching to using the 1.6.3 branch to take advantage of the blending modes you guys have implemented. A bit curious on how you derived the overlay operation btw, the implementation I have currently written is basically taken from the Wikipedia page on alpha compositing but yours seems different at a glance.

[PhantomShiftGuh]

[Studio Beta] Introducing CreateDataModelContent: Convert editable mesh and image data into static content

Hi Creators, We’re excited to introduce AssetService:CreateDataModelContentAsync, a new API that allows you to “bake” editable mesh and image data into static Content for the lifetime of the session when you no longer need to edit them. This static content works just like any URI-backed mesh or image asset with an asset ID (rbxassetid://<Asset ID>): They can be used to drive MeshContent and TextureContent properties on instances like MeshPart. They can replicate between the server and clie…

Mostly posting this here as a reminder to my future self, but this change is actually pretty huge for this project. My original approach for dealing with the memory constraints, which I briefly detailed in the project README, is to create a compositing system that dynamically re-composites layers above and below the layer that’s currently being edited to reduce the number of necessary EditableImages from N-layers to just 4 (two dedicated to editing, two for layers composited above and below). However, at least for basic alpha-based overlaying of layers, this basically completely removes any need to use EditableImages for the layers that aren’t being actively edited. Of course, if I do implement layer blend modes (which I likely won’t, it would probably perform horribly), I would still need to implement such a system myself since it’d basically only be useful for overlay operations.

TLDR for this next paragraph, this change might make multiplayer drawing easier but likely won’t ever be available in the public place version due to moderation.
This does also theoretically make multiplayer editing simpler (in my head I imagine something like Drawpile but players can only edit their own layers, for simplicity), but this opens the whole can of worms that is content moderation. In an ideal world I would just implement some password-based room system and display a warning like, “Hey, this is a room potentially full of strangers, and they can draw things that may be inappropriate,” but my current understanding of Roblox’s moderation policy is that developers need to play a pretty active part in policing user actions whenever free-form content creation is involved. I imagine limiting it to perhaps “friends only” might lessen this burden, but I’m unsure about the situation if, say, a friend does draw something actually “inappropriate” and one of their friends reports them (whether as a joke or as a serious action). I still think multiplayer editing would be a fun problem to work on for the technical aspect, but likely will never be exposed in the public place because of the infrastructure required for content moderation. One persistent thought I’ve had is making a “Studio-only” version of the project that users themselves are responsible for when using, but Roblox can arguably view this as sidestepping the rules as opposed to complying with them. This also defeats part of the point of this project which is showcasing what sort of experiences you can develop in Roblox despite limitations that are present.

[PhantomShiftGuh]

So a small progress update on this, I’ve implemented a very basic compression scheme that seems to be yielding pretty good compression ratios, at least with the very basic paintings I’ve tested it with. Full disclosure, in case it isn’t clear, I’m not super familiar with image formats or anything. As such, implementing a full-on established image format like PNG, JPEG, etc. in Luau was pretty daunting so I decided to keep it dead simple. The basic steps are:

1. Convert the interleaved RGBA buffer data into planar data (i.e. partitioned by channel, RRR… GGG… BBB… AAA…)
2. Delta encode each individual channel so that the compressed representation can use less than 8 bits per value
3. Compress using Zstd (EncodingService:CompressBuffer(buf, Enum.CompressionAlgorithm.Zstd)

On a completely blank layer this turns the base ~4MB size of a 1024x1024 image into just 147 bytes. Of course this is the best case scenario, and I haven’t tried, for example, loading a full complex image and compressing it. But at the very least with some basic paintings I’ve made to test the feature I’ve observed at lowest ~25:1 (down to almost 160KB) compression (which is obviously a high estimate since they’re very simple images with few colors). Frankly If I’ve managed to get even close to 4:1 for complex images I’ll be pretty happy since it’d put them in around the 1MB range.

Example painting (~25:1 compression ratio)

Aside from some optimization on the compression procedure itself, my next steps for this is implementing client-side encryption so that I can eventually store the images either in datastores (if budget allows) or on an external server.

Storing Images on an External Server!?

Frankly the new datastore limits are a bit too restrictive for the game’s current use-case (storing images) even with good compression. Even if I gave players a budget of maybe 4MB (with the assumption that visitors who never play again or rarely play essentially “subsidizes” the budget for more active players) this would probably only fit a couple of fairly complex canvases. This is why I’m specifically planning on using client-side encryption to ensure 1) privacy for users and 2) completely remove liability for hosters (whether that be myself, someone who makes a project based on this one, or some service that we use to offload storage onto).

On the other hand, one big hope is that the API described here completely erases the issue of image storage:

@FGmm_r2

Within published experiences, we are also working on a way to upload in-experience editable and data model content directly to a new inventory; scoped to and owned by your experience itself.

[PhantomShiftGuh]

Again partly as a reminder to myself because I don’t have much time to work on the project at the moment, but I was doing some further reading on image compression and came across the QOI (Quite Okay Image) format. It’s a super simple format that appears to achieve close-to PNG performance in terms of compression ratio (typically slightly worse, but nonetheless comparable) yet (de)compresses significantly faster. The compression speed is of little consequence to me (though still a nice bonus) since my main use-case is “image archival”, but my impression is that given that the format performs similarly to PNG even on photographic content, it’s likely a significantly more robust solution than the nonsense I threw together in that last post and the solution I will use moving forward.

But the good news in fact does not stop there! First off, I don’t need to implement it myself: someone else (based Subaru fan btw ) already implemented it in Luau and is pretty much a drop-in solution. Secondly, and I think most interestingly, generic compression algorithms can be applied to QOI images, allowing compressed QOI to exceed PNG in compression ratio in some cases, which means I can likely use Zstd compression for free benefits. Later on when I actually do have time to work on the project I’ll make sure to do some benchmarking (with actual images, not just my shoddy paintings), but for now I’m actually quite excited to continue working on the project (if only I had the time ).

I will note that a common critique of the format (at least at the time that it was announced) is that its simplicity has sacrificed too much in terms of features (color space management, other metadata) but all of those extras are wholly useless for this program since we’re working with pure 32-bit image data here. Plus, it seems like QOI has a surprising amount of software support, which, while unlikely, could be relevant for future project features .

[seanjosep39 (Sean)]

Author of luau_qoi here, thank you for noticing my prior work! Anyways, this is a really lovely and interesting project! I’ve also been interested in making a painting program for a while (and is actually partly a reason for why I made that QOI implementation) but I’ve never thought about a pen pressure emulation! It’s always so cool to see Roblox developers make software that push the boundaries of what is perceived to be possible in the platform. Great work!