Roblox allows us to filter text using there neural network and years of moderation detection API they’ve built for chat, recently released the source to there AI voice chat moderation and we know there’s AI involved in moderation related to uploading images. With the rise of drawing games and editable images coming soon, is there any plans for functionality related to safety checking in game for player created images in experiences? Right now in drawing games developers add a report button which blocks the image but this after the fact. I’m imagining a function where you can pass an image asset and filters/blocks it.
5 Likes
This is really needed. As for game developers on roblox, you and the player that did said action get punished. And with the blowup of games that relate to users making there own custom videos, images, drawing, it would be needed.
As roblox grows, these games will continue to grow too. And the main problem is, you can’t fully trust your users to report inappropriate drawings. This is such a needed feature, but also could come with downs…I guess we shall see.
2 Likes
What I thought of, but probably hard to do is just Roblox blurring content they think is inappropriate every frame using AI, someone draws something, and boom is blurred on the user’s end. Someone is building a base that looks weird from a certain view, boom Roblox blurs it. Roblox detects some weird text chat and blurs that portion.
Now it will have a lot of false/positives, but I mean so is the current chat filter misunderstanding.
Cuz for example, in some games there are signs and players type a letter/image in each sign to bypass filter. How would you be fixing that issue? The blur thing just came to my mind when I saw it, can’t have all developers in the platform really going after those moderations every time and coding its own “algorithm” to try and find the inappropriate shapes/letters around an open world game.
And can work for the games mentioned here, when the player is shown that image, it blurs out.
This would be super hard to implement. On one hand, moderation for images isn’t that great, and developers could get banned for previously approved images such as UI elements. On the other hand, people can insert bypassed decals into games that allow that.
1 Like
no no roblox is lazy to make moderation improvments
Roblox has plans to implement something like this, they briefly talked about it at RDC 2023
No idea when/if we’ll see further developments on it.
Seems like devs could abuse this to induce a false positive and potentially get a player banned