Currently I am trying to create a machine learning algorithm that utilizes computer vision to create an advanced AI. I wanted to check if it is possible in luau before having to use a python based AI.
Its far fetched but I’m not gonna give up hope just yet
There is not a way to natively scan the pixels on your screen using LuaU, I think Python would be a better approach for this.
Good luck!
Edit: I’m confused on what exactly you’re looking for. If you’re looking for things like colors on specific pixels you could (although highly more intensive and annoying)
Detect any UIs on position and fetch them using PlayerGui:GetGuiObjectsAtPosition (or whatever it’s called) on positions and organize based on ZIndex then fetch the background color or gradient color of the frame. And if it’s an image, then you’d probably need a web proxy and some math to get the relative color on that specific location of the image. It would get even more complicated if it’s using UIRect / image manipulation.
And if there’s no UIs, you could raycast to the point and grab the part color but if it’s a mesh you’d need to do the same thing with the web proxy and calculate the color in that specific location. And that probably doesn’t consider anywhere near every edge case that this could run into.
That’s so like over the top that I think using programming with another language like Python would be much friendlier where I believe you can literally just grab the screen and scan it there.