I am attempting to create a similar effect to viewport frames, but intentionally pixelated and interact-able in the 3D world. I am not focused on efficiency in this case, purely on effect.
I have seen this done before, similar to sircfenner’s custom rendering system, but in the 3D space with SurfaceGuis. Does anyone have an idea on where I could begin?
How many pixels in x and y? If not many, you might be able to use something like ViewportPointToRay and FindPartOnRay with a secondary camera to figure out what color should be assigned to a given pixel in the view (maybe shade color based on surface normals of intersected objects). Then map colors to a 2D array of frames inside the surface gui.
I haven’t tried anything like that, but it might be a place to begin while waiting on a reply from someone who has done this 
. The ViewportFrame can be attached to a SurfaceGui and the canvas size of the SurfaceGui can be reduced way down, but that gives an effect that is more blurry than pixelated. Interesting problem. Hopefully there are some equally interesting solutions to it.
Another way to approach this might be to use an array of objects as a sensor array and use FindPartOnRay from each cell to find the first intersecting part (and it’s color). Then map the colors to frames in the gui.
I used a 10x10 array and a ScreenGui, but it should work with SurfaceGui and other sized arrays. This would be more performant if I used a single object for position and look vector and offset rays from that part instead of creating geometry for each cell of the sensor obj.
Since FindPartOnRay returns surface normals, you might be able to shade colors based on the difference between the ray used for finding parts and the surface normal returned for the point of contact. Incorporating the ClockTime of the lighting object might be something else to try. If the array is small enough and your update slow enough you might be able to do some sort of nearest neighbor averaging to get an anti-aliasing effect. I just used the part color.
The player’s mouse could be used with the gui to control the sensor/camera and accept other input via buttons or whatever.
Thanks a ton for this, it looks exactly like what I need.
Glad that was helpful.
Using UIGridLayout and giving every frame a Name and LayoutOrder from 1 to n (corresponding to names of cells in sensor) made the whole thing less tedious. For some reason I couldn’t get studio to accept number names with leading zeros through a script. Using LayoutOrder let me arrange cell frames in the gui like 1, 2, 3, 4 instead of 1, 10, 100, 11.
