Not if you do it right!
I’ve made a few modules that make this very easy to implement. You would only clone things once!
My most recent handler module also lets you chose the FPS at which it would update object CFrames, so you could make the cubes move at only 30 FPS in the portal view. You could also make the map static, and therefore set the module to 0 FPS and massively save on performance since it’ll clone once and then never touch it.
I believe this is possible to do. As a matter of fact I took a good 30 mins to try and put this together.
It’s not perfect, but I believe I was able to convert the clipping problem to being axis aligned. I imagine with a bit of perspective “fudgery” I could make the clipping frame fit properly (which is why it’s not finished yet). That said the actual rotation aspect is clearly possible in the above video.
I’m very impressed. Didn’t think about that! Care to explain the technical details a bit more in depth?
Assuming I can solve this last hurdle I will do a write up.
You’re a gentleman and a scholar.
Okay, so I have something that’s worth talking about now.
I didn’t go through a nice solution that would make aspect and clipping work nicely. Instead out of laziness I choose to just use a “that’s a big number” solution. This is somewhat problematic because it’s still possible to get that cut off problem (although minimized).
Basically instead of doing the clipping and aspect I just use a large field of view and a stretched x size which doesn’t seem to effect the camera in viewport frames. Unfortunately it does result in some minor blurring. Again, I believe this could be fixed if I took the time 
I’m a little tired of working on this problem, but I know you yourself have a solid track record with viewport frames. I may come back to this problem in a few days, but until then I’ll upload the demo place so the community can have a run at it! 
portal.rbxl (45.8 KB)
Enjoy!
Edit: This isn’t a final solution for the reasons mentioned above, hence no write up yet.
EgoMoose to the rescue, though honestly it’d be so much nicer if non-Euclidean space were achievable in Roblox without hacking it together. I obviously have concerns about ViewportFrames but at the very least it does get that “other dimension” view down pat.
Getting back on topic before someone gets annoyed, let’s discuss this brilliant file you’ve shared.
The most pressing issue is pixelation and blur when close up. If you want players walking through, it needs to work up close too!
(The gif optimizations make this look awesome, but in game it’s just gross)
EgoMoose’s solution is the best we’ve got, and this is a side effect. So, how do we deal with this?
My solution? Game lore. Work with it. Call it “portal sickness”, a phenomenon that occurs when you get too close to a tear in the fabric of space. Add a bit of polish to it using some BlurEffects and particles, and bam: you have a workable product.
The next issue is the behavior at extreme angles. That’s a math issue that’s out of my area of expertise, unfortunately.
Overall, this is the best portal effect I’ve seen so far. As usual, EgoMoose blows everyone away.
There are some issues though.
I will however agree that this is really good.
@boatbomber has already told us the first problem.
But there is also this:
The viewport frame seems to shorten itself at the bottom when you’re close and looking at 90 degrees. Now sure this isn’t exactly a terrible thing. But certainly noticeable.
@EgoMoose What exactly is the module drawClass for? Its not used in the main code and you can remove it without altering functionality. Is it for debugging?
I mentioned it, but because I lack any answers, I didn’t go in detail.
Yes, so quick answer. The drawClass is for debugging so it’s not really needed for functionality.
As for the pixelation problem and the extreme angles issue. It’s my understanding (of my own calculations) that this is because of a low crop aspect ratio.
I’ll explain in more detail the process that led to this calculation:
I figured the best way to start was to go with a case that I knew was 100% possible and then build up from there. This case was when the camera is axis aligned to the surface and only goes forward and backwards.
I figured this out by asking myself what would be the literal perfect position for my camera to be such that I didn’t even need to make an adjustment to the viewportframe? The answer to that is the cframe that causes the top and bottom of the surface to perfectly line up with my camera viewport.
This distance “perfect distance” is calculated as:
local pDist = window.Size.y/2 / math.tan(math.rad(nCamera.FieldOfView)/2)
As for why. It’s all about how cameras use the FOV angle.
Right. So I now knew the ideal distance, but how could I make this work for any distance? Well I could visually see that I just needed to adjust how I scaled the viewport frame to solve for non-perfect distance cases.
So I next calculated the “real distance” of the camera from the surface using the following:
local rDist = (camCF.p - surfaceCF.p):Dot(surfaceCF.LookVector)
I then found the ratio of the real over the perfect and used that to scale the viewport frame’s y size. This fully handles the case of depth.
The next, thing I asked was: How can I extend this such that I’m not yet rotating, but I am translating my camera? I.e. the rotation is still axis aligned but I can now go up and down and left and right.
Well the key here is to move translate the viewport frame, not the camera. So I converted the camera position to object space and found the scale values it would take to move from the center to the camera point projected on the surface.
local p = surfaceCF:PointToObjectSpace(camCF.p)
local s = p / window.Size
vpf.Position = UDim2.new(0.5 - s.x, 0, 0.5 - s.y, 0)
That solves for that translation. The last thing is rotation. So far I’d been using the unmodified camera CFrame and that worked perfectly when it was axis-aligned, not so much when it isn’t. At first I was scratching my head, but then I realized something: how I rotate my camera shouldn’t effect what’s through the portal. With that eureka moment I realized I could just calculate and set a CFrame that was in the camera’s position, but axis aligned.
nCamera.CFrame = CFrame.new(camCF.p) * (surfaceCF - surfaceCF.p) * CFrame.Angles(0, math.pi, 0)
I hope all that makes sense, because now I can talk about the problem, which is this line:
vpf.Size = UDim2.new(4, 0, adjust, 0)
You see when adjust < 1 it seems a number of problems arise. When at extreme angles both adjust < 1 and thus the viewport frame is really small and thus doesn’t fit the whole part (similar thing happens with the x-axis with far away frame translations).
Similarity when you’re really close to the surface the adjust value is also small and if you’re not at an extreme angle it may not see like it’s getting squished cause it matches the camera viewport height, but that’s what’s causing the blur.
So I think the solution is to as I mentioned above do some perspective fudgery to make further away parts seem closer (as mentioned in the video). This should allow us to use a higher aspect value.
Idk it’s def a tricky problem hence why I put it down for now.
Great job on your portal work, EgoMoose! My TARDIS portal demo was based on the exact same principles, though I never resized the viewport due to the same concerns with blurring (not that it worked, if you walked too far away you’d just see blank space).
As it would so happen, in the past week I’ve rewritten the portal with a different method and have reached a near-complete state. It allows for the viewport to maintain quality as it does not need to move nor resize, but issues with perspective come with it that ruin the effect as does blurring.
You can see the warping effect most obviously at the bottom while up close, and in the center while moving left and right, but it is otherwise pretty subtle . I just need some way to distort the viewport’s camera’s rotational matrix so it ‘faces the player’s center of view’, but I don’t understand CFrame matrices enough to do this, or if it’s even technically possible. Might need to ping a CFrame expert like AxisAngles.
This topic has gone quite farther than expected. Thanks to everyone who’s been looking into this. Im currently working on making the cameras connect to different portals, along with the portal clipping.
For the portal, instead of using a world folder in the example, What’s stopping us from simply getting the parts in the players view only to appear in the portal? So that way anything else not visible (such as behind a wall) is not duplicated, reducing lag. As well as that it needs to be live feed, such as a camera. Last part, the blue portal should show what the orange portal see’s, and the orange portal should show what the blue portal see’s and so forth. Once this is done I can start working on the actual gun, and how to smoothin out the transition between portals. This may require raycasting which is not my strong suit. Thanks to everyone once again for helping me with this subject.
So a bit of progress.
I detailed how the aspect ratio was messing things up prior to this, but I think I’ve found a step in the right direction that has ultimately taken us a step back from another problem .
So instead of comparing the “real” and “perfect” distance (which is still a useful metric) and then scaling the vpf’s y-size I figured it may be better to recalculate the camera’s field of view to match such that the real distance would be the perfect distance!
This provides better results when viewing from the side and up close. However, it’s main issue is that our frame doesn’t cover the whole surface again.
Aside from that the blur and weird perspective issues that still happen when very close (although minimized) seem to happen when the rDist/pDist are 0.4 or less. So at the very least we have a measurement of when things start getting blurry and wonky and maybe that can help us hone in on when/how to fix it.
Edit: Oops, forgot placefile.
portal.rbxl (45.8 KB)
Edit2: aha so that 0.4 or less correlates to a field of view of above 120 hence why it doesn’t work. Hmm 
A tricky problem indeed!
The blur/pixels when you’re up close won’t be a problem, I’ll most likely keep it in-game. If you do find a way to fix it feel free.
The mathematically-exact solution is not possible with CFrames and SurfaceGui. I gave the explanation in a place that’s now kind of hard to find, since it’s in the VPF release megathread: ViewportFrame Release
In a nutshell, the matrix you would need for the viewportFrame’s camera view matrix (which we access as Camera.CFrame) in order for the “mirror world” to look correct can’t be represented as a CFrame. It’s the inverse of a perspective projection, which is a general 4x4 affine matrix that a CFrame doesn’t even have enough elements to store.
While you can construct CFrames where the matrix part is not a proper orthonormal rotation matrix, you’re still limited to just rotations, scaling, reflections, and shear transforms. You can’t do a perspective projection.
I feel like I’m so close though! Real shame, but we may have to pack it in…
You should rigorously disprove it for yourself. There is some chance I didn’t have enough coffee that day and did some bad matrix math, and if not, you’ll at least know for sure if you’re chasing your own tail 
@DrAdministration I believe currently @EgoMoose has the best solution for the time being, you should probably give him the solution check 