I hope you meant X and Z axis if you’re talking about world space, since the Y axis usually means height on Roblox. If you wanted to ignore Z, then you’d just be left with the right and up vectors.
All three axes are still going to be relevant if you want to solve this problem, since you’ll need to account for height to determine if the player is on the same floor as the item you’re trying to check the distance to. If you just check across a 2D plane and ignore height, the same problem will occur but instead of a bubble (magnitude check only), the Gui would show regardless of player height so long as their X-Z position is within the X-Z position of the target object. Y is relevant.
You really won’t find an alternative here besides doing hard math yourself, in which you’re still going to need to use magnitude anyway. Magnitude isn’t your problem since magnitude just describes the length of your vector (a scalar quantity), your problem is only using magnitude to determine if the player is in range of the object.
There are tons of ways you can solve this problem. For example: you can determine if the two objects (the target object and the HumanoidRootPart) are in the same region. To do that, you would need to identify the volume of each room (like a brick stretched across the whole floor then put in ReplicatedStorage or so). You can then create a Region3 over that and check if both objects are in that region, but only after a magnitude check to save on computational costs.
Another example: if your floors are of a consistent height (e.g. each floor is 200 studs high), you could probably do something involving dividing the Y of the position of each object and rounding to see what floor it’s on (for example: Y position 200-399 is floor 1 when divided by 200 and rounded down, so if both divisions come out to 1 after rounding, they’re on the same floor). Obviously you’d need some offsets if they’re on unlevelled ground, though.
Many ways to do this. You won’t be able to find something here by searching, you will need to do problem solving. Break your problem down into what the issues are, then get to work on seeing if you can possibly conceptualise resolving the problem. Problem: your distance checks are passing through floors. Think: how can you determine if both items are on the same floor, or how can you check the distance in such a way that it doesn’t pass through floors?
There’s no alternative to magnitude because it is just a scalar describing the length of a vector, nothing can substitute that. Alternatives are applicable to methods where there’s more than one way to do something. There is an alternative regarding checking a quantity between two points, yes, displacement, but that’s a very different concept.
tl;dr Magnitude isn’t your problem, your math is.