If you’ve played Deathrun, there is a radial menu used for emotes, and can be triggered by pressing E. Their system has a dot synced with the cursor (not what I’m asking for), and if your mouse goes beyond the menu (it’s circular), the dot clamps/prevents itself from going beyond the frame.
Given the position of the center of the circle, and its radius, you can check the distance from the frame to the center and if it’s greater than the radius then you can prevent it from going any further if it is.