I’m pretty sure BoxHandleAdornments are selectable and interactive, so all that additional code and memory usage to handle the selections is probably what’s causing the extra lag.
However, I’m not sure if it shouldn’t be causing that much lag if a :Connect was never called to the adornment.