Looks like we are doing the rotation wrong → needs to be new rotation matrix * old rotation matrix transpose, previously we were doing this backwards.
I tested Stravant’s and Garnold’s tests and they both work great now.
In addition I’ll make a post when this makes it to Gametest2 if anyone wants to test it, this can help in catching bugs early and helping us get out a fix earlier, because I want this to work correctly asap.