Personally, I’d do it backwards from what you’re thinking of. Calculate the possible torque by summing up the torque that each thruster can apply by multiplying the distance from the center of mass with the magnitude of the thruster’s force. The easiest way to do this would probably be to group thrusters by the axis of the torque they cause and just sum within those groups. You should have six groups, -X/+X, -Y/+Y, -Z/+Z.
Then apply the torque according to user input up to the max torque of the thrusters. Then you can simply render the effects by figuring out the fractional contribution to the desired torque that each thruster is giving by dividing that thruster’s max torque by the group’s max torque, then you get a number between 0-1 to multiply with the fraction of applied torque divided by max torque for that axis.
By only ever applying torque directly, you’re simplifying the work for the physics engine and also making it more predictable. You also avoid the need to actually make sure the torques balance out, which would probably add an unnecessary art load that no one will notice anyway. You should still try to mostly ensure they balance out for plausibility, but now they don’t need to be perfect.