Implementing Custom Collision System

For unit collision detection there are usually three approaches

  1. Brute force: check every unit against every other unit, which is usually too slow for 100+ units
  2. Spatial hash: categorize each unit into squares and then only check each unit with every other unit in its square and surrounding squares
  3. Quadtree (2d) or octree (3d): Quadtree - Wikipedia and Dynamic Octree System

When you detect a collision, you cant just teleport the unit to be outside the collision range because it might then teleport to collide with another unit. So you move it something like 1/2 the distance and then use substeps to run the simulation (collision check + pushing back) multiple times. Then when a unit gets pushed into another unit, the other unit will accordingly also get pushed back To handle walls you can divide them into the same structure used for unit collision detection.

Also look at

https://gamedev.net/tutorials/programming/general-and-gameplay-programming/swept-aabb-collision-detection-and-response-r3084/

https://www.reddit.com/r/cpp_questions/comments/1h8poij/how_do_i_traverse_through_an_aabb_tree_to_detect/