AI logic in games is usually based on navigation and decision-making models. Typical solutions include finite state machines (FSM) and behavior trees (Behavior Tree), as well as hybrid architectures and code patterns
Final State Machines
A finite state machine is a classic approach where NPC behavior is defined by a set of discrete states and rules for transitions between them. Each state corresponds to a type of behavior (for example, “patrolling,” “following the player,” “attacking,” etc.), and transitions occur based on events or conditions (reaching a target, detecting a player, etc.). Such a machine can easily be represented as a table or an object-oriented solution: a module-script may contain a dictionary of states, each with its own OnEnter, OnUpdate, OnExit functions — where onUpdate runs every frame. (
When the state changes, the corresponding handler functions are called, which simplifies the logic.
A bit simpler: FSM manages the states of an object, each representing a separate behavior, and transitions are defined by incoming events — signals.
For implementation, it’s common to create one “states” module where the current NPC status is stored. For example, there’s a table {Idle = function() … end, Patrol = function() … end, Attack = function() … end} and a variable state. Then in BindToRenderStep this script calls onUpdate of the current state.
A transition can happen with a simple assignment: state = "Attack".
There’s also a more interesting approach (in fact, trying to improve a finite state machine often leads to this) — each State “subclass” has enter(), exit(), update() methods, and the machine holds a reference to the active state object (similar to a more “serious” OOP-architecture). You might also create Translation. Most likely, these objects won’t even need methods, so it could just be create state = function(config) return config end. In my view, this is no worse than tables. Like the Vector library, which creates an actual Vector(not3) object but can also work with simple tables. (vector.method({1,2,3} or smg like this really will work)
FSM Advantages:
They are very simple to build “from scratch” and easy for developers to understand. FSM neatly localizes logic: each state has its own code, making debugging and reuse easier. When refactoring, it’s easy to extract common behavior into base functions and reuse them. If the game has a small set of discrete states, FSM makes the whole picture explicit: “we’re either patrolling, or attacking, or resting.”
FSM Limitations:
At the same time, as the number of states and transitions grows, automata become cumbersome. The number of combinations grows with branching, complicating maintenance. FSMs are “less resistant to changes on the fly,” since any new requirement may require reworking links between states.
In addition, a classic FSM only allows the NPC to be in one state at a time, making parallel actions harder to model. To solve this, hierarchical FSMs are sometimes introduced — for example, one large machine with nested sub-machines — but even so, scalability with a fixed number of states is limited.
Behavior Trees (BT):
A behavior tree is another model where logic is represented as a tree structure of nodes. The root of the tree is a controller, and the nodes can be different:
- Action: represents writing variables or performing some movement.
- Sequence nodes execute child behaviors in order until one returns “Failure,” “Running,” or “Error.” If that doesn’t happen, the node returns “Success.”
- Parallel nodes execute child behaviors until a given number of them return statuses “Failure” or “Success.”
- Selectors execute child behaviors in order until one returns “Success,” “Running,” or “Error.” If that doesn’t happen, the node returns “Failure.”
- Condition: some check returning a boolean — true or false.
- Inverter nodes act like a logical NOT operator.
Such trees are usually designed as data — often with nested tables or specific template prototype modules. For example, you can describe a tree as:
{
type="Selector",
children={
{type="Sequence", children={action1, action2}},
conditionX,
actionY
}
}
And the engine recursively “ticks” nodes each frame.
I explained this poorly. I don’t really understand how they work myself, I only have a basic idea.
BT Advantages:
BTs are convenient for complex and diverse behaviors. They are modular (one branch of the tree = one action scenario), easy to read in visual/tree form, and flexible to modify during development.
For large NPCs, trees significantly simplify development by breaking complex behaviors into understandable pieces, making scaling easier.
Indeed, if an NPC in a game needs to react to many conditions and tasks (for example, “pull the lever,” “avoid the pit,” “activate defense”), the tree orders these checks into a clear scheme.
The convenience of the modular BT architecture is also that if necessary you can “swap” a branch on the fly without breaking the rest of the logic.
BT Limitations:
The main drawback is implementation complexity. You need to design a set of nodes and provide handling for each step (node status: Running/Success/Failure). This requires writing the “framework” of the tree and lots of small functions. If NPC behavior is trivial (say, just walking back and forth), a tree may be unnecessary overengineering.
Combined Approaches:
Often FSM and BT are used together. For example, you can take FSM as the base and embed BT inside specific states, or the other way around.
This makes sense when part of the logic is better described with one approach, and part — with the other.
For example:
Idle → Chase → Attack where each state is in FSM, but inside each state is a separate tree.
That is, the FSM switches the current “main” activity (no target — wait, target detected — chase, close range — attack), and inside “attack” the tree defines the sequence of strikes or dodges.
The opposite case: the main tree may contain leaf nodes which essentially run micro-FSMs (for example, small automata to control animations or local transitions).
The choice depends on the task: if overall behavior is discrete with few states, an FSM on top is simpler; if the NPC must react to many external factors with nested logic, then BT is the “main” structure, and FSM is delegated at the lower level for specific actions.
Optimization (not only speed):
When developing multiple NPCs, it’s important not to copy logic for each. It’s useful to move shared parts into separate modules. For example, you can create a single AI controller or dispatcher that updates multiple NPCs at once via timers or events. This reduces code duplication and allows centralized changes in behavior.
For reusability, data patterns are applied: tables of configurations or data-scripts (like routes, “vision” parameters, game roles).
Performance should also be considered: it’s better not to force each NPC to endlessly run heavy computations every frame. Instead, respond to events (target position changed, alarm triggered, etc.) or update with different intervals. In other words, planning updates reduces load when many NPCs are present.
FSM or Behavior Tree?
The choice depends on goals and behavior complexity. FSM is preferable when NPC behavior is relatively simple and discrete — it guarantees clear and fast implementation. BT is better for more “alive” and flexible characters — when you need to handle many conditions, easily add new tasks, and change action order. One piece of advice from the community: there is no universal “best” solution — always choose the tool for the task.
Combined approaches often give the best results, and good architecture (centralized controllers, modularity, and “blackboards”) helps ensure scalability and easy maintenance for all types of NPCs.
(Blackboard: A common auxiliary structure is the “blackboard” — a shared set of variables (dictionary, attributes, etc.) available to all nodes of the tree. It stores the state of the environment or NPC (targets, timers, etc.) and allows nodes to exchange data.)
- 1
- 2
- 3
- 4
- 5
Also will be useful
ECS in luau and how to cook it: Click here
Pseudo-OOP luau — prototypal programming: why theres no “classic” OOP Click here