-- Create a new table
mat = {}
-- Loop all matrix values
for i = 1, N do
mat[i] = {} -- Create a new row
for j = 1, M do
mat[i][j] = 0
end
end
-- N = is the number of rows
-- M = is the number of columns
-- i = is used to keep track of row indexes
-- j = is used to keep track of col indexes

You’re attempting to use the N and M variable when you didn’t define it. Here’s a fix.

N = 5 -- change number to change the number of rows
M = 5 -- change number to change the number of columns
mat = {}
for i = 1, N do
mat[i] = {}
for j = 1, M do
mat[i][j] = 0
end
end