Watch as I draw on the screen and it predicts what digit I have drawn!
How it works
I’m using a 2 layer feed forward fully connected network with sigmoid activation, cross entropy & softmax cost, stochastic gradient descent. I’m achieving about 92% accuracy (which is bad in MNIST world). It’s not hard to get it up to 99.7% accuracy using convolutional neural nets, but I’m doing all the math by hand in order to learn more linear algebra & how ML works on low level. Doing all the forward/back propagation myself.
I trained it on 60,000 MNIST digits over several epochs (in Python), and then exported the weights & biases to Roblox. I wrote some matrix utility code (dot/mul, add, sigmoid) to do the forward propagation in Roblox. Then I wrote a little canvas painting thing and put it together. Every ~0.2 seconds I update the classification probabilities. It’s really fast.
100 neurons in hidden layer. I tried 30 too. Hyperparameters ¯\_(ツ)_/¯
EDIT: So, layer sizes go 784 (28*28) → 100 → 10 (labels/# of digits)
Right now the main issue is that the MNIST dataset was created by fitting squares around digits, so when I draw digits that aren’t perfectly centered or bounded, it doesn’t recognize them as well. If I did some pre-processing on the dataset and add random rotation/scaling to digits, it’d handle this scenario better. The edge detection that CNN’s bring is far superior, though.
Oh okay, guess you’ll need more hidden units if you want to detect reasonable position/rotation/scaling invariance? That seems pretty good though still.
How do you prune connections by the way? Every epoch, or after training, or?
I probably won’t need more hidden units. Maybe another layer, but really this is best solved with a convolutional network. What do you mean by pruning? Dropout? I don’t have any dropout layers implemented.
I did something similar in a machine learning course and I remember requiring quite a few more hidden units in proportion to input, but maybe it’s because you have some differences w.r.t. cost function and propagation method… either way I’m not a machine learning expert or anything, I was just curious how you set up your network
With pruning I just meant whether you cut away connections that fall below a certain weight threshold to reduce the processing that you need to do in training/testing steps, but I guess with this size network you probably don’t even need to do that yet, so forget about that question
That looks awesome!
I like the human-like body. Is it possible to get that model? I tried to set the sizes and proportions right, but I just couldn’t get it.
Resizing the number to make its largest axis length equal to the size of the bounding box (and proportionally resizing the smaller axis) and then centering the number should reduce the complexity of the samples.
goals
