The tutorials are already available in the documentation. Let me know if you’re curious about anything but isn’t included under the tutorial section.
2 Likes
Hey, I just checked your neural network tutorial and I think it would be great to include the little details like how the bias neuron adds one neuron to the neuron count. That stuff isn’t intuitive to people who are new to the library because I think people would usually assume that adding an input layer with 2 neurons in it and bias set to true counts as 2 neurons instead of 3.
1 Like
I did a test with the most basic neural network possible and purposefully trained it with only one possible input and output.
local Library = require(script.Parent['DataPredict - Release Version 1.2'])
local NeuralNet = Library.Models.NeuralNetwork.new(1,0.01)
NeuralNet:addLayer(1,true,'None',Library.Optimizers.AdaptiveMomentEstimation.new())
NeuralNet:addLayer(1,false,'None')
--NeuralNet:createLayers({1,3,3,2},'ReLU',Optimizer)
NeuralNet:setClassesList({0})
local ModifiedModel = Library.Others.GradientDescentModifier.new(NeuralNet)
local featureMatrix = {}
local labelVectorLogistic = {}
for i = 1,100 do
table.insert(labelVectorLogistic,{5})
table.insert(featureMatrix,{2.5,1})
end
ModifiedModel:train(featureMatrix,labelVectorLogistic)
print(ModifiedModel:predict({{2.5,1}},true)) -- Should be 5
However, there seems to be some issues with the cost. The cost seems to be increasing instead of decreasing.
16:24:55.401 Iteration: 1 Cost: 29.2073946855965 - Server - BaseModel:149
16:24:55.402 Data Number: 1 Final Cost: 29.2073946855965
- Server - GradientDescentModifier:159
16:24:55.402 Iteration: 1 Cost: 29.58692672689677 - Server - BaseModel:149
16:24:55.402 Data Number: 2 Final Cost: 29.58692672689677
- Server - GradientDescentModifier:159
16:24:55.403 Iteration: 1 Cost: 29.96890876791703 - Server - BaseModel:149
16:24:55.403 Data Number: 3 Final Cost: 29.96890876791703
- Server - GradientDescentModifier:159
16:24:55.404 Iteration: 1 Cost: 30.35334080865729 - Server - BaseModel:149
16:24:55.404 Data Number: 4 Final Cost: 30.35334080865729
- Server - GradientDescentModifier:159
16:24:55.404 Iteration: 1 Cost: 30.740222849117547 - Server - BaseModel:149
16:24:55.404 Data Number: 5 Final Cost: 30.740222849117547
- Server - GradientDescentModifier:159
16:24:55.405 Iteration: 1 Cost: 31.129554889297818 - Server - BaseModel:149
16:24:55.405 Data Number: 6 Final Cost: 31.129554889297818
- Server - GradientDescentModifier:159
16:24:55.405 Iteration: 1 Cost: 31.52133692919808 - Server - BaseModel:149
16:24:55.406 Data Number: 7 Final Cost: 31.52133692919808
- Server - GradientDescentModifier:159
16:24:55.406 Iteration: 1 Cost: 31.915568968818338 - Server - BaseModel:149
16:24:55.406 Data Number: 8 Final Cost: 31.915568968818338
- Server - GradientDescentModifier:159
16:24:55.406 Iteration: 1 Cost: 32.3122510081586 - Server - BaseModel:149
16:24:55.407 Data Number: 9 Final Cost: 32.3122510081586
- Server - GradientDescentModifier:159
16:24:55.407 Iteration: 1 Cost: 32.71138304721885 - Server - BaseModel:149
16:24:55.407 Data Number: 10 Final Cost: 32.71138304721885
- Server - GradientDescentModifier:159
16:24:55.407 Iteration: 1 Cost: 33.11296508599912 - Server - BaseModel:149
16:24:55.408 Data Number: 11 Final Cost: 33.11296508599912
- Server - GradientDescentModifier:159
16:24:55.408 Iteration: 1 Cost: 33.51699712449938 - Server - BaseModel:149
16:24:55.408 Data Number: 12 Final Cost: 33.51699712449938
- Server - GradientDescentModifier:159
16:24:55.408 Iteration: 1 Cost: 33.923479162719644 - Server - BaseModel:149
16:24:55.408 Data Number: 13 Final Cost: 33.923479162719644
- Server - GradientDescentModifier:159
16:24:55.409 Iteration: 1 Cost: 34.332411200659905 - Server - BaseModel:149
16:24:55.409 Data Number: 14 Final Cost: 34.332411200659905
- Server - GradientDescentModifier:159
16:24:55.409 Iteration: 1 Cost: 34.74379323832017 - Server - BaseModel:149
16:24:55.410 Data Number: 15 Final Cost: 34.74379323832017
- Server - GradientDescentModifier:159
16:24:55.410 Iteration: 1 Cost: 35.15762527570043 - Server - BaseModel:149
16:24:55.410 Data Number: 16 Final Cost: 35.15762527570043
- Server - GradientDescentModifier:159
16:24:55.411 Iteration: 1 Cost: 35.57390731280069 - Server - BaseModel:149
16:24:55.411 Data Number: 17 Final Cost: 35.57390731280069
- Server - GradientDescentModifier:159
16:24:55.412 Iteration: 1 Cost: 35.99263934962095 - Server - BaseModel:149
16:24:55.412 Data Number: 18 Final Cost: 35.99263934962095
- Server - GradientDescentModifier:159
16:24:55.412 Iteration: 1 Cost: 36.413821386161224 - Server - BaseModel:149
16:24:55.412 Data Number: 19 Final Cost: 36.413821386161224
- Server - GradientDescentModifier:159
16:24:55.413 Iteration: 1 Cost: 36.83745342242148 - Server - BaseModel:149
16:24:55.413 Data Number: 20 Final Cost: 36.83745342242148
- Server - GradientDescentModifier:159
16:24:55.413 Iteration: 1 Cost: 37.26353545840174 - Server - BaseModel:149
16:24:55.413 Data Number: 21 Final Cost: 37.26353545840174
- Server - GradientDescentModifier:159
16:24:55.414 Iteration: 1 Cost: 37.692067494102005 - Server - BaseModel:149
16:24:55.414 Data Number: 22 Final Cost: 37.692067494102005
- Server - GradientDescentModifier:159
16:24:55.415 Iteration: 1 Cost: 38.12304952952226 - Server - BaseModel:149
16:24:55.415 Data Number: 23 Final Cost: 38.12304952952226
- Server - GradientDescentModifier:159
16:24:55.415 Iteration: 1 Cost: 38.55648156466252 - Server - BaseModel:149
16:24:55.415 Data Number: 24 Final Cost: 38.55648156466252
- Server - GradientDescentModifier:159
16:24:55.416 Iteration: 1 Cost: 38.99236359952279 - Server - BaseModel:149
16:24:55.416 Data Number: 25 Final Cost: 38.99236359952279
- Server - GradientDescentModifier:159
16:24:55.416 Iteration: 1 Cost: 39.43069563410305 - Server - BaseModel:149
16:24:55.416 Data Number: 26 Final Cost: 39.43069563410305
- Server - GradientDescentModifier:159
16:24:55.417 Iteration: 1 Cost: 39.871477668403315 - Server - BaseModel:149
16:24:55.417 Data Number: 27 Final Cost: 39.871477668403315
- Server - GradientDescentModifier:159
16:24:55.417 Iteration: 1 Cost: 40.314709702423585 - Server - BaseModel:149
16:24:55.417 Data Number: 28 Final Cost: 40.314709702423585
- Server - GradientDescentModifier:159
16:24:55.418 Iteration: 1 Cost: 40.76039173616385 - Server - BaseModel:149
16:24:55.418 Data Number: 29 Final Cost: 40.76039173616385
- Server - GradientDescentModifier:159
16:24:55.418 Iteration: 1 Cost: 41.2085237696241 - Server - BaseModel:149
16:24:55.418 Data Number: 30 Final Cost: 41.2085237696241
- Server - GradientDescentModifier:159
16:24:55.419 Iteration: 1 Cost: 41.65910580280436 - Server - BaseModel:149
16:24:55.419 Data Number: 31 Final Cost: 41.65910580280436
- Server - GradientDescentModifier:159
16:24:55.419 Iteration: 1 Cost: 42.11213783570462 - Server - BaseModel:149
16:24:55.420 Data Number: 32 Final Cost: 42.11213783570462
- Server - GradientDescentModifier:159
16:24:55.420 Iteration: 1 Cost: 42.56761986832488 - Server - BaseModel:149
16:24:55.420 Data Number: 33 Final Cost: 42.56761986832488
- Server - GradientDescentModifier:159
16:24:55.421 Iteration: 1 Cost: 43.02555190066514 - Server - BaseModel:149
16:24:55.421 Data Number: 34 Final Cost: 43.02555190066514
- Server - GradientDescentModifier:159
16:24:55.421 Iteration: 1 Cost: 43.4859339327254 - Server - BaseModel:149
16:24:55.421 Data Number: 35 Final Cost: 43.4859339327254
- Server - GradientDescentModifier:159
16:24:55.422 Iteration: 1 Cost: 43.94876596450566 - Server - BaseModel:149
16:24:55.422 Data Number: 36 Final Cost: 43.94876596450566
- Server - GradientDescentModifier:159
16:24:55.422 Iteration: 1 Cost: 44.41404799600593 - Server - BaseModel:149
16:24:55.423 Data Number: 37 Final Cost: 44.41404799600593
- Server - GradientDescentModifier:159
16:24:55.423 Iteration: 1 Cost: 44.88178002722619 - Server - BaseModel:149
16:24:55.423 Data Number: 38 Final Cost: 44.88178002722619
- Server - GradientDescentModifier:159
16:24:55.423 Iteration: 1 Cost: 45.35196205816646 - Server - BaseModel:149
16:24:55.424 Data Number: 39 Final Cost: 45.35196205816646
- Server - GradientDescentModifier:159
16:24:55.424 Iteration: 1 Cost: 45.824594088826714 - Server - BaseModel:149
16:24:55.424 Data Number: 40 Final Cost: 45.824594088826714
- Server - GradientDescentModifier:159
16:24:55.424 Iteration: 1 Cost: 46.29967611920698 - Server - BaseModel:149
16:24:55.424 Data Number: 41 Final Cost: 46.29967611920698
- Server - GradientDescentModifier:159
16:24:55.425 Iteration: 1 Cost: 46.777208149307235 - Server - BaseModel:149
16:24:55.425 Data Number: 42 Final Cost: 46.777208149307235
- Server - GradientDescentModifier:159
16:24:55.425 Iteration: 1 Cost: 47.2571901791275 - Server - BaseModel:149
16:24:55.425 Data Number: 43 Final Cost: 47.2571901791275
- Server - GradientDescentModifier:159
16:24:55.426 Iteration: 1 Cost: 47.739622208667754 - Server - BaseModel:149
16:24:55.426 Data Number: 44 Final Cost: 47.739622208667754
- Server - GradientDescentModifier:159
16:24:55.426 Iteration: 1 Cost: 48.224504237928016 - Server - BaseModel:149
16:24:55.426 Data Number: 45 Final Cost: 48.224504237928016
- Server - GradientDescentModifier:159
16:24:55.427 Iteration: 1 Cost: 48.71183626690828 - Server - BaseModel:149
16:24:55.427 Data Number: 46 Final Cost: 48.71183626690828
- Server - GradientDescentModifier:159
16:24:55.427 Iteration: 1 Cost: 49.20161829560855 - Server - BaseModel:149
16:24:55.427 Data Number: 47 Final Cost: 49.20161829560855
- Server - GradientDescentModifier:159
16:24:55.428 Iteration: 1 Cost: 49.69385032402881 - Server - BaseModel:149
16:24:55.428 Data Number: 48 Final Cost: 49.69385032402881
- Server - GradientDescentModifier:159
16:24:55.428 Iteration: 1 Cost: 50.188532352169055 - Server - BaseModel:149
16:24:55.428 Data Number: 49 Final Cost: 50.188532352169055
- Server - GradientDescentModifier:159
16:24:55.429 Iteration: 1 Cost: 50.68566438002933 - Server - BaseModel:149
16:24:55.429 Data Number: 50 Final Cost: 50.68566438002933
- Server - GradientDescentModifier:159
16:24:55.429 Iteration: 1 Cost: 51.18524640760959 - Server - BaseModel:149
16:24:55.429 Data Number: 51 Final Cost: 51.18524640760959
- Server - GradientDescentModifier:159
16:24:55.430 Iteration: 1 Cost: 51.68727843490985 - Server - BaseModel:149
16:24:55.430 Data Number: 52 Final Cost: 51.68727843490985
- Server - GradientDescentModifier:159
16:24:55.430 Iteration: 1 Cost: 52.19176046193011 - Server - BaseModel:149
16:24:55.430 Data Number: 53 Final Cost: 52.19176046193011
- Server - GradientDescentModifier:159
16:24:55.431 Iteration: 1 Cost: 52.69869248867037 - Server - BaseModel:149
16:24:55.431 Data Number: 54 Final Cost: 52.69869248867037
- Server - GradientDescentModifier:159
16:24:55.431 Iteration: 1 Cost: 53.208074515130626 - Server - BaseModel:149
16:24:55.431 Data Number: 55 Final Cost: 53.208074515130626
- Server - GradientDescentModifier:159
16:24:55.432 Iteration: 1 Cost: 53.71990654131089 - Server - BaseModel:149
16:24:55.432 Data Number: 56 Final Cost: 53.71990654131089
- Server - GradientDescentModifier:159
16:24:55.432 Iteration: 1 Cost: 54.23418856721115 - Server - BaseModel:149
16:24:55.432 Data Number: 57 Final Cost: 54.23418856721115
- Server - GradientDescentModifier:159
16:24:55.432 Iteration: 1 Cost: 54.750920592831406 - Server - BaseModel:149
16:24:55.433 Data Number: 58 Final Cost: 54.750920592831406
- Server - GradientDescentModifier:159
16:24:55.433 Iteration: 1 Cost: 55.27010261817167 - Server - BaseModel:149
16:24:55.433 Data Number: 59 Final Cost: 55.27010261817167
- Server - GradientDescentModifier:159
16:24:55.434 Iteration: 1 Cost: 55.79173464323193 - Server - BaseModel:149
16:24:55.434 Data Number: 60 Final Cost: 55.79173464323193
- Server - GradientDescentModifier:159
16:24:55.434 Iteration: 1 Cost: 56.31581666801219 - Server - BaseModel:149
16:24:55.434 Data Number: 61 Final Cost: 56.31581666801219
- Server - GradientDescentModifier:159
16:24:55.435 Iteration: 1 Cost: 56.84234869251245 - Server - BaseModel:149
16:24:55.435 Data Number: 62 Final Cost: 56.84234869251245
- Server - GradientDescentModifier:159
16:24:55.435 Iteration: 1 Cost: 57.37133071673272 - Server - BaseModel:149
16:24:55.435 Data Number: 63 Final Cost: 57.37133071673272
- Server - GradientDescentModifier:159
16:24:55.436 Iteration: 1 Cost: 57.90276274067298 - Server - BaseModel:149
16:24:55.436 Data Number: 64 Final Cost: 57.90276274067298
- Server - GradientDescentModifier:159
16:24:55.436 Iteration: 1 Cost: 58.436644764333245 - Server - BaseModel:149
16:24:55.436 Data Number: 65 Final Cost: 58.436644764333245
- Server - GradientDescentModifier:159
16:24:55.437 Iteration: 1 Cost: 58.972976787713506 - Server - BaseModel:149
16:24:55.437 Data Number: 66 Final Cost: 58.972976787713506
- Server - GradientDescentModifier:159
16:24:55.438 Iteration: 1 Cost: 59.51175881081377 - Server - BaseModel:149
16:24:55.438 Data Number: 67 Final Cost: 59.51175881081377
- Server - GradientDescentModifier:159
16:24:55.438 Iteration: 1 Cost: 60.05299083363403 - Server - BaseModel:149
16:24:55.438 Data Number: 68 Final Cost: 60.05299083363403
- Server - GradientDescentModifier:159
16:24:55.439 Iteration: 1 Cost: 60.59667285617429 - Server - BaseModel:149
16:24:55.439 Data Number: 69 Final Cost: 60.59667285617429
- Server - GradientDescentModifier:159
16:24:55.439 Iteration: 1 Cost: 61.142804878434546 - Server - BaseModel:149
16:24:55.439 Data Number: 70 Final Cost: 61.142804878434546
- Server - GradientDescentModifier:159
16:24:55.440 Iteration: 1 Cost: 61.69138690041482 - Server - BaseModel:149
16:24:55.440 Data Number: 71 Final Cost: 61.69138690041482
- Server - GradientDescentModifier:159
16:24:55.440 Iteration: 1 Cost: 62.24241892211508 - Server - BaseModel:149
16:24:55.440 Data Number: 72 Final Cost: 62.24241892211508
- Server - GradientDescentModifier:159
16:24:55.441 Iteration: 1 Cost: 62.795900943535344 - Server - BaseModel:149
16:24:55.441 Data Number: 73 Final Cost: 62.795900943535344
- Server - GradientDescentModifier:159
16:24:55.441 Iteration: 1 Cost: 63.3518329646756 - Server - BaseModel:149
16:24:55.441 Data Number: 74 Final Cost: 63.3518329646756
- Server - GradientDescentModifier:159
16:24:55.442 Iteration: 1 Cost: 63.91021498553586 - Server - BaseModel:149
16:24:55.442 Data Number: 75 Final Cost: 63.91021498553586
- Server - GradientDescentModifier:159
16:24:55.442 Iteration: 1 Cost: 64.47104700611612 - Server - BaseModel:149
16:24:55.442 Data Number: 76 Final Cost: 64.47104700611612
- Server - GradientDescentModifier:159
16:24:55.443 Iteration: 1 Cost: 65.03432902641639 - Server - BaseModel:149
16:24:55.443 Data Number: 77 Final Cost: 65.03432902641639
- Server - GradientDescentModifier:159
16:24:55.443 Iteration: 1 Cost: 65.60006104643664 - Server - BaseModel:149
16:24:55.443 Data Number: 78 Final Cost: 65.60006104643664
- Server - GradientDescentModifier:159
16:24:55.444 Iteration: 1 Cost: 66.16824306617693 - Server - BaseModel:149
16:24:55.444 Data Number: 79 Final Cost: 66.16824306617693
- Server - GradientDescentModifier:159
16:24:55.444 Iteration: 1 Cost: 66.73887508563719 - Server - BaseModel:149
16:24:55.444 Data Number: 80 Final Cost: 66.73887508563719
- Server - GradientDescentModifier:159
16:24:55.445 Iteration: 1 Cost: 67.31195710481745 - Server - BaseModel:149
16:24:55.445 Data Number: 81 Final Cost: 67.31195710481745
- Server - GradientDescentModifier:159
16:24:55.445 Iteration: 1 Cost: 67.88748912371771 - Server - BaseModel:149
16:24:55.445 Data Number: 82 Final Cost: 67.88748912371771
- Server - GradientDescentModifier:159
16:24:55.446 Iteration: 1 Cost: 68.46547114233798 - Server - BaseModel:149
16:24:55.446 Data Number: 83 Final Cost: 68.46547114233798
- Server - GradientDescentModifier:159
16:24:55.446 Iteration: 1 Cost: 69.04590316067824 - Server - BaseModel:149
16:24:55.446 Data Number: 84 Final Cost: 69.04590316067824
- Server - GradientDescentModifier:159
16:24:55.446 Iteration: 1 Cost: 69.6287851787385 - Server - BaseModel:149
16:24:55.446 Data Number: 85 Final Cost: 69.6287851787385
- Server - GradientDescentModifier:159
16:24:55.447 Iteration: 1 Cost: 70.21411719651876 - Server - BaseModel:149
16:24:55.447 Data Number: 86 Final Cost: 70.21411719651876
- Server - GradientDescentModifier:159
16:24:55.447 Iteration: 1 Cost: 70.80189921401902 - Server - BaseModel:149
16:24:55.448 Data Number: 87 Final Cost: 70.80189921401902
- Server - GradientDescentModifier:159
16:24:55.448 Iteration: 1 Cost: 71.39213123123928 - Server - BaseModel:149
16:24:55.448 Data Number: 88 Final Cost: 71.39213123123928
- Server - GradientDescentModifier:159
16:24:55.448 Iteration: 1 Cost: 71.98481324817953 - Server - BaseModel:149
16:24:55.448 Data Number: 89 Final Cost: 71.98481324817953
- Server - GradientDescentModifier:159
16:24:55.449 Iteration: 1 Cost: 72.5799452648398 - Server - BaseModel:149
16:24:55.449 Data Number: 90 Final Cost: 72.5799452648398
- Server - GradientDescentModifier:159
16:24:55.449 Iteration: 1 Cost: 73.17752728122007 - Server - BaseModel:149
16:24:55.449 Data Number: 91 Final Cost: 73.17752728122007
- Server - GradientDescentModifier:159
16:24:55.449 Iteration: 1 Cost: 73.77755929732032 - Server - BaseModel:149
16:24:55.450 Data Number: 92 Final Cost: 73.77755929732032
- Server - GradientDescentModifier:159
16:24:55.450 Iteration: 1 Cost: 74.38004131314058 - Server - BaseModel:149
16:24:55.450 Data Number: 93 Final Cost: 74.38004131314058
- Server - GradientDescentModifier:159
16:24:55.450 Iteration: 1 Cost: 74.98497332868084 - Server - BaseModel:149
16:24:55.450 Data Number: 94 Final Cost: 74.98497332868084
- Server - GradientDescentModifier:159
16:24:55.451 Iteration: 1 Cost: 75.59235534394111 - Server - BaseModel:149
16:24:55.451 Data Number: 95 Final Cost: 75.59235534394111
- Server - GradientDescentModifier:159
16:24:55.451 Iteration: 1 Cost: 76.20218735892136 - Server - BaseModel:149
16:24:55.452 Data Number: 96 Final Cost: 76.20218735892136
- Server - GradientDescentModifier:159
16:24:55.452 Iteration: 1 Cost: 76.81446937362162 - Server - BaseModel:149
16:24:55.452 Data Number: 97 Final Cost: 76.81446937362162
- Server - GradientDescentModifier:159
16:24:55.452 Iteration: 1 Cost: 77.42920138804189 - Server - BaseModel:149
16:24:55.453 Data Number: 98 Final Cost: 77.42920138804189
- Server - GradientDescentModifier:159
16:24:55.453 Iteration: 1 Cost: 78.04638340218214 - Server - BaseModel:149
16:24:55.453 Data Number: 99 Final Cost: 78.04638340218214
- Server - GradientDescentModifier:159
16:24:55.453 Iteration: 1 Cost: 78.6660154160424 - Server - BaseModel:149
16:24:55.453 Data Number: 100 Final Cost: 78.6660154160424
- Server - GradientDescentModifier:159
I have the same results when adding/removing Adam as the optimizer. I’ve tried these learning rates: 0.001 and 0.00025 and the cost has the same trend.
1 Like
Mate, that’s literally normal. What you’re literally trying to do is something equivalent to linear regression. But that will not work because the output values are mapped to 0 and 1, and so will have hard time training the model parameters.
Also, that data has a number of dimension space, so the model may never find the best parameters. You’re better off using more data for that.
1 Like
I used no activation function for each of the layers. How is it possible for the output values to be mapped to 0 and 1? Even if that were the case the cost would likely be the same instead of increasing.
That code is repeatedly adding {2.5,1} to the featureMatrix and {5} to the labelVectorLogistic array. The only possible feature vector in the data is {2.5,1} and again the only possible label is {5}. Almost any neural network should be able to do this.
Also, sorry if the variable names are confusing, I was just too lazy to change them.
1 Like
Neural network is specifically made to handle classes.
Use linear regression if you want to do these kind of things. Less complicated.
1 Like
This was just a test to see if the optimizer is working correctly. I was suspicious after the cost of my sigmoid neural network didn’t seem to be decreasing but rather staying at ~0.2. I put the same parameters into my neural network framework with the only thing being changed is the optimizer and the cost decreased for mine. Here is the code I used for your library:
local Library = require(script.Parent['DataPredict - Release Version 1.2'])
local NeuralNet = Library.Models.NeuralNetwork.new(10,0.001)
NeuralNet:addLayer(1,true,'Tanh',Library.Optimizers.AdaptiveMomentEstimation.new())
NeuralNet:addLayer(4,true,'Tanh')
NeuralNet:addLayer(4,true,'Tanh')
NeuralNet:addLayer(1,false,'Sigmoid')
--NeuralNet:createLayers({1,3,3,2},'ReLU',Optimizer)
NeuralNet:setClassesList({0})
local ModifiedModel = Library.Others.GradientDescentModifier.new(NeuralNet,'MiniBatch',10,true)
local featureMatrix = {}
local labelVectorLogistic = {}
--[[local featureMatrix = {
{ 0, 0},
{10, 2},
{-3, -2},
{-12, -22},
{ 2, 2},
{ 1, 1},
{-11, -12},
{ 3, 3},
{-2, -2},
}
local labelVectorLogistic = {
{1},
{1},
{0},
{0},
{1},
{1},
{0},
{1},
{0}
}--]]
for i = 1,100 do
local Random1 = math.random(-1,1)
local Random2 = math.random(-1,1)
local TrueLabel = 0
if Random1 >= 0 and Random2 >= 0 then
TrueLabel = 1
end
table.insert(labelVectorLogistic,{TrueLabel})
table.insert(featureMatrix,{Random1,Random2})
end
print(featureMatrix)
print(labelVectorLogistic)
ModifiedModel:train(featureMatrix,labelVectorLogistic)
print(ModifiedModel:predict({{90, 90}},true))
19:44:25.590 Iteration: 1 Cost: 0.24163349439079018 - Server - BaseModel:149
19:44:25.598 Iteration: 2 Cost: 0.24163349439079018 - Server - BaseModel:149
19:44:25.606 Iteration: 3 Cost: 0.24163349439079018 - Server - BaseModel:149
19:44:25.613 Iteration: 4 Cost: 0.24163349439079018 - Server - BaseModel:149
19:44:25.620 Iteration: 5 Cost: 0.24163349439079018 - Server - BaseModel:149
19:44:25.627 Iteration: 6 Cost: 0.24163349439079018 - Server - BaseModel:149
19:44:25.632 Iteration: 7 Cost: 0.24163349439079018 - Server - BaseModel:149
19:44:25.642 Iteration: 8 Cost: 0.24163349439079018 - Server - BaseModel:149
19:44:25.659 Iteration: 9 Cost: 0.24163349439079018 - Server - BaseModel:149
19:44:25.669 Iteration: 10 Cost: 0.24163349439079018 - Server - BaseModel:149
19:44:25.669 Epoch: 1 Final Cost: 0.24163349439079018
- Server - GradientDescentModifier:127
19:44:25.681 Iteration: 1 Cost: 0.24161504661667968 - Server - BaseModel:149
19:44:25.691 Iteration: 2 Cost: 0.24161504661667968 - Server - BaseModel:149
19:44:25.700 Iteration: 3 Cost: 0.24161504661667968 - Server - BaseModel:149
19:44:25.709 Iteration: 4 Cost: 0.24161504661667968 - Server - BaseModel:149
19:44:25.717 Iteration: 5 Cost: 0.24161504661667968 - Server - BaseModel:149
19:44:25.725 Iteration: 6 Cost: 0.24161504661667968 - Server - BaseModel:149
19:44:25.733 Iteration: 7 Cost: 0.24161504661667968 - Server - BaseModel:149
19:44:25.772 Iteration: 8 Cost: 0.24161504661667968 - Server - BaseModel:149
19:44:25.786 Iteration: 9 Cost: 0.24161504661667968 - Server - BaseModel:149
19:44:25.795 Iteration: 10 Cost: 0.24161504661667968 - Server - BaseModel:149
19:44:25.796 Epoch: 2 Final Cost: 0.24161504661667968
- Server - GradientDescentModifier:127
19:44:25.803 Iteration: 1 Cost: 0.24159661713215652 - Server - BaseModel:149
19:44:25.811 Iteration: 2 Cost: 0.24159661713215652 - Server - BaseModel:149
19:44:25.821 Iteration: 3 Cost: 0.24159661713215652 - Server - BaseModel:149
19:44:25.830 Iteration: 4 Cost: 0.24159661713215652 - Server - BaseModel:149
19:44:25.839 Iteration: 5 Cost: 0.24159661713215652 - Server - BaseModel:149
19:44:25.844 Iteration: 6 Cost: 0.24159661713215652 - Server - BaseModel:149
19:44:25.852 Iteration: 7 Cost: 0.24159661713215652 - Server - BaseModel:149
19:44:25.862 Iteration: 8 Cost: 0.24159661713215652 - Server - BaseModel:149
19:44:25.870 Iteration: 9 Cost: 0.24159661713215652 - Server - BaseModel:149
19:44:25.876 Iteration: 10 Cost: 0.24159661713215652 - Server - BaseModel:149
19:44:25.876 Epoch: 3 Final Cost: 0.24159661713215652
- Server - GradientDescentModifier:127
19:44:25.884 Iteration: 1 Cost: 0.24157820596622326 - Server - BaseModel:149
19:44:25.889 Iteration: 2 Cost: 0.24157820596622326 - Server - BaseModel:149
19:44:25.894 Iteration: 3 Cost: 0.24157820596622326 - Server - BaseModel:149
19:44:25.901 Iteration: 4 Cost: 0.24157820596622326 - Server - BaseModel:149
19:44:25.907 Iteration: 5 Cost: 0.24157820596622326 - Server - BaseModel:149
19:44:25.914 Iteration: 6 Cost: 0.24157820596622326 - Server - BaseModel:149
19:44:25.921 Iteration: 7 Cost: 0.24157820596622326 - Server - BaseModel:149
19:44:25.928 Iteration: 8 Cost: 0.24157820596622326 - Server - BaseModel:149
19:44:25.936 Iteration: 9 Cost: 0.24157820596622326 - Server - BaseModel:149
19:44:25.943 Iteration: 10 Cost: 0.24157820596622326 - Server - BaseModel:149
19:44:25.943 Epoch: 4 Final Cost: 0.24157820596622326
- Server - GradientDescentModifier:127
19:44:25.948 Iteration: 1 Cost: 0.24155981314786956 - Server - BaseModel:149
19:44:25.955 Iteration: 2 Cost: 0.24155981314786956 - Server - BaseModel:149
19:44:25.961 Iteration: 3 Cost: 0.24155981314786956 - Server - BaseModel:149
19:44:25.965 Iteration: 4 Cost: 0.24155981314786956 - Server - BaseModel:149
19:44:25.973 Iteration: 5 Cost: 0.24155981314786956 - Server - BaseModel:149
19:44:25.978 Iteration: 6 Cost: 0.24155981314786956 - Server - BaseModel:149
19:44:25.985 Iteration: 7 Cost: 0.24155981314786956 - Server - BaseModel:149
19:44:25.991 Iteration: 8 Cost: 0.24155981314786956 - Server - BaseModel:149
19:44:25.995 Iteration: 9 Cost: 0.24155981314786956 - Server - BaseModel:149
19:44:26.003 Iteration: 10 Cost: 0.24155981314786956 - Server - BaseModel:149
19:44:26.003 Epoch: 5 Final Cost: 0.24155981314786956
- Server - GradientDescentModifier:127
19:44:26.008 Iteration: 1 Cost: 0.24154143870607453 - Server - BaseModel:149
19:44:26.014 Iteration: 2 Cost: 0.24154143870607453 - Server - BaseModel:149
19:44:26.020 Iteration: 3 Cost: 0.24154143870607453 - Server - BaseModel:149
19:44:26.026 Iteration: 4 Cost: 0.24154143870607453 - Server - BaseModel:149
19:44:26.030 Iteration: 5 Cost: 0.24154143870607453 - Server - BaseModel:149
19:44:26.038 Iteration: 6 Cost: 0.24154143870607453 - Server - BaseModel:149
19:44:26.043 Iteration: 7 Cost: 0.24154143870607453 - Server - BaseModel:149
19:44:26.050 Iteration: 8 Cost: 0.24154143870607453 - Server - BaseModel:149
19:44:26.055 Iteration: 9 Cost: 0.24154143870607453 - Server - BaseModel:149
19:44:26.062 Iteration: 10 Cost: 0.24154143870607453 - Server - BaseModel:149
19:44:26.062 Epoch: 6 Final Cost: 0.24154143870607453
- Server - GradientDescentModifier:127
19:44:26.068 Iteration: 1 Cost: 0.2415230826698038 - Server - BaseModel:149
19:44:26.072 Iteration: 2 Cost: 0.2415230826698038 - Server - BaseModel:149
19:44:26.079 Iteration: 3 Cost: 0.2415230826698038 - Server - BaseModel:149
19:44:26.084 Iteration: 4 Cost: 0.2415230826698038 - Server - BaseModel:149
19:44:26.093 Iteration: 5 Cost: 0.2415230826698038 - Server - BaseModel:149
19:44:26.099 Iteration: 6 Cost: 0.2415230826698038 - Server - BaseModel:149
19:44:26.104 Iteration: 7 Cost: 0.2415230826698038 - Server - BaseModel:149
19:44:26.109 Iteration: 8 Cost: 0.2415230826698038 - Server - BaseModel:149
19:44:26.117 Iteration: 9 Cost: 0.2415230826698038 - Server - BaseModel:149
19:44:26.123 Iteration: 10 Cost: 0.2415230826698038 - Server - BaseModel:149
19:44:26.124 Epoch: 7 Final Cost: 0.2415230826698038
- Server - GradientDescentModifier:127
19:44:26.129 Iteration: 1 Cost: 0.24150474506801242 - Server - BaseModel:149
19:44:26.134 Iteration: 2 Cost: 0.24150474506801242 - Server - BaseModel:149
19:44:26.141 Iteration: 3 Cost: 0.24150474506801242 - Server - BaseModel:149
19:44:26.148 Iteration: 4 Cost: 0.24150474506801242 - Server - BaseModel:149
19:44:26.154 Iteration: 5 Cost: 0.24150474506801242 - Server - BaseModel:149
19:44:26.161 Iteration: 6 Cost: 0.24150474506801242 - Server - BaseModel:149
19:44:26.165 Iteration: 7 Cost: 0.24150474506801242 - Server - BaseModel:149
19:44:26.172 Iteration: 8 Cost: 0.24150474506801242 - Server - BaseModel:149
19:44:26.177 Iteration: 9 Cost: 0.24150474506801242 - Server - BaseModel:149
19:44:26.182 Iteration: 10 Cost: 0.24150474506801242 - Server - BaseModel:149
19:44:26.182 Epoch: 8 Final Cost: 0.24150474506801242
- Server - GradientDescentModifier:127
19:44:26.187 Iteration: 1 Cost: 0.2414864259296416 - Server - BaseModel:149
19:44:26.191 Iteration: 2 Cost: 0.2414864259296416 - Server - BaseModel:149
19:44:26.199 Iteration: 3 Cost: 0.2414864259296416 - Server - BaseModel:149
19:44:26.204 Iteration: 4 Cost: 0.2414864259296416 - Server - BaseModel:149
19:44:26.212 Iteration: 5 Cost: 0.2414864259296416 - Server - BaseModel:149
19:44:26.218 Iteration: 6 Cost: 0.2414864259296416 - Server - BaseModel:149
19:44:26.223 Iteration: 7 Cost: 0.2414864259296416 - Server - BaseModel:149
19:44:26.229 Iteration: 8 Cost: 0.2414864259296416 - Server - BaseModel:149
19:44:26.234 Iteration: 9 Cost: 0.2414864259296416 - Server - BaseModel:149
19:44:26.241 Iteration: 10 Cost: 0.2414864259296416 - Server - BaseModel:149
19:44:26.242 Epoch: 9 Final Cost: 0.2414864259296416
- Server - GradientDescentModifier:127
19:44:26.248 Iteration: 1 Cost: 0.2414681252836223 - Server - BaseModel:149
19:44:26.253 Iteration: 2 Cost: 0.2414681252836223 - Server - BaseModel:149
19:44:26.261 Iteration: 3 Cost: 0.2414681252836223 - Server - BaseModel:149
19:44:26.266 Iteration: 4 Cost: 0.2414681252836223 - Server - BaseModel:149
19:44:26.270 Iteration: 5 Cost: 0.2414681252836223 - Server - BaseModel:149
19:44:26.278 Iteration: 6 Cost: 0.2414681252836223 - Server - BaseModel:149
19:44:26.283 Iteration: 7 Cost: 0.2414681252836223 - Server - BaseModel:149
19:44:26.291 Iteration: 8 Cost: 0.2414681252836223 - Server - BaseModel:149
19:44:26.296 Iteration: 9 Cost: 0.2414681252836223 - Server - BaseModel:149
19:44:26.304 Iteration: 10 Cost: 0.2414681252836223 - Server - BaseModel:149
19:44:26.304 Epoch: 10 Final Cost: 0.2414681252836223
Is this a bug or did I just do something wrong in the code?
1 Like
Overfitting. Or it has already converged. Or both.
Also you for got to add 1 to your classes list.
1 Like
I’m not an expert but I think it’s a bit strange that the cost barely changes every 10 iterations. I don’t think that is overfitting. I also don’t think it could converge in just 100 iterations.
I tried adding 1 to the classes list and the same thing happens.
2 Likes
We’re talking about neural networks. They are designed to capture complex pattern in the data.
You might see the cost fluctuating using LeakyReLU, instead of tanh.
1 Like
I’ve tried LeakyReLU and the same thing happens.
1 Like
Same reason here.
Also, I forgot to mention you need an extra column for bias (value of 1) at the very left by adding one.
1 Like
Extra column? Where? Can you point to a specific line? Also, if it is overfitting why is the training cost so high? Also, I’m noticing patterns in the cost. It seems to be decreasing but verrrrrrrrrry slowly.
1 Like
In your feature matrix, it needs an extra column to take into account of bias.
So when you set the first layer to have bias, you need value of ones in the first column.
The cost values are independent from the overfitting.
At this point, I recommend you to take an online course on machine learning. Too may gaps on your knowledge.
1 Like
Extra column…? There are already 2 columns? I set it to add 1 to the featureVector in the for loop but it just gives me the “Input Layer has 2 neurons but feature matrix has 3 features” error.
local Library = require(script.Parent['DataPredict - Release Version 1.2'])
local NeuralNet = Library.Models.NeuralNetwork.new(10,0.01)
NeuralNet:addLayer(1,true,'LeakyReLU')
NeuralNet:addLayer(4,true,'LeakyReLU')
NeuralNet:addLayer(4,true,'LeakyReLU')
NeuralNet:addLayer(1,false,'Sigmoid')
--NeuralNet:createLayers({1,3,3,2},'ReLU',Optimizer)
NeuralNet:setClassesList({1})
local ModifiedModel = Library.Others.GradientDescentModifier.new(NeuralNet,'MiniBatch',10,true)
local featureMatrix = {}
local labelVectorLogistic = {}
--[[local featureMatrix = {
{ 0, 0},
{10, 2},
{-3, -2},
{-12, -22},
{ 2, 2},
{ 1, 1},
{-11, -12},
{ 3, 3},
{-2, -2},
}
local labelVectorLogistic = {
{1},
{1},
{0},
{0},
{1},
{1},
{0},
{1},
{0}
}--]]
for i = 1,100 do
local Random1 = math.random(-1,1)
local Random2 = math.random(-1,1)
local TrueLabel = 0
if Random1 >= 0 and Random2 >= 0 then
TrueLabel = 1
end
table.insert(labelVectorLogistic,{TrueLabel})
table.insert(featureMatrix,{1,Random1,Random2})
end
print(featureMatrix)
print(labelVectorLogistic)
ModifiedModel:train(featureMatrix,labelVectorLogistic)
print(ModifiedModel:predict({{90, 90}},true))
1 Like
All I can say, it is recommended that you take machine learning course for full understanding of neural networks, especially when it comes to bias values. No matter how much I will explain it, it will only cause confusion without sufficient knowledge.
1 Like
I understand how neural networks work I just don’t understand how your library works.
1 Like
No, you certainly don’t exactly understand how neural network works. Nor machine learning in general. If you ask about why the cost is too high when overfitting, that already marks that you are extremely confident in your machine learning knowledge.
Pay attention to feature matrix, why can it accept two columns when you specified one + bias neuron. One of those columns are supposed to be for bias value (which is 1). This should have been common sense for people who are familiar with bias values.
2 Likes
Hahaha, this thread is so long. Maybe it should be taken to DMs.
1 Like
