DataPredict [Release 1.21] - General Purpose Machine Learning And Deep Learning Library (Learning AIs, Generative AIs, and more!)

The tutorials are already available in the documentation. Let me know if you’re curious about anything but isn’t included under the tutorial section.

2 Likes

Hey, I just checked your neural network tutorial and I think it would be great to include the little details like how the bias neuron adds one neuron to the neuron count. That stuff isn’t intuitive to people who are new to the library because I think people would usually assume that adding an input layer with 2 neurons in it and bias set to true counts as 2 neurons instead of 3.

1 Like

I did a test with the most basic neural network possible and purposefully trained it with only one possible input and output.

local Library = require(script.Parent['DataPredict  - Release Version 1.2'])
local NeuralNet = Library.Models.NeuralNetwork.new(1,0.01)

NeuralNet:addLayer(1,true,'None',Library.Optimizers.AdaptiveMomentEstimation.new())
NeuralNet:addLayer(1,false,'None')
--NeuralNet:createLayers({1,3,3,2},'ReLU',Optimizer)

NeuralNet:setClassesList({0})

local ModifiedModel = Library.Others.GradientDescentModifier.new(NeuralNet)

local featureMatrix = {}
local labelVectorLogistic = {}

for i = 1,100 do
	table.insert(labelVectorLogistic,{5})
	table.insert(featureMatrix,{2.5,1})
end

ModifiedModel:train(featureMatrix,labelVectorLogistic)

print(ModifiedModel:predict({{2.5,1}},true)) -- Should be 5

However, there seems to be some issues with the cost. The cost seems to be increasing instead of decreasing.

 16:24:55.401  Iteration: 1		Cost: 29.2073946855965  -  Server - BaseModel:149
  16:24:55.402  Data Number: 1		Final Cost: 29.2073946855965
  -  Server - GradientDescentModifier:159
  16:24:55.402  Iteration: 1		Cost: 29.58692672689677  -  Server - BaseModel:149
  16:24:55.402  Data Number: 2		Final Cost: 29.58692672689677
  -  Server - GradientDescentModifier:159
  16:24:55.403  Iteration: 1		Cost: 29.96890876791703  -  Server - BaseModel:149
  16:24:55.403  Data Number: 3		Final Cost: 29.96890876791703
  -  Server - GradientDescentModifier:159
  16:24:55.404  Iteration: 1		Cost: 30.35334080865729  -  Server - BaseModel:149
  16:24:55.404  Data Number: 4		Final Cost: 30.35334080865729
  -  Server - GradientDescentModifier:159
  16:24:55.404  Iteration: 1		Cost: 30.740222849117547  -  Server - BaseModel:149
  16:24:55.404  Data Number: 5		Final Cost: 30.740222849117547
  -  Server - GradientDescentModifier:159
  16:24:55.405  Iteration: 1		Cost: 31.129554889297818  -  Server - BaseModel:149
  16:24:55.405  Data Number: 6		Final Cost: 31.129554889297818
  -  Server - GradientDescentModifier:159
  16:24:55.405  Iteration: 1		Cost: 31.52133692919808  -  Server - BaseModel:149
  16:24:55.406  Data Number: 7		Final Cost: 31.52133692919808
  -  Server - GradientDescentModifier:159
  16:24:55.406  Iteration: 1		Cost: 31.915568968818338  -  Server - BaseModel:149
  16:24:55.406  Data Number: 8		Final Cost: 31.915568968818338
  -  Server - GradientDescentModifier:159
  16:24:55.406  Iteration: 1		Cost: 32.3122510081586  -  Server - BaseModel:149
  16:24:55.407  Data Number: 9		Final Cost: 32.3122510081586
  -  Server - GradientDescentModifier:159
  16:24:55.407  Iteration: 1		Cost: 32.71138304721885  -  Server - BaseModel:149
  16:24:55.407  Data Number: 10		Final Cost: 32.71138304721885
  -  Server - GradientDescentModifier:159
  16:24:55.407  Iteration: 1		Cost: 33.11296508599912  -  Server - BaseModel:149
  16:24:55.408  Data Number: 11		Final Cost: 33.11296508599912
  -  Server - GradientDescentModifier:159
  16:24:55.408  Iteration: 1		Cost: 33.51699712449938  -  Server - BaseModel:149
  16:24:55.408  Data Number: 12		Final Cost: 33.51699712449938
  -  Server - GradientDescentModifier:159
  16:24:55.408  Iteration: 1		Cost: 33.923479162719644  -  Server - BaseModel:149
  16:24:55.408  Data Number: 13		Final Cost: 33.923479162719644
  -  Server - GradientDescentModifier:159
  16:24:55.409  Iteration: 1		Cost: 34.332411200659905  -  Server - BaseModel:149
  16:24:55.409  Data Number: 14		Final Cost: 34.332411200659905
  -  Server - GradientDescentModifier:159
  16:24:55.409  Iteration: 1		Cost: 34.74379323832017  -  Server - BaseModel:149
  16:24:55.410  Data Number: 15		Final Cost: 34.74379323832017
  -  Server - GradientDescentModifier:159
  16:24:55.410  Iteration: 1		Cost: 35.15762527570043  -  Server - BaseModel:149
  16:24:55.410  Data Number: 16		Final Cost: 35.15762527570043
  -  Server - GradientDescentModifier:159
  16:24:55.411  Iteration: 1		Cost: 35.57390731280069  -  Server - BaseModel:149
  16:24:55.411  Data Number: 17		Final Cost: 35.57390731280069
  -  Server - GradientDescentModifier:159
  16:24:55.412  Iteration: 1		Cost: 35.99263934962095  -  Server - BaseModel:149
  16:24:55.412  Data Number: 18		Final Cost: 35.99263934962095
  -  Server - GradientDescentModifier:159
  16:24:55.412  Iteration: 1		Cost: 36.413821386161224  -  Server - BaseModel:149
  16:24:55.412  Data Number: 19		Final Cost: 36.413821386161224
  -  Server - GradientDescentModifier:159
  16:24:55.413  Iteration: 1		Cost: 36.83745342242148  -  Server - BaseModel:149
  16:24:55.413  Data Number: 20		Final Cost: 36.83745342242148
  -  Server - GradientDescentModifier:159
  16:24:55.413  Iteration: 1		Cost: 37.26353545840174  -  Server - BaseModel:149
  16:24:55.413  Data Number: 21		Final Cost: 37.26353545840174
  -  Server - GradientDescentModifier:159
  16:24:55.414  Iteration: 1		Cost: 37.692067494102005  -  Server - BaseModel:149
  16:24:55.414  Data Number: 22		Final Cost: 37.692067494102005
  -  Server - GradientDescentModifier:159
  16:24:55.415  Iteration: 1		Cost: 38.12304952952226  -  Server - BaseModel:149
  16:24:55.415  Data Number: 23		Final Cost: 38.12304952952226
  -  Server - GradientDescentModifier:159
  16:24:55.415  Iteration: 1		Cost: 38.55648156466252  -  Server - BaseModel:149
  16:24:55.415  Data Number: 24		Final Cost: 38.55648156466252
  -  Server - GradientDescentModifier:159
  16:24:55.416  Iteration: 1		Cost: 38.99236359952279  -  Server - BaseModel:149
  16:24:55.416  Data Number: 25		Final Cost: 38.99236359952279
  -  Server - GradientDescentModifier:159
  16:24:55.416  Iteration: 1		Cost: 39.43069563410305  -  Server - BaseModel:149
  16:24:55.416  Data Number: 26		Final Cost: 39.43069563410305
  -  Server - GradientDescentModifier:159
  16:24:55.417  Iteration: 1		Cost: 39.871477668403315  -  Server - BaseModel:149
  16:24:55.417  Data Number: 27		Final Cost: 39.871477668403315
  -  Server - GradientDescentModifier:159
  16:24:55.417  Iteration: 1		Cost: 40.314709702423585  -  Server - BaseModel:149
  16:24:55.417  Data Number: 28		Final Cost: 40.314709702423585
  -  Server - GradientDescentModifier:159
  16:24:55.418  Iteration: 1		Cost: 40.76039173616385  -  Server - BaseModel:149
  16:24:55.418  Data Number: 29		Final Cost: 40.76039173616385
  -  Server - GradientDescentModifier:159
  16:24:55.418  Iteration: 1		Cost: 41.2085237696241  -  Server - BaseModel:149
  16:24:55.418  Data Number: 30		Final Cost: 41.2085237696241
  -  Server - GradientDescentModifier:159
  16:24:55.419  Iteration: 1		Cost: 41.65910580280436  -  Server - BaseModel:149
  16:24:55.419  Data Number: 31		Final Cost: 41.65910580280436
  -  Server - GradientDescentModifier:159
  16:24:55.419  Iteration: 1		Cost: 42.11213783570462  -  Server - BaseModel:149
  16:24:55.420  Data Number: 32		Final Cost: 42.11213783570462
  -  Server - GradientDescentModifier:159
  16:24:55.420  Iteration: 1		Cost: 42.56761986832488  -  Server - BaseModel:149
  16:24:55.420  Data Number: 33		Final Cost: 42.56761986832488
  -  Server - GradientDescentModifier:159
  16:24:55.421  Iteration: 1		Cost: 43.02555190066514  -  Server - BaseModel:149
  16:24:55.421  Data Number: 34		Final Cost: 43.02555190066514
  -  Server - GradientDescentModifier:159
  16:24:55.421  Iteration: 1		Cost: 43.4859339327254  -  Server - BaseModel:149
  16:24:55.421  Data Number: 35		Final Cost: 43.4859339327254
  -  Server - GradientDescentModifier:159
  16:24:55.422  Iteration: 1		Cost: 43.94876596450566  -  Server - BaseModel:149
  16:24:55.422  Data Number: 36		Final Cost: 43.94876596450566
  -  Server - GradientDescentModifier:159
  16:24:55.422  Iteration: 1		Cost: 44.41404799600593  -  Server - BaseModel:149
  16:24:55.423  Data Number: 37		Final Cost: 44.41404799600593
  -  Server - GradientDescentModifier:159
  16:24:55.423  Iteration: 1		Cost: 44.88178002722619  -  Server - BaseModel:149
  16:24:55.423  Data Number: 38		Final Cost: 44.88178002722619
  -  Server - GradientDescentModifier:159
  16:24:55.423  Iteration: 1		Cost: 45.35196205816646  -  Server - BaseModel:149
  16:24:55.424  Data Number: 39		Final Cost: 45.35196205816646
  -  Server - GradientDescentModifier:159
  16:24:55.424  Iteration: 1		Cost: 45.824594088826714  -  Server - BaseModel:149
  16:24:55.424  Data Number: 40		Final Cost: 45.824594088826714
  -  Server - GradientDescentModifier:159
  16:24:55.424  Iteration: 1		Cost: 46.29967611920698  -  Server - BaseModel:149
  16:24:55.424  Data Number: 41		Final Cost: 46.29967611920698
  -  Server - GradientDescentModifier:159
  16:24:55.425  Iteration: 1		Cost: 46.777208149307235  -  Server - BaseModel:149
  16:24:55.425  Data Number: 42		Final Cost: 46.777208149307235
  -  Server - GradientDescentModifier:159
  16:24:55.425  Iteration: 1		Cost: 47.2571901791275  -  Server - BaseModel:149
  16:24:55.425  Data Number: 43		Final Cost: 47.2571901791275
  -  Server - GradientDescentModifier:159
  16:24:55.426  Iteration: 1		Cost: 47.739622208667754  -  Server - BaseModel:149
  16:24:55.426  Data Number: 44		Final Cost: 47.739622208667754
  -  Server - GradientDescentModifier:159
  16:24:55.426  Iteration: 1		Cost: 48.224504237928016  -  Server - BaseModel:149
  16:24:55.426  Data Number: 45		Final Cost: 48.224504237928016
  -  Server - GradientDescentModifier:159
  16:24:55.427  Iteration: 1		Cost: 48.71183626690828  -  Server - BaseModel:149
  16:24:55.427  Data Number: 46		Final Cost: 48.71183626690828
  -  Server - GradientDescentModifier:159
  16:24:55.427  Iteration: 1		Cost: 49.20161829560855  -  Server - BaseModel:149
  16:24:55.427  Data Number: 47		Final Cost: 49.20161829560855
  -  Server - GradientDescentModifier:159
  16:24:55.428  Iteration: 1		Cost: 49.69385032402881  -  Server - BaseModel:149
  16:24:55.428  Data Number: 48		Final Cost: 49.69385032402881
  -  Server - GradientDescentModifier:159
  16:24:55.428  Iteration: 1		Cost: 50.188532352169055  -  Server - BaseModel:149
  16:24:55.428  Data Number: 49		Final Cost: 50.188532352169055
  -  Server - GradientDescentModifier:159
  16:24:55.429  Iteration: 1		Cost: 50.68566438002933  -  Server - BaseModel:149
  16:24:55.429  Data Number: 50		Final Cost: 50.68566438002933
  -  Server - GradientDescentModifier:159
  16:24:55.429  Iteration: 1		Cost: 51.18524640760959  -  Server - BaseModel:149
  16:24:55.429  Data Number: 51		Final Cost: 51.18524640760959
  -  Server - GradientDescentModifier:159
  16:24:55.430  Iteration: 1		Cost: 51.68727843490985  -  Server - BaseModel:149
  16:24:55.430  Data Number: 52		Final Cost: 51.68727843490985
  -  Server - GradientDescentModifier:159
  16:24:55.430  Iteration: 1		Cost: 52.19176046193011  -  Server - BaseModel:149
  16:24:55.430  Data Number: 53		Final Cost: 52.19176046193011
  -  Server - GradientDescentModifier:159
  16:24:55.431  Iteration: 1		Cost: 52.69869248867037  -  Server - BaseModel:149
  16:24:55.431  Data Number: 54		Final Cost: 52.69869248867037
  -  Server - GradientDescentModifier:159
  16:24:55.431  Iteration: 1		Cost: 53.208074515130626  -  Server - BaseModel:149
  16:24:55.431  Data Number: 55		Final Cost: 53.208074515130626
  -  Server - GradientDescentModifier:159
  16:24:55.432  Iteration: 1		Cost: 53.71990654131089  -  Server - BaseModel:149
  16:24:55.432  Data Number: 56		Final Cost: 53.71990654131089
  -  Server - GradientDescentModifier:159
  16:24:55.432  Iteration: 1		Cost: 54.23418856721115  -  Server - BaseModel:149
  16:24:55.432  Data Number: 57		Final Cost: 54.23418856721115
  -  Server - GradientDescentModifier:159
  16:24:55.432  Iteration: 1		Cost: 54.750920592831406  -  Server - BaseModel:149
  16:24:55.433  Data Number: 58		Final Cost: 54.750920592831406
  -  Server - GradientDescentModifier:159
  16:24:55.433  Iteration: 1		Cost: 55.27010261817167  -  Server - BaseModel:149
  16:24:55.433  Data Number: 59		Final Cost: 55.27010261817167
  -  Server - GradientDescentModifier:159
  16:24:55.434  Iteration: 1		Cost: 55.79173464323193  -  Server - BaseModel:149
  16:24:55.434  Data Number: 60		Final Cost: 55.79173464323193
  -  Server - GradientDescentModifier:159
  16:24:55.434  Iteration: 1		Cost: 56.31581666801219  -  Server - BaseModel:149
  16:24:55.434  Data Number: 61		Final Cost: 56.31581666801219
  -  Server - GradientDescentModifier:159
  16:24:55.435  Iteration: 1		Cost: 56.84234869251245  -  Server - BaseModel:149
  16:24:55.435  Data Number: 62		Final Cost: 56.84234869251245
  -  Server - GradientDescentModifier:159
  16:24:55.435  Iteration: 1		Cost: 57.37133071673272  -  Server - BaseModel:149
  16:24:55.435  Data Number: 63		Final Cost: 57.37133071673272
  -  Server - GradientDescentModifier:159
  16:24:55.436  Iteration: 1		Cost: 57.90276274067298  -  Server - BaseModel:149
  16:24:55.436  Data Number: 64		Final Cost: 57.90276274067298
  -  Server - GradientDescentModifier:159
  16:24:55.436  Iteration: 1		Cost: 58.436644764333245  -  Server - BaseModel:149
  16:24:55.436  Data Number: 65		Final Cost: 58.436644764333245
  -  Server - GradientDescentModifier:159
  16:24:55.437  Iteration: 1		Cost: 58.972976787713506  -  Server - BaseModel:149
  16:24:55.437  Data Number: 66		Final Cost: 58.972976787713506
  -  Server - GradientDescentModifier:159
  16:24:55.438  Iteration: 1		Cost: 59.51175881081377  -  Server - BaseModel:149
  16:24:55.438  Data Number: 67		Final Cost: 59.51175881081377
  -  Server - GradientDescentModifier:159
  16:24:55.438  Iteration: 1		Cost: 60.05299083363403  -  Server - BaseModel:149
  16:24:55.438  Data Number: 68		Final Cost: 60.05299083363403
  -  Server - GradientDescentModifier:159
  16:24:55.439  Iteration: 1		Cost: 60.59667285617429  -  Server - BaseModel:149
  16:24:55.439  Data Number: 69		Final Cost: 60.59667285617429
  -  Server - GradientDescentModifier:159
  16:24:55.439  Iteration: 1		Cost: 61.142804878434546  -  Server - BaseModel:149
  16:24:55.439  Data Number: 70		Final Cost: 61.142804878434546
  -  Server - GradientDescentModifier:159
  16:24:55.440  Iteration: 1		Cost: 61.69138690041482  -  Server - BaseModel:149
  16:24:55.440  Data Number: 71		Final Cost: 61.69138690041482
  -  Server - GradientDescentModifier:159
  16:24:55.440  Iteration: 1		Cost: 62.24241892211508  -  Server - BaseModel:149
  16:24:55.440  Data Number: 72		Final Cost: 62.24241892211508
  -  Server - GradientDescentModifier:159
  16:24:55.441  Iteration: 1		Cost: 62.795900943535344  -  Server - BaseModel:149
  16:24:55.441  Data Number: 73		Final Cost: 62.795900943535344
  -  Server - GradientDescentModifier:159
  16:24:55.441  Iteration: 1		Cost: 63.3518329646756  -  Server - BaseModel:149
  16:24:55.441  Data Number: 74		Final Cost: 63.3518329646756
  -  Server - GradientDescentModifier:159
  16:24:55.442  Iteration: 1		Cost: 63.91021498553586  -  Server - BaseModel:149
  16:24:55.442  Data Number: 75		Final Cost: 63.91021498553586
  -  Server - GradientDescentModifier:159
  16:24:55.442  Iteration: 1		Cost: 64.47104700611612  -  Server - BaseModel:149
  16:24:55.442  Data Number: 76		Final Cost: 64.47104700611612
  -  Server - GradientDescentModifier:159
  16:24:55.443  Iteration: 1		Cost: 65.03432902641639  -  Server - BaseModel:149
  16:24:55.443  Data Number: 77		Final Cost: 65.03432902641639
  -  Server - GradientDescentModifier:159
  16:24:55.443  Iteration: 1		Cost: 65.60006104643664  -  Server - BaseModel:149
  16:24:55.443  Data Number: 78		Final Cost: 65.60006104643664
  -  Server - GradientDescentModifier:159
  16:24:55.444  Iteration: 1		Cost: 66.16824306617693  -  Server - BaseModel:149
  16:24:55.444  Data Number: 79		Final Cost: 66.16824306617693
  -  Server - GradientDescentModifier:159
  16:24:55.444  Iteration: 1		Cost: 66.73887508563719  -  Server - BaseModel:149
  16:24:55.444  Data Number: 80		Final Cost: 66.73887508563719
  -  Server - GradientDescentModifier:159
  16:24:55.445  Iteration: 1		Cost: 67.31195710481745  -  Server - BaseModel:149
  16:24:55.445  Data Number: 81		Final Cost: 67.31195710481745
  -  Server - GradientDescentModifier:159
  16:24:55.445  Iteration: 1		Cost: 67.88748912371771  -  Server - BaseModel:149
  16:24:55.445  Data Number: 82		Final Cost: 67.88748912371771
  -  Server - GradientDescentModifier:159
  16:24:55.446  Iteration: 1		Cost: 68.46547114233798  -  Server - BaseModel:149
  16:24:55.446  Data Number: 83		Final Cost: 68.46547114233798
  -  Server - GradientDescentModifier:159
  16:24:55.446  Iteration: 1		Cost: 69.04590316067824  -  Server - BaseModel:149
  16:24:55.446  Data Number: 84		Final Cost: 69.04590316067824
  -  Server - GradientDescentModifier:159
  16:24:55.446  Iteration: 1		Cost: 69.6287851787385  -  Server - BaseModel:149
  16:24:55.446  Data Number: 85		Final Cost: 69.6287851787385
  -  Server - GradientDescentModifier:159
  16:24:55.447  Iteration: 1		Cost: 70.21411719651876  -  Server - BaseModel:149
  16:24:55.447  Data Number: 86		Final Cost: 70.21411719651876
  -  Server - GradientDescentModifier:159
  16:24:55.447  Iteration: 1		Cost: 70.80189921401902  -  Server - BaseModel:149
  16:24:55.448  Data Number: 87		Final Cost: 70.80189921401902
  -  Server - GradientDescentModifier:159
  16:24:55.448  Iteration: 1		Cost: 71.39213123123928  -  Server - BaseModel:149
  16:24:55.448  Data Number: 88		Final Cost: 71.39213123123928
  -  Server - GradientDescentModifier:159
  16:24:55.448  Iteration: 1		Cost: 71.98481324817953  -  Server - BaseModel:149
  16:24:55.448  Data Number: 89		Final Cost: 71.98481324817953
  -  Server - GradientDescentModifier:159
  16:24:55.449  Iteration: 1		Cost: 72.5799452648398  -  Server - BaseModel:149
  16:24:55.449  Data Number: 90		Final Cost: 72.5799452648398
  -  Server - GradientDescentModifier:159
  16:24:55.449  Iteration: 1		Cost: 73.17752728122007  -  Server - BaseModel:149
  16:24:55.449  Data Number: 91		Final Cost: 73.17752728122007
  -  Server - GradientDescentModifier:159
  16:24:55.449  Iteration: 1		Cost: 73.77755929732032  -  Server - BaseModel:149
  16:24:55.450  Data Number: 92		Final Cost: 73.77755929732032
  -  Server - GradientDescentModifier:159
  16:24:55.450  Iteration: 1		Cost: 74.38004131314058  -  Server - BaseModel:149
  16:24:55.450  Data Number: 93		Final Cost: 74.38004131314058
  -  Server - GradientDescentModifier:159
  16:24:55.450  Iteration: 1		Cost: 74.98497332868084  -  Server - BaseModel:149
  16:24:55.450  Data Number: 94		Final Cost: 74.98497332868084
  -  Server - GradientDescentModifier:159
  16:24:55.451  Iteration: 1		Cost: 75.59235534394111  -  Server - BaseModel:149
  16:24:55.451  Data Number: 95		Final Cost: 75.59235534394111
  -  Server - GradientDescentModifier:159
  16:24:55.451  Iteration: 1		Cost: 76.20218735892136  -  Server - BaseModel:149
  16:24:55.452  Data Number: 96		Final Cost: 76.20218735892136
  -  Server - GradientDescentModifier:159
  16:24:55.452  Iteration: 1		Cost: 76.81446937362162  -  Server - BaseModel:149
  16:24:55.452  Data Number: 97		Final Cost: 76.81446937362162
  -  Server - GradientDescentModifier:159
  16:24:55.452  Iteration: 1		Cost: 77.42920138804189  -  Server - BaseModel:149
  16:24:55.453  Data Number: 98		Final Cost: 77.42920138804189
  -  Server - GradientDescentModifier:159
  16:24:55.453  Iteration: 1		Cost: 78.04638340218214  -  Server - BaseModel:149
  16:24:55.453  Data Number: 99		Final Cost: 78.04638340218214
  -  Server - GradientDescentModifier:159
  16:24:55.453  Iteration: 1		Cost: 78.6660154160424  -  Server - BaseModel:149
  16:24:55.453  Data Number: 100		Final Cost: 78.6660154160424
  -  Server - GradientDescentModifier:159

I have the same results when adding/removing Adam as the optimizer. I’ve tried these learning rates: 0.001 and 0.00025 and the cost has the same trend.

1 Like

Mate, that’s literally normal. What you’re literally trying to do is something equivalent to linear regression. But that will not work because the output values are mapped to 0 and 1, and so will have hard time training the model parameters.

Also, that data has a number of dimension space, so the model may never find the best parameters. You’re better off using more data for that.

1 Like

I used no activation function for each of the layers. How is it possible for the output values to be mapped to 0 and 1? Even if that were the case the cost would likely be the same instead of increasing.

That code is repeatedly adding {2.5,1} to the featureMatrix and {5} to the labelVectorLogistic array. The only possible feature vector in the data is {2.5,1} and again the only possible label is {5}. Almost any neural network should be able to do this.

Also, sorry if the variable names are confusing, I was just too lazy to change them.

1 Like

Neural network is specifically made to handle classes.

Use linear regression if you want to do these kind of things. Less complicated.

1 Like

This was just a test to see if the optimizer is working correctly. I was suspicious after the cost of my sigmoid neural network didn’t seem to be decreasing but rather staying at ~0.2. I put the same parameters into my neural network framework with the only thing being changed is the optimizer and the cost decreased for mine. Here is the code I used for your library:

local Library = require(script.Parent['DataPredict  - Release Version 1.2'])
local NeuralNet = Library.Models.NeuralNetwork.new(10,0.001)

NeuralNet:addLayer(1,true,'Tanh',Library.Optimizers.AdaptiveMomentEstimation.new())
NeuralNet:addLayer(4,true,'Tanh')
NeuralNet:addLayer(4,true,'Tanh')
NeuralNet:addLayer(1,false,'Sigmoid')
--NeuralNet:createLayers({1,3,3,2},'ReLU',Optimizer)

NeuralNet:setClassesList({0})

local ModifiedModel = Library.Others.GradientDescentModifier.new(NeuralNet,'MiniBatch',10,true)

local featureMatrix = {}
local labelVectorLogistic = {}
--[[local featureMatrix = {

	{ 0,  0},
	{10, 2},
	{-3, -2},
	{-12, -22},
	{ 2,  2},
	{ 1,  1},
	{-11, -12},
	{ 3,  3},
	{-2, -2},

}

local labelVectorLogistic = {

	{1},
	{1},
	{0},
	{0},
	{1},
	{1},
	{0},
	{1},
	{0}

}--]]

for i = 1,100 do
	local Random1 = math.random(-1,1)
	local Random2 = math.random(-1,1)
	local TrueLabel = 0
	
	if Random1 >= 0 and Random2 >= 0 then
		TrueLabel = 1
	end
	
	table.insert(labelVectorLogistic,{TrueLabel})
	table.insert(featureMatrix,{Random1,Random2})
end

print(featureMatrix)
print(labelVectorLogistic)

ModifiedModel:train(featureMatrix,labelVectorLogistic)

print(ModifiedModel:predict({{90, 90}},true))
  19:44:25.590  Iteration: 1		Cost: 0.24163349439079018  -  Server - BaseModel:149
  19:44:25.598  Iteration: 2		Cost: 0.24163349439079018  -  Server - BaseModel:149
  19:44:25.606  Iteration: 3		Cost: 0.24163349439079018  -  Server - BaseModel:149
  19:44:25.613  Iteration: 4		Cost: 0.24163349439079018  -  Server - BaseModel:149
  19:44:25.620  Iteration: 5		Cost: 0.24163349439079018  -  Server - BaseModel:149
  19:44:25.627  Iteration: 6		Cost: 0.24163349439079018  -  Server - BaseModel:149
  19:44:25.632  Iteration: 7		Cost: 0.24163349439079018  -  Server - BaseModel:149
  19:44:25.642  Iteration: 8		Cost: 0.24163349439079018  -  Server - BaseModel:149
  19:44:25.659  Iteration: 9		Cost: 0.24163349439079018  -  Server - BaseModel:149
  19:44:25.669  Iteration: 10		Cost: 0.24163349439079018  -  Server - BaseModel:149
  19:44:25.669  Epoch: 1			Final Cost: 0.24163349439079018
  -  Server - GradientDescentModifier:127
  19:44:25.681  Iteration: 1		Cost: 0.24161504661667968  -  Server - BaseModel:149
  19:44:25.691  Iteration: 2		Cost: 0.24161504661667968  -  Server - BaseModel:149
  19:44:25.700  Iteration: 3		Cost: 0.24161504661667968  -  Server - BaseModel:149
  19:44:25.709  Iteration: 4		Cost: 0.24161504661667968  -  Server - BaseModel:149
  19:44:25.717  Iteration: 5		Cost: 0.24161504661667968  -  Server - BaseModel:149
  19:44:25.725  Iteration: 6		Cost: 0.24161504661667968  -  Server - BaseModel:149
  19:44:25.733  Iteration: 7		Cost: 0.24161504661667968  -  Server - BaseModel:149
  19:44:25.772  Iteration: 8		Cost: 0.24161504661667968  -  Server - BaseModel:149
  19:44:25.786  Iteration: 9		Cost: 0.24161504661667968  -  Server - BaseModel:149
  19:44:25.795  Iteration: 10		Cost: 0.24161504661667968  -  Server - BaseModel:149
  19:44:25.796  Epoch: 2			Final Cost: 0.24161504661667968
  -  Server - GradientDescentModifier:127
  19:44:25.803  Iteration: 1		Cost: 0.24159661713215652  -  Server - BaseModel:149
  19:44:25.811  Iteration: 2		Cost: 0.24159661713215652  -  Server - BaseModel:149
  19:44:25.821  Iteration: 3		Cost: 0.24159661713215652  -  Server - BaseModel:149
  19:44:25.830  Iteration: 4		Cost: 0.24159661713215652  -  Server - BaseModel:149
  19:44:25.839  Iteration: 5		Cost: 0.24159661713215652  -  Server - BaseModel:149
  19:44:25.844  Iteration: 6		Cost: 0.24159661713215652  -  Server - BaseModel:149
  19:44:25.852  Iteration: 7		Cost: 0.24159661713215652  -  Server - BaseModel:149
  19:44:25.862  Iteration: 8		Cost: 0.24159661713215652  -  Server - BaseModel:149
  19:44:25.870  Iteration: 9		Cost: 0.24159661713215652  -  Server - BaseModel:149
  19:44:25.876  Iteration: 10		Cost: 0.24159661713215652  -  Server - BaseModel:149
  19:44:25.876  Epoch: 3			Final Cost: 0.24159661713215652
  -  Server - GradientDescentModifier:127
  19:44:25.884  Iteration: 1		Cost: 0.24157820596622326  -  Server - BaseModel:149
  19:44:25.889  Iteration: 2		Cost: 0.24157820596622326  -  Server - BaseModel:149
  19:44:25.894  Iteration: 3		Cost: 0.24157820596622326  -  Server - BaseModel:149
  19:44:25.901  Iteration: 4		Cost: 0.24157820596622326  -  Server - BaseModel:149
  19:44:25.907  Iteration: 5		Cost: 0.24157820596622326  -  Server - BaseModel:149
  19:44:25.914  Iteration: 6		Cost: 0.24157820596622326  -  Server - BaseModel:149
  19:44:25.921  Iteration: 7		Cost: 0.24157820596622326  -  Server - BaseModel:149
  19:44:25.928  Iteration: 8		Cost: 0.24157820596622326  -  Server - BaseModel:149
  19:44:25.936  Iteration: 9		Cost: 0.24157820596622326  -  Server - BaseModel:149
  19:44:25.943  Iteration: 10		Cost: 0.24157820596622326  -  Server - BaseModel:149
  19:44:25.943  Epoch: 4			Final Cost: 0.24157820596622326
  -  Server - GradientDescentModifier:127
  19:44:25.948  Iteration: 1		Cost: 0.24155981314786956  -  Server - BaseModel:149
  19:44:25.955  Iteration: 2		Cost: 0.24155981314786956  -  Server - BaseModel:149
  19:44:25.961  Iteration: 3		Cost: 0.24155981314786956  -  Server - BaseModel:149
  19:44:25.965  Iteration: 4		Cost: 0.24155981314786956  -  Server - BaseModel:149
  19:44:25.973  Iteration: 5		Cost: 0.24155981314786956  -  Server - BaseModel:149
  19:44:25.978  Iteration: 6		Cost: 0.24155981314786956  -  Server - BaseModel:149
  19:44:25.985  Iteration: 7		Cost: 0.24155981314786956  -  Server - BaseModel:149
  19:44:25.991  Iteration: 8		Cost: 0.24155981314786956  -  Server - BaseModel:149
  19:44:25.995  Iteration: 9		Cost: 0.24155981314786956  -  Server - BaseModel:149
  19:44:26.003  Iteration: 10		Cost: 0.24155981314786956  -  Server - BaseModel:149
  19:44:26.003  Epoch: 5			Final Cost: 0.24155981314786956
  -  Server - GradientDescentModifier:127
  19:44:26.008  Iteration: 1		Cost: 0.24154143870607453  -  Server - BaseModel:149
  19:44:26.014  Iteration: 2		Cost: 0.24154143870607453  -  Server - BaseModel:149
  19:44:26.020  Iteration: 3		Cost: 0.24154143870607453  -  Server - BaseModel:149
  19:44:26.026  Iteration: 4		Cost: 0.24154143870607453  -  Server - BaseModel:149
  19:44:26.030  Iteration: 5		Cost: 0.24154143870607453  -  Server - BaseModel:149
  19:44:26.038  Iteration: 6		Cost: 0.24154143870607453  -  Server - BaseModel:149
  19:44:26.043  Iteration: 7		Cost: 0.24154143870607453  -  Server - BaseModel:149
  19:44:26.050  Iteration: 8		Cost: 0.24154143870607453  -  Server - BaseModel:149
  19:44:26.055  Iteration: 9		Cost: 0.24154143870607453  -  Server - BaseModel:149
  19:44:26.062  Iteration: 10		Cost: 0.24154143870607453  -  Server - BaseModel:149
  19:44:26.062  Epoch: 6			Final Cost: 0.24154143870607453
  -  Server - GradientDescentModifier:127
  19:44:26.068  Iteration: 1		Cost: 0.2415230826698038  -  Server - BaseModel:149
  19:44:26.072  Iteration: 2		Cost: 0.2415230826698038  -  Server - BaseModel:149
  19:44:26.079  Iteration: 3		Cost: 0.2415230826698038  -  Server - BaseModel:149
  19:44:26.084  Iteration: 4		Cost: 0.2415230826698038  -  Server - BaseModel:149
  19:44:26.093  Iteration: 5		Cost: 0.2415230826698038  -  Server - BaseModel:149
  19:44:26.099  Iteration: 6		Cost: 0.2415230826698038  -  Server - BaseModel:149
  19:44:26.104  Iteration: 7		Cost: 0.2415230826698038  -  Server - BaseModel:149
  19:44:26.109  Iteration: 8		Cost: 0.2415230826698038  -  Server - BaseModel:149
  19:44:26.117  Iteration: 9		Cost: 0.2415230826698038  -  Server - BaseModel:149
  19:44:26.123  Iteration: 10		Cost: 0.2415230826698038  -  Server - BaseModel:149
  19:44:26.124  Epoch: 7			Final Cost: 0.2415230826698038
  -  Server - GradientDescentModifier:127
  19:44:26.129  Iteration: 1		Cost: 0.24150474506801242  -  Server - BaseModel:149
  19:44:26.134  Iteration: 2		Cost: 0.24150474506801242  -  Server - BaseModel:149
  19:44:26.141  Iteration: 3		Cost: 0.24150474506801242  -  Server - BaseModel:149
  19:44:26.148  Iteration: 4		Cost: 0.24150474506801242  -  Server - BaseModel:149
  19:44:26.154  Iteration: 5		Cost: 0.24150474506801242  -  Server - BaseModel:149
  19:44:26.161  Iteration: 6		Cost: 0.24150474506801242  -  Server - BaseModel:149
  19:44:26.165  Iteration: 7		Cost: 0.24150474506801242  -  Server - BaseModel:149
  19:44:26.172  Iteration: 8		Cost: 0.24150474506801242  -  Server - BaseModel:149
  19:44:26.177  Iteration: 9		Cost: 0.24150474506801242  -  Server - BaseModel:149
  19:44:26.182  Iteration: 10		Cost: 0.24150474506801242  -  Server - BaseModel:149
  19:44:26.182  Epoch: 8			Final Cost: 0.24150474506801242
  -  Server - GradientDescentModifier:127
  19:44:26.187  Iteration: 1		Cost: 0.2414864259296416  -  Server - BaseModel:149
  19:44:26.191  Iteration: 2		Cost: 0.2414864259296416  -  Server - BaseModel:149
  19:44:26.199  Iteration: 3		Cost: 0.2414864259296416  -  Server - BaseModel:149
  19:44:26.204  Iteration: 4		Cost: 0.2414864259296416  -  Server - BaseModel:149
  19:44:26.212  Iteration: 5		Cost: 0.2414864259296416  -  Server - BaseModel:149
  19:44:26.218  Iteration: 6		Cost: 0.2414864259296416  -  Server - BaseModel:149
  19:44:26.223  Iteration: 7		Cost: 0.2414864259296416  -  Server - BaseModel:149
  19:44:26.229  Iteration: 8		Cost: 0.2414864259296416  -  Server - BaseModel:149
  19:44:26.234  Iteration: 9		Cost: 0.2414864259296416  -  Server - BaseModel:149
  19:44:26.241  Iteration: 10		Cost: 0.2414864259296416  -  Server - BaseModel:149
  19:44:26.242  Epoch: 9			Final Cost: 0.2414864259296416
  -  Server - GradientDescentModifier:127
  19:44:26.248  Iteration: 1		Cost: 0.2414681252836223  -  Server - BaseModel:149
  19:44:26.253  Iteration: 2		Cost: 0.2414681252836223  -  Server - BaseModel:149
  19:44:26.261  Iteration: 3		Cost: 0.2414681252836223  -  Server - BaseModel:149
  19:44:26.266  Iteration: 4		Cost: 0.2414681252836223  -  Server - BaseModel:149
  19:44:26.270  Iteration: 5		Cost: 0.2414681252836223  -  Server - BaseModel:149
  19:44:26.278  Iteration: 6		Cost: 0.2414681252836223  -  Server - BaseModel:149
  19:44:26.283  Iteration: 7		Cost: 0.2414681252836223  -  Server - BaseModel:149
  19:44:26.291  Iteration: 8		Cost: 0.2414681252836223  -  Server - BaseModel:149
  19:44:26.296  Iteration: 9		Cost: 0.2414681252836223  -  Server - BaseModel:149
  19:44:26.304  Iteration: 10		Cost: 0.2414681252836223  -  Server - BaseModel:149
  19:44:26.304  Epoch: 10			Final Cost: 0.2414681252836223

Is this a bug or did I just do something wrong in the code?

1 Like

Overfitting. Or it has already converged. Or both.

Also you for got to add 1 to your classes list.

1 Like

I’m not an expert but I think it’s a bit strange that the cost barely changes every 10 iterations. I don’t think that is overfitting. I also don’t think it could converge in just 100 iterations.

I tried adding 1 to the classes list and the same thing happens.

2 Likes

We’re talking about neural networks. They are designed to capture complex pattern in the data.

You might see the cost fluctuating using LeakyReLU, instead of tanh.

1 Like

I’ve tried LeakyReLU and the same thing happens.

1 Like

Same reason here.

Also, I forgot to mention you need an extra column for bias (value of 1) at the very left by adding one.

1 Like

Extra column? Where? Can you point to a specific line? Also, if it is overfitting why is the training cost so high? Also, I’m noticing patterns in the cost. It seems to be decreasing but verrrrrrrrrry slowly.

1 Like

In your feature matrix, it needs an extra column to take into account of bias.

So when you set the first layer to have bias, you need value of ones in the first column.

The cost values are independent from the overfitting.

At this point, I recommend you to take an online course on machine learning. Too may gaps on your knowledge.

1 Like

Extra column…? There are already 2 columns? I set it to add 1 to the featureVector in the for loop but it just gives me the “Input Layer has 2 neurons but feature matrix has 3 features” error.

local Library = require(script.Parent['DataPredict  - Release Version 1.2'])
local NeuralNet = Library.Models.NeuralNetwork.new(10,0.01)

NeuralNet:addLayer(1,true,'LeakyReLU')
NeuralNet:addLayer(4,true,'LeakyReLU')
NeuralNet:addLayer(4,true,'LeakyReLU')
NeuralNet:addLayer(1,false,'Sigmoid')
--NeuralNet:createLayers({1,3,3,2},'ReLU',Optimizer)

NeuralNet:setClassesList({1})

local ModifiedModel = Library.Others.GradientDescentModifier.new(NeuralNet,'MiniBatch',10,true)

local featureMatrix = {}
local labelVectorLogistic = {}
--[[local featureMatrix = {

	{ 0,  0},
	{10, 2},
	{-3, -2},
	{-12, -22},
	{ 2,  2},
	{ 1,  1},
	{-11, -12},
	{ 3,  3},
	{-2, -2},

}

local labelVectorLogistic = {

	{1},
	{1},
	{0},
	{0},
	{1},
	{1},
	{0},
	{1},
	{0}

}--]]

for i = 1,100 do
	local Random1 = math.random(-1,1)
	local Random2 = math.random(-1,1)
	local TrueLabel = 0
	
	if Random1 >= 0 and Random2 >= 0 then
		TrueLabel = 1
	end
	
	table.insert(labelVectorLogistic,{TrueLabel})
	table.insert(featureMatrix,{1,Random1,Random2})
end

print(featureMatrix)
print(labelVectorLogistic)

ModifiedModel:train(featureMatrix,labelVectorLogistic)

print(ModifiedModel:predict({{90, 90}},true))
1 Like

All I can say, it is recommended that you take machine learning course for full understanding of neural networks, especially when it comes to bias values. No matter how much I will explain it, it will only cause confusion without sufficient knowledge.

1 Like

I understand how neural networks work I just don’t understand how your library works.

1 Like

No, you certainly don’t exactly understand how neural network works. Nor machine learning in general. If you ask about why the cost is too high when overfitting, that already marks that you are extremely confident in your machine learning knowledge.

Pay attention to feature matrix, why can it accept two columns when you specified one + bias neuron. One of those columns are supposed to be for bias value (which is 1). This should have been common sense for people who are familiar with bias values.

2 Likes

Hahaha, this thread is so long. Maybe it should be taken to DMs.

1 Like

On line 198 in your QLearningNeuralNetwork, you need to change self.epsilon to self.currentEpsilon.

Original code:

Fixed code:

4 Likes