Hilo! Currently I am using the raw magnitude distance as an input to tell the car how far eachraycast went for each side, so the value can reach well up to 50-100, I was wondering if there was a way to shorten this value since inverseDistance causes it to not perform that well compared to raw Raycast magnitude Distance.
PS: The car reaches the curve more often now! (nevermind, still likes to crash into the wall)
Apply log function to the distance.
Hey, I just realized maybe it’s better to detect if the car has reached the destination or not instead of distance. 0 for not reaching the destination, and 1 is when you reached the destination.
I don’t think the magnitude of the distance carry much weight compared to the orientation of the target location. That should make training more faster.
Hmm, the thing is the AI doesn’t seem to know where exactly to go, I think I have an idea, why not place him somewhere easier to drive at, somewhere there’s no turns, this way I think I can teach him to avoid the walls first, because that is most likely what is confusing him, once he reaches the goal, I can elevate his difficulty more and more, adding turns then eventually a 4-way.
What do you think?
Yeah, that concept is already exists in research papers. They call it “Curriculum Learning”, which has proven to work.
That being said, don’t forget to save the model parameters to a text file that you can bring them anywhere out side Roblox. It seems like you’re going to spend days working on this and it would be bad if you lose all the progress. :3
You can find the examples in the sword fighting AIs code.
I also realised a mistake I made which highly likely is the cause for most of the lack of learning, the vehicle applies realistic physics when it accelerates or reverses, and since I rely on my raycasts to be precise enough to be able to hit the short sidewalk, most of the time the rays shoot just above the sidewalk and dont hit anything, returning nil or hit the ground, to solve this lazy issue, I decided to bring in ‘Ray Walls’ that I am currently moduling into every single of my road pieces to absorb raycasts adequately and restrict them from not hitting anything, making the raycast more precise and accurate to hopefully improve training, I am guessing this will greatly improve what agent I have now since he does not need to fight against rough or noisy input data anymore.
I thought you would realize that later when I saw your videos :>
All I could suggest is to make them point a little more downward.
Wait… jus curious am I bugging or did u disable us from directly adding layers to Double Expected SARSA
Also, still get these issues from DeepDoubleExpectedSARSAV2
Happens from this line of code
ClippedPPO has issues as well:
I don’t think there is a way to set the classesList
I managed to somewhat brute-force fix it:
But I get another error after that
From here
Same as well for DeepQlearning and DeepDoubleQlearning
Ah I forgot to mention that you now need to use :setModel() and put your neural network there.
If I had left it unchanged there, I expect adding future codes will become much more difficult and time consuming.
What about the errors? D: they were occurring after I had :setModel()
Are the models still giving outputs despite the errors?
They give output once before erroring and refusing to provide further output.
Was there any change to how you collect the states?
For example you collect the state every heartbeat instead of timed interval?
Because if it is, I can confirm that the neural network calculation isn’t fast enough to catch up with it.
Ah okay. Try to remove the experience replay part. I’m pretty sure that one isn’t there previously.
still the same errorr
That’s very odd. I can’t replicate the error at all. I have to look into your code base if I want to fix this.
Meanwhile, I can see you’re doing Genetic Algorithm, maybe you can check if your code is having issues?
Oh, the genetic algorithm UI was for my old implementation, I repurposed it to just keep track of how long the car has been living for
Edit: did you update MatrixL?
local sStorage = game:GetService('ServerStorage')
local repStorage = game:GetService('ReplicatedStorage')
local DataStore = game:GetService('DataStoreService')
local Gizmo = require(repStorage.Gizmo)
workspace:SetAttribute("GizmosEnabled", true)
local ActorModelStorage = DataStore:GetDataStore('ActorModelStorage')
local CriticModelStorage = DataStore:GetDataStore('CriticModelStorage')
local DataPredict = require(repStorage.DataPredict)
local InputDataEvent = script.InputData
local SaveParamsEvent = script.Save
local NPCHumanoids = workspace.NPCHumanoids
local PathfinderCourse = workspace.PathfinderCourse
local CityMap = workspace.CityMap
local Car = sStorage["2014 Chevy Caprice PPV"]
local Start = CityMap.Start
local Goal = CityMap.Goal
local raycastParams = RaycastParams.new()
raycastParams.FilterType = Enum.RaycastFilterType.Exclude
raycastParams.FilterDescendantsInstances = {Car}
local RayParts = Car.RayParts
--print('Saved Model Parameters: ', ModelStorage:GetAsync(script.StorageID.Value))
local MatrixL = require(repStorage.DataPredict.MatrixL)
local maxRewardArrayLength = 100
local maxCurrentArrayLength = 100
local isRewardedArray = {}
local currentAccuracyArray = {}
local classesList = {'IncreaseThrottle', 'IncreaseLeftSteer', 'DecreaseThrottle', 'IncreaseRightSteer'}
local Optimizer = DataPredict.Optimizers.AdaptiveMomentEstimation.new()
local Reg = DataPredict.Others.Regularization.new()
local Attempts = 0
local Controller = require(Car.Controls)
--[[local function buildNPC()
local npcClone = NPC:Clone()
npcClone.Parent=NPCHumanoids
npcClone:PivotTo(Start.CFrame)
Controller = require(npcClone.Controls)
return npcClone
end]]
local function buildActor()
local Actor = DataPredict.Models.NeuralNetwork.new(1)
if not script.LoadModel.Value then
Actor:setModelParametersInitializationMode("LeCunUniform")
Actor:addLayer(9, true, 'ReLU', 0.001)
Actor:addLayer(6, true, 'ReLU', 0.001)
Actor:addLayer(4, false, 'StableSoftmax', 0.001)
Actor:setClassesList(classesList)
else
print('Loading Actor...')
Actor:setModelParameters(ActorModelStorage:GetAsync(script.StorageID.Value))
end
if script.LoadModel.Value==true then
Actor:setModelParameters(ActorModelStorage:GetAsync(script.StorageID.Value))
end
SaveParamsEvent.Event:Connect(function()
local ModelParams = Actor:getModelParameters()
ActorModelStorage:SetAsync(script.StorageID.Value, ModelParams)
end)
return Actor
end
local function buildCritic()
local Critic = DataPredict.Models.NeuralNetwork.new(1)
if not script.LoadModel.Value then
Critic:setModelParametersInitializationMode("LeCunUniform")
Critic:addLayer(9, true, 'ReLU', 0.001)
Critic:addLayer(4, true, 'ReLU', 0.001)
Critic:addLayer(1, false, 'Sigmoid', 0.001)
Critic:setClassesList({1,2,3,4})
else
print('Loading Critic...')
Critic:setModelParameters(CriticModelStorage:GetAsync(script.StorageID.Value))
end
if script.LoadModel.Value==true then
Critic:setModelParameters(CriticModelStorage:GetAsync(script.StorageID.Value))
end
SaveParamsEvent.Event:Connect(function()
local ModelParams = Critic:getModelParameters()
CriticModelStorage:SetAsync(script.StorageID.Value, ModelParams)
end)
return Critic
end
local function buildModel()
local ExperienceReplay = DataPredict.ExperienceReplays.UniformExperienceReplay.new(1, nil, 30)
local NeuralNetwork = DataPredict.Models.NeuralNetwork.new(1)
NeuralNetwork:addLayer(9, true, 'ReLU')
NeuralNetwork:addLayer(6, false, 'ReLU')
NeuralNetwork:addLayer(#classesList, false, 'StableSoftmax')
local Model = DataPredict.Models.DeepDoubleQLearningV2.new(1, 0.6)
local StorageID = script.StorageID.Value
--[[Model:setActorModel(buildActor())
Model:setCriticModel(buildCritic())]]
Model:setModel(NeuralNetwork)
local QLearningNeuralNetworkQuickSetup = DataPredict.Others.ReinforcementLearningQuickSetup.new(60,0,0)
--QLearningNeuralNetworkQuickSetup:setExperienceReplay(ExperienceReplay)
QLearningNeuralNetworkQuickSetup:setModel(Model)
QLearningNeuralNetworkQuickSetup:setClassesList(classesList)
--Model:setPrintReinforcementOutput(false)
return QLearningNeuralNetworkQuickSetup
end
local function buildCar()
local BuiltCar = Car:Clone()
BuiltCar.Parent=workspace
BuiltCar:PivotTo(CityMap.Start.CFrame)
return BuiltCar
end
local function generateEnvironmentFeatureVector()
local featureVector1 = MatrixL:createRandomNormalMatrix(1, 3)
local featureVector2 = MatrixL:createRandomNormalMatrix(1, 3)
local environmentFeatureVector = MatrixL:subtract(featureVector1, featureVector2)
environmentFeatureVector[1][1] = 1 -- 1 at first column for bias.
return environmentFeatureVector
end
local function countTrueBooleansInBooleanArray(booleanArray)
local numberOfTrueBooleans = 0
for i, boolean in ipairs(booleanArray) do
if (boolean == true) then
numberOfTrueBooleans += 1
end
end
return numberOfTrueBooleans
end
local function calculateCurrentAccuracy(booleanArray)
local numberOfBooleans = #booleanArray
local numberOfTrueBooleans = countTrueBooleansInBooleanArray(booleanArray)
local currentAccuracy = (numberOfTrueBooleans / numberOfBooleans) * 100
currentAccuracy = math.floor(currentAccuracy)
return currentAccuracy
end
local function calculateAverage(array)
local sum = 0
local average
for i, number in ipairs(array) do
sum += number
end
average = (sum / #array)
return average
end
local function getCurrentAverageAccuracy()
local currentAccuracy
local currentAverageAccuracy
if (#isRewardedArray > maxRewardArrayLength) then
table.remove(isRewardedArray, 1)
currentAccuracy = calculateCurrentAccuracy(isRewardedArray)
table.insert(currentAccuracyArray, currentAccuracy)
end
if (#currentAccuracyArray > maxCurrentArrayLength) then
table.remove(currentAccuracyArray, 1)
currentAverageAccuracy = calculateAverage(currentAccuracyArray)
end
return currentAverageAccuracy
end
-- Function to calculate the angle between two vectors
local function calculateAngle(vector1, vector2)
local dotProduct = vector1:Dot(vector2)
local magnitudeProduct = vector1.Magnitude * vector2.Magnitude
local cosineOfAngle = dotProduct / magnitudeProduct
local angle = math.acos(cosineOfAngle)
return math.deg(angle) -- Convert from radians to degrees
end
-- Function to calculate the required rotation for the NPC to face the target part
local function getTurnAngle(BasePart, targetPart)
local npcPosition = BasePart.Position
local targetPosition = targetPart.Position
-- Calculate the direction vector from the NPC to the target part
local directionToTarget = (targetPosition - npcPosition).unit
-- Get the NPC's current forward direction (assuming NPC is oriented along the Z-axis)
local npcForward = BasePart.CFrame.LookVector
-- Calculate the angle between the NPC's forward direction and the direction to the target
local angle = calculateAngle(npcForward, directionToTarget)
-- Determine if the target is to the left or right of the NPC
local crossProduct = npcForward:Cross(directionToTarget)
if crossProduct.Y < 0 then
angle = -angle -- If the cross product's Y component is negative, the target is to the left
end
return angle
end
function round(number, decimals)
number = number*10^decimals
number = math.floor(number+0.5)
number = number / 10^decimals
return number
end
local function startEnvironment(Model, CarModel)
Controller=require(CarModel.Controls)
--Model:getModel():clearModelParameters()
Attempts+=1
game.StarterGui.GeneticAlgoDebug.Frame.DebugFrame.Attempts.Text = tostring('Attempt, '..Attempts)
local timeAlive = 0
game.StarterGui.GeneticAlgoDebug.Frame.DebugFrame.TimeAlive.Text = tostring('Time Alive:, '..math.round(timeAlive))
local reward = 0
local defaultReward = 0.01
local defaultPunishment = -1
local predictedLabel
local environmentVector = generateEnvironmentFeatureVector()
local lastRotationErrorY
local currRotationErrorY= getTurnAngle(CarModel.DriveSeat, Goal)
local FLRayHit = workspace:Raycast(CarModel.RayParts.FL.Position, CarModel.RayParts.FL.CFrame.LookVector*1000, raycastParams)
local FRRayHit = workspace:Raycast(CarModel.RayParts.FR.Position, CarModel.RayParts.FR.CFrame.LookVector*1000, raycastParams)
local FRayHit = workspace:Raycast(CarModel.RayParts.F.Position, CarModel.RayParts.F.CFrame.LookVector*1000, raycastParams)
local LRayHit = workspace:Raycast(CarModel.RayParts.L.Position, CarModel.RayParts.L.CFrame.LookVector*1000, raycastParams)
local RiRayHit = workspace:Raycast(CarModel.RayParts.Ri.Position, CarModel.RayParts.Ri.CFrame.LookVector*1000, raycastParams)
local RLRayHit = workspace:Raycast(CarModel.RayParts.RL.Position, CarModel.RayParts.RL.CFrame.LookVector*1000, raycastParams)
local RRRayHit = workspace:Raycast(CarModel.RayParts.RR.Position, CarModel.RayParts.RR.CFrame.LookVector*1000, raycastParams)
local RRayHit = workspace:Raycast(CarModel.RayParts.R.Position, CarModel.RayParts.R.CFrame.LookVector*1000, raycastParams)
local FLDist
local FRDist
local LDist
local RiDist
local FDist
local RLDist
local RRDist
local RDist
local previousPosition
local currentPosition = CarModel:GetPivot().Position
local previousinverseGoalDistance
local inverseGoalDistance
local touched = false
if FLRayHit then FLDist=1/FLRayHit.Distance else FLDist=0 end
if FRRayHit then FRDist=1/FRRayHit.Distance else FRDist=0 end
if FRayHit then FDist=1/FRayHit.Distance else FDist=0 end
if LRayHit then LDist=1/LRayHit.Distance else LDist=0 end
if RiRayHit then RiDist=1/RRayHit.Distance else RiDist=0 end
if RLRayHit then RLDist=1/RLRayHit.Distance else RLDist=0 end
if RRRayHit then RRDist=1/RRRayHit.Distance else RRDist=0 end
if RRayHit then RDist=1/RRayHit.Distance else RDist=0 end
local function getRewardValue()
--[[local distanceAdjustmentFactor = 0.1
local rotationAdjustmentFactor = 0.01
local distanceChangeReward = (lastDistancefromGoal - currDistancefromGoal) --* distanceAdjustmentFactor
print('DistanceReward:', distanceChangeReward)
local rotationChangeReward = 0 --(lastRotationErrorY - currRotationErrorY) --* rotationAdjustmentFactor
print('RotationReward:', rotationChangeReward)
local rewardValue = distanceChangeReward + rotationChangeReward]]
local rewardValue = 0 --(currDistancefromGoal < lastDistancefromGoal) or (currDistancefromGoal < 4 and currDistancefromGoal == lastDistancefromGoal)
return rewardValue
end
local goalReached=0
CarModel.Hitbox.Touched:Connect(function(otherPart)
if otherPart.Name=='Goal' then
print('Goal Reached!')
goalReached=1
elseif not otherPart:IsDescendantOf(CarModel) then
print(otherPart, 'Touched!')
touched=true
end
end)
CarModel.Hitbox.TouchEnded:Connect(function(otherPart)
if otherPart.Name=='Goal' then
print('Goal Overshot!!')
goalReached=0
end
end)
local totalReward = 10
game:GetService('RunService'):BindToRenderStep("DrawGizmos", 1, function ()
Gizmo.PushProperty('Color3', Color3.new(1, 0.160784, 0.176471))
if FLRayHit then Gizmo.Ray:Draw(CarModel.RayParts.FL.Position, FLRayHit.Position) end
if FRRayHit then Gizmo.Ray:Draw(CarModel.RayParts.FR.Position, FRRayHit.Position) end
if FRayHit then Gizmo.Ray:Draw(CarModel.RayParts.F.Position, FRayHit.Position) end
if LRayHit then Gizmo.Ray:Draw(CarModel.RayParts.L.Position, LRayHit.Position) end
if RiRayHit then Gizmo.Ray:Draw(CarModel.RayParts.Ri.Position, RiRayHit.Position) end
if RLRayHit then Gizmo.Ray:Draw(CarModel.RayParts.RL.Position, RLRayHit.Position) end
if RRRayHit then Gizmo.Ray:Draw(CarModel.RayParts.RR.Position, RRRayHit.Position) end
if RRayHit then Gizmo.Ray:Draw(CarModel.RayParts.R.Position, RRayHit.Position) end
Gizmo.PushProperty('Color3', Color3.new(1, 0.576471, 0.152941))
if FLRayHit then Gizmo.Sphere:Draw(CFrame.new(FLRayHit.Position), 1, 20, 360) end
if FRRayHit then Gizmo.Sphere:Draw(CFrame.new(FRRayHit.Position), 1, 20, 360) end
if FRayHit then Gizmo.Sphere:Draw(CFrame.new(FRayHit.Position), 1, 20, 360) end
if LRayHit then Gizmo.Sphere:Draw(CFrame.new(LRayHit.Position), 1, 20, 360) end
if RiRayHit then Gizmo.Sphere:Draw(CFrame.new(RiRayHit.Position), 1, 20, 360) end
if RLRayHit then Gizmo.Sphere:Draw(CFrame.new(RLRayHit.Position), 1, 20, 360) end
if RRRayHit then Gizmo.Sphere:Draw(CFrame.new(RRRayHit.Position), 1, 20, 360) end
if RRayHit then Gizmo.Sphere:Draw(CFrame.new(RRayHit.Position), 1, 20, 360) end
Gizmo.PushProperty('Color3', Color3.new(0.8, 1, 0.14902))
if FLRayHit then Gizmo.Text:Draw(CarModel.RayParts.FL.Position, round(FLDist, 2)) end
if FRRayHit then Gizmo.Text:Draw(CarModel.RayParts.FR.Position, round(FRDist, 2)) end
if FRayHit then Gizmo.Text:Draw(CarModel.RayParts.F.Position, round(FDist, 2)) end
if LRayHit then Gizmo.Text:Draw(CarModel.RayParts.L.Position, round(LDist, 2)) end
if RiRayHit then Gizmo.Text:Draw(CarModel.RayParts.Ri.Position, round(RiDist, 2)) end
if RLRayHit then Gizmo.Text:Draw(CarModel.RayParts.RL.Position, round(RLDist, 2)) end
if RRRayHit then Gizmo.Text:Draw(CarModel.RayParts.RR.Position, round(RRDist, 2)) end
if RRayHit then Gizmo.Text:Draw(CarModel.RayParts.R.Position, round(RDist, 2)) end
Gizmo.PushProperty('Color3', Color3.new(1, 0, 0.0156863))
if RRayHit then Gizmo.Text:Draw(CarModel.DriveSeat.Position, CarModel.DriveSeat.AssemblyLinearVelocity.Magnitude) end
end)
Controller['Ignition'](true)
while task.wait(0.01) do
previousPosition=currentPosition
currentPosition=CarModel:GetPivot().Position
timeAlive+=0.1
game.StarterGui.GeneticAlgoDebug.Frame.DebugFrame.TimeAlive.Text = tostring('Time Alive:, '..math.round(timeAlive))
FLRayHit = workspace:Raycast(CarModel.RayParts.FL.Position, CarModel.RayParts.FL.CFrame.LookVector*1000, raycastParams)
FRRayHit = workspace:Raycast(CarModel.RayParts.FR.Position, CarModel.RayParts.FR.CFrame.LookVector*1000, raycastParams)
FRayHit = workspace:Raycast(CarModel.RayParts.F.Position, CarModel.RayParts.F.CFrame.LookVector*1000, raycastParams)
LRayHit = workspace:Raycast(CarModel.RayParts.L.Position, CarModel.RayParts.L.CFrame.LookVector*1000, raycastParams)
RiRayHit = workspace:Raycast(CarModel.RayParts.Ri.Position, CarModel.RayParts.Ri.CFrame.LookVector*1000, raycastParams)
RLRayHit = workspace:Raycast(CarModel.RayParts.RL.Position, CarModel.RayParts.RL.CFrame.LookVector*1000, raycastParams)
RRRayHit = workspace:Raycast(CarModel.RayParts.RR.Position, CarModel.RayParts.RR.CFrame.LookVector*1000, raycastParams)
RRayHit = workspace:Raycast(CarModel.RayParts.R.Position, CarModel.RayParts.R.CFrame.LookVector*1000, raycastParams)
currRotationErrorY = getTurnAngle(CarModel.DriveSeat, Goal)
local DistancefromGoal = (Goal.Position - CarModel:GetPivot().Position).Magnitude
previousinverseGoalDistance = inverseGoalDistance
inverseGoalDistance = DistancefromGoal>0 and 1/DistancefromGoal or 1
local OldLDist = LDist
local OldRDist = RDist
if FLRayHit then FLDist=FLRayHit.Distance else FLDist=0 end
if FRRayHit then FRDist=FRRayHit.Distance else FRDist=0 end
if FRayHit then FDist=FRayHit.Distance else FDist=0 end
if LRayHit then LDist=LRayHit.Distance else LDist=0 end
if RiRayHit then RiDist=RiRayHit.Distance else RiDist=0 end
if RLRayHit then RLDist=RLRayHit.Distance else RLDist=0 end
if RRRayHit then RRDist=RRRayHit.Distance else RRDist=0 end
if RRayHit then RDist=RRayHit.Distance else RDist=0 end
--if RDist==OldRDist and CarModel.DriveSeat.Throttle==1 then reward+=0.5 end
if LDist==OldLDist and CarModel.DriveSeat.Throttle==1 and CarModel.DriveSeat.AssemblyLinearVelocity.Magnitude > 5 then reward+=5
elseif LDist==OldLDist and CarModel.DriveSeat.Throttle==1 then reward+=0.5 end
environmentVector = {
{
1,
FLDist, FDist, FRDist, LDist, RiDist,-- RLDist, RDist, RRDist,
currRotationErrorY, goalReached,
math.clamp(CarModel.DriveSeat.ThrottleFloat, -1, 1), math.clamp(CarModel.DriveSeat.SteerFloat, -1, 1)
}
}
print('Review: ', environmentVector, reward)
reward = 0 --getRewardValue(environmentVector, predictedLabel)
if touched then
reward=defaultPunishment
print('Punishment: ', reward)
--Model:getActorModel():reset()
--Model:getCriticModel():reset()
reward+=inverseGoalDistance
elseif (currentPosition-previousPosition).Magnitude > 0 and CarModel.DriveSeat.Throttle==1 then
reward+=inverseGoalDistance
end
if goalReached==1 and CarModel.DriveSeat.Throttle<=0 then
reward+=5
end
totalReward+=reward
predictedLabel, confidence = Model:reinforce(environmentVector, reward)
print('PredictedLabel = ', predictedLabel)
--print('PREDICTED RESULT: ', predictedLabel, 'REWARDED: ', isRewarded)
print('Reward: ', reward, confidence)
Controller[predictedLabel]()
--table.insert(isRewardedArray, isRewarded)
--currentAverageAccuracy = getCurrentAverageAccuracy()
--if (currentAverageAccuracy ~= nil) then print(currentAverageAccuracy) end
if touched then
CarModel:Destroy()
game:GetService('RunService'):UnbindFromRenderStep("DrawGizmos")
break
end
end
startEnvironment(Model, buildCar())
end
local function run()
local QDNModel = buildModel()
local CarModel = buildCar()
startEnvironment(QDNModel, CarModel)
end
run()
Here is the code
Oh wait. This is the problem with your code.
local FLDist
local FRDist
local LDist
local RiDist
local FDist
local RLDist
local RRDist
local RDist
Should have at least set some values if it cant get a ray hit.
Also update the library, I just fixed the experience replays.
Still the same, in the while loop I factored this in already as well