Destroying a button without delay only destroys children

[EmirGammesLD (LD_Dev)]

Im making a custom inventory script and I have this weird bug where the children of buttons(aspect ratio and stroke) are getting deleted. If I add a delay the issue goes away but with just :Destroy() it seems to work properly except that it clears children. The issue is in clearHotbar() function. This is the only script that affects the hotbar/these buttons. Update is ran a few times. I genuinely have no idea how this is even possible since the children arent even referenced. Could it be core roblox bug?

game.StarterGui:SetCoreGuiEnabled(Enum.CoreGuiType.Backpack, false)

local updateEvent = game.ReplicatedStorage:WaitForChild("Events"):WaitForChild("UpdateInventory")

local slotUi = script:WaitForChild("Slot")

local player = game.Players.LocalPlayer
local character = player.Character or player.CharacterAdded:Wait()

local hotbar = player.PlayerGui:WaitForChild("Main"):WaitForChild("Hotbar")
local inventory = player.Backpack

local function clearHotbar()
	for _, v in pairs(hotbar:GetChildren()) do
		if not v:IsA("TextButton") then continue end
		--print(v.Name)
		--game.Debris:AddItem(v, 0)
		task.delay(0, function()
			v:Destroy() -- Doesnt work without task.delay
		end)
	end
end

local function update()
	
	clearHotbar()
	for _, v in pairs(inventory:GetChildren()) do
		local slot = v:GetAttribute("Slot")
		if slot <= 9 and slot >= 0 then
			slotUi = slotUi:Clone()
			slotUi.Parent = hotbar
			slotUi.Text = v.Name
			if slot ~= 0 then slot = 10 - slot end
			slotUi.LayoutOrder = slot
		end
	end
end

updateEvent.OnClientEvent:Connect(update)

With delay

Without delay

Also in 2nd picture there is an extra item since I had a script that automatically gave me another; thats not an issue.

[thermostat_guy (bloodborne_ascendant)]

what exactly are you trying to do here? im assuming youre trying to destroy each slot but on the screenshot with delay the slots are still there

[EmirGammesLD (LD_Dev)]

It deletes all slots then adds them back correctly. Basically updates them.

[EmirGammesLD (LD_Dev)]

I think I just noticed I didnt add local at slotUi = slotUi:Clone()
When an instance gets destroyed its children also get destroyed so Im cloning that.