Server changes causing problems with client inventory

At the start of the game, I go through all the slots and call this function below. This basically handles all slots on startup and listens for any changes made on the server. The main point to look at is the ChildAdded and ChildRemoved for the SlotInfo. I use these to check if these elements have been moved. However, I have a dragging system which is causing problems.

function SlotCreator.Setup(slot, folder, line)
	-- Get the specific data folder
	local Folder = Data:FindFirstChild(folder)
	if not Folder then return end
	
	local SlotInfo
	
	if folder == "Backpack" then
		SlotInfo = Folder[line]:FindFirstChild(slot.Name)
		if not SlotInfo then return end
	else
		-- Hotbar
		SlotInfo = Folder:FindFirstChild(slot.Name)
		if not SlotInfo then return end
	end
	
	if #SlotInfo:GetChildren() > 0 then
		-- Item equipped in this slot
		FillSlot(slot, SlotInfo)
	else
		-- Nothing equipped in this slot
		EmptySlot(slot)
	end
	
	-- Check for slot being filled / emptied
	SlotInfo.ChildAdded:Connect(function()
		-- Slot has been filled with an item
		FillSlot(slot, SlotInfo)
	end)
	
	SlotInfo.ChildRemoved:Connect(function()
		-- Clear slot (it's been emptied)
		EmptySlot(slot)
	end)
end

When dragging, I swap the frames over on the client, then tell the server that they’ve been swapped and where to, so changes can be made to the players data.

local OriginalFrame = container.Parent
-- Move the frames on the client
container.Parent = frame.Parent
frame.Parent = OriginalFrame
					
-- Get parent and positions
local NewParent, OldParent = container.Parent.Parent.Name, OriginalFrame.Parent.Name
local NewPosition, OldPosition = container.Parent.Name, frame.Parent.Name
					
-- Tell server item has moved
MoveItem:FireServer(NewParent, NewPosition, OldParent, OldPosition)

The server then swaps the children around. However, this then causes the ChildRemoved/ChildAdded functions to be fired, which then causes a mountain of problems, where one of the frames gets ‘cleared’

local function ItemMoved(player, newParent, newPosition, oldParent, oldPosition)
	local Data = player:FindFirstChild("Data")
	if not Data then return end
	
	local Backpack = Data:FindFirstChild("Backpack")
	if not Backpack then return end
	
	local Hotbar = Data:FindFirstChild("Hotbar")
	if not Hotbar then return end
	
	local NewParentLocation = newParent == "Hotbar" and Hotbar[newPosition] or Backpack[newParent][newPosition]
	local OldParentLocation = oldParent == "Hotbar" and Hotbar[oldPosition] or Backpack[oldParent][oldPosition]
	
	-- Get the stuff from the old parent and new parent
	local OldParentData = OldParentLocation:GetChildren()
	local NewParentData = NewParentLocation:GetChildren()

	for _, v in pairs(OldParentData) do
		v.Parent = NewParentLocation
	end

	for _, v in pairs(NewParentData) do
		v.Parent = OldParentLocation
	end
end

Example of the problem


IGNORE THE ERROR!! The error message isn’t the problem here. It’s that ChildRemoved is being called, and so it’s clearing the frame.

If I comment out the MoveItem:FireServer() line, then it works perfectly. However, I need the server to know of these changes.