How to read a list with 1 Million Objects without lag

Help and Feedback Scripting Support
1

Recently I’ve been making a chunk / render system, it’s pretty basic, but I’ve ran into a problem.
When over 100k items in a list it begins to lag.

Any ideas on how to read lists quicker with less lag?

This what I used to read the lists…

while true do
wait(0.1)
StarFolder:ClearAllChildren()
local int = 0

for i = 1,#Positions do
	
	local starPosition = Positions[i]

	local Mag = (starPosition - Camera.CFrame.Position).Magnitude
	if Mag < RenderDistance then
		int += 1
		local Star = GenFolder.Star:Clone()
		Star.Position = starPosition
		Star.Parent = StarFolder
		Star.Color = Colors[i]
		Star.Size = Vector3.new(Sizes[i],Sizes[i],Sizes[i])
	end
	
	
	
end

end

1 Like
2

There’s a couple things you can do here. First, would be to not recreate every instance each loop, just reuse them instead, and use chunks so you’re not scanning through the whole list every frame.

1 Like
3

aight thanks a lot, didn’t expect a response so quick.

1 Like
4

I agree with Pow.
To give a little more info on how you could identify this yourself in the future, I wrote a little explanation.

You are doing N iterations roughly every 0.1 seconds, where N is the number of items. We’ll say that the data setting step is k cost, and the cloning part is c cost. This gives your algorithm what we’ll say is N * (k+c) cost.

To make your program faster, you need to target those three variables. His suggestion on chunking targets N by reducing the number of objects you need to consider. Not recreating every instance each loop eliminates c. You don’t run the creation step at all each loop. You definitely want to do that.

Reducing k is more up to you. There are ways I could see you using instance streaming here, or you could just keep a certain number of models on hand already loaded (and just set position as needed, only using about 1/4 your current k).

Try thinking about the way you want to unload stars out of a certain range next.

EDIT:

Do factor in the number of repetitions too, totally slipped my mind. Your user isn’t traveling through the render distance every 0.1 secs, so you realistically don’t need to check that often. Thats a multiplier on top of N(k+c). I recommend checking once every second instead.

2 Likes
5

Instead of waiting 0.1 seconds every single time, you could always make it so that it waits 0.1 seconds every 100 or 1000 loops

6

Ok so here’s a newer version of the script.

It go over the objects already loaded, and unloads ones out of range.
It also waits 0.1 seconds every section of items .

while true do

local StarsInView = {}

for _, star in pairs(StarFolder:GetChildren()) do
	if star:IsA("BasePart") then
		local Mag = (star.Position - Camera.CFrame.Position).Magnitude
		
		
		if Mag > RenderDistance then
			star:Destroy()
		else
			table.insert(StarsInView,star.Position)
		end
		
		
	end
end

local int = 0

for i = 1,#Positions do
	
	local starPosition = Positions[i]
	
	int += 1

	local Mag = (starPosition - Camera.CFrame.Position).Magnitude
	if Mag < RenderDistance and not table.find(StarsInView,starPosition)then
		int += 1
		local Star = GenFolder.Star:Clone()
		Star.Position = starPosition
		Star.Parent = StarFolder
		Star.Color = Colors[i]
		Star.Size = Vector3.new(Sizes[i],Sizes[i],Sizes[i])
	end
	
	if int == NumberOfStars / 50 then
		int = 0
		wait(0.1)
	end
	
end

end

7

Hey, if you don’t find the best solution for your usage within a day, I’ll be attempting to write one in the back for fun because this problem is very interesting to me :slight_smile:

8

aight cool, dev forum is surprisingly helpful ngl. :smiley:

9

There are a couple optimizations you can do. Considering youre reading and writing from so many tables at once, something basic you can do is just simply for loop through the Positions table. No need to make a separate loop that indexes into the table.
Basically this:

for i, starPosition in pairs(Positions) do
-- Run your code from here.
end

It’ll provide a decent uplift.

Secondly, as other people mentioned here try not to keep cloning these instances over and over. Cache some of the older ones and re-use them for later. This will save you tons of resources.

Thirdly, a simple hack you can do is to not keep re-reading a million times from the Camera. Interactions with the DataModel are slow. So why not reference the camera cframe position only ONCE outside of the for loop? This should provide good numbers.

Fourthly, When setting the sizes, youre re-reading the same table three times for the exact same value. Simply make the Sizes[i] into a singular local variable and use that instead for the three vector axis.

And lastly, im not sure how youve done this whole little chunk system of yours. But i wonder if you can utilize an octree system to more efficiently pick the right chunks to generate these instances in rather than to just loop through every chunk by default.

1 Like