How to Properly Draw Frames of Animation to EditableImage

[erqilz]

1. What do you want to achieve?
   I’m trying to render an animation to an EditableImage, with a certain delay after every frame that can vary from frame to frame.

2. What is the issue?
   The animation will sometimes render too slowly (likely due to the time it takes to render to the EditableImage taking time in itself). You see, I’m task.wait()ing for that set delay after every time I draw, but it doesn’t account for that EditableImage draw time.

3. What solutions have you tried so far?
   I tried tracking the time before I render the frame and the time after rendering the frame and getting the difference, then subtracting that from the time I task.wait(). Unfortunately, this still doesn’t work, going too slowly for some animations. To test, I task.wait()ed for a very short time like 0.00001 seconds and the animation rendered at a speed almost true to the original, so I know it is possible for the animation to render fast. A completely different thing I’ve tried is making a separate EditableImage for each frame and then switching them under the Decal, but for some reason that doesn’t draw many of the pixels as it animates.

Here’s my current solution that goes too slow for some animations with short delays:

local editableImage = (editable image)
local pixels = {}
local numFrames = (number of frames)
local frameIndex = 1

while true do
	local delay = howLongDelayIsForThisFrameInSecondsFunction(frameIndex)
	
	local start = os.clock()     -- track beginning time?
    
    -- both of the below functions probably take a decent amount of time
	putPixelsOntoPixelsArray(frameIndex, pixels)
	editableImage:WritePixels(Vector2.zero, editableImage.Size, pixels)

    local difference = os.clock() - start     -- should get the time lost when doing the above operations
	
	frameIndex = frameIndex % numFrames + 1
	
    task.wait(delay - difference)     -- theoretically, subtracting the above lost time should compensate for it, but it doesn't. 
    -- Fast playing animations (as in, with short delays) are still rendered/drawn slow.
    -- Remember, I know it's possible for them to play at the right speed, but I need a way to do delay after each frame that will work for all animations
end

[theseformasentence (kaziel)]

task.wait() is limited by the clients framerate, so the minimum delay for someone running 60 fps is 1/60th. On the server, the minimum delay is the same as the heartbeat delay (1/60).
Calling task.wait therefore leads to inconsistencies, like if a player is running at 22 fps, and your animation is 60 fps, you animation will play around 3x slower using task.wait.

Instead, if we increment some time variable by the frame delay (DeltaTime), then just round that number, we can use that to index the table of each frame, or better, benchmark the start time.

This is a basic function that takes your editableimage, animation-name (has to be unique), frame-rate, and animation-frames, then runs through each frame (accounting for the appropriate time), and stops at the end, obviously if there aren’t any errors

local RS = game:GetService([[RunService]])

local ActiveAnims = 0
--I’m not sure if DebugId can be retrieved in a non-studio application, so if it cant
--You’ll need a new solution for unique names of parts

local function InterpolateAnimationOnEditableImage(editableImage: EditableImage, frameRate: number, animationFrames: {})
    local startTime = os.clock()
    local name = editableimage:GetDebugId()
    ActiveAnims += 1
    RS:BindToRenderStep(name, 305+ActiveAnims, function(deltaTime)
        local elapsedTime = os.clock() - startTime
        local frameIndex = elapsedTime * frameRate
        local frame = animationFrames[math.floor(frameIndex)]
        if frame == nil and math.ceil(frameIndex) >= #animationFrames then RS:UnbindFromRenderStep(name) ActiveAnims -= 1 return end
        EditableImage:WritePixels(Vector3.zero, EditableImage.Size, frame) 
    end)
end

I would highly recommend against using high resolution frames using this system, since this is all ran serial rather than parralel. Running hundreds of animations at the same time is bound to lag your clients.

[erqilz]

Thanks for this response, very interesting. I have two questions:

1. Why did you choose the number 305 for priority?
2. The frame rate/delay before next frame can differ from frame to frame, so like it shows in the post, I need to use a function with the frame’s index as an argument to get the amount to delay. The problem is, I don’t think I can use the frameIndex to pass into the function to get the frameRate when the frameRate needs the frameIndex. What I mean is, frameIndex is initialized with frameRate, but frameRate needs to be intialized with frameIndex. How would I go about making this work?

[theseformasentence (kaziel)]

1. 305 is just an arbitrary number. A lot of processes don’t continue for a while past 300 and many process are free, its mainly occupied by camera work.
2. Framerate can be calculated by dividing 1 by the interval, returning the frames per second (1 / 0.016 ~~ 60, equivalent to 1 / (1/60)). If you want to just get the interval, deltaTime provided in ::BindToRenderStep gives you that interval.
   It is really difficult to make a wait function for a highly specific interval like 0.00392, and doing so absorbs most of the computers power (freezing other stuff in the process). To my knowledge, multi-threading will still pause the CPU / GPU if any of the cores take longer than the standard interval as we don’t have explicit control over info sent to the CPU and GPU, but even so nothing will be visible until the next frame finishes rendering.

[erqilz]

Here’s my code:

local pixels = {}
local numFrames = numFrames()
local frameRate = 1 / (frameInfo(1).delay / 100)
local startTime = os.clock()
RunService:BindToRenderStep("Test", 305, function(deltaTime)
	local elapsed = os.clock() - startTime
	local frameIndex = elapsed * frameRate
	local ceiledIndex = math.ceil(frameIndex)
	frameRate = 1 / (frameInfo(ceiledIndex).delay / 100)
	putPixelsOntoArray(ceiled, pixels)
	editableImage:WritePixels(Vector2.zero, editableImage.Size, pixels)
	
	if ceiledIndex == numFrames then
		startTime = os.clock()
	end
end)

(For your information, I’m dividing the delay by 100 because I get it in hundredths of a second, for example if the delay is 0.02 seconds I’ll get 2 from the function, so I have to divide by 100 to change it to 0.02)

However, I noticed 3 things:

1. The animation starts at a high frame. It never starts at frame 1 like it should. It usually starts around 22 or 23, then once it goes to the next iteration of the animation loop it starts back from 1 like it should.
2. In the first complete iteration of the animation loop (as in from the start frame to the last in the animation) the frame number goes up by 2s, but every iteration after that the frame number goes up by 1s, which is correct I think. What causes this?
3. Doesn’t seem to be a problem, but I noticed every number with 3 at the end comes twice (so the frame is drawn twice) which I don’t know if this should happen. By that I mean numbers like 3, 13 and 23.
4. Just a question, what is deltaTime for here? Should I use it?

Also, could you please tell me what I should change in my code or if there’s any other/better ways to animate an EditableImage?

I’m going to mark your previous response as answer, but I’ve only tested this on one animation so far and the speed seems about right from what I remember but I’m not sure.

[theseformasentence (kaziel)]

1. The index used could possibly start at a negative index, making it loop back to one of the ending frames. You would have to check for how it starts at the end, which im assuming it starts where you index the data. Its possible that due to number rounding, startTime is larger than os.clock() for the first frame, and math.ceil would round it to -1, you could switch to math.floor and add one to the index, so it isnt trying to index -1 or 0.
2. Not enough information.
3. This is probably because the elapsedTime still remains on the same index. Drawing the image twice is counterproductive, so you could store the current index and exit the function early, so you can wait for the next frame. 13 and 23 might just be coincidences.
4. deltaTime is just the interval between the current and last frame (not the interval to the next frame though)

A possible hole is that if you did change the frameRate mid animation, the new frameIndex would be lower or higher than the last frame.

Lets say the first frameRate is 24, and at our 7th index, we switch it to 15. Our elapsedTime remains relatively the same, but our frameIndex has changed from 7 to 4. Instead of using a delay to control the distance between frames, it would make more sense to just use the time based on some framerate, like “Draw A 0.5 seconds after start, and Draw B 1.5 seconds after start” rather then “Draw A in 0.5 seconds, then Draw B 1 second later.

This is how standard animation is handled, it uses keyframes that are scheduled, but doesn’t have to happen every frame, only change when the keyframes are changed.

This is what I would suggest:

local currentIndex = 1
local startTime = os.clock()

function getFrameIndex(time, frames)
    local newIndex = nil
    for index,frame in frames do
        if frame.Time <= time then newFrame = index end
    end
    return newIndex
end

function onAnimationFinish()
    --From here you can set startTime to os.clock() to loop, or unbind the animation
end

function animate(editableimage, frames)
    startTime = os.clock()
    RS:BindToRenderStep(“Test”,305,function(deltaTime)
        local elapsedTime = os.clock() - startTime
        local newIndex = getFrameIndex(elapsedTime, frames)
        if newIndex ~= currentIndex then
            currentIndex = newIndex
            editableimage:WritePixels(Vector2.zero,editableimage.Size,frames[currentIndex].Value)
        elseif newIndex == #frames then
            onAnimationFinish()
        end 
    end)
end

Correction: I said at some point you could use the time based on the framerate, but if your frame data just looks likes this:

local frames = {
[1]={Time=0,Value={1,1,1,0}
[2]={Time=1.5,Value={1,0,0,0}}
}

you wouldn’t need a framerate feature since it ultimately depends on your monitors refresh rate. If you made them equal distance, it would make it look like it is at a certain framerate. the Value index is just the pixel data.

[erqilz]

Unfortunately, I realized nothing works. This method you’re giving me right now is painfully slow and that’s not even how I store delays so I had to do a bunch of weird stuff to make it work that way. Your first method doesn’t work for varying frame delays. I tried to make a simple as possible approach, but this ends up having the same issue as what I made this post for in the first place:

local frameIndex = 1

local timeSinceLastFrame = os.clock()
RunService:BindToRenderStep("test", 300, function(deltaTime)
	local elapsed = os.clock() - timeSinceLastFrame
	local delayTime = decoder:frameInfo(frameIndex).delay
	if elapsed >= delayTime then
		decoder:decodeAndBlitFrameRGBA(frameIndex, pixels)
		editableImage:WritePixels(Vector2.zero, editableImage.Size, pixels)
			
		frameIndex += 1

		if frameIndex > numFrames then
			frameIndex = 1
		end
		
		timeSinceLastFrame = os.clock()
	end
end)

I even tried accumulating deltaTime in a variable and resetting it on frame change but that’s really slow for some reason too, though I thought it’d be faster. I’m really confused as to what to do at this point, it’s been hours.

[erqilz]

I found a solution. Thank you so much @5uphi. Here it is:

local totalTime = 0

RunService.Heartbeat:Connect(function(deltaTime)
	totalTime += deltaTime
	
	local delayTime = frameInfo(frameIndex).delay
	
	while totalTime >= delayTime do
		totalTime -= delayTime
		
		decodeAndBlitFrameRGBA(frameIndex, pixels)
		editableImage:WritePixels(Vector2.zero, editableImage.Size, pixels)
		
		frameIndex = frameIndex % numFrames + 1
	end
end)

I wonder why Heartbeat is used and not RenderStepped. Anyways, thanks for your help. You telling me that task.wait is dependent on frame rate is what made me find @5uphi’s amazing video. I’m marking your first reply as answer. But @5uphi, if you see this, thank you so much.