Setting part.MaterialVariant uses insane amounts of CPU time

When trying to make parts, I need to remove ALL shading from some. I have tried using :clone() but its slow as hell (80ms CPU time) and using Instance.new is faster (70ms CPU time)

I just noticed that when just removing part.MaterialVariant brings the CPU time down to 50ms. Is there a way to get around that?

Just blanket removing it makes it appear wrong

With MaterialVariant:

Without: