i = "<Instance> (HelloWorld)"
local getValue = string.gsub(i, "%(", "")
getValue = string.gsub(getValue, "%)", "")
print(getValue) -- <Instance> HelloWorld
It’s easiest to do this with two operations conceptually. Though I suppose you could do it with another method as well.
Your problem is you are matching against a capture pattern of <Instance> and then replacing that with "" or in otherwords, no charater, which doesn’t affect parenthesis at all.