Rate Limited Webhook Util for Discord (Not a proxy or a workaround)

Just a quick notice: This will not get your webhooks working. Rather, it will greatly help prevent future blocks from Discord, and greatly improve your webhook speeds by preventing Discord from issuing extra penalty limits on you for repeated request errors.

Rate Limited Webhook API for Discord

Hello! :smile:
I wanted to write this quick resource here to encourage the use of proper rate limits for Discord Webhooks. It’s implemented in a future-proof way and supports various fallbacks and rate limiting methods to make it as effective as possible.

Discord issues additionally penalty limits if you don’t follow their rate limits properly, so, once the block is lifted this will ensure you’re following their requested rate limits, even if they change in the future.

Download

Source code
-- Services
local HttpService = game:GetService("HttpService")

-- Module
local DiscordWebhook = {
	-- Rate limit states
	State = {
		Default = {
			Limit = 1,
			Remaining = 1,
			ResetTime = os.time(),
			ResetDelay = 1
		}
	},
	Buckets = {},
	CleanupTime = os.time(),
	
	Discarded = {} -- Discarded Webhook URLs (404s)
}

-- Default state metatable
DiscordWebhook.DefaultState = {__index = DiscordWebhook.State.Default}

-- How many seconds buckets should be kept in memory for before being cleared (Keep 30-60 seconds, super low values are *not* better)
local BUCKET_LIFETIME = 30

-- Crude domain check to make sure you're not accidentally requesting non discord URLs
function DiscordWebhook:CheckDomain(url)
	return string.find(url, "discord.com", nil, true) or not string.find(url, "discordapp.com", nil, true)
end

-- Cleans up rate limit buckets
function DiscordWebhook:CleanupBuckets()
	-- Return if cleaned up too recently
	if os.time() - self.CleanupTime < BUCKET_LIFETIME then
		return
	end
	
	local markedCleanupTime = false
	
	-- Check state expiration and clean up any old states outside the lifetime
	local states = self.State
	for bucketId, state in pairs(states) do
		-- Clear expired buckets
		local expires = state.Expires
		if expires and os.time() > expires then
			states[bucketId] = nil
		end
	end
	
	-- Clear unused buckets
	local buckets = self.Buckets
	for webhookUrl, bucketId in pairs(buckets) do
		-- Clear removed buckets
		if not states[bucketId] then
			buckets[webhookUrl] = nil
		end
	end
	
	self.CleanupTime = os.clock()
end

-- Drop in replacement for HttpService:PostAsync
function DiscordWebhook:PostAsync(webhookUrl, data)
	-- Fall back to real PostAsync for non Discord URLs
	if not self:CheckDomain(webhookUrl) then
		return HttpService:PostAsync(webhookUrl, data)
	end
	
	-- Make the request
	local response = self:RequestAsync({
		Url = webhookUrl,
		Method = "POST",
		Body = data
	})
	return response.Body
end

-- Drop in replacement for HttpService:RequestAsync
function DiscordWebhook:RequestAsync(request)
	if request and type(request) == "table" then
		local webhookUrl = rawget(request, "Url")
		if self:CheckDomain(webhookUrl) then
			-- If the URL was discarded, return the cached response
			local discardedResponse = self.Discarded[webhookUrl]
			if discardedResponse then
				return discardedResponse
			end
			
			-- Make a request via HttpService
			local success, response = pcall(function()
				-- Repeatedly retry until response ready
				while true do
					-- Wait for the lock time if locked
					if self.LockedFor then
						wait(self.LockedFor)
					end

					-- Bucket id
					local bucketId = self.Buckets[webhookUrl]
					if bucketId then
						local curState = self.States[bucketId]
						
						-- If no remaining requests
						if curState.Remaining == 0 then
							-- Wait until the reset time & reset the request count
							local resetTime = curState.ResetTime
							if resetTime then
								wait(os.time() - resetTime)
								-- Reset the request count to the limit
								curState.Remaining = curState.Limit
								-- Increase the reset time by the reset delay
								curState.ResetTime += curState.ResetDelay
							end
						end
						
						-- Subtract a request from the count
						curState.Remaining -= 1
					end
					
					-- Clean up expired buckets
					self:CleanupBuckets()
					
					-- Make a request
					local response = HttpService:RequestAsync(request)
					
					-- Handle various error codes
					if response.StatusCode == 404 then
						-- Discard 404s as per Discord request
						self.Discarded[webhookUrl] = response
					elseif response.StatusCode == 401 or response.StatusCode == 403 then
						-- Discard 401s and 403s since they mean the webhook is useless
						self.Discard[webhookUrl] = response
					elseif response.StatusCode == 429 then
						-- We're being rate limited, make sure to respect Retry-After header
						
						-- Get Retry-After
						local retryAfter = tonumber(response.Headers["Retry-After"])
						
						-- Set lock state
						self.LockedFor = retryAfter
						-- Wait for retry period
						wait(retryAfter)
						-- Clear lock state
						self.LockedFor = nil
						
						-- Do a retry
						continue
					else
						-- Get response headers
						local headers = response.Headers

						-- Get rate limit header info
						local bucketId = headers["X-RateLimit-Bucket"]
						if bucketId then
							local limit = tonumber(headers["X-RateLimit-Limit"])
							local remaining = tonumber(headers["X-RateLimit-Remaining"])
							local resetTime = tonumber(headers["X-RateLimit-Reset"])
							local resetDelay = tonumber(headers["X-RateLimit-Reset-After"])

							-- Get current state or assign new one
							local curState = self.State[bucketId]
							if not curState then
								curState = setmetatable({
									Expires = os.time() + BUCKET_LIFETIME
								}, self.DefaultState)
								self.State[bucketId] = curState
							end

							-- Update state
							curState.Limit = limit
							curState.Remaining = remaining
							curState.ResetTime = resetTime
							curState.ResetDelay = resetDelay

							-- Store bucket id
							self.Buckets[webhookUrl] = bucketId

							-- Perform rate limiting on the current thread
							if remaining == 0 then
								-- Set lock state
								self.LockedFor = resetDelay
								-- Wait for reset delay
								wait(resetDelay)
								-- Clear lock state
								self.LockedFor = nil
							end
						end
					end
					return response
				end
			end)
			
			-- Use a fallback response on failure
			if not success then
				return {
					Success = false,
					StatusCode = 555,
					StatusMessage = "DiscordWebhook fallback error (" .. tostring(response) .. ")",
					Headers = {},
					Body = "{}"
				}
			end
			return response
		end
	end
	
	-- Fall back to real RequestAsync
	return HttpService:RequestAsync(request)
end

return DiscordWebhook

Usage

Using this tool is simple.
Just require it like so, and replace your HttpService:PostAsync or HttpService:RequestAsync usage. (Please note that GetAsync and other HttpService items are not implemented since they aren’t used for webhook posting, just those two are)

local DiscordWebhook = require(path.To.DiscordWebhook)

DiscordWebhook:PostAsync(webhookUrl, webhookData)
-- or
DiscordWebhook:RequestAsync(webhookRequest)

The module will automatically attempt to detect if you accidentally use a non Discord url, and pass requests through normally.

Technical Details

(See https://discord.com/developers/docs/topics/rate-limits)

The module keeps track of each bucket by its ID as specified by Discord. Every bucket can have a unique state and the bucket id is cached per webhook. Buckets and cached webhook urls are discarded after a set amount of time (default is 30 seconds).

All states are updated after each webhook request made to Discord, as all information about the rate limit state is returned in addition to timing info. States are also calculated, so, if a request fails, the state should remain up to date. Previous timing info will be used in that case.

Discord requests that requests returning a 404 code are discarded and requests are not repeated. This module respects this by caching the initial response and returning it on duplicate urls, as well as implementing this behaviour for 401s and 403s.

This module additionally respects the standard error 429 Retry-After header as a fallback to the standard Discord limits, and performs a retry for the request correctly.

20 Likes

Although this is a good tutorial, this doesn’t help with the current situation. No requests sent from Roblox are accepted at all.

This module could be improved by automatically converting the Discord url to a proxy url.

This project is not designed to be a proxy or workaround to the Discord ban, its intention is purely oriented around Discord eventually unblocking requests from Roblox servers.

The point of the project is (in the long run) to discourage the use of proxies to circumvent the ban, I won’t be adding a proxy URL. Proxies additionally do not typically forward the response headers that this module requires for rate limiting.

4 Likes

I see no reason for Discord to unblock requests just from the way it currently is; not everyone is going to use this module, thus rendering it useless. Unblocking Roblox just based off a promise “I swear we won’t break your ToS!” Isn’t going to cut it.
A much more plausible and likely solution would be if Roblox were to add rate limits themselves, for instance by checking which site you are sending requests to.

2 Likes

Discord probably won’t even enable Roblox requests for maybe a few years just due to the amount of people spamming the API

The case also sank when people spammed the legacy api (discordapp.com), so to see this module actually work, it’ll be a miracle

The only way to send data from Roblox to webhooks directly is by proxies now

Although if they ever do stuff like this, and if this module works again, I will implement it into a project of mine

The goal of this isn’t really to go “Discord! Please unblock us! :sob:” the goal is that when and if they do, some people will start actually implementing rate limiting properly. I by no means expect this to get used a lot, but at the very least, it creates a presence for people to be doing rate limiting properly.

I tried to go above and beyond with the module while keeping it extremely simple for people to drop in, because not all contributors to the issue are experienced programmers, so, creating a free, high quality resource that only requires an extremely basic level of experience to use lowers the gap from potentially seeming difficult for some people, to being easily accessible, and importantly, being easily accessible to get the “high quality” version of rate limiting.

I honestly don’t expect any big contributors to use this resource, or to know it exists, but, the more people are aware that proper rate limiting is important, and the more people who can potentially use it, the better. A 1% impact is something in this case, whereas in other cases that might not be true.

This was mentioned by a Roblox staff member in a bug report post on this, and, the likely scenario is that Roblox just replaces the use of this module in the future anyways. However, this is still a physical presence on the devforum for proper Discord rate limiting, and, that’s really one of the important things.

In conclusion, I don’t think this module will just magically solve everything, in fact, I don’t think it will solve anything. But, with the way that the devforum works, it at least creates awareness and lowers the experience level required for people to be following the standards.

2 Likes

If your goal is to create awareness, you should create a community tutorial instead. You yourself stated this resource won’t solve anything, but I’d rather meant to create awareness, and as such a tutorial explaining what rate limiting is would be a better solution instead.

I said it may not.

Regardless, I’ve explained why I made the resource, and you aren’t obligated to agree with my reasoning. But trying to convince me my resource is bad just because you think it is is extremely unhelpful. I’m not going to take action against something I’ve put some genuine effort into just because you disagree and are adamant about it.

I don’t want to continue this argument any further, as its not productive.

1 Like