(Text Generation) JSON_SCHEMA Response Type fails to force JSON compliance

When using the JSON_SCHEMA ResponseFormatType with TextGenerator, it sometimes returns plain text instead of valid JSON. This occurs inconsistently across different prompts depending on whether the AI is able to produce a valid response or not.

There is a workaround where you can add a strict rule to the SystemPrompt like You must return JSON format as requested which seems to increase the success rate, but this should not be necessary if JSON_SCHEMA is specified especially since this still is not guaranteed to force JSON compliance.

Minimal repro code:

local HttpService = game:GetService("HttpService")

local JSONFormat = {
	name = "Item",
	description = "Generated item data",
	schema = {
		type = "object",
		properties = {
			name = {
				type = "string",
				description = "Short in-game display name of the item"
			},
			description = {
				type = "string",
				description = "Clear description of the object based on its name"
			}
		},
		required = {"name", "description"}
	}
}

local JSONSystemPrompt = [[
<Examples>
User: a dog
Response: {
	"name": "Dog",
	"description": "four-legged mammal shape with golden fur, rounded ears, and a stubby tail"
}
</Examples>
]]

local function CleanPrompt(text: string): string
	local cleaned = string.gsub(text, "Content generated by artificial intelligence, may make mistakes%.?", "")
	return string.match(cleaned, "^%s*(.-)%s*$") or ""
end

local function GenerateJSONPrompt(prompt: string)
	local TextGenerator = Instance.new("TextGenerator")
	TextGenerator.Parent = workspace
	TextGenerator.SystemPrompt = JSONSystemPrompt

	local success, response = pcall(function()
		return TextGenerator:GenerateTextAsync({
			UserPrompt = prompt,
			ResponseFormatType = "JSON_SCHEMA",
			JsonSchemaStr = HttpService:JSONEncode(JSONFormat)
		})
	end)

	TextGenerator:Destroy()

	if success then
		local cleaned = CleanPrompt(response.GeneratedText)
		print("Prompt:", prompt)
		print("Result:", cleaned)
		print("---")
	else
		warn(success, response)
	end
end

local phrases = {"hi there", "ice cream sandwich", "soda cup", "twenty one pilots"}

for _, phrase in phrases do
	GenerateJSONPrompt(phrase)
	task.wait(1)
end

Example outputs:

Prompt: hi there
Result:
Hi! How are you doing today? :blush:

Prompt: ice cream sandwich
Result:
{
“name”: “Ice Cream Sandwich”,
“description”: “a dessert consisting of two thin slices of cookie or wafer with a layer of ice cream in between”
}

Prompt: soda cup
Result:
{
“name”: “Soda Cup”,
“description”: “clear plastic cup with a circular base, narrow neck, and a metal pull-tab lid, typically containing carbonated liquid”
}

Prompt: twenty one pilots
Result:
Twenty One Pilots is an American rock band formed in 2009, consisting of Josh Dun and Tyler Joseph. They are known for their blend of alternative rock, pop, and hip-hop influences, as well as their dynamic live performances and use of visual storytelling through music videos and social media. Their songs often explore themes of mental health, identity, and personal growth.

Visual aid:

Expected behavior

JSON_SCHEMA should always force JSON compliance OR throw an error if it is unable to provide JSON instead of giving plaintext

A private message is associated with this bug report

5 Likes