When using the JSON_SCHEMA ResponseFormatType with TextGenerator, it sometimes returns plain text instead of valid JSON. This occurs inconsistently across different prompts depending on whether the AI is able to produce a valid response or not.
There is a workaround where you can add a strict rule to the SystemPrompt like You must return JSON format as requested which seems to increase the success rate, but this should not be necessary if JSON_SCHEMA is specified especially since this still is not guaranteed to force JSON compliance.
Minimal repro code:
local HttpService = game:GetService("HttpService")
local JSONFormat = {
name = "Item",
description = "Generated item data",
schema = {
type = "object",
properties = {
name = {
type = "string",
description = "Short in-game display name of the item"
},
description = {
type = "string",
description = "Clear description of the object based on its name"
}
},
required = {"name", "description"}
}
}
local JSONSystemPrompt = [[
<Examples>
User: a dog
Response: {
"name": "Dog",
"description": "four-legged mammal shape with golden fur, rounded ears, and a stubby tail"
}
</Examples>
]]
local function CleanPrompt(text: string): string
local cleaned = string.gsub(text, "Content generated by artificial intelligence, may make mistakes%.?", "")
return string.match(cleaned, "^%s*(.-)%s*$") or ""
end
local function GenerateJSONPrompt(prompt: string)
local TextGenerator = Instance.new("TextGenerator")
TextGenerator.Parent = workspace
TextGenerator.SystemPrompt = JSONSystemPrompt
local success, response = pcall(function()
return TextGenerator:GenerateTextAsync({
UserPrompt = prompt,
ResponseFormatType = "JSON_SCHEMA",
JsonSchemaStr = HttpService:JSONEncode(JSONFormat)
})
end)
TextGenerator:Destroy()
if success then
local cleaned = CleanPrompt(response.GeneratedText)
print("Prompt:", prompt)
print("Result:", cleaned)
print("---")
else
warn(success, response)
end
end
local phrases = {"hi there", "ice cream sandwich", "soda cup", "twenty one pilots"}
for _, phrase in phrases do
GenerateJSONPrompt(phrase)
task.wait(1)
end
Example outputs:
Prompt: hi there
Result:
Hi! How are you doing today? ![]()
Prompt: ice cream sandwich
Result:
{
“name”: “Ice Cream Sandwich”,
“description”: “a dessert consisting of two thin slices of cookie or wafer with a layer of ice cream in between”
}
Prompt: soda cup
Result:
{
“name”: “Soda Cup”,
“description”: “clear plastic cup with a circular base, narrow neck, and a metal pull-tab lid, typically containing carbonated liquid”
}
Prompt: twenty one pilots
Result:
Twenty One Pilots is an American rock band formed in 2009, consisting of Josh Dun and Tyler Joseph. They are known for their blend of alternative rock, pop, and hip-hop influences, as well as their dynamic live performances and use of visual storytelling through music videos and social media. Their songs often explore themes of mental health, identity, and personal growth.
Visual aid:
Expected behavior
JSON_SCHEMA should always force JSON compliance OR throw an error if it is unable to provide JSON instead of giving plaintext
A private message is associated with this bug report
