This code is a reference to interact with huggingface API’s hosted Mistral 7B instruct model, this setup works for other models such as Zephyr 7B, and other models with the same recipe.
This required some coding implementations to do correctly and I would like to share the results.
local module = {}
local HttpService = game:GetService("HttpService")
local endpoint = "https://api-inference.huggingface.co/models/mistralai/Mistral-7B-Instruct-v0.3" -- Replace with your actual endpoint
local apiKey = "Bearer " -- Replace with your actual API key
local function format_history(str)
-- find the assistant response in the string
local start = string.find(str, "<|user|>")
local finish = string.len(str)
-- local finish = string.find(str, "</s>", start)
local response = string.sub(str, start + 8, finish)
-- return the response in a code block format
return "" .. response .. ""
end
local function format_response(str)
-- find the last assistant response in the string
local start = string.find(str, "<|assistant|>", nil, true)
if not start then return str end -- If "<|assistant|>" is not found, return nil
-- Find the last occurrence by searching backwards from the end of the string
local last_start = start
while start do
last_start = start
start = string.find(str, "<|assistant|>", start + 1, true)
end
-- Calculate the end of the last "<|assistant|>" tag
local finish = string.len(str)
-- Extract the response after the last "<|assistant|>"
local response = string.sub(str, last_start + 13, finish)
return response
end
function module.query(input, system_message,history)
local system="<|system|>\n "..system_message.. "</s>\n"
-- {
if history==nil then
history=""
else history="<|user|>\n "..history
end
local npcdata={inputs = system..history.."</s>\n<|user|>\n "..input.."</s>\n<|assistant|>\n",
max_new_tokens = 512,
do_sample = true,
temperature = 0.7,
top_k = 50,
top_p = 0.95
}
local response = HttpService:RequestAsync({
Url = endpoint,
Method = "POST",
Headers = {
["Content-Type"] = "application/json",
["Authorization"] = apiKey
},
Body = HttpService:JSONEncode(npcdata),
})
local result=HttpService:JSONDecode(response.Body)
local response=format_response(result[1].generated_text)
local history=format_history(result[1].generated_text)
print(response)
print(history)
return response,history
end
return module
Also have this to share a model that should make setting up this as a chatbot very easy!
This code connects to a players input message, and fires it to the client to display as a chat message.
The code is intentionally very basic so that developers can build upon it
Chatbotsetup.rbxm (4.8 KB)
You can test this function in the command line by some variation of this. In the example I was testing the chathistory implementation.
local mod=require(workspace.ChatbotSetup.Chatbot.ZephyrOrpo:Clone()) mod.query("This is a test, can you remember the question I just asked you?","You are a helpful chatbot.", "Hello, would you like to join me on a adventure?</s>\n<|assistant|>\nHello! I'd love to join you on an adventure! Just tell me where we're going and what we'll be doing, and I'll do my best to help you along the way. Let's get started!")
in this the top is the received message from the LLM and the bottom is the chat history.
Hope that is easy to understand have a great day!
Will be hooking this up to procedurally generated NPCS wih its own identity, awareness, RAG.
You can check out the project here to see what is possible with AI on ROBLOX!
Some other Code I documented working with several other AI models such as Google Gemma, Zero-shot classification, summarization, and more can be found in the link below.
Chatbot & LLM Artificial Intelligence Model API Code Documentation FREE (Open Source) and Other Useful APIs - Resources / Community Resources - Developer Forum | Roblox
On a side note I’m still trying to figure out how to get it to use tool calls since it is advertised to be able to do so! Hopefully an update on that will be coming soon.