Mistral 7b V.3 Instruct Inference Luau Code and Example -- 32K Context, Prompt, Server Message, and Chat history (Tools coming soon?)

This code is a reference to interact with huggingface API’s hosted Mistral 7B instruct model, this setup works for other models such as Zephyr 7B, and other models with the same recipe.
This required some coding implementations to do correctly and I would like to share the results.

local module = {}
local HttpService = game:GetService("HttpService")
local endpoint = "https://api-inference.huggingface.co/models/mistralai/Mistral-7B-Instruct-v0.3" -- Replace with your actual endpoint
local apiKey = "Bearer " -- Replace with your actual API key

local function format_history(str)
		-- find the assistant response in the string
		local start = string.find(str, "<|user|>")
		local finish = string.len(str)
		-- local finish = string.find(str, "</s>", start)
		local response = string.sub(str, start + 8, finish)
		-- return the response in a code block format
		return "" .. response .. ""
end

local function format_response(str)
	-- find the last assistant response in the string
	local start = string.find(str, "<|assistant|>", nil, true)
	if not start then return str end  -- If "<|assistant|>" is not found, return nil

	-- Find the last occurrence by searching backwards from the end of the string
	local last_start = start
	while start do
		last_start = start
		start = string.find(str, "<|assistant|>", start + 1, true)
	end

	-- Calculate the end of the last "<|assistant|>" tag
	local finish = string.len(str)

	-- Extract the response after the last "<|assistant|>"
	local response = string.sub(str, last_start + 13, finish)

	return response
end


function module.query(input, system_message,history)
	local system="<|system|>\n "..system_message.. "</s>\n"
	
	--	{
	if history==nil then
		history=""
	else history="<|user|>\n "..history	
	end
	
	local npcdata={inputs = system..history.."</s>\n<|user|>\n "..input.."</s>\n<|assistant|>\n",		
		max_new_tokens = 512,
		do_sample = true,
		temperature = 0.7,
		top_k = 50,
		top_p = 0.95
	}
	
	local response = HttpService:RequestAsync({
		Url = endpoint,
		Method = "POST",
		Headers = {
			["Content-Type"] = "application/json",
			["Authorization"] = apiKey
		},
		Body = HttpService:JSONEncode(npcdata),
	})
	local result=HttpService:JSONDecode(response.Body)
	local response=format_response(result[1].generated_text)
	local history=format_history(result[1].generated_text)
	print(response)
	print(history)
	return response,history
end

return module

Also have this to share a model that should make setting up this as a chatbot very easy!
This code connects to a players input message, and fires it to the client to display as a chat message.
The code is intentionally very basic so that developers can build upon it

Chatbotsetup.rbxm (4.8 KB)

You can test this function in the command line by some variation of this. In the example I was testing the chathistory implementation.

local mod=require(workspace.ChatbotSetup.Chatbot.ZephyrOrpo:Clone()) mod.query("This is a test, can you remember the question I just asked you?","You are a helpful chatbot.", "Hello, would you like to join me on a adventure?</s>\n<|assistant|>\nHello! I'd love to join you on an adventure! Just tell me where we're going and what we'll be doing, and I'll do my best to help you along the way. Let's get started!")

in this the top is the received message from the LLM and the bottom is the chat history.
image

Hope that is easy to understand have a great day!

Will be hooking this up to procedurally generated NPCS wih its own identity, awareness, RAG.
You can check out the project here to see what is possible with AI on ROBLOX!

Some other Code I documented working with several other AI models such as Google Gemma, Zero-shot classification, summarization, and more can be found in the link below.
Chatbot & LLM Artificial Intelligence Model API Code Documentation FREE (Open Source) and Other Useful APIs - Resources / Community Resources - Developer Forum | Roblox

On a side note I’m still trying to figure out how to get it to use tool calls since it is advertised to be able to do so! Hopefully an update on that will be coming soon.

1 Like