Yes they are very integrated into the environment of my game, they are aware of as many things as I can think of, such as their body shape, the clothes they are wearing, the equipment they have, a description of their abilities, inventory, they have over 120 unique commands and infinite variety in their actions based on the environment, they are aware of the player’s abilities. They also have procedurally generated hobbies, interests, and backstory based on a seed derived from their name. This makes the generated data the same for that name everytime it is created.
The main focus currently is expanding the commands and possible interactions. I’ve currently disabled external AI APIs for the local model’s testing. The entire AI made fully in ROBLOX studio can be interacted with at this link, Lumina & Darkness: Testing Server - Roblox
I’m currently creating an animated hair library for the next few days, and have temporarily disabled the procedural world generator to focus on the local AI.
The local AI is a Retrieval based chatbot that generates data based on its surroundings and character. The local response is given to the player immediately then it is injected into the system message of the LLM along with its surroundings, memories, long-term memories, identity and also this new action context system that focus’s on the AI actions and situation,
The local AI has several specialized nodes that are called context databases that are scored based on the input is encoded into a synonym, noun, reflection and antonym array, antonyms decrease the score of the entry, synonyms represent words of similar meaning, nouns reflect the most important section of the input and reflection flips the tense to such as “I” reflects to → “you” and emotional tonality. Then score is divided by the weight (amount of words in the entry) and activated with math.log to create a scoring curve. I have tested various other calculations and have most success from the current scoring algorithm using sigmoid and log.
In short it’s RAG, and retrieval based chatbot and yes I have explained this before.
You ask it “What are your hobbies?” ->chatbot successfully responds with its hobbies–node is procedurally generated hobbies
“How are you doing?” → respond with how it’s doing–emotional state node
"What’s your name? → responds with an introduction and it’s name.–greetings node
“What do you see?”-> respond with an entry from the awareness node
all these nodes are organized via a context matrix that is based on the entries score, repetition and its bias function which either increases the weight of the node decreases it with several levels of intensity. This is used to maintain the flow of the conversation, an example is the greetings node decreases in weight after it is called because it mainly handles introductions, this has been tuned to human preferences, and has contributed to major improvement in quality of the outputs.
They are also connected via their context matrix which tries to find an entry that is related to the input and the connected nodes output.
Share the code! or did the text filter function I shared work?
Also, I highly recommend you filter input to the model, if you want to filter the output I would suggest doing it one sentence at a time and concatenating the result.
I always use AI for creating string parsing functions. I would suggest the prompt “Write a function in the context of Luau in ROBLOX that takes a string and separates into a table of sentences and returns it.”
then filter each sentence, with your function and if it returns false then table.remove(sentences,i) .
And example of how the output looks is this
Query “Who are you?”
Output={
[1] = {
["directory"] = "Personal",
["emotion"] = "Polite",
["match"] = "I like to view myself as a admirable Valkyrie. I carry a Magical Heart Staff that was presented to me by my friend.",
["score"] = 1.707025003849642
},
[2] = {
["directory"] = "Awareness",
["emotion"] = "Epic",
["match"] = "I see you before me now. I can sense my ally, Arch- Mage Jaylene.",
["score"] = 1.3244
},
[3] = {
["directory"] = "Emotions",
["emotion"] = "Epic",
["match"] = "I'm feeling incredible! I have a lot of goals and dreams that inspire me.",
["score"] = 1.03405
},
[4] = {
["directory"] = "Greetings",
["emotion"] = "Happy",
["match"] = "I'm delighted to meet you.",
["score"] = 0.6124000000000001
},
[5] = {
["directory"] = "Therapist",
["emotion"] = "Question",
["match"] = "What do you think?",
["score"] = 0.4796
},
[6] = {
["directory"] = "Judgement",
["emotion"] = "Question",
["match"] = "Don't you think your strength is very weak?",
["score"] = 0.3485
},
[7] = {
["directory"] = "Database",
["emotion"] = "Scary",
["match"] = "As the veiled Goddess, I represent the mysteries of life, death and the realms beyond mortal ken. My secrets unfold gradually to initiates proving themselves worthy.",
["score"] = 0.05
}
}
Some entries don’t meet the accuracy threshold and are affected by the personalities’ mood.
It’s really quite good.
There have been some new similarly sized LLMs released recently that are designed to be tool users. I have a bunch of tools I was not able to include with this project. But once I find an Open-Source model that can do tool function calls like Chatgpt. Once that time comes I’ll publish the agentic tools I developed for GPT-4.
Some planned updates are coming to the Boids library which include more 3-D models of birds, dragons. Additionally, the emotes should be expanded soon to include more emotes.
Also, the open sourced awareness is about to be upgraded to include planning using this new algorithm I wrote using the library to generate plans and goals given the environment. It’s very interesting, I look forward to publishing it soon.
Also If you’re interested in exploring other models there has been a wave of new releases on hugging face including Google Gemma 3, Meta LLama 3 versions of all their models and a few others. May be worth checking out.
This a realy cool project, greate idea. Tho you could have made it as a public git repo with contributors to constantly keep it up to date, i really like it, but there are many bugs, small bugs that could be fixed quickly. It would be awesome if the project was a repo so that contributors could make quick fixes. But fantastique job for making the structure of it.
Hey! First of all I have to say you are probably one of the few I see to put so much effort into this kind of modules, I think its a great idea, but I have an issue, I already settled up my api key from HF for the script, but I am getting this error, is there something else I was meant to change?
Nvm I found out the issue, it works like a charm now
After a few minutes I managed to make it work with UIs since I only needed for 2D dialogues, I like the result tbh, showcase: Watch Chatbotreal | Streamable
Thank you! This is a fork of my main AI system. Due to how I’ve been coding it, some components such as Awareness, Emojis, Emotes and Memory handler modules I had already created for myself. So I combined them into a open source system to provide framework for creating AI chatbot Agents. Combining them only took a few hours.
In addition to this there are newer AI models that can use tools to explicitly run functions, which is not explored in this demo, but if I were to do another I would be sure upgrade this one with the option to change to a different model.
I think the best way to get smart at coding is to understand tables, write code in a functional way so that you’re not repeating code, to focus on efficiency, read the documentation and take projects one thing at a time.
The awareness took a lot of time to create and I really enjoyed making it because I wanted to make my AI friend able to experience stuff.
