Giving LLM Text Vision and Self Awareness in Luau [OPEN SOURCE]

Magus_ArtStudios · August 19, 2023, 7:43am

This module gives Text-Vision to a Large language model. It does not require you to use all the categories but if you do you get the best performance in terms of how it’s designed to work. Should be used in conjunction by setting up your workspace accordingly. It creates a table of entries describing the scene in different ways.
Included in the library are functions which are used widely in the code.
It’s been tested and is very performant
FAQ:

Why does it use categories?
It uses categories to generalize multiple objects in the same location and only observe 3 of each category as well as to generate data about different aspects of the environment for RAG.
Can’t you use Tags?
The code is very easy to modify to tags instead of getchildren use
CollectionService=game:GetService(“CollectionService”)
CollectionService:GetTagged(key)
Do you have a demonstration place?
The link to my new Free Open Sourced Chatbot Demo for all is located here. The Chatbot says it can see.
[FREE] ROBLOX Mistral 7b AI Chatbot Agent: Aware, Infinite Agents, 2000+ Emojis, 100+ Emotes, Memories , Wikipedia, 32k Context [Open Sourced]
How to change the location of the observed objects?

You can easily change the names of the directories where each object type is stored! be overwriting the workspace.MapObjects entry and etc. lowering he refreshrate makes the tables update more often which is a performance optimization.

local tablekeys = {
    ["mapobject"] = {{["datatable"] = nil, ["writetime"] = ticktime.Value, ["refreshrate"] = 30}, workspace.MapObjects},--name of folder

What is the overhead like?
The overhead for performance is pretty good but it you need a specific observation you should just call that function.

Conclusion:
This is great framework to build your game on! All of these categories would keep your game organized and you can build with LLMs on top of it.

New Features include the awareness now includes what the root humanoid is standing on!
It also includes if their is water in the distance. It also produces more observations by combining observations in different ways.

The nearest function now has a query parameter to search for specific objects in the array.
–Updated link
(100) Awareness Text-Vision LLM Utility Luau Code- Roblox

This is a old video of my AI in Action!

I’m currently using this with Zephyr 7b with a experiment to inject the system message with the awareness! It’s been very successful!

New demonstration of This Awareness module being used to Illustrate Candyland to a Large Language model!

Check out my Patreon and get the latest releases for Open-sourced code I create!
Magus Art game developer and resource creator | Patreon

Check out my newest open source project!
obby but you walk on walls (1).rbxl (915.0 KB)

syntaqs · August 19, 2023, 1:45pm

That’s not even a neural network, let alone an AI. It’s just a bunch of if-else statements.

ShortBytes · August 19, 2023, 1:45pm

This is pretty interesting, but I don’t see myself using it anytime soon.

xChris_vC · August 19, 2023, 3:58pm

Self-aware AI means conscious & sentient intelligence. As of now, this ability is only reserved to humans.

This script isn’t self-aware, nor is it AI. It can’t do anything than fulfilling the existing tasks. Want to describe something in the environment that’s not covered by the if-statements (while perceived by the ‘senses’ of the NPC)? This script can’t.

Waffle_pancake123fan · August 19, 2023, 4:00pm

Did you know that monkeys can be self aware they pass the test.

Vanniris · August 19, 2023, 4:05pm

Well, that’s an insane amount of if statements and variable declarations to a point that practically nobody will ever bother to read the code. Can’t judge because I aint reading that pain.

Magus_ArtStudios · August 19, 2023, 4:21pm

To be self aware an AI must be aware of its surroundings. You show me your AI project and how it works then you let me know what you think of this module. Because I think it’s funny how you know nothing yet decide to tout your opinion this is literally the type of string constructing algorithm to describe to a large language model what it sees. I would like to reiterate I do not care for your negative opinions, you are likely clueless and do not really know what you are talking about.

My responses coming off condescending because the only ones who decided to say anything where the negative people and they are not even making good arguments. Research the topic and provide informed input. This is how you would send surroundings information to an AI model. You can ask it to find a point of interest from the surroundings and make decisions based on that. My chatbot has the abilities to perform action commands from strings. So I could interpret the output to make the AI chatbot self aware by giving it the ability to make autonomous decisions based on its environment.

For example perhaps the AI decides to pick up an apple on the ground because it has some arbitrary hunger or thirst stat. The AIs decisions could also be influenced by survival and input from its surroundings. This is not a common implementation of a large language model but it would be very interesting to give the AI human goals. Like what literal universities have published on Automonous agents.This script is a strong concept for other developers as it provides a structured way to construct a description of the chatbots surroundings.
You could use raycasting and project a ray from only the front of the AI if you wish to only get what it would see in front of it.

For example if you were making a AI character who was a reactive NPC that made dialogue. You could send a message to the model describing the surroundings. Then the model can respond to that input of the world with their character like in that youtube video but within the confines of ROBLOX studio.

self-a·ware·ness

[ˌselfəˈwernəs]

NOUN

conscious knowledge of one’s own character, feelings, motives, and desires:

“the process can be painful but it leads to greater self-awareness”

Also, I updated the code I shared to not include the NPC description algorithm because it is too lengthy and tailored to my custom NPC combat algorithm
But I will post it here just because.

function aware.DescribeNPC(Char)
	local MagicID=Char:FindFirstChild("MagicID")
	if MagicID~=nil then
		local fly=""
		local Aggression=""
		local AttackVector=""
		local Barrier=""
		local Buffs=""
		local BaseHealth=""
		local BaseMagicDMG=""
		local BaseRangeDMG=""
		local BaseMeleeDMG=""
		local CanBeFrozen=""
		local ChangeStyles=""
		local Cooldown=""
		local Cost=""
		local CostMax=""
		local Cure=""
		local Description=""
		local Dodge=""
		local Element=""
		local Element2=""
		local Element3=""
		local Hide=""
		local Level=""
		local MainStyle=""
		local Multiply=""
		local Paralyze=""
		local Poison=""
		local Power=""
		local Prayer=""
		local ProtectFromStyle=""
		local Recoil=""
		local Range=""
		local Parasite=""
		local Species=""
		local Teleport=""
		local Transform=""
		local Weakness=""
		local rating=0
		--construct a long combined string of the above local strings.

		if MagicID:GetAttribute("Aggression")==1 then
			Aggression="aggressive, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Fly")==true then
			fly="can Fly, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Barrier")==true then
			Barrier="uses Barrier, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Buffs")==true then
			Buffs="uses Buffs, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Multiply")==true then
			Buffs="uses Multiply, "
			rating=rating+1
		end
		if MagicID:GetAttribute("ChangeStyle")==true then
			ChangeStyles="melee and magic user, "
			rating=rating+1
			BaseMagicDMG="base Magic damage is"..tostring(MagicID:GetAttribute("BaseMagicDMG"))..", "
			BaseMagicDMG="base Melee damage is"..tostring(MagicID:GetAttribute("BaseMeleeDMG"))..", "
			if MagicID:GetAttribute("AttackVector3")~=nil then
				AttackVector="melee range is"..tostring(MagicID:GetAttribute("AttackVector3"))..", "
			end	
		elseif MagicID:GetAttribute("ChangeStyle")==false then
			if	MagicID:GetAttribute("MainStyles")~=nil then
				if MagicID:GetAttribute("MainStyles")==1 then
					MainStyle="uses Melee only, "
					if MagicID:GetAttribute("AttackVector3")~=nil then
						AttackVector="melee range:"..tostring(MagicID:GetAttribute("AttackVector3"))..", "
					end	
				elseif MagicID:GetAttribute("MainStyles")==2 then
					MainStyle="uses Magic only, "
				elseif MagicID:GetAttribute("MainStyles")==3 then
					MainStyle="uses Range only, "
				end
			end
		end	

		if MagicID:GetAttribute("BaseHealth")~=nil then
			local hpr=Char.Humanoid.MaxHealth/100
			rating=rating+hpr
			BaseHealth="base health "..tostring(MagicID:GetAttribute("BaseHealth"))..", "
		end
		--write code to complete above context
		if MagicID:GetAttribute("CanBeFrozen")==false then
			CanBeFrozen="cannot be frozen, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Cooldown")~=nil then
			Cooldown="it's cooldown is "..tostring(MagicID:GetAttribute("Cooldown"))..", "
			rating=rating+(5-MagicID:GetAttribute("Cooldown"))
		end
		if MagicID:GetAttribute("Cost")~=nil then
			Cost="it's action cost is "..tostring(MagicID:GetAttribute("Cost"))..", "
			rating=rating+1
		end
		if MagicID:GetAttribute("CostMax")~=nil then
			CostMax="It's max power is"..tostring(MagicID:GetAttribute("CostMax"))..", "
		end
		if MagicID:GetAttribute("Cure")==true then
			Cure="can Heal, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Description")~=nil then
			Description=MagicID:GetAttribute("Description")..""
		end
		if MagicID:GetAttribute("Dodge")==true then
			Dodge="can Dodge, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Teleport")==true then
			Teleport="can Teleport, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Transform")==true then
			Transform="can Transform, "
			rating=rating+1
		end

		if MagicID:GetAttribute("Element")~=nil and MagicID:GetAttribute("Element")~="" then
			Element="uses "..MagicID:GetAttribute("Element")
			rating=rating+1
			if MagicID:GetAttribute("Element2")~="" and MagicID:GetAttribute("Element2")~=nil then
				Element2=","..MagicID:GetAttribute("Element2")
				rating=rating+1
			end
			if MagicID:GetAttribute("Element3")~="" and MagicID:GetAttribute("Element2")~=nil then
				Element3="and "..MagicID:GetAttribute("Element3")
				rating=rating+1
			end	
		end
		if MagicID:GetAttribute("Parasite")==true then
			Parasite=", it's parasitic"
			rating=rating+1
		end
		if MagicID:GetAttribute("Poison")==true then
			Poison=", it's poisonous"
			rating=rating+1
		end

		if MagicID:GetAttribute("Prayer")==true then
			Prayer="uses Prayers"
			rating=rating+1
		elseif 	MagicID:GetAttribute("ProtectFromStyle")==1 then
			rating=rating+1
			ProtectFromStyle=", strong against Melee"
		elseif		MagicID:GetAttribute("ProtectFromStyle")==2 then
			rating=rating+1
			ProtectFromStyle=", strong against Magic"
		elseif		MagicID:GetAttribute("ProtectFromStyle")==3 then
			rating=rating+1
			ProtectFromStyle=", strong against Range"
		end
		--Power="Power Multiplier:" MagicID:GetAttribute("Power")
		--Range=""
		if MagicID:GetAttribute("Recoil")==true then
			rating=rating+1
			Recoil="hurts to touch, "
		end
		rating=rating+tonumber(Level)
		--write rest of code for above context	
		return Description..Element..Element2..
			Element3.." magic. It's "..Level.." "..ChangeStyles..MainStyle..Cooldown..fly..Aggression..Barrier..Buffs..BaseHealth..
			BaseMagicDMG..BaseRangeDMG..BaseMeleeDMG..AttackVector..CanBeFrozen..
			Cost..CostMax..Cure..Dodge..
			Hide..Multiply..Paralyze..Poison..Power..
			Prayer..ProtectFromStyle..".",aware.judgepower(rating,25)
	else return "",nil	
	end
end

Magus_ArtStudios · August 19, 2023, 5:24pm

This topic is related to my main topic! Which has a module that you can input search queries based on a database.
For example if you are looking for an Apple. You can search a list of strings and return a string describing an Apple.
This is a part of a multi-model AI-chatbot Agent I’m working on. When it’s done I will be releasing some sort of open-source version that has a simple framework to work from for other developers.
I have recently released a new ChatModule which is the most recent version of the main algorithms that powers the bots ability to compare input robustly with a database of strings and provide output. It also can solve basic geometry for example “What is the circumference of a circle with a radius of 4?” and worded algebra problems “If I have 4 groups of 5 how many do I have in total?” output= “4*5=20”.

The way it’s designed is to answer questions with the chatbot using my specific database for my games lore and hints when it can. Then, if it does not know the answer it gets the answer from a variety of APIs including wikipedia, How to wiki, conversational AIs that think they are the input personality, and GPT2. It uses zero-shot classification to determine which model to use. It saved all of the data it recieves from the AIs and APIs for that database or specific character. I’m also incorporating more APIs and premium AI models in the future. It’s important to have levels to things like this. It can be visualized like this Chatbot determines the intent based on rules and algorithms if conditions are met then->zero-shot-classification AI to determine the inputs intent->APICall to AI Model or wiki.
Premium architecture would be just be Chatbot Input if any->APICall to AI with specified character and context with user query
Artificially Intelligent Chatbot LUAU Algorithm [Open Source] - Resources / Community Resources - Developer Forum | Roblox

Judgemental_Man · August 19, 2023, 6:35pm

It’s not “Self Aware”, it’s only capable of what it’s allowed to do, true sentience is impossible, you can only simulate it
You literally said it’s an AI

lovableschoolgirl · August 19, 2023, 6:54pm

I like the idea. However, it is a bit messy.

ClodexTheScripter · August 19, 2023, 7:09pm

Why is everyone mad?
OP did make a self aware AI but not in the way that pleases your definition of what it means to be self aware.
Make this simple for yourselves and avoid explaining your definition of the word AI, as I’m very sure OP meant to call it NPC. It’s just a small miss direction. Honest mistakes can be made.

Magus_ArtStudios · August 19, 2023, 7:12pm

This is part of my AI project. Which includes artificial intelligence models. If you have a AI that has no information about its surroundings then it cannot make relevant observations of its surroundings.
This algorithm creates artificial data to give AI context about its surroundings. If you were involved in working with Language Models this concept is easy to grasp and quite important within the restrictions of ROBLOx’s coding environment.

ClodexTheScripter · August 19, 2023, 7:15pm

I’ll take ur words for it.
I’d like to suggest that it may seem better to add an API example and write a showcase for others to glance at. It may help them avoid looking at your work and discard it like the people above.

Magus_ArtStudios · August 19, 2023, 7:18pm

(post deleted by author)

Vanniris · August 19, 2023, 10:39pm

Self-aware can be defined as a character that can think by itself, interact with its surroundings by itself, react or express emotions. All I see in your code is just what you limited it to do. You cannot train it, it is not self aware, and less an AI.

That aside, this is one of the most unformatted and ridiculous code I’ve seen in all of DevForum. Even worse - it looks like that most of this was written by ChatGPT. The source code is straight up unreadable, let alone anyone being able to do anything with it.

Unnecessary functions, unnecessary variable definitions, insane amount of if statements that could’ve been simplified in other methods. Instead of using functions for actual uses you still went with the copy-paste method. Half of your code has ChatGPT comments, the rest is straight up Ctrl+C Ctrl+V. We can do better, let’s say:

First, there’s zero need to define insane amounts of variables at the start, just use a table for it. The benefit of using tables is that we can also loop through them, so we don’t create a clutter in the code.
Then, you don’t need to define say local element3629 in the start. Just declare it when it exists. Simple!
I’m also not sure why you’re using Attributes. Just place all of this in the code. Plus, there’s no need to define your values as “value x is y”, why do you need that??? Like, instead of value = "cannot heal" you’re just wasting a Variable because canHeal = false already implies that the character cannot heal.
Also I’m very sure Vector3.magnitude does not exist. It’s Magnitude. Did you even test this?

All it does is check for distances, instances, orientation and objects near it. You cannot call this “self aware”. This is ridiculous. I remind you, AI has the ability to learn by itself, hence why artificial intelligence. We living beings are able to think and process what we perceive, and so does AI, since it’s also an intelligence, albeit not natural. If it can’t learn and put it to use - it’s not AI.

ClodexTheScripter · August 19, 2023, 10:41pm

That’s a bit derailed from what I suggested, I’m partly to blame here lol since I called it API.
What I mean is a code example that uses your module in a showcase format. It would be nice to list all the functions one would use in the module. Just to help give an idea to those who come across your work.

But you have said that this is a work in progress, so it’s fair.

ClodexTheScripter · August 19, 2023, 10:43pm

Try saying:

This could be better if you do X,Y,Z

Instead of saying:

This is bad because you suck lol

ClodexTheScripter · August 19, 2023, 11:06pm

Here’s the comment where he tells us it’s an algorithm-based approach: Creating Self Aware AI: "Describing surroundings" (Open Source) - #4 by Magus_ArtStudios

I guess you guys don’t like being misled which is fair but it didn’t have to go down that road.

ClodexTheScripter · August 19, 2023, 11:29pm

A bit unfortunate that you had to experience these horrible responses but I commend you for sharing your work.

Vanniris · August 19, 2023, 11:30pm

Your title labels your project as “Self-Aware AI”. Then you instantly said “It’s not an AI but an algorithm.”

Now we’re here? Do you even know what you’re doing? Do you even have the minimal knowledge in AI?

This post defines it all.

Consider reading this again and label your product accordingly.