Giving LLM Text Vision and Self Awareness in Luau [OPEN SOURCE]

[Magus_ArtStudios]

This module gives Text-Vision to a Large language model. When you set up your workspace accordingly. It creates a table of entries describing the scene in different ways.
UPDATE

Giving LLM Text Vision and Self Awareness in Luau [OPEN SOURCE]

New update to the module! The update is the awareness now includes a generalized description of the terrain! --[[--{ [1] = "I observe that, I am walking on basalt.", [2] = "What I see is the environment has a meager amount of ground, a meager amount of cracked lava, a sparse amount of limestone, clusters of sand, a ton of rock, and heaps of basalt.", [3] = "I am observing I believe, we are in an area called Island of Morgan, the center…

--[[ {
                    [1] = "I notice, a medium-small plant called upside down Mint Chocolate cone is kind of near to the west.",
                    [2] = "I glimpse a Grape Lollipop is extremely near behind me to the right, and their are a quartet of trees that are carved or decorated.",
                    [3] = "I perceive, a piece of tiny rubble embedded with a Halloween Cupcake with Sprinkles is kind of near to the northwest.",
                    [4] = "So I notice, a medium-small plant called upside down Mint Chocolate cone is kind of near to the west, and a Grape Lollipop is very near behind me to the right. .",
                    [5] = "I am walking on a cherry heart cake, and a chunk of mini rock embedded with a Halloween Cupcake with Sprinkles is kind of not far to the northwest.",
                    [6] = "Their are a quartet of trees that form the scene.",
                    [7] = "I glimpse a medium-small plant called upside down Mint Chocolate cone is somewhat near westward, a Grape Lollipop is very close behind me to the right, and their are a quartet of trees that are carved or decorated. A piece of tiny rubble embedded with a Halloween Cupcake with Sprinkles is somewhat near to the northwest, I notice, a medium-small plant called upside down Mint Chocolate cone is somewhat near westward, and a Grape Lollipop is very close behind me to the right.  I also am aware, I am walking on a cherry heart cake, and a chunk of tiny rock embedded with a Halloween Cupcake with Sprinkles is somewhat near to the northwest. ",
                    [8] = "A medium-small plant called upside down Mint Chocolate cone is pretty near to the west",
                    [9] = "I am walking on a cherry heart cake. What I notice is a medium-small plant called upside down Mint Chocolate cone is kind of close to the west, a Grape Lollipop is very close behind me to the right, a quartet of trees that form the scene, and a chunk of tiny rock embedded with a Halloween Cupcake with Sprinkles is kind of close to the northwest. "
                 }  -  Client - Awareness:3509 ]]--

I use this module for other things as well it has a very efficient array handler that updates based on a refresh rate to prevent spamming of duplicate arrays.

This is great framework to build your game on! All of these categories would keep your game organized and you can build with LLMs on top of it.

You can easily change the names of the directories where each object type is stored!

local tablekeys = {
    ["mapobject"] = {{["datatable"] = nil, ["writetime"] = ticktime.Value, ["refreshrate"] = 30}, workspace.MapObjects},--name of folder

New Features include the awareness now includes what the root humanoid is standing on!
It also includes if their is water in the distance. It also produces more observations by combining observations in different ways.

The nearest function now has a query parameter to search for specific objects in the array.
–Updated link
(100) Awareness Text-Vision LLM Utility Luau Code- Roblox

This is a old video of my AI in Action!

Multi-Model AI #shorts #artificialintelligence #roblox #game #roleplay #education #ai

I’m currently using this with Zephyr 7b with a experiment to inject the system message with the awareness! It’s been very successful!

Storytelling Dancing AI Agent Party Member Zephyr 7B with Emojis

New demonstration of This Awareness module being used to Illustrate Candyland to a Large Language model!

The code is too long for this post but here is part of it.

Check out my Patreon and get the latest releases for Open-sourced code I create!
Magus Art game developer and resource creator | Patreon

[syntaqs (syntax)]

That’s not even a neural network, let alone an AI. It’s just a bunch of if-else statements.

[EncodedTerabyte (Tera)]

This is pretty interesting, but I don’t see myself using it anytime soon.

[xChris_vC (Chris)]

Self-aware AI means conscious & sentient intelligence. As of now, this ability is only reserved to humans.

This script isn’t self-aware, nor is it AI. It can’t do anything than fulfilling the existing tasks. Want to describe something in the environment that’s not covered by the if-statements (while perceived by the ‘senses’ of the NPC)? This script can’t.

[Waffle_pancake123fan]

Did you know that monkeys can be self aware they pass the test.

[Vanniris (Vanni)]

Well, that’s an insane amount of if statements and variable declarations to a point that practically nobody will ever bother to read the code. Can’t judge because I aint reading that pain.

[Magus_ArtStudios]

To be self aware an AI must be aware of its surroundings. You show me your AI project and how it works then you let me know what you think of this module. Because I think it’s funny how you know nothing yet decide to tout your opinion this is literally the type of string constructing algorithm to describe to a large language model what it sees. I would like to reiterate I do not care for your negative opinions, you are likely clueless and do not really know what you are talking about.

My responses coming off condescending because the only ones who decided to say anything where the negative people and they are not even making good arguments. Research the topic and provide informed input. This is how you would send surroundings information to an AI model. You can ask it to find a point of interest from the surroundings and make decisions based on that. My chatbot has the abilities to perform action commands from strings. So I could interpret the output to make the AI chatbot self aware by giving it the ability to make autonomous decisions based on its environment.

For example perhaps the AI decides to pick up an apple on the ground because it has some arbitrary hunger or thirst stat. The AIs decisions could also be influenced by survival and input from its surroundings. This is not a common implementation of a large language model but it would be very interesting to give the AI human goals. Like what literal universities have published on Automonous agents.This script is a strong concept for other developers as it provides a structured way to construct a description of the chatbots surroundings.
You could use raycasting and project a ray from only the front of the AI if you wish to only get what it would see in front of it.

For example if you were making a AI character who was a reactive NPC that made dialogue. You could send a message to the model describing the surroundings. Then the model can respond to that input of the world with their character like in that youtube video but within the confines of ROBLOX studio.

self-a·ware·ness

[ˌselfəˈwernəs]

NOUN

1. conscious knowledge of one’s own character, feelings, motives, and desires:

“the process can be painful but it leads to greater self-awareness”

Also, I updated the code I shared to not include the NPC description algorithm because it is too lengthy and tailored to my custom NPC combat algorithm
But I will post it here just because.

function aware.DescribeNPC(Char)
	local MagicID=Char:FindFirstChild("MagicID")
	if MagicID~=nil then
		local fly=""
		local Aggression=""
		local AttackVector=""
		local Barrier=""
		local Buffs=""
		local BaseHealth=""
		local BaseMagicDMG=""
		local BaseRangeDMG=""
		local BaseMeleeDMG=""
		local CanBeFrozen=""
		local ChangeStyles=""
		local Cooldown=""
		local Cost=""
		local CostMax=""
		local Cure=""
		local Description=""
		local Dodge=""
		local Element=""
		local Element2=""
		local Element3=""
		local Hide=""
		local Level=""
		local MainStyle=""
		local Multiply=""
		local Paralyze=""
		local Poison=""
		local Power=""
		local Prayer=""
		local ProtectFromStyle=""
		local Recoil=""
		local Range=""
		local Parasite=""
		local Species=""
		local Teleport=""
		local Transform=""
		local Weakness=""
		local rating=0
		--construct a long combined string of the above local strings.

		if MagicID:GetAttribute("Aggression")==1 then
			Aggression="aggressive, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Fly")==true then
			fly="can Fly, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Barrier")==true then
			Barrier="uses Barrier, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Buffs")==true then
			Buffs="uses Buffs, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Multiply")==true then
			Buffs="uses Multiply, "
			rating=rating+1
		end
		if MagicID:GetAttribute("ChangeStyle")==true then
			ChangeStyles="melee and magic user, "
			rating=rating+1
			BaseMagicDMG="base Magic damage is"..tostring(MagicID:GetAttribute("BaseMagicDMG"))..", "
			BaseMagicDMG="base Melee damage is"..tostring(MagicID:GetAttribute("BaseMeleeDMG"))..", "
			if MagicID:GetAttribute("AttackVector3")~=nil then
				AttackVector="melee range is"..tostring(MagicID:GetAttribute("AttackVector3"))..", "
			end	
		elseif MagicID:GetAttribute("ChangeStyle")==false then
			if	MagicID:GetAttribute("MainStyles")~=nil then
				if MagicID:GetAttribute("MainStyles")==1 then
					MainStyle="uses Melee only, "
					if MagicID:GetAttribute("AttackVector3")~=nil then
						AttackVector="melee range:"..tostring(MagicID:GetAttribute("AttackVector3"))..", "
					end	
				elseif MagicID:GetAttribute("MainStyles")==2 then
					MainStyle="uses Magic only, "
				elseif MagicID:GetAttribute("MainStyles")==3 then
					MainStyle="uses Range only, "
				end
			end
		end	

		if MagicID:GetAttribute("BaseHealth")~=nil then
			local hpr=Char.Humanoid.MaxHealth/100
			rating=rating+hpr
			BaseHealth="base health "..tostring(MagicID:GetAttribute("BaseHealth"))..", "
		end
		--write code to complete above context
		if MagicID:GetAttribute("CanBeFrozen")==false then
			CanBeFrozen="cannot be frozen, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Cooldown")~=nil then
			Cooldown="it's cooldown is "..tostring(MagicID:GetAttribute("Cooldown"))..", "
			rating=rating+(5-MagicID:GetAttribute("Cooldown"))
		end
		if MagicID:GetAttribute("Cost")~=nil then
			Cost="it's action cost is "..tostring(MagicID:GetAttribute("Cost"))..", "
			rating=rating+1
		end
		if MagicID:GetAttribute("CostMax")~=nil then
			CostMax="It's max power is"..tostring(MagicID:GetAttribute("CostMax"))..", "
		end
		if MagicID:GetAttribute("Cure")==true then
			Cure="can Heal, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Description")~=nil then
			Description=MagicID:GetAttribute("Description")..""
		end
		if MagicID:GetAttribute("Dodge")==true then
			Dodge="can Dodge, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Teleport")==true then
			Teleport="can Teleport, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Transform")==true then
			Transform="can Transform, "
			rating=rating+1
		end

		if MagicID:GetAttribute("Element")~=nil and MagicID:GetAttribute("Element")~="" then
			Element="uses "..MagicID:GetAttribute("Element")
			rating=rating+1
			if MagicID:GetAttribute("Element2")~="" and MagicID:GetAttribute("Element2")~=nil then
				Element2=","..MagicID:GetAttribute("Element2")
				rating=rating+1
			end
			if MagicID:GetAttribute("Element3")~="" and MagicID:GetAttribute("Element2")~=nil then
				Element3="and "..MagicID:GetAttribute("Element3")
				rating=rating+1
			end	
		end
		if MagicID:GetAttribute("Parasite")==true then
			Parasite=", it's parasitic"
			rating=rating+1
		end
		if MagicID:GetAttribute("Poison")==true then
			Poison=", it's poisonous"
			rating=rating+1
		end

		if MagicID:GetAttribute("Prayer")==true then
			Prayer="uses Prayers"
			rating=rating+1
		elseif 	MagicID:GetAttribute("ProtectFromStyle")==1 then
			rating=rating+1
			ProtectFromStyle=", strong against Melee"
		elseif		MagicID:GetAttribute("ProtectFromStyle")==2 then
			rating=rating+1
			ProtectFromStyle=", strong against Magic"
		elseif		MagicID:GetAttribute("ProtectFromStyle")==3 then
			rating=rating+1
			ProtectFromStyle=", strong against Range"
		end
		--Power="Power Multiplier:" MagicID:GetAttribute("Power")
		--Range=""
		if MagicID:GetAttribute("Recoil")==true then
			rating=rating+1
			Recoil="hurts to touch, "
		end
		rating=rating+tonumber(Level)
		--write rest of code for above context	
		return Description..Element..Element2..
			Element3.." magic. It's "..Level.." "..ChangeStyles..MainStyle..Cooldown..fly..Aggression..Barrier..Buffs..BaseHealth..
			BaseMagicDMG..BaseRangeDMG..BaseMeleeDMG..AttackVector..CanBeFrozen..
			Cost..CostMax..Cure..Dodge..
			Hide..Multiply..Paralyze..Poison..Power..
			Prayer..ProtectFromStyle..".",aware.judgepower(rating,25)
	else return "",nil	
	end
end

[Magus_ArtStudios]

This topic is related to my main topic! Which has a module that you can input search queries based on a database.
For example if you are looking for an Apple. You can search a list of strings and return a string describing an Apple.
This is a part of a multi-model AI-chatbot Agent I’m working on. When it’s done I will be releasing some sort of open-source version that has a simple framework to work from for other developers.
I have recently released a new ChatModule which is the most recent version of the main algorithms that powers the bots ability to compare input robustly with a database of strings and provide output. It also can solve basic geometry for example “What is the circumference of a circle with a radius of 4?” and worded algebra problems “If I have 4 groups of 5 how many do I have in total?” output= “4*5=20”.

The way it’s designed is to answer questions with the chatbot using my specific database for my games lore and hints when it can. Then, if it does not know the answer it gets the answer from a variety of APIs including wikipedia, How to wiki, conversational AIs that think they are the input personality, and GPT2. It uses zero-shot classification to determine which model to use. It saved all of the data it recieves from the AIs and APIs for that database or specific character. I’m also incorporating more APIs and premium AI models in the future. It’s important to have levels to things like this. It can be visualized like this Chatbot determines the intent based on rules and algorithms if conditions are met then->zero-shot-classification AI to determine the inputs intent->APICall to AI Model or wiki.
Premium architecture would be just be Chatbot Input if any->APICall to AI with specified character and context with user query
Artificially Intelligent Chatbot LUAU Algorithm [Open Source] - Resources / Community Resources - Developer Forum | Roblox

[Judgemental_Man (MivunC)]

1. It’s not “Self Aware”, it’s only capable of what it’s allowed to do, true sentience is impossible, you can only simulate it

2. You literally said it’s an AI

[lovableschoolgirl (lovable)]

I like the idea. However, it is a bit messy.

[ClodexTheScripter (Clodex)]

Why is everyone mad?
OP did make a self aware AI but not in the way that pleases your definition of what it means to be self aware.
Make this simple for yourselves and avoid explaining your definition of the word AI, as I’m very sure OP meant to call it NPC. It’s just a small miss direction. Honest mistakes can be made.

[Magus_ArtStudios]

This is part of my AI project. Which includes artificial intelligence models. If you have a AI that has no information about its surroundings then it cannot make relevant observations of its surroundings.
This algorithm creates artificial data to give AI context about its surroundings. If you were involved in working with Language Models this concept is easy to grasp and quite important within the restrictions of ROBLOx’s coding environment.

[ClodexTheScripter (Clodex)]

I’ll take ur words for it.
I’d like to suggest that it may seem better to add an API example and write a showcase for others to glance at. It may help them avoid looking at your work and discard it like the people above.

[Magus_ArtStudios]

(post deleted by author)

[Vanniris (Vanni)]

Self-aware can be defined as a character that can think by itself, interact with its surroundings by itself, react or express emotions. All I see in your code is just what you limited it to do. You cannot train it, it is not self aware, and less an AI.

That aside, this is one of the most unformatted and ridiculous code I’ve seen in all of DevForum. Even worse - it looks like that most of this was written by ChatGPT. The source code is straight up unreadable, let alone anyone being able to do anything with it.

Unnecessary functions, unnecessary variable definitions, insane amount of if statements that could’ve been simplified in other methods. Instead of using functions for actual uses you still went with the copy-paste method. Half of your code has ChatGPT comments, the rest is straight up Ctrl+C Ctrl+V. We can do better, let’s say:

• First, there’s zero need to define insane amounts of variables at the start, just use a table for it. The benefit of using tables is that we can also loop through them, so we don’t create a clutter in the code.
• Then, you don’t need to define say local element3629 in the start. Just declare it when it exists. Simple!
• I’m also not sure why you’re using Attributes. Just place all of this in the code. Plus, there’s no need to define your values as “value x is y”, why do you need that??? Like, instead of value = "cannot heal" you’re just wasting a Variable because canHeal = false already implies that the character cannot heal.
• Also I’m very sure Vector3.magnitude does not exist. It’s Magnitude. Did you even test this?

All it does is check for distances, instances, orientation and objects near it. You cannot call this “self aware”. This is ridiculous. I remind you, AI has the ability to learn by itself, hence why artificial intelligence. We living beings are able to think and process what we perceive, and so does AI, since it’s also an intelligence, albeit not natural. If it can’t learn and put it to use - it’s not AI.

[ClodexTheScripter (Clodex)]

That’s a bit derailed from what I suggested, I’m partly to blame here lol since I called it API.
What I mean is a code example that uses your module in a showcase format. It would be nice to list all the functions one would use in the module. Just to help give an idea to those who come across your work.

But you have said that this is a work in progress, so it’s fair.

[ClodexTheScripter (Clodex)]

Try saying:

This could be better if you do X,Y,Z

Instead of saying:

This is bad because you suck lol

[ClodexTheScripter (Clodex)]

Here’s the comment where he tells us it’s an algorithm-based approach: Creating Self Aware AI: "Describing surroundings" (Open Source) - #4 by Magus_ArtStudios

I guess you guys don’t like being misled which is fair but it didn’t have to go down that road.

[ClodexTheScripter (Clodex)]

A bit unfortunate that you had to experience these horrible responses but I commend you for sharing your work.

[Vanniris (Vanni)]

Your title labels your project as “Self-Aware AI”. Then you instantly said “It’s not an AI but an algorithm.”

Now we’re here? Do you even know what you’re doing? Do you even have the minimal knowledge in AI?

This post defines it all.

@syntaqs

That’s not even a neural network, let alone an AI. It’s just a bunch of if-else statements.

Consider reading this again and label your product accordingly.