Giving LLM Text Vision and Self Awareness in Luau [OPEN SOURCE]

Resources Community Resources
#39

If I were to even consider arguing with you I would just tell you this. An AI needs to see right???
How would you get Chat GPT API to know its surroundings? You could tell it where it is and whats its surroundings look like. Or you can just leave it as a talking head who has no actual idea of what is going on around it. Either is useful! But do not come here trying to dis on my project kid.

image
image1029×577 71 KB

This is the same method NVIDIA used to input information to GPT-4 to create autonomous agents in minecraft.
Minecraft AI - NVIDIA uses GPT-4 to create a SELF-IMPROVING :exploding_head: autonomous agent. - YouTube

4 Likes
#40

I have updated and completed the aware module.
It now requires the ChatModule linked here
Artificially Intelligent Chatbot LUAU Algorithm [Open Source] - Resources / Community Resources - Developer Forum | Roblox

The required module is located here

local aware = {}
local chatmodule=require(script.Parent.ChatModule)
syntheticdatasynonms={
	{"very close","quite close","extremely close"},
	{"close","near","not far"},
	{"kind of close","pretty near","somewhat near"},

	{" tiny "," mini "," minuscule "},
	{" small "," little "," petite "},

	{" large "," big ","huge"},
	{"gigantic","colossal","massive"},

	{"a few","some","a couple"},
	{"many","numerous","several"},
	{"lots of","tons of","a good number of"},

	{"There is","Over there is"},
	{"I see","I can see"},
	{"Nearby is","Close by is"},

	{"Additionally","Furthermore","Moreover"},
	{"Also","In addition","Plus"},
	{"However","But","Though"},

	{"to the north","northward","northern"},
	{"to the south","southward","southern"},
	{"to the east","eastward","eastern"},

	{"to the west","westward","western"}
}
function aware.GetCombatDescription(Char)
	local MagicID=Char:FindFirstChild("MagicID")
	if MagicID~=nil then
		local fly=""
		local Aggression=""
		local AttackVector=""
		local Barrier=""
		local Buffs=""
		local BaseHealth=""
		local BaseMagicDMG=""
		local BaseRangeDMG=""
		local BaseMeleeDMG=""
		local CanBeFrozen=""
		local ChangeStyles=""
		local Cooldown=""
		local Cost=""
		local CostMax=""
		local Cure=""
		local Description=""
		local Dodge=""
		local Element=""
		local Element2=""
		local Element3=""
		local Hide=""
		local Level=""
		local MainStyle=""
		local Multiply=""
		local Paralyze=""
		local Poison=""
		local Power=""
		local Prayer=""
		local ProtectFromStyle=""
		local Recoil=""
		local Range=""
		local Parasite=""
		local Species=""
		local Teleport=""
		local Transform=""
		local Weakness=""
		--construct a long combined string of the above local strings.

		if MagicID:GetAttribute("Aggression")==1 then
			Aggression="aggressive, "
		end
		if MagicID:GetAttribute("Fly")==true then
			fly="can Fly, "
		end
		if MagicID:GetAttribute("Barrier")==true then
			Barrier="uses Barrier, "
		end
		if MagicID:GetAttribute("Buffs")==true then
			Buffs="uses Buffs, "
		end
		if MagicID:GetAttribute("Multiply")==true then
			Buffs="uses Multiply, "
		end
		if MagicID:GetAttribute("ChangeStyle")==true then
			ChangeStyles="melee and magic user, "
			BaseMagicDMG="base Magic DMG:"..tostring(MagicID:GetAttribute("BaseMagicDMG"))..", "
			BaseMagicDMG="base Melee DMG:"..tostring(MagicID:GetAttribute("BaseMeleeDMG"))..", "
			if MagicID:GetAttribute("AttackVector3")~=nil then
				AttackVector="Melee Range:"..tostring(MagicID:GetAttribute("AttackVector3"))..", "
			end	
			--if MagicID:GetAttribute("MainStyles")~=nil then
			--	if MagicID:GetAttribute("MainStyles")==1 then
			--		MainStyle="Melee focused, "
			--	elseif MagicID:GetAttribute("MainStyles")==2 then
			--		MainStyle="Magic focused, "
			--	elseif MagicID:GetAttribute("MainStyles")==3 then
			--		MainStyle="Range focused, "
			--	end
			--end 

		elseif MagicID:GetAttribute("ChangeStyle")==false then
			if	MagicID:GetAttribute("MainStyles")~=nil then
				if MagicID:GetAttribute("MainStyles")==1 then
					MainStyle="Melee only, "
					if MagicID:GetAttribute("AttackVector3")~=nil then
						AttackVector="Melee Range:"..tostring(MagicID:GetAttribute("AttackVector3"))..", "
					end	
				elseif MagicID:GetAttribute("MainStyles")==2 then
					MainStyle="Magic only, "
				elseif MagicID:GetAttribute("MainStyles")==3 then
					MainStyle="Range only, "
				end
			end
		end	

		if MagicID:GetAttribute("BaseHealth")~=nil then
			BaseHealth="Base Health "..tostring(MagicID:GetAttribute("BaseHealth"))..", "
		end
		--write code to complete above context
		if MagicID:GetAttribute("CanBeFrozen")==false then
			CanBeFrozen="cannot be Frozen, "
		end
		if MagicID:GetAttribute("Cooldown")~=nil then
			Cooldown="Cooldown:"..tostring(MagicID:GetAttribute("Cooldown"))..", "
		end
		if MagicID:GetAttribute("Cost")~=nil then
			Cost="Power Cost:"..tostring(MagicID:GetAttribute("Cost"))..", "
		end
		if MagicID:GetAttribute("CostMax")~=nil then
			CostMax="Max Power:"..tostring(MagicID:GetAttribute("CostMax"))..", "
		end
		if MagicID:GetAttribute("Cure")==true then
			Cure="can Heal, "
		end
		if MagicID:GetAttribute("Description")~=nil then
			Description=MagicID:GetAttribute("Description")..""
		end
		if MagicID:GetAttribute("Dodge")==true then
			Dodge="can Dodge, "
		end
		if MagicID:GetAttribute("Teleport")==true then
			Teleport="can Teleport, "
		end
		if MagicID:GetAttribute("Transform")==true then
			Transform="can Transform, "
		end

		if MagicID:GetAttribute("Element")~=nil and MagicID:GetAttribute("Element")~="" then
			Element=" /"..MagicID:GetAttribute("Element")
			if MagicID:GetAttribute("Element2")~="" and MagicID:GetAttribute("Element2")~=nil then
				Element2="/"..MagicID:GetAttribute("Element2")
			end
			if MagicID:GetAttribute("Element3")~="" and MagicID:GetAttribute("Element2")~=nil then
				Element3="/"..MagicID:GetAttribute("Element3")
			end	
		end
		if MagicID:GetAttribute("Parasite")==true then
			Parasite="Parasitic, "
		end
		if MagicID:GetAttribute("Poison")==true then
			Poison="Poisonous, "
		end

		if MagicID:GetAttribute("Prayer")==true then
			Prayer="uses Prayers, "
		elseif 	MagicID:GetAttribute("ProtectFromStyle")==1 then
			ProtectFromStyle="strong against Melee, "
		elseif		MagicID:GetAttribute("ProtectFromStyle")==2 then
			ProtectFromStyle="strong against Magic, "
		elseif		MagicID:GetAttribute("ProtectFromStyle")==3 then
			ProtectFromStyle="strong against Range, "
		end
		--Power="Power Multiplier:" MagicID:GetAttribute("Power")
		--Range=""
		if MagicID:GetAttribute("Recoil")==true then
			Recoil="hurts to touch, "
		end

		--write rest of code for above context	
		return Description..Element..Element2..
			Element3.." Type."..Level..ChangeStyles..MainStyle..Cooldown..fly..Aggression..Barrier..Buffs..BaseHealth..
			BaseMagicDMG..BaseRangeDMG..BaseMeleeDMG..AttackVector..CanBeFrozen..
			Cost..CostMax..Cure..Dodge..
			Hide..Multiply..Paralyze..Poison..Power..
			Prayer..ProtectFromStyle
	end
end


function aware.DescribeNPC(Char)
	local MagicID=Char:FindFirstChild("MagicID")
	if MagicID~=nil then
		local fly=""
		local Aggression=""
		local AttackVector=""
		local Barrier=""
		local Buffs=""
		local BaseHealth=""
		local BaseMagicDMG=""
		local BaseRangeDMG=""
		local BaseMeleeDMG=""
		local CanBeFrozen=""
		local ChangeStyles=""
		local Cooldown=""
		local Cost=""
		local CostMax=""
		local Cure=""
		local Description=""
		local Dodge=""
		local Element=""
		local Element2=""
		local Element3=""
		local Hide=""
		local Level=""
		local MainStyle=""
		local Multiply=""
		local Paralyze=""
		local Poison=""
		local Power=""
		local Prayer=""
		local ProtectFromStyle=""
		local Recoil=""
		local Range=""
		local Parasite=""
		local Species=""
		local Teleport=""
		local Transform=""
		local Weakness=""
		local rating=0
		--construct a long combined string of the above local strings.

		if MagicID:GetAttribute("Aggression")==1 then
			Aggression="aggressive, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Fly")==true then
			fly="can Fly, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Barrier")==true then
			Barrier="uses Barrier, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Buffs")==true then
			Buffs="uses Buffs, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Multiply")==true then
			Buffs="uses Multiply, "
			rating=rating+1
		end
		if MagicID:GetAttribute("ChangeStyle")==true then
			ChangeStyles="melee and magic user, "
			rating=rating+1
			BaseMagicDMG="base Magic damage is"..tostring(MagicID:GetAttribute("BaseMagicDMG"))..", "
			BaseMagicDMG="base Melee damage is"..tostring(MagicID:GetAttribute("BaseMeleeDMG"))..", "
			if MagicID:GetAttribute("AttackVector3")~=nil then
				AttackVector="melee range is"..tostring(MagicID:GetAttribute("AttackVector3"))..", "
			end	
		elseif MagicID:GetAttribute("ChangeStyle")==false then
			if	MagicID:GetAttribute("MainStyles")~=nil then
				if MagicID:GetAttribute("MainStyles")==1 then
					MainStyle="uses Melee only, "
					if MagicID:GetAttribute("AttackVector3")~=nil then
						AttackVector="melee range:"..tostring(MagicID:GetAttribute("AttackVector3"))..", "
					end	
				elseif MagicID:GetAttribute("MainStyles")==2 then
					MainStyle="uses Magic only, "
				elseif MagicID:GetAttribute("MainStyles")==3 then
					MainStyle="uses Range only, "
				end
			end
		end	

		if MagicID:GetAttribute("BaseHealth")~=nil then
			local hpr=Char.Humanoid.MaxHealth/100
			rating=rating+hpr
			BaseHealth="base health "..tostring(MagicID:GetAttribute("BaseHealth"))..", "
		end
		--write code to complete above context
		if MagicID:GetAttribute("CanBeFrozen")==false then
			CanBeFrozen="cannot be frozen, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Cooldown")~=nil then
			Cooldown="it's cooldown is "..tostring(MagicID:GetAttribute("Cooldown"))..", "
			rating=rating+(5-MagicID:GetAttribute("Cooldown"))
		end
		if MagicID:GetAttribute("Cost")~=nil then
			Cost="it's action cost is "..tostring(MagicID:GetAttribute("Cost"))..", "
			rating=rating+1
		end
		if MagicID:GetAttribute("CostMax")~=nil then
			CostMax="It's max power is"..tostring(MagicID:GetAttribute("CostMax"))..", "
		end
		if MagicID:GetAttribute("Cure")==true then
			Cure="can Heal, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Description")~=nil then
			Description=MagicID:GetAttribute("Description")..""
		end
		if MagicID:GetAttribute("Dodge")==true then
			Dodge="can Dodge, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Teleport")==true then
			Teleport="can Teleport, "
			rating=rating+1
		end
		if MagicID:GetAttribute("Transform")==true then
			Transform="can Transform, "
			rating=rating+1
		end

		if MagicID:GetAttribute("Element")~=nil and MagicID:GetAttribute("Element")~="" then
			Element="uses "..MagicID:GetAttribute("Element")
			rating=rating+1
			if MagicID:GetAttribute("Element2")~="" and MagicID:GetAttribute("Element2")~=nil then
				Element2=","..MagicID:GetAttribute("Element2")
				rating=rating+1
			end
			if MagicID:GetAttribute("Element3")~="" and MagicID:GetAttribute("Element2")~=nil then
				Element3="and "..MagicID:GetAttribute("Element3")
				rating=rating+1
			end	
		end
		if MagicID:GetAttribute("Parasite")==true then
			Parasite=", it's parasitic"
			rating=rating+1
		end
		if MagicID:GetAttribute("Poison")==true then
			Poison=", it's poisonous"
			rating=rating+1
		end

		if MagicID:GetAttribute("Prayer")==true then
			Prayer="uses Prayers"
			rating=rating+1
		elseif 	MagicID:GetAttribute("ProtectFromStyle")==1 then
			rating=rating+1
			ProtectFromStyle=", strong against Melee"
		elseif		MagicID:GetAttribute("ProtectFromStyle")==2 then
			rating=rating+1
			ProtectFromStyle=", strong against Magic"
		elseif		MagicID:GetAttribute("ProtectFromStyle")==3 then
			rating=rating+1
			ProtectFromStyle=", strong against Range"
		end
		--Power="Power Multiplier:" MagicID:GetAttribute("Power")
		--Range=""
		if MagicID:GetAttribute("Recoil")==true then
			rating=rating+1
			Recoil="hurts to touch, "
		end
		rating=rating+tonumber(Level)
		--write rest of code for above context	
		return Description..Element..Element2..
			Element3.." magic. It's "..Level.." "..ChangeStyles..MainStyle..Cooldown..fly..Aggression..Barrier..Buffs..BaseHealth..
			BaseMagicDMG..BaseRangeDMG..BaseMeleeDMG..AttackVector..CanBeFrozen..
			Cost..CostMax..Cure..Dodge..
			Hide..Multiply..Paralyze..Poison..Power..
			Prayer..ProtectFromStyle..".",aware.judgepower(rating,25)
	else return "",nil	
	end
end
	local function tablehandler(location)
		-- Get the folder from the workspace
		local tableobj={}
		local mapObjectsFolder = location
		-- Check if the folder exists
		if mapObjectsFolder then
			-- Loop through the objects in the folder
			for _, obj in pairs(mapObjectsFolder:GetChildren()) do
				-- Add them to the stable
				table.insert(tableobj, obj)
			end
			-- Connect a function to the ChildAdded event of the folder
			mapObjectsFolder.ChildAdded:Connect(function(child)
				-- Add the new object to the stable
				table.insert(tableobj, child)
			end)
			-- Connect a function to the ChildRemoved event of the folder
			mapObjectsFolder.ChildRemoved:Connect(function(child)
				-- Find the index of the removed object in the stable
				local index = table.find(tableobj, child)
				-- Remove it from the stable
				table.remove(tableobj, index)
			end)
		end
		-- Return the stable
		return tableobj
	end

	-- Call the function and assign it to a variable
	mapObjects = tablehandler(workspace.MapObjects)
	plants = tablehandler(workspace.Plants)
	enemys=tablehandler(workspace.Enemys)
	rubble = tablehandler(workspace.Rubble)
	trees=tablehandler(workspace.Trees)
	chests=tablehandler(workspace.Chests)
	grounditems = tablehandler(workspace.grounditems)
	houses=tablehandler(workspace.Houses)
	playerstable=tablehandler(game.Players)
	dungeons=tablehandler(workspace.DungeonRender)
npcs = tablehandler(workspace.NPCS)
function getnearest(list,root,dist,getone)
	local list=list
	local pos=root.Position
	local torso = nil
	local dist = 30
	local temp = nil
	local human = nil
	local temp2 = nil
	local amnt=0
	local scope=nil
	local list=list	
	if list~=nil then
		for x = 1, #list do
			temp2 = list[x]
			if temp2==root.Parent then
				temp=nil
			elseif temp2:IsA("BasePart") then
				temp = temp2
			elseif temp2:IsA("Model") then
				if temp2.PrimaryPart~=nil then
					temp=temp2.PrimaryPart
				else 
					temp=temp2:FindFirstChildOfClass("BasePart")	
				end	
			elseif temp2:IsA("Player") then
				temp=temp2.Character:FindFirstChild("HumanoidRootPart")
			elseif temp2:IsA("Folder") then
				local m=temp2:FindFirstChildOfClass("Model")
				if m then
					temp=m:FindFirstChildOfClass("BasePart")
				end	
				end
			if (temp ~= nil)  then
				if (temp.Position - pos).magnitude < dist then
					amnt=amnt+1
					torso = temp2
					dist = (temp.Position - pos).magnitude
					scope = temp.Size
					if getone==true then
						return torso,scope
					end
				end
			end
		end
	end
	return torso,amnt,scope
end

function aware.judgedistance(position, position2, maxDist)

	local distVector = (position - position2).Magnitude

	local judgedDist = {0.2, 0.4, 0.6, 0.8, 1, 1.5, 2} 
	local judgeStrings = {{"very close"}, {"close"}, {"kind of close"},{"not far"}, {"kinda far"}, {"far"}, {"very far"}}

	local index = 1
	local distanceJudge = ""
	repeat
		if distVector <= maxDist * judgedDist[index] then
			distanceJudge = judgeStrings[index]
			break
		end
		index = index + 1
	until index > #judgedDist

	-- Get direction
	local dirVector = (position2 - position).Unit -- Unit vector pointing to position2
	local angle = math.atan2(dirVector.Z, dirVector.X) -- Angle in radians
	local dirName = ""

	if angle > math.rad(157.5) or angle < math.rad(-157.5) then
		dirName = "south"
	elseif angle > math.rad(112.5) then 
		dirName = "southwest"
	elseif angle > math.rad(67.5) then
		dirName = "west"
	elseif angle > math.rad(22.5) then
		dirName = "northwest"
	elseif angle > math.rad(-22.5) then
		dirName = "north"
	elseif angle > math.rad(-67.5) then
		dirName = "northeast"
	elseif angle > math.rad(-112.5) then
		dirName = "east"
	else
		dirName = "southeast"
	end

	return distanceJudge,dirName
end

function aware.judgeamnt(amnt,judgestrings)
	local judgedistribution={1,2,3,4,5,6,8}
	local judgestrings={{"a single"},{"a couple"},{"a few"},{"a quartet"},{"several"},{"a bunch"},{"a lot"}}
	local index=1

	repeat
		local distvector=amnt
		if distvector<=judgedistribution[index] then
			return judgestrings[index][1]
		end	
	until index>=#judgedistribution	
end

function aware.judgething(size, bigSize,judgeSizes,judgeStrings)
	local index = 1
	local sizeJudge = ""
	for i, judge in ipairs(judgeSizes) do
		if size <= bigSize * judge then
			local sizeJudge2 = judgeStrings[i]
			sizeJudge=sizeJudge2[math.random(1,#sizeJudge)]
			break
		end
	end
	return sizeJudge
end

function aware.judgesize(size, bigSize)
	local judgeSizes = {0.1, 0.25, 0.5, 0.75, 1, 1.5, 2}
	local judgeStrings = {{"tiny"}, {"small"}, {"medium-small"}, {"medium"}, {"medium-large"}, {"large"}, {"huge"}}
	local sizeJudge= aware.judgething(size, bigSize,judgeSizes,judgeStrings)
	return sizeJudge
end

function aware.judgepower(power, bigpower)
	local judgeSizes = {0.1, 0.25, 0.5, 0.75, 1, 1.5, 2}
	local judgeStrings = {{"very weak"}, {"weak"}, {"kind of weak"}, {"average"}, {"kind of strong"}, {"strong"}, {"powerful"}}
	local sizeJudge= aware.judgething(power, bigpower,judgeSizes,judgeStrings)
	return sizeJudge
end

function aware.judgeobject(instance)
	if instance:FindFirstChild("Humanoid")~=nil then
		return instance.Humanoid.DisplayName
	elseif instance:IsA("folder")~=nil then	
		return instance.Name
	elseif instance:IsA("BasePart") then
		local s=instance:FindFirstChild("Species")
		if s~=nil then
			return s.Value	
		end
	elseif instance:IsA("Model") then
		return instance.Name
	end
end


function aware.getsurroundingobjects(root)
	local closestPlant, numPlants,plantsize = getnearest(plants, root, 40)
	local closestTree, numTrees = getnearest(trees, root, 80) 
	local closestStructure, numStructures,structuresize = getnearest(mapObjects, root, 160)
	local closestEnemy, numEnemies = getnearest(enemys, root, 60)
	local closestNPC, numNPCs = getnearest(npcs, root, 60)
	local closestPlayer, numPlayers = getnearest(playerstable, root, 80)
	local closestRubble, numRubble = getnearest(rubble, root, 20) 
	local closestChest, numChests = getnearest(chests, root, 30)
	local closestGroundItem, numGroundItems = getnearest(grounditems, root, 40)
	local closestHouse, numHouses,housesize = getnearest(houses, root, 100)
	local closestDungeon, numDungeons,dungsize = getnearest(dungeons, root, 200)
	local pos=root.Position


	
	local function describeTrees()
		if numTrees > 0 then
			local treeText = aware.judgeamnt(numTrees).." of trees surround the area"
			return treeText			
		end
	end

	local function describePlants()
		if numPlants > 0 then  
			local plantText = aware.judgeamnt(numPlants).." of plants are scattered around"
			return plantText
		end
	end

	local function describeRubble()
		if numRubble > 0 then
			local rubbleText = aware.judgeamnt(numRubble).." of pieces of rubble lay on the ground" 
			return rubbleText
		end
	end

	local function describeEnemies()
		if numEnemies > 0 then
			local enemyText = aware.judgeamnt(numEnemies).." of enemies are lurking nearby"
			return enemyText
		end
	end

	local function describeNPCs()
		if numNPCs > 0 then
			local npcText = aware.judgeamnt(numNPCs).." of strangers are in the area"
			return npcText
		end
	end

	local function describePlayers()
		if numPlayers > 0 then
			local playerText = aware.judgeamnt(numPlayers).." of players are nearby" 
			return playerText
		end
	end

	local function describeChests()
		if numChests > 0 then
			local chestText = aware.judgeamnt(numChests).." of chests are scattered around"
			return chestText
		end
	end

	local function describeGroundItems()
		if numGroundItems > 0 then
			local groundItemText = aware.judgeamnt(numGroundItems).." of items lay on the ground"
			return groundItemText
		end
	end 

	local function describeHouses()
		if numHouses > 0 then
			local houseText = aware.judgeamnt(numHouses).." of houses stand nearby"
			return houseText
		end
	end

	local function describeDungeons()
		if numDungeons > 0 then
			local dungeonText = aware.judgeamnt(numDungeons).." of areas around here."
			return dungeonText
		end
	end
	local function describestructures()
		if numStructures > 0 then
			local structureText = aware.judgeamnt(numDungeons).." of interesting places to check out around here."
			return structureText
		end
	end

	local function describeSurroundings(pos)

		local info = aware.getsurroundingobjects(pos)

		local descStrings = {}
local opener={"Their are ","I see ","Surrounding us are "}
		table.insert(descStrings, ""..describeTrees())
		table.insert(descStrings, ""..describePlants())
		table.insert(descStrings, ""..describeRubble())
		table.insert(descStrings, ""..describeEnemies())
		table.insert(descStrings, ""..describeNPCs())
		table.insert(descStrings, ""..describePlayers())
		table.insert(descStrings, ""..describeChests())
		table.insert(descStrings, ""..describeGroundItems())
		table.insert(descStrings, ""..describeHouses())
		table.insert(descStrings, ""..describeDungeons())

		-- Remove any nil values
		local finalStrings = {}
		local paragraphstring=""
		for i,str in ipairs(descStrings) do
			if str ~= "" then
				--table.insert(finalStrings, opener[math.random(1,#opener)]..str..".") 
				paragraphstring=paragraphstring..str..", "
			end
		end
		if paragraphstring~="" then
			paragraphstring=opener[math.random(1,#opener)]..paragraphstring.."."
			table.insert(finalStrings,paragraphstring)
		end
		-- Join the strings together

		return finalStrings

	end
	local function describeSurroundings2(pos)

		local info = aware.getsurroundingobjects(pos)
		local function describeRubble2(context)

			local rubbleText = ""

			if closestRubble then
				local dist, dir = aware.judgedistance(pos, closestRubble.Position, 20)
				local size = aware.judgesize(closestRubble.Size)
				rubbleText = "piece of " ..size.. " rubble is " ..dist.. " to the " ..dir
			end

			if numRubble > 1 then
				if context==nil then
					return rubbleText,aware.judgeamnt(numRubble).. " of pieces of rubble lay on the ground" 	
				else
					rubbleText = rubbleText .. context.. aware.judgeamnt(numRubble).. " of pieces of rubble lay on the ground"
				end	
			end

			return rubbleText

		end


		local function describeChests2(context)

			local chestText = ""

			if closestChest then
				local dist, dir = aware.judgedistance(pos, closestChest.PrimaryPart.Position, 30)
				--local size = aware.judgesize(closestChest.Size)
				chestText = "chest is " ..dist.. " to the " ..dir
			end

			if numChests > 1 then
				if context==nil then
					return chestText,aware.judgeamnt(numChests).." of chests are scattered around" 	
				else
					chestText = chestText .. context .. aware.judgeamnt(numChests).." of chests are scattered around" 
				end
			end	

			return chestText

		end


		local function describeGroundItems2(context)

			local groundItemText = ""

			if closestGroundItem then
				local dist, dir = aware.judgedistance(pos, closestGroundItem.Item:FindFirstChildOfClass("BasePart").Position, 40)
				local size = aware.judgesize(closestGroundItem.Size)
				groundItemText = aware.judgeobject(closestGroundItem)" is " ..dist.. " to the " ..dir
			end

			if numGroundItems > 1 then
				if context==nil then
					return groundItemText,aware.judgeamnt(numGroundItems).." of items lay on the ground"	
				else
					groundItemText = groundItemText .. context .. aware.judgeamnt(numGroundItems).." of items lay on the ground"
				end	
			end

			return groundItemText

		end


		local function describeHouses2(context)

			local houseText = ""

			if closestHouse then
				local dist, dir = aware.judgedistance(pos, closestHouse.Position, 100)
				local size = aware.judgesize(closestHouse.Size)
				houseText = "" ..size.. " house is " ..dist.. " to the " ..dir
			end

			if numHouses > 1 then
				if context==nil then
					return houseText,aware.judgeamnt(numHouses).." of other houses stand nearby"	
				else
					houseText = houseText .. context .. aware.judgeamnt(numHouses).." of other houses stand nearby"
				end
			end	

			return houseText

		end

		local function describestructures2(context)
			if numStructures > 0 then
				local structureText = aware.judgeamnt(numDungeons).." of interesting places to check out around here"
				return structureText
			end
			local dungeonText=""
			if closestDungeon then
				local dist, dir = aware.judgedistance(pos, closestStructure.PrimaryPart.Position, 200)
				local size = aware.judgesize(closestStructure.PrimaryPart.Size)
				dungeonText = aware.judgeobject(closestStructure).." is "..dist.. " to the " ..dir
			end

			if numStructures > 1 then
				if context==nil then
					return dungeonText,aware.judgeamnt(numStructures)" of interesting places to check out around here" 	
				else
					dungeonText = dungeonText .. context..aware.judgeamnt(numStructures)" of interesting places to check out around here" 
				end	
			end

		end

		local function describeDungeons2(context)

			local dungeonText = ""

			if closestDungeon then
				local dist, dir = aware.judgedistance(pos, closestDungeon:FindFirstChildOfClass("BasePart").Position, 200)
				local size = aware.judgesize(closestDungeon.Size)
				dungeonText = aware.judgeobject(closestDungeon).." is "..dist.. " to the " ..dir
			end

			if numDungeons > 1 then
				if context==nil then
					return dungeonText,aware.judgeamnt(numDungeons)" of areas around here" 	
				else
					dungeonText = dungeonText .. context..aware.judgeamnt(numDungeons)" of areas around here" 
				end	
			end

			return dungeonText

		end
		local function describeTrees2(context)

			local treeText = ""

			if closestTree then
				local dist, dir = aware.judgedistance(pos, closestTree.PrimaryPart.Position, 160) 
				treeText = aware.judgeobject(closestTree).." is " ..dist.. " to the " ..dir
			end

			if numTrees > 1 then
				if context==nil then
					return treeText, aware.judgeamnt(numTrees).." of other trees surround the area" 	
				else
					treeText = treeText ..context.. aware.judgeamnt(numTrees).." of other trees surround the area"
				end	
			end

			return treeText

		end


		local function describePlants2(context)

			local plantText = ""

			if closestPlant then
				local dist, dir = aware.judgedistance(pos, closestPlant.Position, 40)
				local size = aware.judgesize(closestPlant.Size)
				plantText =size.. " "..aware.judgeobject(closestPlant).." is " ..dist.. " to the " ..dir
			end

			if numPlants > 1 then
				if context==nil then
					return plantText,  aware.judgeamnt(numPlants).." of other plants are scattered around"	
				else
					plantText = plantText ..context.. aware.judgeamnt(numPlants).." of other plants are scattered around"
				end	
			end

			return plantText

		end


		local function describeEnemies2(context)

			local enemyText = ""

			if closestEnemy then
				local dist, dir = aware.judgedistance(pos, closestEnemy.HumanoidRootPart.Position, 60)
				enemyText = aware.judgeobject(closestEnemy).. " is " ..dist.. " to the " ..dir
			end

			if numEnemies > 1 then
				if context==nil then
					return enemyText,aware.judgeamnt(numEnemies).." enemies are lurking nearby"
				else
					enemyText = enemyText ..context.. aware.judgeamnt(numEnemies).." enemies are lurking nearby"
				end
			end	

			return enemyText

		end

		local function describePlayers2(context)

			local enemyText = ""

			if closestPlayer then
				local dist, dir = aware.judgedistance(pos, closestPlayer.Character.HumanoidRootPart.Position, 80)
				enemyText = aware.judgeobject(closestEnemy).. " is " ..dist.. " to the " ..dir
			end

			if numPlayers > 1 then
				if context==nil then
					return enemyText,aware.judgeamnt(numPlayers).." of players are nearby" 	
				else
					enemyText = enemyText ..context.. aware.judgeamnt(numPlayers).." of players are nearby" 
				end	
			end

			return enemyText

		end
		local function describeNPCs2(context)

			local npcText = ""

			if closestNPC then
				local dist, dir = aware.judgedistance(pos, closestNPC.HumanoidRootPart.Position, 80)
				npcText = aware.judgeobject(closestNPC).." is " ..dist.. " to the " ..dir
			end

			if numNPCs > 1 then
				if context==nil then
					return npcText,aware.judgeamnt(numNPCs).." of strangers are in the area"	
				else
					npcText = npcText ..context.. aware.judgeamnt(numNPCs).." strangers are in the area"
				end	
			end

			return npcText

		end
		local descStrings = {}
		local opener={"Their are ","I see ","Surrounding us are "}
		table.insert(descStrings, ""..describeTrees2(", and their are "))
		table.insert(descStrings, ""..describePlants2(", and their are "))
		table.insert(descStrings, ""..describeRubble2(", and their are "))
		table.insert(descStrings, ""..describeEnemies2(", and their are "))
		table.insert(descStrings, ""..describeNPCs2(", and their are "))
		table.insert(descStrings, ""..describePlayers2(", and their are "))
		table.insert(descStrings, ""..describeChests2(", and their are "))
		table.insert(descStrings, ""..describeGroundItems2(", and their are "))
		table.insert(descStrings, ""..describeHouses2(", and their are "))
		table.insert(descStrings, ""..describeDungeons2(", and their are "))
		local descStrings2 = {}
		local opener2={"Their is ","","I can see "}
		
		local tre,tree2=describeTrees2()
		local plan,plan2=describePlants2()
		local rub,rub2=describeRubble2()
		local enem,enem2=describeEnemies2()
		local npc1,npc2=describeNPCs2()
		local plr1,plr2=describePlayers2()
		local chest,ches2=describeChests2()
		local g1,g2=describeGroundItems2()
		local hous,house=describeHouses2()
		local dung,dungeon=describeDungeons2()
		local struc,struc2=describestructures2()
		local function DetermineGoal()
			local interest=""
			
			if closestPlayer then
				interest=interest.." I can see that "..plr1..""		
			end
			if enem then
				if interest~="" then
					interest=interest..", w"
				else
					interest="W"
				end
				interest=interest.."atch out, their is "..enem..""		
			end
			if npc1 then
				if closestPlayer and interest~="" then
					interest=interest..", and"
				end 
				interest=interest.." "..npc1..""
			end
			if dung then
				--if not npc1 and interest~="" then
				--	interest=interest..", we "
				--else 
				--	interest=interest.." We "
				--end
				if interest~="" then
					interest=interest.."."
				end
				interest=interest.." We are in a place called "..aware.judgeobject(closestDungeon)..""
				
			end
			if struc then
				if interest~="" then
					interest=interest.."."
				end
				interest=interest.." Their is a "..struc
			end
			if chest then
				if interest~="" then
					interest=interest.."."
				end
				interest=interest.." I notice a "..chest..""
			end
			if g1 then
				if interest~="" then
					interest=interest..","
				end
				interest=interest..g1
			end
			if hous then
				if struc and interest~="" then
					interest=interest.."."
				end
				interest=interest.." I can see a "..hous..""
			end
			if tree2 then
				if interest~="" then
					interest=chatmodule.randomizeString(interest..". Additionally, t")
				else 
					interest=interest.."T"
				end
				interest=interest.."heir are "..tree2
			end
			if plan2 then
				if interest~="" then
					interest=interest..","
				else
					interest=interest.." Their are "
				end
				interest=interest..""..plan2..""
			end
			if interest~="" then
				interest=interest.."."	
			end
			return interest
		end
		--example 	enemyText = aware.judgeobject(closestEnemy.Parent).. " is " ..dist.. " to the " ..dir
		--return enemyText,aware.judgeamnt(numPlayers).." of players are nearby" 	
		--	enemyText = enemyText ..context.. aware.judgeamnt(numPlayers).." of players are nearby" 

		-- Remove any nil values
		local finalStrings = {}
		local paragraphstring=""
		for i,str in ipairs(descStrings) do
			if str ~= "" then
				table.insert(finalStrings, opener[math.random(1,#opener)]..str..".") 
				paragraphstring=paragraphstring..str..", "
			end
		end
		table.insert(finalStrings,DetermineGoal())
		if paragraphstring~="" then
			paragraphstring=paragraphstring.."."
			table.insert(finalStrings,paragraphstring)
		end
		-- Join the strings together

		return finalStrings

	end
	local d1=describeSurroundings(pos)
	local d2=describeSurroundings2(pos)
	local syntheticdata= function()
		for i,str in ipairs(d1) do
		if str ~= "" then
				table.insert(d2, str) 
			--paragraphstring=paragraphstring..str..", "
		end
		end
		for i,str in ipairs(d2) do
			if str ~= "" then
				d2[i]=chatmodule.randomizeStringLight(str,syntheticdatasynonms)
				--paragraphstring=paragraphstring..str..", "
			end
		end
		
		return d2
	end
	
	return syntheticdata()
	--,
	--{
	--	closestPlant = closestPlant,  
	--	numPlants = numPlants,
	--	plantsize = plantsize,

	--	closestTree = closestTree,
	--	numTrees = numTrees,

	--	closestStructure = closestStructure,
	--	numStructures = numStructures,
	--	structuresize = structuresize,

	--	closestEnemy = closestEnemy,
	--	numEnemies = numEnemies, 

	--	closestNPC = closestNPC,
	--	numNPCs = numNPCs,

	--	closestPlayer = closestPlayer,
	--	numPlayers = numPlayers,

	--	closestRubble = closestRubble,
	--	numRubble = numRubble,

	--	closestChest = closestChest,
	--	numChests = numChests,

	--	closestGroundItem = closestGroundItem, 
	--	numGroundItems = numGroundItems,

	--	closestHouse = closestHouse,
	--	numHouses = numHouses,
	--	housesize,

	--	closestDungeon = closestDungeon,
	--	numDungeons = numDungeons,
	--	dungsize = dungsize
	--}
end

return aware
2 Likes
#41

While I kinda disagree with OP here, this assumption is wrong.

Deep learning is a subset of machine learning (neural nets), which is a subset of artifical intelligience. This means that machine learning falls under AI, but not all AI is machine learning!

image

Francois Chollet’s “Deep Learning with Python” covers this very well as it’s first chapter, if you’re interested about the specifics.

#42

I know its incorrect; my point was the OP shouldn’t be aggressive towards those who believe so

2 Likes
#43

Artificial intelligence encompasses Machine learning functions and rule based algorithms that do specific things. What you are sternly referring to as AI is also an LLM. Bing for example uses a combination of rule based search algorithms and AI to create user outputs. Very similar to my architecture which I have been developing in parallel.
If you research the best kind of AI’s they use a combination of Machine learning and rule based systems.
From what I’ve learned recently I would be looking into turning strings into excutable code. So the AI can write its own functions.
A LLM or chatbot needs ways to know its surroundings. With this module the AI can see its surroundings.
With the chatmodule. Their can even be simple rule based systems to create weights that determines the bots motivations. For example say the goal was picking apples and putting them in your inventory.
The bots I’ve created have randomly generated bodies and they share a combat/action module with the player and have the same data structure as the players. I did this just because it was the best way. But they have a 42 item inventory, 3 combat styles of weapons, armors, halos, shields, magic books etc and a 32 attribute object that determines the AI’s combat behavior. My AI system learns from AIs and incorporates it into it’s own data and saves it for that specific personality or database.

image
image293×778 28.5 KB

image

The custom chatsystem I developed extracts the emotional tonality from the string and plays atmospheric sounds when it starts output. It parses the response into sentences and characters and uses a simple letter to sound thing that gives them their own spoken language (Alas their is not any text to sound modules on roblox). It also puts emojis in the string that uses my robust completequery function to determine the emoji.
Theirs a lot more to it. But soon it will be open sourced so others can use it as well.

RobloxScreenShot20230811_230203086
RobloxScreenShot20230811_2302030861208×810 108 KB

1 Like
#44

I have done a semisuccessful test of this script! :slight_smile: It needs a little tweaking. But this is so cool. :slight_smile:

image
image954×390 49.8 KB

3 Likes
#45

image

1 Like
#47

Now when you bring it all together we get this

image
image918×473 50.9 KB

image
image435×593 68.5 KB

So the bot now combines its situational data with its databases by doing a search query with each individual entry from the awareness module. For example a combination would be “I notice a Darkness Juggernaut lurking northwest,” This would call the bestiary database and combine that string with this.
image
image1129×72 16.1 KB

So if you were to ask the bot what it see it would say.
“I notice a Darkness Juggernaut lurking northwest, a massive and powerful beast that can crush anything in its path. It has thick armor and spikes that can deflect most attacks. It is loyal to the dark cloud and follows its commands without question.”

1 Like
#48

Wow, I’m impressed by your work here, how long did it take you to train the LLM model?

#49

I’m using a variety of hugging face APIs and my personality based chatbot algorithm. So the algorithm constructs context for LLMs so they can expand the response.
image
For example if you were to input the first entry in this table to a LLM for example this free to use interference GPT2 model api from huggingface, it would write something based on that input context.
If you were to use a premium model like GPT 3.5 or host some cloned model on huggingface like LLama 7b-13b you would be able to make the AI think it is that character and it would be able to become an autonomous agent.

function GPT2(inputText)
-- Import the HttpService


-- Define the API URL
--local API_URL = "https://api-inference.huggingface.co/models/gpt2"
	local API_URL ="https://api-inference.huggingface.co/models/gpt2-large"
-- Define the headers with your authorization key
local headers = {
	["Authorization"] = Bearerkey
}

-- Define a function to query the API with a payload
local function query(payload)
	-- Encode the payload as a JSON string
	local jsonPayload = HttpService:JSONEncode(payload)
	-- Send a POST request to the API URL with the headers and the payload
	local response = HttpService:PostAsync(API_URL, jsonPayload, Enum.HttpContentType.ApplicationJson, false, headers)
	-- Decode the response as a JSON table
	local jsonResponse = HttpService:JSONDecode(response)
	-- Return the JSON table
	return jsonResponse
end
	
-- Define your input text
	
-- Query the API with your input text as the inputs field
	local output = query({
		["inputs"] = inputText,
		["max_length"] = 200, -- This will limit the output to 150 tokens
		["num_return_sequences"] =1, -- This will generate only one output
		["temperature"] = 2-- This will increase the randomness of the output
	})
	
	-- or
	local generatedText = output[1]["generated_text"]
print(output)
	-- Print the string
	return generatedText
-- Print the output

end

Their are over 100 personalities in this module they are called based on the characters name. It does a search query to get the name. I constructed the personalities by providing a template to LLMs and a prompt for the character. They have 5 contextual databases which are their personal preferences and flair. So the chatbot can influence/trick a LLM by inserting these entries into the previous generated response table of an LLM.
2 Likes
#51

I finished this module! I have updated the parent posts code and included a two links to the respective modules.
I just saw this news article relating to this modules use case
“Some of the limitations of large language models have been put down to them not being able to understand the physical world. Maybe it’s possible they can become more intelligent if they can embody and experience a digital one.”
OpenAI snaps up role-playing game dev as first acquisition (msn.com)
Here is a link to my completed open source module. If you wish to utilize the randomizestring function then It requires my chatmodule script.

#52

Was this inspired by the video or by this post:
[Release Version 1.2] DataPredict - Machine And Deep Learning Library (Q-Learning NNs with Experience Replay, Retrainable Models and more!) - Resources / Community Resources - Developer Forum | Roblox

Instead of comparing your module with other people try to improve it and make it a self-learning AI.

1 Like
#53

I have published a video on youtube demonstrating a use case of this algorithm!
In the video I am demonstrating with my AI personality based chatbot system.

#54

No it wasn’t :slight_smile: This module is for a LLM it’s not a machine learning function, but you can utilize it in ROBLOX studio.

1 Like
#55

How long have you been on Roblox? It’s been superseded by Magnitude, but it still exists.

I told that to many people who had NPC movement scripts.

Very neat, but some of the execution of it is done a little poorly, but it’s not like I could do any better.

It would be also cool if the AI was somehow indicated some logic (like the player sword deals 5 damage, enemy does 6) to tell the player what the best option is to do next.

1 Like
#56

Yes! Well it’s a work in progress. For the application of my chatbot, I create a table of observations, then do a search query of the observations with the goal. For example the player can ask what do you see?

I wrote the constructor string functions this way because it was easiest for me. They are only 3-9 lines each. but looking at it it would be so easy to make them into modular functions. I just wanted to be tedious about the explanations for specific types of objects to minimize the repetitive nature of machine constructed strings. Which also is why I used my chatmodule function. This allows the paragraphs to have their sentence structures reversed or even randomized. and allows for robust navigation with the data.

For my application, I get a table of these observations. So if the player asks, about a tree or a plant the NPC would describe the closest plantss and a description of the amount of other plants this logic applies to all the constructors.

I admit the structure is tailored to my use case. But it provides a template, and if you were to read it and review it seriously, modifications can be made, and the structure improved for your use case.

But at its root it represents an idea, and by sharing ideas we open the door for better ideas. :slight_smile:

Concerning your combat module you have some good points. But if you check out my chatmodule it has capabilities for getting strings robustly from tables and combining sentences. For example I use this module to do a search query with its personality and general knowledge databases. So if it sees something it constructs the tables into new tables, it also recognizes the player and replaces that section with a personal reference to the player.

In this example the data is used to recognize enemies and give a lore description of the enemy and combine that string to the observation. I have disabled API calls in this section because these are just entries and not actually used. They can be associated and input in an AI. I’m currently working on a action module. Which is going to be a list of strings,that are commands, then I could potentially have a LLM AI-Model tell it what to do by constructing a formatted prompt with the context of their abilities listed by the initial function called DescribeNPC which describes the abilities of my combat AIs based on their 32 parameter “MagicID” object.
A LLM wouldn’t be able to handle real-time combat. But it could make interesting decisions and input, based on the context. Having a way to recognize these goals is second nature to this architecture I’m creating.

function PersonalChatLogic(query,Greetings,inquiry,IDK,Database,wisdom,persona,context,responses,player,npc)
	--local answer = AnswerBox.Text -- Get the user's answer from the TextBox
	--go through stack in order of most common context
	local Greetings,inquiry,IDK,DataBase,wisdom=Greetings,inquiry,IDK,Database,wisdom
	local Result=nil	
	--
	local Awaretable=awareness.getsurroundingobjects(npc.HumanoidRootPart)
	print(persona)
	print(Awaretable)
	local plrref={{tostring(player).." is","you are" }}
		
	local npcnam=npc.Humanoid.DisplayName
	local Greetwithname={{persona,npcnam}}
	local function ContextAwareness()
		if #responses>0 then
			local str1={cm.ReduceQuery(responses[#responses])}
			local str2={cm.ReduceQuery(context[#context])}
		end
	end
	local index=1
	local answer=nil
	repeat
		local Result=nil
		answer=Awaretable[index]

	if answer ~="" and answer~=nil then -- Change the correct answer as needed	
		if Greetings==nil or inquiry==nil or IDK==nil or Database==nil or wisdom==nil then
			Greetings,inquiry,IDK,Database,wisdom=chatbot.getperson(persona)
			repeat wait() until Greetings~=nil
		end
		--cm.CompleteQuery(str,database,filter,repetitive,randomize,context,reverse)
		local reverse=false
		if cm.mathrandom(1,2)==1 then
			reverse=true
		end
		local Result1=nil
		if Result1==nil then	
			print("searching greetings")
			Result1=cm.CompleteQuery(answer, Greetings,false,true,false,false,reverse)
		end		
		if Result==nil then
			Result=cm.SearchQuery(answer,Bestiary,false,true)
			if Result~=nil then
			
				local npcname=Result		
				local body={" Let me tell you a story of "..npcname.."... "," The darkness comes in many forms such as the entity "..npcname..". "," I've seen this one myself... I heard it speak its name... It's like it was inside my mind..."..npcname.." I'll never forget its name... Nor will I forget its story engrained in my mind. "," You will encounter many entities... Perhaps even the one known as "..npcname..", their is a legend of that destructive force... "," Maybe you have seen it ... it goes by the name "..npcname..". "," You must be careful in your travels... They tell tales around here of the creature known as "..npcname..". Listen to this tale... "," Be wary if you encounter "..npcname.."."}
				local description=DescribeNPC(Result)
				--intro[mathrandom(1,#intro)]..
				Result=body[cm.mathrandom(1,#body)]..description
			end
		end
		if Result==nil then
			print("searching database")
			Result=cm.CompleteQuery(answer,Database,true,true,false,true,reverse)
		end
		if Result==nil then	
			print("searching wisdom")
			Result=cm.CompleteQuery(answer,wisdom,true,true,true,true,reverse)
		end
		
		
		if Result==nil and Result1~=nil then
			Result=Result1
			elseif Result~=nil and Result1~=nil then
			Result=Result1..". "..Result
		end	
		if Result~=nil then
			Result= cm.randomizeString(Result)
		else
		end	
	else 		
		end
		if Result~=nil then
			local rs=cm.randomizeStringLight(Awaretable[index],plrref,2) 
			Awaretable[index]=rs..". "..Result.."."
		end
		index=index+1
	until answer==nil	
	print(Awaretable)
	
	return Awaretable
end
2 Likes
#57

Here’s a more in depth example of my chatbot in action, you can also test it here. But be aware it saves all of your chatdata to train itself. For a little hint the longer your queries are the longer the responses and more unique the responses will be. It utilizes a variety of AI models in different contexts.

You can argue machine learning is the only way but machine learning is a lot like how we code. I also guarantee you cannot train a effective AI model on ROBLOX using machine learning. Mine is a combination of utilizing both, I utilize zero-shot reasoning in a very liberal matter, their is more than just zero-shot-reasoning as an option for determining the emotional tonality of a string as well as determining logic based determinations such as determining what constritutes a search query.

I have recently updated my chatmodule it has much work done in this field of research. I also did an update today that increases its performance by 33%. It has functions for extracting emotional tonality from strings and a very robust english search query function that leverages a table of thousands of words grouped into synonymous words and phrases that can replace each other in any context. This is diffferent than simple synonym grouping, this is for vernacular. You cannot get this kind of subtlty from the understanding of AI chatbots, it also has the ability to randomize the output with this massive table. It’s all available as an open source module here. It’s very powerful but it’s only part of my chatbot system. In the future I will be releasing the full multi-model architecture. You can see some of the code for my API calls in the bottom of this module. You may have to remove certain dependencies in this module But it will work fine for most of its functions besides displaystring.

1 Like
#63

instead of constructing strings and passing them to some random NLP you can just train ur model to understand vertexes and associate them with pre inputted variables but of course it won’t work well without thousands or million of pre inputted data of vertexes that represent random forms, and even then ur AI might have very bad accuracy at guessing forms or even associating them with objects
also a true NLP that could actually roleplay as a character while still having good accuracy will eat a minimum of 4-5 gb of ram (quantized model), and I’m sure u wouldn’t be able to run that on a roblox server anyways, and to have it online at some external website would require you to pay 2k$+ for a good server that would produce a nice tokens/s value
edit: another good way of doing this would be a basic LLM that will have the input generated by BLIP ( Bootstrapping Language-Image Pre-training for Unified Vision), problem is that you would need a render from the AI’s character and here I have no idea if you can extract it in any way outside of roblox to do image processing

1 Like
#65

This is very similar to the concept of BLIP ( Bootstrapping Language-Image Pre-training for Unified Vision), But instead its an algorithmic approach based on already labeled data. In that context the AI recognizes and labels the data for input to an LLM but in this context the data is already labeled and this is a set of conditions for the string construction of each observation. This is stored in an array and combined with its database to make relevant observations about things it has information about. Or provide more character flair and context the output string.
The end result is like reading a paragraph from a story book.
In conclusion you end up with a automated system that allows the AI to see by using text, similar to how you visualize a book as you read it. Deeper information can be constructed based on goals and searching the observations table with the goal, to find the goal in the 3D world, using only machine generated Text to an LLM.
If you’re going to use a AI model you can use Open AIs GPT-4 API or you can hug a open source model such as LLama 2. These models are very good at roleplaying and can make decisions based on personality motivations and input. My architecture can decode the commands by creating strings called by a table of keywords or table of phrases while also leveraging synonyms to have construct more robust synonymic understanding of english, to excute the commands of an LLM. Their are also a variety of free to use interference models on huggingface that you can also access via APIs.

The way I’m building this AI system has been from the ground up. I’ve also researched modern methods and discovered the AI system I’m creating is akin to the the best kind of AI that uses a combination of database querying and AI models to construct it’s output. This is similar to what Microsoft has been doing with Bing for the past few months. They also use algorithm constructed strings and context to communicate with an AI chatbot. For example “Hello, I am bing,” is the the starting string along with the character context “You are helpful and educational chatbot…etc”. They also use web-based search querying and summarization.
The main problem with modern AI is that researchers lack of understanding of the blackbox’s decision making process. Algorithmic approaches to the english language can improve outputs. For example mine does not repeat itself and has algorithms for removing repeating sentences in the case of a malfunctioning AI, it also randomizes the output with synoynmous phrases and words which allows the AI to create more varied output. It also has helper functions that can recognize worded algebraic expressions and solves them, which is like the code interpreter modern AI use to evaluate its math. It stores all of the contexts of different personalities and includes them in a blacklist for that session. Each personality has an emotional state generated by evaluating its databases with my getemotion function. As players interact with the AI this emotional state changes based on the players input and characters output.

#70

But in short, thats what this module is doing, basically generating the label data similar to what a vision module does. Except this one constructs it into a paragraph and is not using machine learning since that would be inefficient when I already had the architecture in my game. I’ve designed my game using algorithmic approaches as often as possible to reduce complexity and increase versatility and to create systems that allow for easy additions. So the all the label data has already been generated,