Using this library to make vector database of a large dataset is very easy but it can take a while to process the data
The tables where then cosntructed with this function
function index.GetVectorData(key)
if not book then
book=require(script.FantasyDialogue)
end
if not keydata then
keydata=require(script.VectorDatabase)
end
local returndata={}
if keydata[key] then
for i,v in keydata[key] do
for t,o in v do
table.insert(returndata,book[i][o])
end
end
end
return returndata
end
But 3000 entries is a lot so here is a sampling function. Which picks 300 random numbers without repeating
-- Define the function
function pick_random_numbers(min, max)
-- Check if the input is valid
if min > max or max - min < 99 then
return nil, "Invalid range"
end
-- Create an array of numbers from min to max
local numbers = {}
for i = min, max do
table.insert(numbers, i)
end
-- Shuffle the array using Fisher-Yates algorithm[^1^][1]
for i = #numbers, 2, -1 do
local j = math.random(i)
numbers[i], numbers[j] = numbers[j], numbers[i]
end
--print(numbers)
-- Return the first 100 elements of the shuffled array
return table.unpack(numbers, 1, 300)
end
This allows us to sample the data and decrease the overhead of querying 3000 entries.
function index.GetVectorData(key)
if not book then
book=require(script.FantasyDialogue)
end
if not keydata then
keydata=require(script.VectorDatabase)
end
local returndata={}
if keydata[key] then
for i,v in keydata[key] do
for t,o in v do
table.insert(returndata,book[i][o])
end
end
end
local result = {pick_random_numbers(1, (#returndata))}
-- print(emotiond[emotion])
local quantizeddata={}
for o,i in result do
table.insert(quantizeddata,returndata[o])
end
return quantizeddata
end
Finally here is an example of how a context database would look constructed by this function.
It is utilizing the sampling function to get 100 random entries from the table without repeating.
The entire array is 3000 entries. So this is a small representation of the data. I could quantitfy it further but for each response I indexed all of the responses that passed the minimum threshold. instead of just getting the best match. I could quantify the function further by just getting respone #1 which was ordered upon generation from most accurate to least.
["What are you feeling?"]= {
[1] = "See that Wilred gets the send-off he deserves.",
[2] = "See that big red cross east of Sullen? That's where the treasure's buried! What do you think it is?",
[3] = "See for yourself.",
[4] = "Fear notI will be sure to intervene should you begin to fall under its spell! I would never leave a familiar to such a fate, and especially not a familiar created by Azem!",
[5] = "Fear not, my fellow inspector, for I am a master of disguise! Have you any information regarding the duelist himself? A description?",
[6] = "Fear not, Forename! I will be standing watch here, ready to leap to your defense should either you or Tsubame come to harm!",
[7] = "Fear and anxiety are beginning to take their toll upon the citizenry, Forename. For their sake, I ask that you aid us in this investigation. ",
[8] = "Ask and ye shall receive! We can ambush that wizened dragoon, take him hostage, and demand an exchange!",
[9] = "Stubborn brutes...",
[10] = "Stubborn as ever, I see. No matter. In due time you will be mine... One way or another.",
[11] = "Seein' as how you're on good terms with the goblins' leaders, do you think you might have a word with Slowfix for me about me becomin' an official citizen of Idyllshire?",
[12] = "Sai-Lewq himself has some training as a warrior. He would guard her as if she were a princess, and he her gallant knight... I'm sorry, it's just... ",
[13] = "Farewell, Forename. When next we meet, I look forward to fighting with you on the same side of the battlefield.",
[14] = "Farewell, brother. May Rhalgr watch over you.",
[15] = "Farewell, [Name], and thank you for everything!",
[16] = "Zenos yae Galvus",
[17] = "Weren't jus' somethin' as simple as havin' a piece o' steel ye've grown accustomed to. Those weapons were like extensions o' the Braves' bodies.",
[18] = "Unfortunately, some of the residents did not take too kindly to our presence. They pelted us with clumps of a strange-smelling powder and we were forced to run! Terrible business, I tell you.",
[19] = "Unfortunately, no. Unlike weapons, there are no substitutes for proper protection...",
[20] = "Unfortunately, a number of the Blessed were poisoned while fending off the soldiers.",
[21] = "Unfortunately the answer is no, I have not seen the Lightwarden. I patrol the ruins' perimeter regularly, and if it were hiding nearby, I would know about it.",
[22] = "Less hearteningly, practical trials of transference methods suited for use with the vessel have yet to yield satisfactory results...to put it mildly.",
[23] = "Speaking of the captain, wherever did he run off to? I must deliver my gift before E'lahmui does, lest she be the one invited to work the lever tonight...",
[24] = "Speaking of personal, I mustn't keep you long, lest the king think I'm trying to steal away their precious sapling.",
[25] = "Speaking of different ways, have you paid your respects to Singing Stormcloud? She's one to appreciate a ceremony or three, if you know what I mean.",
[26] = "Speaking of commissions, I think you've proven yourself sufficiently skilled to handle orders without my supervision.",
[27] = "Speaking of bloodI have a proposal for the next stage of the expedition.",
[28] = "Speaking for myself, what I wear now better suits my tastes, even if it is of inferior make.",
[29] = "Splendid. I require several of the reddish soma herbs which grow among the shrubbery to the northeast. With both of us working in tandem, I wager we should have enough gathered before the next bell.",
[30] = "Splendid, splendid! We shall examine those wear patterns and make the necessary adjustments to ensure that your perfected tool is as perfect as can be.",
[31] = "Splendid! Then let us first make for Ser Vaindreau's Grace. I think that you, too, will be keen to lay your eyes on a location which I believe to be the pride of the Firmament.",
[32] = "Splendid! Fortunately, I have already ascertained a suitable location, not far from here, where the view of the One River should provide the perfect backdrop.",
[33] = "Splendid! Before we begin, however, you must gather the spices we will use with your own hands. It's the only way to attune the tea to your destiny.",
[34] = "Splendid! Absolutely Splendid! Thank you ever so much.",
[35] = "Splendid specimens of horsebirdflesh, would you not agree? In the hands of a skilled jockey, any one of them could take first place at the Saucer.",
[36] = "Halone above... Magnificent, simply magnificent! If you will allow me to transfer the remains of Saint Finnea...",
[37] = "Precisely, Master Gerolt. Have you any advice to give us regarding this endeavor?",
[38] = "Precisely! See? I knew I was onto something here!",
[39] = "Precisely where and how to allocate the necessary funds is, of course, another question...",
[40] = "Precisely how this project has continued despite such turmoil, and under whose auspices, remains a mystery.",
[41] = "Precisely how far beyond compare had his skills become, I simply had to seenay, test for myself. So I made my way back to Ul'dah.",
[42] = "Fifteen alligator pears, and just in the nick of time! You have spared us Miounne's wrath.",
[43] = "Part o' the deal was we didn't ask no questions. Even the merchant didn't know her real name. It was odd, aye, but we needed the coin an' I didn't rightly care either way.",
[44] = "Know that anything less than perfection will be deemed unacceptable.",
[45] = "Know that 'tis for no other purpose but to neutralize the tower that we have journeyed to this land.",
[46] = "Got her hackles up, that one. My ma always said, every dog needs a little pettin', a little treatin'. A bit o' salt beef from a new friend? Might do, might do. How about it, sweet thing, are ye up for talking to her?",
[47] = "Got all that, lad? Seein' to them three tasks'll take ye on a little jaunt 'round the city, an' 'elp ye work out where everythin' is. Now, I've best be gettin' on. Run along, eh?",
[48] = "Got a plan, do you? Far be it from me to doubt the Warrior of Darkness. Show us what you've got in mind, and we'll back you up to the best of our ability.",
[49] = "An' look at that serpent head! It's a symbol o' good fortune over here, ain't it!? Not to mention how such an original design'll catch the eye of yer prospective employers!",
[50] = "An' I think ye'll find that Lambard gets his orders from the imperial base in northern Thanalan. Castrum Meridianum. That's yer best bet, I reckon.",
[51] = "An' here's where we part ways, son.",
[52] = "An' here you go! I reckon this'll serve you well enough.",
[53] = "An' he's never goin' to survive duty at the bridge now that work on the Victory has a flood o' deliveries comin' in an' out every bloody day. Wyrstmaga better harden up, or start lookin' for a quieter line o' work.",
[54] = "The visiting samurai is in front of the Quicksand. You'll recognize her by her dignified air.",
[55] = "The tempered that we were able to treat will be taken to the citystates for observation. As soon as the Alliance is satisfied that they're fully recovered, they will be free to return home.",
[56] = "The technological secrets which were stolen were contained in a similar crystal. This particular sample has been loaned to me by a healing institute in Sharlayan.",
[57] = "The tale begins in Onokoro. I trust you will escort me there?",
[58] = "The table is set! And so I bid you raise a glass with me, that together we might toast your spectacular accomplishment!",
[59] = "The sylphs slain in Thanalan... Gods, ye were one of 'em!",
[60] = "The survivors speak of an army of thralls loose in the facility, and we must assume that these followers continue their efforts to awaken the eikon.",
[61] = "The surest way to limit casualties is to reach our destination without drawing attention to ourselves. Perhaps afterwards we can try again to convince Jullus that the tempered can be saved...",
[62] = "The supplies that your men recovered have been prepared for transport, in accordance with your wishes.",
[63] = "The sun sets, and the port is cloaked in shadow. If there were a perfect time to strike...",
[64] = "The Sun is disinclined to entertain you at length. Speak your purpose in coming.",
[65] = "By thine account, the Fourteenth seat of the Convocation was known as Azem. At the title's distinctly familiar ring, mine imagination could not fail but run wild.",
[66] = "By the Twelve... Did the beast just eat that island!?",
[67] = "By the time we realized we were overmatched, it was too late. I saw Lanbyrd fall, and when the eater set her sights on the others, I fled.",
[68] = "By the time that spinning wheel feels like it belongs in your hand, I daresay I'll have thought of a new task for you. Until then, my boy.",
[69] = "By the time I realized how low he had sunk through his lust for gold, it was too late. If I had only paid more attention... If I hadn't been so absorbed in my own studies... I owe it to him to help him in whatever way I can!",
[70] = "By the same token, however, this means we will receive no support from the dragoons. We fight this battle alone.",
[71] = "By the same individualor individualsthat released its captives and set them upon you, no doubt. Such knowledge avails us little, however, if we cannot ascertain their identity.",
[72] = "By the onion, you're Forename! You couldn't have come at a better time. A friend of mine sent word that he needs a capable adventurer, you see, and I naturally thought of you.",
[73] = "By the Navigator, Forename Surname! Tales of your heroism have been an inspiration to us all. It is truly an honor to fight at your side.",
[74] = "By the looks of you, you are more inclined to adventuring than the average person, yes? Apologies for the imposition, but could you tell me if you've come across a creature such as the one before me on your journeys?",
[75] = "By the looks of it you have gone to great lengths to ensure my safety upon arriving on the island. Time and time again you impress me with your resolve to provide the best of services to your customers. I must say the future between your organization and ours is bright.",
[76] = "Cor, look at all them halberds! I've never seen such a well-armed building before!",
[77] = "Cor, it's cold out here!",
[78] = "Cor, it's been too long since I've had a chance to whip up a batch of Nashu's Delights. You can be sure I'll be putting these little darlings to good use next time!",
[79] = "Cor! Inspector Hildibrand, you look amazing!",
[80] = "Then...we are seeing the same view.",
[81] = "Then...unlike the worlds that have been rejoined, mine still...",
[82] = "Truth be told, I did not expect this investigation to become so involved. Of course, I would like to resolve matters here without resorting to outside help, but...",
[83] = "Truth be told, Estinien's tactless observations have saved me from disaster more than once, and I can well understand Alphinaud's affection for him. He is a friend for whom I would gladly",
[84] = "Mine oath is unbending as steel, little goddess! Thy ruse will but add to mine enjoyment!",
[85] = "Mine apologies for the delay. Minfilia and I had a private matter to discuss.",
[86] = "Search high, search low~ ",
[87] = "Man and beastman, united by an undying devotion to the denizens of the depthsis that water in my eye, or am I welling up? In any event, we've effected a change on the oceans today, a change that will, in turn, come to have consequences for generations to come.",
[88] = "Pray ask Lord Haurchefant to direct us to the meeting place when you are ready.",
[89] = "Pray allow me to accompany thee on thy reconnaissance. An arcane perspective may prove needful.",
[90] = "Pray accompany me back inside, Forename.",
[91] = "Pray accept my humble verse into your heart. Songs give the listener fresh eyes to see, and mayhap mine own will allow you to look upon your deeds anew...",
[92] = "Pray accept my apologies, Neddard. You turned to us for help, and yet your grandson now lies before you, broken and battered...",
[93] = "Something on your mind, Forename?",
[94] = "Something must have given them cause to venture this far. More than like, the missing sylph elder's among them. Here, I've marked the tract where they were sighted on your map. Go see if there's anything to be seen.",
[95] = "Something is playing havoc with the air currents. Such a disturbance bespeaks corruption.",
[96] = "Something definitely happened here. I see several people who look like they might be injured. Let's give them a hand and find out what we missed.",
[97] = "Something comes! Be careful!",
[98] = "Likewise! Long voyage notwithstanding, you all seem none the worse for wear.",
[99] = "Saying that we're all family, Lady Ameliance shares the contents of her children's letters, you see.",
[100] = "Dear gods... These wounds were not intended to kill, but to torture and maim... Does their savagery know no bounds!? Enemies or no, they go too far!"
}
The output of this model for each query is based on score, emotion score, word count, weight and context weight, and repetition it can be represented like this [0,0,0,0,0,0] which is 6 dimensions of features. These features are measured in the input. and used to score the entries in the table. Sometimes if the context node allows the response from the node is weighed as 1/6 of the input and added to the score of the response to give reward for relevancy. It also weighs higher strings with an emotion matching the previous response higher.
