A calculator programmed correctly is inherently true as it follows the basic rules of arithmetic; a LLM has truth as only one of its objectives (and probably the most challenging one to get right).
Using ChatGPT for information is really the equivalent of having a fancy display for a search engine - it forms answers based on the Internet results it was trained on, however it is unable to see that these results are often inaccurate or outdated when asked something as technical and niche as Roblox development.
The most important thing here is that ChatGPT is not a developer. It has no real experience with Roblox, and all it can do is “mush” together answers from the Internet without critically evaluating their accuracy. Likewise, it not only gets things wrong, it lies confidently and will just invent facts. That kind of deceptive behaviour would be grounds for removal from this community, but for OpenAI this is not uncommon.
Example of ChatGPT being dangerously wrong about Roblox
For context, I just logged into https://chat.openai.com and asked it the following question:
On Roblox, should I use a HopperBin when I’m developing tools?
The correct answer to this is no. HopperBins are extremely old and very deprecated. The Tool instance should always be used instead.
Here’s what ChatGPT said:
Yes, you should use a HopperBin when developing tools in Roblox. HopperBins are a useful tool in Roblox that allow players to quickly and easily equip and unequip tools, without having to go through the Inventory menu.
Using a HopperBin also makes it easier for you to test and iterate on your tool, as you can quickly equip and unequip it to see how it performs. This can save you a lot of time and make the development process more efficient.
To create a HopperBin, you can go to the Explorer window in Roblox Studio, select the Workspace, and then right-click and choose “Insert Object”. From there, select “HopperBin” and give it a name.
Once you’ve created your HopperBin, you can add your tool to it by dragging it from the Explorer window and dropping it onto the HopperBin. You can then test your tool by selecting the HopperBin in-game and equipping the tool from there.
For example, if I asked you if I should use a deprecated HopperBin for tools on Roblox, then you’d probably say no (or that you don’t know), but ChatGPT may say yes and then lie about the reasoning.
It also is known to just invent facts and references when it can’t find relevant information, which is called “AI hallucination”. Here is some interesting discussion and reading about that:
According to that Datanami article, ChatGPT gets around 20% of its answers factually wrong.
Here’s a good rule of thumb about when you should use ChatGPT:
(from https://twitter.com/kareem_carr/status/1619743014813052930)
To cut to the chase, ChatGPT may look fantastic, but it’s nowhere near ready to be relied on to help aspiring game developers with problems that they’re seeking help with.
If users want to ask ChatGPT a question, they should consent to this themselves and they can do so freely on the OpenAI website.
