So I’m just learning about HTTPService and I wanted to try it out. I’ve gotten the hang of it. But I’m running into a problem where on certain URLs i’m unable to use JSONDecode because I run into the error (“Can’t parse json”)
I’ve looked online and while other people have had the same error as me they didn’t have the same problem and their solutions didn’t work for me.
Right now i’m trying to send a http get request to twitter (now known as X) as a test. Then I use JSONDecode to convert the JSON string into a luau table. And I get the error saying cant parse json.
I tried reddit and reddit works completely fine but certain websites like X doesn’t.
I’ve tried things like using JSON Encode to convert the response into a json string (even though its already a string) and then using json decode. Problem is that it doesn’t turn it into a luau table so that I can easily extract the data I want from the website using luau dictionary keys.
And just using the response body is too difficult since it’s a huge amount of characters and again its hard to get the data I want from that string.
Here’s my code, this isnt for production so that’s why I haven’t wrapped stuff in pcalls:
local HTTPService = game:GetService("HttpService")
local URL = "https://x.com/home.json"
local Response = HTTPService:GetAsync(URL)
local Data = HTTPService:JSONDecode(Response) -- Can't parse JSON error here
print(Data)
So i’m still new to HTTPService so correct me if i’m wrong but I believe the problem is that luau is basically saying it cant convert the JSON string into a luau table. I guess this might be due to the way twitter’s website is coded? Compared to reddit?
I would really like to avoid having to use a JSON string instead which will be much more difficult
Ok update basically I decided to use a different URL instead of x.com/home because that’s different for every client and if you’re not logged in it sends u to the login page.
I tried using https://x.com/Roblox_RTC but yeah the same error happens whether I added .json at the end or not.
When making a GET request to that url without authentication and with the default headers, you get an error HTML page response “this page doesn’t exist”. However this is expected from the webpage so the status code is 200 (meaning Roblox wont raise an error). Obviously if you try to treat an html page response as a JSON object the JSONDecode function will error because Response isn’t valid json (it doesn’t follow the json format).
Please check if the url you’re trying to call needs additional header information (such as content type, authentication cookie, etc) and if the method type is a GET request or not (for example if it says POST you should use PostAsync instead).
Personally when I test url endpoints I follow an “add all then remove” method. Basically I add in all the cookies, all the headers, all url parameters, everything to make it identical with the default request the browser makes. Then I slowly removes pieces of information and only keep the headers, credentials and parameters that are needed for the request.
A good tool to figure out how the browser fetches information is the good old inspect element. There you can see if information is loaded within the HTML page itself or it’s loaded afterwards and then dynamically appended to the webpage. For example Roblox in particular loads default webpages and then attaches the information to them (through API calls). In a situation like that you don’t need to fetch the webpage, but the endpoints that deliver the information.
Most websites don’t serve JSON files. You need to test with a URL that actually serves you a JSON formatted file. And if you’re targeting a site that does actually serve JSON, just because you can see it in your browser doesn’t mean the Roblox server will get served the same file. There are cookies, authorizations, etc. that determine what response any particular requestor gets. You can dump the actual response to the Output window to see what you get. But yeah, the Roblox server is not logged into X.com as a valid user.
If you try with a url that actually returns json, if you put that url in your browser, you should get something like this
Url: http://api.open-notify.org/astros.json
This is how it looks in firefox, firefox sees that the data is json, and has a special format to display json (instead of trying to display a webpage). It may look different in another browser. If you get the raw data, you get the json string, that JsonDecode() is meant to parse
That’s the problem, I want the json data from twitter. So that I can convert it into a luau table and then easily get and use the data I want.
I think due to privacy reasons twitter is preventing me from getting json data. Is there any way possible for me to get that data I want from twitter specifically with http service or no?
Yeah I realised later using a json validator that it is a JSON object but an invalid one. Sadly I can only gather that invalid information from the url.
I did try using request async though, because as the documentation says getasync is good for a quick simple retrieval but request sync has more options.
Im going to play around with the parameters a bit more when I get up. But I already tried using requestasync with just the url and no other options like headers or methods and I received a luau table.
While it is a luau table it was still kind of unusable information, not a table in the way you’d want it to be..
If I play around with the parameters more I think I could get what I want. I’ll let everyone know.
Check this to figure out which endpoint you have to use for your desired data. Twitter API requires an auth key so you would need to setup an account and all.
Yeah no if you want to use X for API requests you need to have at the very least their Bearer token, an x-guest-token and their x-client-transaction-id. Possibly a valid user-agent as well. Alternatively you can use your own login Cookie but they have an annoying rate limit that you will quickly reach. That’s why I’m always logged off when I bot their site.
x-client-transaction-id is a bit trickier to add per request since it’s constructed on their website per request but there’s a few Github pages you can use that have reverse engineered it.
Alternatively make a few get requests and collect your own x-client-transaction-id’s that end with the letter A, remove that letter and you can append A through P to them and cycle through them. They are rate limited as well but you can make about 3 or 4 requests per x-client-transaction-id per minute or so.
Twitter is not a good website to play with their api, i think they’ve made it paid to prevent ai being trained on their data (unless that was reddit, or both are doing that now?)
Other websites have free apis, roblox’s own api is free (although a good portion you can’t access from Roblox servers). Free apis are (or were) the norm. They are also usually easier to deal with if they don’t have the access tokens and authorization stuff
I am assuming you are trying to get like your most recent twitter post or something?
