Network Streaming

Feature Requests Engine Features
#1

Currently the Roblox HTTP API is limited. The request method takes in a message, and a response is returned. While this works for many use cases it falls short when developers need:

  • Status updates or intermediate results from a server
  • Real-time external applications

For these use cases, developers need to have the Roblox game server listening for one or more updates from an external server. Essentially, streaming into and out of Roblox.

To get around this, developers must currently break messages into multiple HTTP requests and use a combination of polling and long-polling. Long-polling is difficult to enable in many servers, and most current APIs do not support it. Polling is inefficient (and quickly eats the HTTP request limit) and can cause performance issues on both Roblox and external servers. In short, there isn’t a good work around.

What is my use case?

I would like it for my navigation mesh generator so I can stream the map as it is generated, but I don’t want to maintain many large buffer files waiting for requests that may never arrive. Users would love to see the map being generated and streaming would be much easier for me to use. My use case is basically intermediate results and status updates.

You can learn more about my project on the Discord server (Polaris-Nav). Here is a picture of an example mesh on the default obstacle course which the Roblox pathfinder cannot generate a complete mesh for:

20211208_093523
20211208_0935231024×664 68.2 KB

What other use cases are possible?

  • The Lua Chat in the Roblox App could use it, if it wasn’t already implemented in the background and just made unavailable to developers. (real-time external application)
  • Leaderboards (real-time external application)
  • Matchmaking (status updates)
  • Permission systems (status updates)
  • Pathfinding (intermediate results)

Possible Implementations

Streaming can be enabled using HTTP 1.1 chunking or sockets. Allowing TCP socket access comes with security issues (users could easily write a HTTP implementation with insecure headers). UDP sockets are a great option, but is not reliable (good for some use cases, very bad for others). HTTP 1.1 chunking (which all HTTP 1.1 compliant clients should be implementing anyways) is an excellent choice. It doesn’t pose any security issues and is reliable.

No matter the underlying method, the API exposed to Lua should follow the standard every other networking library has:

  • A way to create a chunked request / socket
  • A method to send chunks / messages
  • A callback / event when messages are received (NOT as a result from a sent message)
  • A way to send the request or close the socket

Let me handle this for you

Think of it has a “trusted developer” role, a sort of middle ground between open-sourcing Roblox to submit pull requests and keeping Roblox closed-source so only the team with limited resources can add features.
Since:

  1. This feature would be very helpful for my business
  2. Many Roblox developers would love it
  3. Most features take a long time to implement on here, but I need it now
  4. I’ve worked at Roblox before
  5. I have extensive experience developing networking clusters and web servers

It would be beneficial for everyone if I was hired as a contractor / partner to develop this feature and submit it for review. I’ve submitted a “Senior Navigation Engineer” application, but please reach out to me directly in a DM if you don’t know who is handling my application and what the status is.

65 Likes
Reliability types for RemoteEvent
#2

Feature requests should focus on problems, not on proposed solutions or technical ways to solve an issue: How to post a Feature Request

You might want to re-title your post after the specific issue you are having, rather than a specific solution for your issue. There may be other ways to solve your problem rather than such a low-level solution that would require lots of custom implementation from the developer to solve the problem.

1 Like
#3

I would change the title, but it would break the existing links I’ve made to it. Enabling streaming enables events by controlling the timing, and bidirectional streaming is the definition of sockets, so I think the other use cases are okay lumped in here.

Besides HTTP chunking, TCP or UDP sockets would also fit the bill. They will require the same machinery I suggested and enable the same benefits. Sockets would be a dream come true for every developer, but chunking support seems more realistic due to security concerns.

I think the above is worth justifying HTTP chunking in the title, but I’ll add notes about the other ways to implement it.

2 Likes
#4

Slugs are ignored in Discourse URLs, it’s just for SEO. :slightly_smiling_face:

https://devforum.roblox.com/t/something-or-other/1580879

For that reason feel free to fix the title + layout of your post to be about the issue (your use cases, what can you not implement currently) rather than so prominently about your specific proposed solution, which Roblox may not be interested in implementing.

5 Likes
#5

I don’t understand this section of your post. Are you offering some sort of service for game developers on Roblox? For streaming to be implemented, it must be done at the engine level and then exposed through the Lua API.

2 Likes
#6

The post is addressed to Roblox employees. I’m available for Roblox to hire (perhaps by contract) to implement this feature and expose it via the Lua API.

I’ve edited the post for clarity.

2 Likes
#7

There are definitely numerous use cases for this, in my opinion roblox would ideally simply provide a way of opening raw tcp streams to any specified host/ip and port, without touching http.

As far as the security issues go that you mentioned, tcp wont be insecure in any way, and also wont present a new attack vector (the latter assumes roblox blocks local-net connections).

1 Like
#8

Allowing all TCP connections would open up security issues: most hosting platforms are responsible for ensuring that they are not hosting attacks. Allowing any outgoing TCP request would allow for:

Spoofing smtp emails (& other smtp headers)
Proxying cross origin requests (& other http headers)
Spamming emails
Port scanning
Many zero-day attacks
Attack surfaces on virtually every application

Since signing up and server hosting is free, the result would be a huge influx of malicious users and Roblox servers being blacklisted in many locations. Probably many lawsuits as well.

Mitigating all of this would be a huge effort. You can block specific ports but detecting when an attack is taking place is very difficult.

UDP has less issues, but they still exist.

HTTP chunking allows streaming, while ensuring that the application on Roblox is talking to a HTTP server, greatly narrowing the attack targets onto a surface that is generally harder to attack. It doesn’t pose additional issues over existing HTTP request, and implementing it should not requires any changes in networking configuration or architecture. It’s a small, secure software change that gets exposed to the Lua VM to a great benefit for the developer community.