Yeah, something like this. I understand that the less keywords added the better, so I don’t necessarily think it needs to be as advanced as TypeScript’s import system. But it could be something in the type expression context only, like type ExternalModuleNamespace = import(path.to.module), which lets you access that module’s namespace.
Personally, I think it would be less bloaty if you didn’t even import types with require statements, but instead required that you use an additional import type definition in addition to/instead of requiring the module you want types from, although that might be a little bit more verbose.
You should make a bug report for this as it could definitely be related. Roblox does a lot of changes to BuiltInPlugins (and StandaloneBuiltInPlugins which is used for a few plugins) behind the scenes, and these are almost never documented publicly.
Also, what is DataModel:GetEngineFeature? This sounds related to Fast Flags, but I’m not sure.
Hype!!! This probably improves the efficiency of my compression algorithm quite a bit as one of the biggest slowdowns is GC! Lastly, does this imply __gc could become available to developers again in the future?
I know what __gc is, and I know why it was restricted, and I heavily disagree that “there is no reason to have __gc in Lua.” We have access to create userdatas (through newproxy), and having __gc so we can reliably cleanup our complexly nested userdatas and tables would be amazing, not to mention its uses for diagnosing memory leaks. Think about it, if you could just ask your code to tell you when your data was GCed finding memory leaks would be immensely faster from a developer point of view as you’d never need to go out of your way with __mode, yielding, and watcher threads. While the function has no defined security context currently, if Roblox is making changes to its security right now, there is a potential for it to get its own context, or a potential for user code to be securely executed.
I don’t see how __gc would be useful for diagnosing memory leaks, other than seeing that you have a memory leak (which isn’t super helpful).
What would you need __gc to clean up? When the userdata is garbage collected, all weak references are removed, so using a weak keyed table should be good for storing any references which the userdata has to other values.
There’s nothing “reliable” about using __gc to do that either. Your code should continue to function correctly even if __gc takes an arbitrarily long time to get called on some objects, which makes it useless for actually managing anything game logic related.
Well, memory leaks can be extremely hard to locate in a lot of situations. You can easily tell one exists usually, but locating where the memory leak is occuring when you have hundreds of thousands of lines of code is a whole other issue. If you could just flip a switch and see exactly when different things are GCed that’d make diagnosing memory leaks much easier as we could know exactly when something is getting GCed. And I suppose yes, you can usually use __mode to achieve anything you’d be able to do with __gc, but I find it kind of difficult that I am not able to gaurantee a dereference when I know something has been GCed rather than relying on __mode to work as I think it should, plus the way you’d need to do this usually involves nesting tables in more tables so that you can isolate certain keys for __mode, which leads to messier code. If you want one keyed value to get dereferenced rather than a full table to have weak values, you just can’t do this without creating a subtable with its values weak.
Additionally, @tnavarts what I meant by “reliably” was more that there is a gaurantee that something has been cleaned up when you expected it to be cleaned up. The __mode metamethod has undefined behaviour. There’s no way to truly gaurantee something can actually be GCed, whereas with access to __gc you can be 100% sure your code is functioning as intended.
I have to chime in and say that __gc would be incredibly useful to diagnose Garbage Collection issues.
Being able to certify that a table is in fact being GC by linking this metamethod to a notification would be incredibly useful when testing complex scripts.
Put the things you want to inspect into a weak table and verify that the weak table isn’t growing unboundedly large. In fact, doing it that way is likely less error prone and more useful than using the __gc approach, since it also gives you a full record of exactly what objects have leaked to inspect for common usage patterns. You could even associate the objects in the weak table with the stack trace at their point of creation to help you trace down an issue.
I suppose this is another example of the “can you” vs “should you” situations. It’s similar to client security. Can you detect property changes and kick the player on the client? Yes. Should you? Probably not since it can easily be worked around. I don’t agree with doing anything clientside to stop exploits, but many people, even well known developers still have this opinion and I don’t really think that’s a bad thing, as client side security checks do offer a few benefits that server side security doesn’t do as well, especially for exploiters who don’t really know what they are doing. There are still situations where it can be useful, and people will always disagree on where the line should be drawn on what is considered to be “worth it”, but I think there’s no reason to make it impossible to do it one way as there are downsides and upsides to both.
I agree with the example you’ve provided here, __mode definitely offers some extremely useful information over __gc in this case, but like I had given as an example before, while you can still do anything you want with __mode that you can do with __gc, there are cases where using it involves creating extra nested tables, and its sometimes not intuitive to use it. I think that if properly used, __gc is very useful even if technically there are many downsides to __gc in most situations. It really depends on the situation, and it depends on what the developer is trying to solve in terms of garbage collection.
In fact, I don’t see why __gc can’t be used in conjunction with __mode to diagnose further issues. If __gc is called, the object can be placed into a weakly keyed table to ensure that the object is really getting gced or some other issue is coming up which is preventing this.
@zeuxcg In this case, I honestly see this as a good thing anyway even though I feel like __gc can be useful, as it means less lua threads need to be created, and less gc overhead which again, is currently one of the biggest slowdowns in my compression algorithm. If gc overhead could be for the most part mitigated, that could mean hundreds of ms of time shaved off of compressions/decompressions in my case as usually my code has thousands of objects to gc which massively cuts down performance according to what I’m seeing in the microprofiler.
I’m going to cover this a bit more in the next Luau recap, probably next week, but the short of it is - I agree there are some uses for __gc but the mechanism is simultaneously insecure for a few different reasons (it’s not just Lua-side privilege escalation, this work was in part motivated by a recent exploit), and hard to implement in a performant manner. So yeah the recommended mechanism (and the only one that Roblox has supported for a decade now) for observing GC behavior is to use weak tables.
Perhaps you should be requesting a compression API instead of __gc. Diagnosing memory leaks is already possible with weak tables and is generally done while debugging.
I’d prefer that the garbage collector doesn’t spend cycles checking my tables’ metatables for a __gc field.
The compression algorithm I was referring to wasn’t related to my use cases for __gc, but, I’d absolutely love a compression API. A feature request was already made to allow us to have access to the same compression used in rbxm files. To be honest, if we had this exposed, this would bring a whole new wave of rbxm processing imo, and it’d be huge for utilities like Rojo. I already shared support for the feature, but, it’s a few years old now so it may have been discarded for some reason or another, or maybe its just extremely low priority.
Additionally, the reason I have been developing my own algorithm for such a long time now is actually because it layers quite well with LZ4! In fact, I’d argue that my algorithm meshes so well when layered with LZ4 that its the perfect duo.
LZ4 is mainly based on repetition. My algorithm does little to no manipulation to bit structure, so not only can it be manipulated and stored directly in memory with a very low cost in a majority of cases, but it actually should end up improving LZ4 efficiency as the input size is significantly lower to begin with without negating most benefits of LZ4.
So, if LZ4 became exposed to developers, this would genuinely be one of the most important Roblox updates to me personally as it would give me a huge incentive for a professional release of my algorithm.
