Os.time() is returning really out of sync values in some game servers, values both in the past, and also in the future. We’ve tracked several cases of this. I have seen up to 5 minutes in the past, and 5 minutes in the future.
We backup players data when they leave, to our own server, and save the timestamp value from os.time() [Not when the request reaches our server. The value that is stored in the MySQL database is the value sent in the post request of os.time()]. I can go in servers and leave, and manually check how out of sync they are. They are in sync in the majority of our game servers, but occasionally, roughly 1% of the time, I will see servers that have values either in the past or future. It’s more common that these out of sync cases are returning values in the past. For instance, of our 5509 game servers that had out of sync time by at least 100 seconds, 762 returned values in the future and 4757 returned values in the past. (By anywhere from 100 seconds to 300 seconds in these cases)
When the server spins up, I’m comparing the value of os.time() with an NTP server, and if it’s significantly out of sync, I track an event - passing in the difference of the actual utc timestamp and the os.time() value. Again, these out of sync cases are where the timestamp is behind by up to 300 seconds, but other less frequent cases where the timestamp is ahead by 300 seconds.
I understand that they’re never guaranteed to be in sync - there always might be some slight delay due to latency, but I would certainly consider it a bug if it is returning values in the future, which is exactly what we are seeing.
Some example servers where we saw os.time() out of sync:
I am not checking on the client. I am referring to calling os.time() in server scripts - sorry if that was unclear. It differs by a matter of minutes, and not hours, and as I mentioned this is only happening in a very small percentage of our servers.
I did notice that we are starting to see servers hosted from different parts of the world now so its possible that this new adjustment is causing the offset. At any rate, its completely understandable that this is an issue.
A temporary fix could be to use messaging service to set a constant time value across all servers. This would create a better sync than what os.time() seems to be giving you, since MSG service is supposed to be “instantaneous”.
FYI it is much better practice to have the MySQL database generate the timestamp in this situation, otherwise you’re mixing timestamps from two sources of truths which you have found will lead to trouble very quickly and also doesn’t make a whole ton of sense. You can specify a column in the schema that automatically is set to the current timestamp upon insert/update.
What you’re saying makes a lot of sense - however there is a specific reason I didn’t use the timestamp as the entry was created or written in my database - it is because I was also saving the same os.time() value to Roblox’s datastores, so I would have needed to ping my server to fetch the timestamp for datastores. So instead I saved the os.time() value to both my external server and roblox’s datastores. The problem became when there were roblox clocks out of sync - more specifically in the case where some would return timestamps in the future. We stopped using os.time for data related stuff a long time ago, but I’m more just bringing this to light, specifically because I would certainly consider the servers returning future time values a bug, and also because this post mentioned that it had been fixed previously: https://devforum.roblox.com/t/os-time-now-better-aligned-across-game-servers/2631
Thank you for bringing this to our attention. Our operations team is investigating some of the reported servers and can confirm that there is clock skew, and they are investigating why our existing mechanism to prevent clock skew didn’t catch this (they are working on the fix).
The operations team is making progress, they have a reasonably good handle on the underlying issue, and they are planning to have this resolved within a week from today (assuming no additional unknown issues crop up).
Awesome! I did notice only a couple now servers reporting out of sync clocks every hour on my end since the 28th, as opposed to the 30-40 average it was at before. Thank you guys so much! I can continue tracking this in the future and let you know if I see anything.
Correct, things are as resolved as they are going to be. The operations team has identified a few “root” causes for this issue and have taken steps to fix them & added some protections to help going forward. As of the last time they did a sweep of the farm the max observed skew was 14 seconds, but I think that sweep was done around the time of my last message (end of August).
Hey @LordRugdumph it looks like this issue has returned. I’ve observed several servers with os.time() out of sync in both the future and in the past. Here are some job ids:
Ahead by 237 seconds: 66b4b563-da56-372a-e2d0-8763eb1dcc99
Ahead by 109 seconds: 6f0bccd4-db79-b417-45d8-8382694409af
Ahead by 238 seconds: ba92d608-c3fd-b818-bda5-4479ee6a749c
Ahead by 237 seconds: 66b4b563-da56-372a-e2d0-8763eb1dcc99
Behind by 86 seconds: 2978479d-ff42-737c-0b23-2cdf3d925ffc
Behind by 52 seconds: 24c22f0e-0d7d-d3bd-e358-b896ec665603
Behind by 86 seconds: 696c0a4e-766c-2665-f95e-bab91f00c0c3
Behind by 86 seconds: b8b01f64-d622-e63d-868a-5e2aa0a4943b
