Implemented Timeouts

probably finally solved. I have no idea why I haven't thought about this earlier, but somehow I didn't think about modsauce and galaxy being on the same machine. Well checking all the root server config adjustments I discovered one missing. The backups (rsync) where set on -c1 which means real-time. Now they are on -c3 (idle) like on every other machine. I also discovered that the servers have been given room for up to 12gb ram which is too much and causes garbage collection performance issues. Depending on the modpack and player count they are now between 4 and 8gb.
I hope that this did solve the problem permanently. Sorry that I haven't thought about this before, I guess the fact that no reports from bteam came (which was on the same box as galaxy) did let me assume it is not related to the whole box.

 

you tell me tomorrow

 

I guess we got to it at the same time. I didn't read it yet as this was the first thing I did today for the server.
Sitting there with the visual vm, atop (as root), server console and the game open at the same time and just watching how things go. (non root accounts don't have access about disk io analytics which is why I didn't discover this earlier)

If this did solve the issue it is caused by a long list of things happening at the same time:

Putting disk usage at around 100%

• Backup running of 2 servers at the same time or daily backup which deletes old data when the server is + 12 days old
• Autosave of another server

Letting the server wait until finished (does normally only cause micro to short lag spikes, but with 100% disk usage and the higher priority of the backup task, it can take ages)

• World is getting loaded or unloaded
• Player login -> force chunk load

Depending on direct chunk unloads due to client connections timing out this can be a vicious circle on player login -> until the disk usage drops

 

please go ahead and let us know. We don't know anything for certain.

 

This is quite unlikely. I did some further changes, lets wait another 24 hours and see if something changed. Than we can look further. (It would be possible to disable this/these plugins but also impact other stuff like homes, spawn, warps..)

 

From what I know the issue is not reproducible this way. Even with a lot of players online there are episodes where the server runs fine for +1 hour and if we know consider that during such a day only a small part of the players would be online..
The thread dumps show that the server is waiting for chunk loading and saving and the disk io of this system is still quite high, compared to the other boxes. From profiling most of it is coming from galaxy than modsauce and as last bteam. As bteam was the most heavy one in terms of disk io of the 1.6.4 packs the new ones do top it. I still assume that it is related to world load/unload and player force chunkloads through player logins/fast traveling or mod messing around with chunk-loading.

 

Please keep on open eye from now until the same time of tomorrow. If you notice any differences let me know, if not, too.