Hello Everyone! How is everyone doing? Good? Well, let me just get right into this; About a week ago the Administration Team and I became aware of the extreme impact the Refined Storage mod was having on our server infrastructure. This is specifically apparent on our ATM8 (All of the Mods 8) server. Below I will go into the finer details of everything but keep in mind this ONLY applies to our 1.19 servers. What is Refined Storage? Chances are if you need to read this section this post doesn't apply to you. However, for the sake of common knowledge, and to make sure everyone is on the same page I will explain a little bit. Refined Storage is a network-based storage system modeled on the same concept as Applied Energistics 2. It allows you to store mass amounts of items in a compact and easily accessible way, and then utilize that storage in ways ranging from auto-crafting to stock-keeping and import/export. However, compared to Applied Energistics 2, Refined Storage has a long history of not being "Server Friendly". This brings us to our current issue where its problematic nature is, frankly, not easily compatible with the most modern server software. History and Reasoning Putting Refined Storage's long history of not playing well on servers aside, we still wanted to give this mod a try on our servers. Sadly, this wasn't fruitful. This became apparent when I copied over our current production All of the Mods 8 server to our development playground to diagnose the very long boot time the issue with the server. How long you say? Well, to the point where our server wrapper we use thinks it has crashed because it is taking so long reading a file. Which file you may ask? The file that takes a long time reading is called refinedstorage_nodes.dat in the world's data folder. From what I understand, digging into that file, it holds all of the patterns in all of the Refined Storage crafters in the world. Once I deleted that file, the server boot time what cut by 2/3rds. This is pretty significant on it's own. All of this probably wouldn't be as bad if ATM8 didn't have the mod "Extra Storage". This is why we didn't really see this issue on our other servers as the Extra Storage mod adds the tiered crafters. You know, the Netherite Crafters that store something like 100 patterns? This alone accelerated the file's growth, allowing us to easily spot the disaster waiting to happen on every 1.19 server currently running Refined Storage. There are also other issues related to refined storage that have been adding up. For example, we have seen several players having their patterns disappear in their crafters altogether. I don't know the exact cause of this, but I strongly believe that this file could be resetting certain parts of its data on server load because of how long the file itself takes to load. This is only a theory because as it is very difficult to replicate, we haven't had many chances to test and diagnose the issue. There is also an issue with Refined Storage that when you have a lot of items in your system, that clients suffer from big FPS drops and client lag. As time goes on, I believe the already extreme impact of Refined Storage is only going to get much worse. As that file grows larger, and as other symptoms of its effects pile up, we will start seeing player progress loss, and even longer load times. These long load times also negatively impact other services and infrastructure. Without having a smooth startup, the servers can expect even more instability during runtime. This means even more bugs, and worse, even more crashes. Hence the decision made here as of this announcement. The staff team and I have talked about our options on what we should do going forward. We ultimately made the decision to disable the Refined Storage mod in its entirety. While announcing this doesn't come lightly, we strongly believe this is the correct decision for the health of our network and the experience of our player-base. Affected Servers Direwolf20 1.19 All of the Mods 8 FTB Skies The Removal Process On July 1st, 2023 we will be disabling all crafting recipes for the Refined Storage and any Refined Storage mod add-ons. We will not be deleting the mod's items from the world or removing anything from your inventories. We will however be deleting this problematic file from the servers over the next couple of days. This means all patterns from any Refined Storage crafters will be deleted. The point of starting with only this step is to slowly remove this mod from our 1.19 servers without causing players to lose items held inside the Refined Storage system while at the same time addressing the immediate threat. During these stages, we suggest players transition to other storage alternatives. Moving forward we will also remove any quests on all packs having to do with Refined Storage. For now, this will only be done on 1.19 packs, but we will inform players if it impacts any other servers. Future Servers with Only Refined Storage In the event that a modpack only has this storage solution, we obviously WILL NOT disable this mod. In the event that this ever happens on a modpack, we will inform you all how we set this server up when the time comes. The Alternative With this all happening we are aware those of you who rely on Refined Storage might be left scrambling for storage solutions. The most obvious alternative is easily the Applied Energistics 2 mod. The AE2 mod offers the same functionality as its Refined Storage counterpart, without the extremely negative costs to server performance. Unlike Refined Storage, the basic AE2 storage cells are restricted in the number of unique item types a single storage cell can hold. However, the Deep Item Storage DisKs can be utilized to mimic this Refined Storage feature. This offers you a balance of both functionalities. Overall, while we don't like to disabling entire mods like this as we want our players to use what a pack has available, but this is all for the sake of keeping our server stable for the community. If you have any questions, please feel free to ask and we will do our best to answer them. -MyM Admin Team
I have to wonder is this because of the unlimited connections that refined storage networks are capable, where AE2 has limits, RS has no cable limits? Or is it more of an auto-crafting thing? Like on exporters (crafting upgrades)? Not that AE2 isn't better or anything, which it is, just kind of leaves a hole in my heart seeing as I grew fond of RS for quick setups that didn't require any planning when it came to expanding it. Sad to see it go. That said, I do notice the huge lags on ATM8. **quickly runs to convert all RS stuff to AE2**
From my tests, it is more on how the mod stores it's data to the world. So it can be a combination of these.
Reminder that today will be the last day before some RS files will be removed, which from understanding could lead to your RS network to stop functioning properly or even stop working altogether. If you haven't migrated yet, I advise you to do this as soon as possible. For those that still need to migrate from RS and play on ATM8: I created 'AE2 kits' ready to be given if needed. This is a basic setup with plenty of storage and a small autocrafting starting point. Contact me anywhere if you want to receive said 'kit' in ATM8.
Hey Everyone! I have removed ALL refined storage stuff, I ended up converting it all to AE2, not as painful as you might think since most things I was able to just hot-swap and duplicate in some way. Seems that AE2 has caught up a bit since the last time I used it, so it wasn't as bad as I thought it would be. Plus it seems to be faster anyway. The only thing I had to change was planning for channel usage, which I really didn't have to do since the controller was central anyway (by habit from using AE so much in the past) FTB Skies was especially easy because the storage drives are essentially the same thanks to that one mod that changes things to 1:1 like RS does. So you won't be having to make a ton of drives like you might think (not limited to 63 item types per drive). That said, I still suggest against putting non-stackable items in the network, and of course never put other storage devices in the network like backpacks or danks, or anything with a large NBT. You can use functional storage armory cabinets for those if you need to be 100% digital (be sure to set priority!). Stuck Items I haven't ran into this issue for some time, but I'll put it out there anyway: When transferring your items to your ME, and you encounter items that are "stuck" in your RS system, you can use this method to get them out: 1. Make a network with a controller, and a chest, with a grid (for verification), disk manipulator, and an external storage/storage bus on the chest. Make sure the controller has power, and that all components are connected by cables or touching one another. 2. Put your drive that has the stuck items into the Disk manipulator so it is exported to the new network. 3. Verify with the terminal that the offending items have been transferred. 4. The stuck items should be exported to the chest where you can easily get them out. (This works on AE2 as well, in case you accidently put a large NBT item into the network and cant get it back out AE2: Storage IO = Disk Manipulator, Terminal = Grid, Storage Bus = External Storage... the rest is pretty much the same.)
The worst part of this is that I have to basically restructure my entire base, since a lot of it won't be compatible with the limited channels. RIP to my seed network, that only required external storage and exporters.
Is using an Item Pipe (no need for a pipe upgrade) from Pipez into an Ender Chest not an option? I have 1 AE2 import bus on an Ender Chest and that can handle all of my farms. Message me when you're online and I'll see if I can help you out
Han came on a couple weeks ago and mentioned potentially having to disable pipez. so the last thing i want is to redo the entire setup, only to then have it completely deleted yet again. it's not that i have a lot going on, it's just large scale and slowly growing so I have to keep lag in scale in mind
