1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

The Sunday Outage

Discussion in 'Announcements' started by Slind, Mar 24, 2015.

Thread Status:
Not open for further replies.
  1. Slind

    Slind Founder

    Messages:
    8,332
    Likes Received:
    3,018
    Local Time:
    1:35 AM
    The Sunday Outage

    On Sunday (22th March) the network had been offline for an entire day as most of you probably noticed. This is announcement is meant to clear out the dust and give you a in depth explanation, to which we didn't have time before and Staff wasn't allowed to talk about to avoid miss information.

    What was the reason for the downtime?
    A DDoS attack was sent against our main server with the global database withholding under many things the Permission-System. Without the permission data the servers them self are not operational for security reasons. With the on-holding of the attack the data center cut off the server (null route) until the attack stopped to prevent the impact on other servers. This is a normal behavior for data centers without proper DDoS protection.

    Why didn’t we drop requests from the attacking IP or did XYZ?
    The attack wasn’t only coming from a single IP. From the short log we got (from the data center) it looks like a bot network with connections from all over the world but with a focus from Kansas. So even if we would have had the control over the router we wouldn't have been able to prevent it fully without enterprise DDoS migration. (Once the server is null routed which happens within a few minutes, we don’t have access to it either and need to wait for it being unlocked once the attack stops.)

    Why don’t we have have DDoS migration?
    • We never had been attacked in this size before
    • Enterprise DDoS migration is expensive
    • Data centers with DDoS migration + latest hardware + single thread optimized CPU’s + monthly/quarterly subscriptions + acceptable support + many big and direct backbones from big carriers (low latency world wide) -> are extremely rare and expensive (the cheapest ones I know are at 3-4 times the monthly cost)
    On a side node, we do have servers with enterprise DDoS protection, but after 1.5 we didn't use them anymore to profit from the saved latency and better carrier connections. As there hasn't been any attack (besides self attacks from mod leaking packet’s exponentially :D) why should we have given those small advantages away for something we didn't need for +1 year?

    Why does someone DDoS MyM?
    This could have many reasons. I personally don’t care!
    Could be a angry player, could be another server, could be just for fun…

    Was it handled poorly or good from our side, when looking back?
    When looking back from my perspective I think we handled it good, at least I don’t know what we could have done better in the rough. So once I got online (half a day after the start of it) with the info mail from the data center in the inbox, the initial focus was about getting the control of the server back and getting our staff team up to date on what is really going on. We decided to sit it out as it might have just have been a script kiddo, another packet leak or whatever. The only thing we could have done would be to turn servers back online with a database backup which would take quite some time to setup and undo later on. Furthermore there was the possibility of dragging the focus on the other servers.
    Once we got the control back we put the servers back online. Unfortunately only a moment later the same happened again, another attack and we were back at the same point without the required database and 1.6 proxy. As it was clear that the attack wasn't over and might strike again once we got the control back. I moved the 1.6 servers to the old machine in a data center with DDoS protection and setup a local copy of the permissions so we could get the servers up again even without the main database, though many things weren't working as you may noticed, but the basic game play possible.
    Well the next day we had control about the main server again and the DDoS either stopped or followed the server into the data center with DDoS migration. All services are back and fully operational.

    What did we change due to it and what is planned?
    As already stated above we did move back to the data center with DDoS migration and are currently working on a multi-master database cluster (Lucid and me) in order to keep server’s operational in case the master drops out.
    Sounds simple, but is quite some work, thoughts, research and testing due to many difficulties that come with multi-master systems, especially without database cluster experience.
    I hope this post answers your questions and sorts the chaos of self interpreted information into an understandable mosaic.

    Thank you all for your patience and baring with us.
     
    Last edited by a moderator: Mar 24, 2015
    Timkoo, Demonica6666, Lux3 and 20 others like this.
  2. bim63743

    bim63743 New Member

    Messages:
    152
    Likes Received:
    10
    Local Time:
    5:35 PM
    Thank you for this info, thanks for letting everyone know and not hiding anything :)
     
  3. Terrails

    Terrails Well-Known Member

    Messages:
    127
    Likes Received:
    8
    Local Time:
    1:35 AM
    Thanks for info Slind.
    You can tell everything to us we will understand you!
     
  4. All_Mighty_POWA

    All_Mighty_POWA Well-Known Member

    Messages:
    149
    Likes Received:
    17
    Local Time:
    12:35 AM
    Thank YOU for being by FAR the best founders of any minecraft server I've ever seen in 6 years! <3 this server
     
    ShadowWolf_Beast and Ketzak like this.
  5. The_Icy_One

    The_Icy_One Procrastinates by doing work

    Messages:
    1,044
    Likes Received:
    210
    Local Time:
    12:35 AM
    You guys are amazing. You provide a free service to hundreds of people, or maybe thousands now, and you still find the time to explain a short(ish) outage. I think you actually have better customer service than Steam. So, basically, thanks :3
     
    Innovea, Ketzak, Terrails and 2 others like this.
  6. Decnav

    Decnav Proud member of the VRMasterRace Patron Tier 2

    Messages:
    413
    Likes Received:
    140
    Local Time:
    7:35 PM
    This is why I like MYM, no reason to hide anything, and you always get the facts right from the top.
     
  7. Mining_Canadian

    Mining_Canadian Well-Known Member

    Messages:
    296
    Likes Received:
    59
    Local Time:
    4:35 PM
    Cheers Slind you guys are truly amazing :) And I am honored to be apart of the staff team!!!
    Thanks to all the Staff as a whole you're all amazing!!!!
    -Mining
     
    Last edited by a moderator: Mar 25, 2015
    Kronnn likes this.
  8. SilentBane

    SilentBane Well-Known Member

    Messages:
    1,030
    Likes Received:
    628
    Local Time:
    6:35 PM
    Coming from someone who works quite closely with both @Slind and @SirWill I just wanted to take the time to thank you both for all the extra effort you put in everyday to make sure that we remain #1. It has been an honor to work with the both of you these past months. And a privelage to call you friends. Heres to many more atop the minecraft community
     
  9. LucidTheStick

    LucidTheStick Well-Known Member

    Messages:
    3,143
    Likes Received:
    1,327
    Local Time:
    7:35 PM
    Well said Charles =)
     
    Shpotee, bobthepig2 and SilentBane like this.
  10. Alicatt

    Alicatt New Member

    Messages:
    46
    Likes Received:
    6
    Local Time:
    6:35 PM
    I wish every server owner took the time to be this transparent. It's rare. Thanks for the info!
     
    Ketzak likes this.
  11. Shpotee

    Shpotee New Member

    Messages:
    49
    Likes Received:
    5
    Local Time:
    12:35 AM
    I can only raise my both hands to back this opinion. I still can't simply understand why you, the Staff, are willing to do some much for so many for free (unsuccessfully rephrasing Churchill). From this place, I'd like to urge the members of our community to donate, even the smallest, symbolic amount - not only will you get some perks (enderchest!) but, first of all, you will show your appreciation which these people surly deserve!
     
    Innovea, The_Icy_One and Ketzak like this.
  12. Ketzak

    Ketzak New Member

    Messages:
    19
    Likes Received:
    11
    Local Time:
    7:35 PM
    Being a newcomer, I had already been impressed with the level of dedication of the staff and the quality of play offered, all without obtrusively asking for donations or restricting items or standard abilities to donor ranks. The community was friendly and immediately accepting of me, and I quickly felt at home and also proceeded to spend WAY too many late nights playing with you guys!

    Having had some previous server administration experiences, I watched how you guys handled both the attack, and the consequential outcry from many players. You did an outstanding job, while still maintaining a professional manner and offering a place for us to play, despite the inconvenience, which puts you guys way up there as a world-class server network--far beyond anything I've seen in the past!

    The transparency offered during and in the wake of the outage is greatly appreciated by all, I'm sure--especially us technical types--it not only demonstrates a level of real care for the player community, but also shows the competence and understanding of your staff, in not only technical aspects, but also the managerial and financial.

    I am proud to be a part of this community, and even more so to be accepted as a part of your team. For any other newcomers here, you've found a wonderful place to play, and one that won't be going anywhere any time soon! Keep up the awesome work, guys! :D
     
  13. xLuarkeNx

    xLuarkeNx Well-Known Member

    Messages:
    73
    Likes Received:
    28
    Local Time:
    7:35 AM
    that's the reason why i am still in forum even i have less time to play with:D
     
    Ketzak likes this.
  14. MayuraSakai

    MayuraSakai Well-Known Member

    Messages:
    15
    Likes Received:
    9
    Local Time:
    1:35 AM
    Thx for the info. I hope this is a one time thing. I don't mind it being down, but it's great to know that the staff is always there to pick up the pieces. Although this shouldn't be even necessary. People have always been jealous of others success. If you ever need anything back from a community member, just shout.

    Greets Mayura.
     
  15. SirStudMuffin

    SirStudMuffin Well-Known Member

    Messages:
    2,365
    Likes Received:
    337
    Local Time:
    12:35 AM
    Do you think it would be possible to create some sort of lab network for the master drives which then goes into a WAN for everything else
     
  16. Nobbie93

    Nobbie93 New Member

    Messages:
    11
    Likes Received:
    1
    Local Time:
    1:35 AM
    Thanks for your hard work staff and very good you did explain what happened in detail and honest, im proud to be a part of this server :)
     
  17. Ketzak

    Ketzak New Member

    Messages:
    19
    Likes Received:
    11
    Local Time:
    7:35 PM
    For a school project, I'm experimenting with creating a resilient, load-balanced Moodle set-up using multiple identical machines with Maria DB Galera Cluster as the synchronized SQL server and GlusterFS as the synchronized data area. I don't know if you are familiar with either of these technologies, but they sound like they might be a potential help in your desire to create a "hot spare" of sorts to take over in the event of a server failure--Galera to keep database-tied data like perms and block logs synced, and Gluster to keep world data synced. Theoretically, downtime could be reduced to little more than the time it takes to start a server and maybe a redirect in the event of a failure or maintenance on one of the machines. Of course, not being overly familiar with your underlying setup, this idea could be totally off base :p
     
  18. Mining_Canadian

    Mining_Canadian Well-Known Member

    Messages:
    296
    Likes Received:
    59
    Local Time:
    4:35 PM
    All this sounds very complicated any I have no idea if it would work or not however It might be a great idea (Honnestly I have no Clue) May I recomend that maybe you post this under suggustion thread ( Suggestions and Feedback | MineYourMind Community ) and maybe tag @Administrator , It might be an awesome idea!

    Anyway Thank you (and welcome to the comunity^^^)

    -Mining
     
  19. Slind

    Slind Founder

    Messages:
    8,332
    Likes Received:
    3,018
    Local Time:
    1:35 AM
    We are using Galera with MariaDB and probably HAProxy for the Multi-Master Cluster.
     
  20. chugga_fan

    chugga_fan ME 4M storage cell of knowledge, all the time

    Messages:
    5,861
    Likes Received:
    730
    Local Time:
    7:35 PM
    XD irony
     
Thread Status:
Not open for further replies.

Share This Page