So, you are wondering what happened to this guy? Why there is no single activity on social media for past 8 days? Is he alive or dead because of his ridiculous (NOT reckless) car driving?
No more guesses! Let me tell you what exactly happened in the last 8 days, which is one of the most expensive lesson and real-time experience that I gained it from a stranger!Cut the crap and tell me what that is this? Kind of book? PDF or hardcopy? C’mon man…..
It’s a massive DDoS (distributed denial of service attack) attack against one of our dedicated server which is hosted under Hivelocity Networks!
Layer 4 ( Syn Flood) Attack
I noticed there was a huge spike in our server CPU, RAM, and the networks and the upstream provider confirmed it’s a Layer 4 SYN flood based attack with a much higher amount of TCP connections. We started getting DDoS alert notification from Splunk, and their report confirmed it’s TCP based SYN Flood.
After talking with Hivelocity , NOC engineers started deploying the DDoS mitigation through their upstream provider! I assume they were using Corero for such emergency DDoS mitigation activities!
Even, it was an unmanaged server, Hivelocity NOC engineers were really helpful to mitigate the situation for hardening the CSF and configuring custom rules within the firewall! However, it didn’t help me to stop the situation! Hivelocity Tech support asked me to contact the sales team or my account manager in order to obtain one of their DDoS mitigation package!
I tried to contact Hivelocity sales via email, Support center, and the Livechat! But, unfortunately, the started up selling their hardware’s and service saying The RAM is full, HD is FULL upgrading is the right choice which is absolutely a pure lie, and it won’t help in this situation.
Then I send a mail to my Hivelocity Account Manager Lee Linton; It’s confirmed he opened my email several times, but he didn’t reply either assign this to any other manager to outreach!
I’m bit disappointed about the outcome, Then I contacted Hivelocity Livechat, and the account Manager Drew Adams helped me to select the right mitigation package in order to stabilize the situation and the mitigation officially started.
Layer 3 (Network Layer) Attack
Based on Hivelocity DDoS Filtering, I thought it was under control! But my assumption is wrong; we started getting attack from ports, mail server IP’s, WHM, etc.! After deep investigation, we found it’s a layer 3 based network layer attack and checking pcap file, Again there is a huge packet loss from the server main IP ! Our Server management company helped in this stage for blocking IP’s, uRPFing IP’s and hardening the source guard to limit IP’s on unverified layers. It started from 256 Mbps and reached until 512 Mbps
The server Management Company and Hivelocity NOC engineers worked very hard to mitigate the situation, and yes they did! It was under control from 01/18/2017 12:48 EST from 01/18/2017 14:24 EST.
At this stage, our newly appointed CTO Saruban .B started collaborating with the NOC and the server management company! Saruban is a kid and very new to this industry neither me, But he is very aggressive, fast learner and positive thinker! So, we both started investigating the root causes for this issue and we ended up without any clue and then started learning about the OSI model, DDoS mitigation strategies to prevent it in future.
Layer 7 (Web Application layer) Attack
People who know me aware I’m an avid traveler and even I don’t know where I will be in tomorrow, especially in weekends, I will be in somewhere near to the beach or hiking mountains.
Okay, Fine! I thought it’s all done and started digging booking.com to find out the best weekend deal chill with my family! That time Saruban B said, he isn’t able to access the web and mail servers! I told him it’s a simple IP block just restart the router or check VPN; he said he tried all these basic DNS flushing activities, but it didn’t help him.
Same time, I received a mail from our server management company saying; the server is getting Level 7 Web Application Layer (WAF) attack! I just replied them saying “you know how to mitigate it right? Please do the needful “because I need to plan my weekend adventureJ! I received the reply saying this is kind of application layer level attack and the server is not responding!
This is something weird! I just tried to access the server via IPMI and it is not responding at all and even the DC isn’t able to find it! They did a hard reboot and the server back online after 7 minutes and then found there is a massive flood in consuming the CPU and the RAM.
The server management company said nothing they can do until we get professional DDoS mitigation plan! Seriously? Oh, then my weekend plans? Yep, it’s ruined!
The situation is now getting heated! We started getting advice from the industry experts and as per their advice decided to move this data to another server with proper mitigation plan and we did it!
The (Simple, Yet Powerful) Action Plan
We sent a Global email to all our hosting customers who are hosted on Layer24 server about the current situation (Didn’t hide anything) and our action plan. We already have few servers from multiple providers like SingleHop, Softlayer, Rackspace and Limestonenetworks and OVH. We started collecting quote from them and selected one. Data migration had been started, but the server was responding 600 times slower! Which mean 1GB CPmove is taking 2 hours to move another Cpanel server! So, we decided to move based on the priority and used different types of migration strategies, and we made it. Collected quote from major DDoS mitigation providers Like Incapsula, Arbor Networks, CloudFlare Enterprise and shortlisted one and negotiated with them about the features and protocols. Now, all our dedicated servers are under one of the world Top Rated DDoS protection company surveillance.
When there is a Layer 7,4 Attack in place; your firewall, load balancer, and the fancy routers will not help you! It doesn’t matter how techie you are or you are a doctorate in Cyber Security or ethical hacking! The only way to resolve this situation is finding out best DDoS mitigation provider and let them handle the situation! If you choose the cheap one, you will get what you paid!
Privilegeserver Technologies isn’t a typical shared hosting, nor it’s not for anyone! I started this company because there is a need for my Digital marketing clients that’s “High performance” and optimize and manage their account by ourselves while they focus on growing their business. As per my business plan, this year I decided to focus on Extreme SEO business activities to avoid distractions.
Within last 6 days, there is NO sleep, NO Trips, NO meetings at all! But all I learned is one of the most expensive and precious lesson, which I “CAN’T” afford or cannot learn from the universities! Yes, that’s a real time experience! We didn’t give UP, We have mitigated the situation, maybe it will happen again, and the server may rarely go offline! But, I will find a way to mitigate the solution rather giving up!!! Because, I’m still alive and breathing!
Now It’s Your Turn, That’s all about our DDos attack nightmare and mitigation strategies that we applied. And now I’d like to hear from you:
- Do you have any questions about this post or any better suggestions?
- Maybe you have a great tip that I didn’t emphasize it here.
- Either way, let me know by leaving a comment below right now.