So, you are wondering what happened to this guy? Why has there is no single activity on social media for the past 8 days? Is he alive or dead because of his ridiculous (NOT reckless) car driving? Well, I’m alive and ready to drive with no destination. Yep! I’m gonna make you all jealous again by publishing selfies! So, get ready to unfollow or block me from your channels while you are stuck with the workload.
No more guesses! Let me tell you what exactly happened in the last 8 days, which is one of the most expensive lessons and real-time experience that I gained it from a stranger! Cut the crap and tell me what that is? Kind of book? PDF or hardcopy? C’mon man…! it’s a massive DDoS (distributed denial of service attack) attack against one of our dedicated servers which are hosted under Hivelocity Networks!
Layer 4 ( Syn Flood) Attack
I noticed there was a huge spike in our server CPU, RAM, and the networks and the upstream provider confirmed it’s a Layer 4 SYN flood-based attack with a much higher amount of TCP connections. We started getting DDoS alert notifications from Splunk, and their report confirmed its TCP-based SYN Flood. After talking with Hivelocity, NOC engineers started deploying the DDoS mitigation through their upstream provider! I assume they were using Corero for such emergency DDoS mitigation activities!
Time:01/18/2017 12:12 ESTnn Host: 126.96.36.199nn Impact: 251 Mbps / 403115 pps nn Attacks: Service Flood to http at 251 Mbps / 403115 ppsnn Sflow stats: Unblocked:75 EP:75 Total:288
Even, it was an unmanaged server, Hivelocity NOC engineers were really helpful to mitigate the situation for hardening the CSF and configuring custom rules within the firewall! However, it didn’t help me to stop the situation! Hivelocity Tech support asked me to contact the sales team or my account manager in order to obtain one of their DDoS mitigation packages!
I tried to contact Hivelocity sales via email, the Support center, and Livechat! But, unfortunately, they started up selling their hardware’s and service saying The RAM is full, HD is FULL upgrading is the right choice which is absolutely a pure lie, and it won’t help in this situation.
Then I send a mail to my Hivelocity Account Manager Lee Linton; It’s confirmed he opened my email several times, but he didn’t reply either assign this to any other manager to outreach! I’m a bit disappointed about the outcome, Then I contacted Hivelocity Livechat, and the account Manager Drew Adams helped me to select the right mitigation package in order to stabilize the situation and the mitigation officially started.
Layer 3 (Network Layer) Attack
Based on Hivelocity DDoS Filtering, I thought it was under control! But my assumption is wrong; we started getting attacks from ports, mail server IPs, WHM, etc.! After deep investigation, we found it’s a layer 3 based network layer attack and checking pcap file, Again there is a huge packet loss from the server main IP! Our Server management company helped in this stage for blocking IPs, uRPFing IP’s and hardening the source guard to limit IPs on unverified layers. It started from 256 Mbps and reached 512 Mbps
The server Management Company and Hivelocity NOC engineers worked very hard to mitigate the situation, and yes they did! It was under control from 01/18/2017 12:48 EST from 01/18/2017 14:24 EST.
At this stage, our newly appointed CTO Saruban. B started collaborating with the NOC and the server management company! Saruban is a kid and very new to this industry neither me, But he is very aggressive a fast learner and a positive thinker! So, we both started investigating the root causes for this issue and we ended up without any clue and then started learning about the OSI model, DDoS mitigation strategies to prevent it in the future.
Layer 7 (Web Application layer) Attack
People who know me are aware I’m an avid traveler and even I don’t know where I will be tomorrow, especially on weekends, I will be somewhere near to the beach or hiking mountains.
Okay, Fine! I thought it’s all done and started digging booking.com to find out the best weekend deal to chill with my family! That time Saruban B said, he isn’t able to access the web and mail servers! I told him it’s a simple IP block just restart the router or check VPN; he said he tried all these basic DNS flushing activities, but it didn’t help him.
Same time, I received a mail from our server management company saying; the server is getting a Level 7 Web Application Layer (WAF) attack! I just replied to them saying “you know how to mitigate it right? Please do the needful “because I need to plan my weekend adventure! I received the reply saying this is a kind of application layer level attack and the server is not responding!
This is something weird! I just tried to access the server via IPMI and it is not responding at all and even the DC isn’t able to find it! They did a hard reboot and the server back online after 7 minutes and then found there is a massive flood in consuming the CPU and the RAM.
The server management company said nothing they can do until we get a professional DDoS mitigation plan! Seriously? Oh, then my weekend plans? Yep, it’s ruined!
The situation is now getting heated! We started getting advice from the industry experts and as per their advice decided to move this data to another server with a proper mitigation plan and we did it!
The (Simple, Yet Powerful) Action Plan
We sent a Global email to all our hosting customers who are hosted on the Layer24 server about the current situation (Didn’t hide anything) and our action plan. We already have few servers from multiple providers like SingleHop, Softlayer, Rackspace and Limestonenetworks, and OVH. We started collecting quotes from them and selected one. Data migration had been started, but the server was responding 600 times slower! This means 1GB CPmove is taking 2 hours to move another Cpanel server! So, we decided to move based on the priority and used different types of migration strategies, and we made it. Collected quotes from major DDoS mitigation providers Like Incapsula, Arbor Networks, Cloudflare Enterprise and shortlisted one and negotiated with them about the features and protocols. Now, all our dedicated servers are under one of the world’s Top Rated DDoS protection company surveillance.
When there is a Layer 7,4 Attack in place; your firewall, load balancer, and the fancy routers will not help you! It doesn’t matter how techie you are or you are a doctorate in Cyber Security or ethical hacking! The only way to resolve this situation is to find out the best DDoS mitigation provider and let them handle the situation! If you choose the cheap one, you will get what you paid!
Privilegeserver Technologies isn’t a typical shared hosting, nor it’s not for anyone! I started this company because there is a need for my Digital marketing clients that’s “High performance” and optimize and manage their accounts by ourselves while they focus on growing their business. As per my business plan, this year I decided to focus on Extreme SEO business activities to avoid distractions.
Within the last 6 days, there is NO sleep, NO Trips, NO meetings at all! But all I learned is one of the most expensive and precious lessons, which I “CAN’T” afford or cannot learn from the universities! Yes, that’s a real-time experience! We didn’t give UP, We have mitigated the situation, maybe it will happen again, and the server may rarely go offline! But, I will find a way to mitigate the solution rather than giving up!!! Because I’m still alive and breathing!
Now It’s Your Turn, That’s all about our DDoS attack nightmare and mitigation strategies that we applied. And now I’d like to hear from you:
- Do you have any questions about this post or any better suggestions?
- Maybe you have a great tip that I didn’t emphasize here.
- Either way, let me know by leaving a comment below right now.
Disclaimer: I am a part of the organizing committee for Startup Weekend Jaffna. Written below is my opinion about the event in its genuine and unbiased form. The view expressed in this post is purely based on a culmination of my experience of attending the event in another country and the experience of some of my colleagues and business associates as well.
Greetings and hello everyone. I know there is a lot of buzz happening in Sri Lanka about ‘some’ event called Startup Weekend Jaffna or #SWJ that will take place from the 24th to 27th of June in Jaffna. Many of you have been shooting me emails and voice messages to know more about the prospect of the event, Therefore, I decided to write up a quick blog to answer all your questions.
I plan to cover some of the topics of interest, like, What the event is about, what is the prospect of such an event and who should attend it, how we made it possible in Sri Lanka and some commonly asked doubts that most inquiries have.
What is Startup Weekend All About?
The essence of Startup Weekend is that people with an idea for a startup or individuals who want to collaborate with another startup(s) meet up under one roof over the weekend and work together to get something cooking.
Before I dwell further into this blog, I would like to mention just one thing about Startup Weekend that you should know. It doesn’t matter if you are a serial entrepreneur, a budding startup, or just an investor look for tie-ups. If you are motivated enough about your product, service, or vision, you will easily meld into the crowd at #SWJaffna next weekend.
How Did We Make it Happen in Sri Lanka?
Well, this one’s a long-long tale, and it’s impossible for me to write in full detail. However, I will ‘rewind-n-relive’ the most valuable minutes of this journey, some of which, have been truly inspiring and life-changing.
It was during last year when I was in Malaysia on a business tour that the idea of bringing Startup Weekend to Sri Lanka had struck me. I was attending the CAYE Asia Summit in Malaysia and on the last day the Summit was held at MAGIC Center (Malaysian Global Innovation and Creativity Center). Call it my luck, or destiny but Startup Weekend was also running parallel in the same building (JACKPOT).
As a startup Entrepreneur, I had always dreamt of attending such an event. Startup Weekend was a dream for me. I wanted to participate in the event after the summit ended and it was the second time I felt lucky that day when the #SWM organizing team let us attend the event as a guest from Startup Weekend MaGIC. I just attended #SWM for an hour with one of my colleagues Puwaneswaran who also participated CAYE Asia Summit.
I made the best use of my time at the event by indulging in conversation with various Attendees, Malaysian Entrepreneurs, and Members of the Organizing Team. One of the lead organizers introduced me to Lalitha Wemel (Regional Manager, Startup Programs for South East Asia and Oceania). I expressed my desire to bring Startup Weekend to Sri Lanka to Mrs. Lalitha, and she gave me the reference of Anurag Maloo (Regional Manager for South & Central Asia).
When I connected with Anurag Maloo, he openly expressed his concern about the Sri Lankan scene. Within minutes of discussion, I came to know that many “Entrepreneurs” had asked him to get #SW to Sri Lanka, but he had not found anyone motivated enough to bring such a big event to the country. Anurag told me that most people merely inquired about the event, and when the time came to take action, they were nowhere to be found.
With such a pathetic and frustrating first experience, it was tough to convince him. However, I was hell-bent on explaining my vision. I wanted Startup Weekend to come to Sri Lanka. I shared my ideas with Anurag and explained to him why #SWSL would be a very good idea and how I could help him make it possible. Upon further discussion, his team interviewed me and verified my credentials, and gave me a green signal to get the planning started for the event.
Upon receiving the green signal from Anurag Maloo, I immediately summoned my most trusted business associates and friends to tell them about the event – Thavaruban Thangarajah, Chamara Peiris, and Robinson Prashanthan. They are not merely business associates or friends for me, they are people I can trust even with my eyes closed. With them in place, it was in a matter of few hours that I was able to form the highly motivated team that was eager to see the event of such great standing, take place in our proud Country.
Why Did We Decide Jaffna, Instead of all the Other Cities?
Upon receiving the green signal from Anurag Maloo, I immediately summoned my most trusted business associates and friends to tell them about the event. These were people I could trust even with my eyes closed. It was in a matter of few hours that I had formed a highly motivated team that was eager to see the event of such great standing, take place in our proud Country.
After countless hours of brainstorming, we defined Jaffna as the host city for the event, as it is a growing city and it’s rapidly recovering post-war. The reason we never considered Colombo is because already there are lots of Hackathons and Startup Events in this town every year. We did not want to make Startup Weekend a part of any other event and thus considered between Kandy and Jaffna. Now, as Kandy is already marked by SLASSCOM for KandyITBPM, we decided to stick to Jaffna.
Who Should Attend Startup Weekend?
It doesn’t matter if you are looking to kick-start your entrepreneurial journey or co-found another company. And, it doesn’t even matter if you just have an idea. Since you will be sharing your idea with business evangelists from various industries like Technology, Education, Health, Disability, Automobile,s and more, you can rest assure that you will be receiving mission-critical feedback about the highs and lows of your master plan.
My Opinion About Startup Weekend
“The primary reason why most startups ‘CLOSEUP’ is because they fail to scrutinize the bigger picture of their business strategy. Most novice entrepreneurs are too busy looking at the upsides and advantages of their game plan, that they completely overlook or become blind to evident and irrecoverable business potholes. Startup Weekend is like a face-off with the mirror for everyone, rookie or seasoned professionals alike, as we all are learning.”
The idea of having to face your own internal junk scares everyone to death. There is lots of talent in Sri Lanka and lots of bright people who can shape the future of our country. But, most of them are too shy to come out of the closet.
People don’t like their ideas being dissected or analyzed. The fear of sharing their next best idea with everyone, or for that matter being made fun of because their idea is not that amusing, is too overwhelming than their desire to create a unique and evergreen product.
For anyone looking to start off their business, or be a part of one, my advice would be to head over to Startup Weekend Jaffna and pitch yourself in front of the crowd and face your demons. In the end, it won’t matter if you have created an ‘awesome product for an invisible problem.’ It is best to get yourself validated, call BS early and move on to the next idea.
As an Entrepreneur, it is entirely acceptable to make mistakes, but what is not acceptable is to continue doing what you are doing, even when you know you are wrong.” You should go for Startup Weekend and share your insights about, a vast and profitable market that may be lying untapped, a market in which you have identified a model customer, with a unique problem that has not yet been solved, and discuss your solution.
Meet people, go co-founder dating, discuss unique strategies, talk about your team. Let people know that you already have interested investors (if any). Try to explain a conceivable business model along with your customer research. You can also show off a prototype of your product and service and demonstrate some traction if you are already in business.
For people who are just looking to invest in other Startups and build their connections in the industry, Startup Weekend is a unique treat. One of the best parts about #SWJ is that the organizing committee is getting an incredible lineup of advisors and mentors from various industries like marketing, finance, health, technology, and more. It doesn’t matter if you don’t have a startup idea, you can still visit startup weekend to seek out, possible and lucrative business opportunities and entrepreneurs who want to collaborate with you to extend their idea into a thriving and ever-growing business model.
Startup or Not – You Should Attend #SWJaffna
The bottom line is, whether you are an investor or an entrepreneur a college graduate, or anything in between, the 54 Hours of #SWJ will be the most memorable and astonishing moments of your life, and by the end you will have “a lot” to take away from the event. It is an energizing and refreshing experience. If you want to register for the Startup Weekend that will happen from the 24th to the 26th of this month in Jaffna, you can visit the following link to book your tickets. Click Here.
Or, have you attended Startup Weekend in any other country? Don’t forget to share your experience about the event in the comments section below 🙂