Imagine yourself operating a VM hosting provider.
You have innumerable lots of customers, the vast majority of them having just a small - or, at worse, none at all - technical ability and knowledge to ensure that their VMs are running smoothly, without causing any issues either to you or to the larger Internet community as a whole.
Being a good netizen requires technical knowledge that is not so widely disseminated or given attention at nowadays, where the popular trend getting momentum is "anybody can fire up an Ubuntu and be happy with not ever having to deal with UNIX internals".
Now, imagine one of your customers VMs was causing trouble - not only to you, but to another ISP, and their customers too.
What would you do?
Most of us technically inclined people, would be quick to answer: "Shut them down so you alleviate the issue, and deal with them later, or not deal with them at all - they're not going to understand technical matters". Most probably, I myself would be amongst those giving the same exact answer when faced with this question.
But I wasn't.
To make it worse, this time, I was part of the problem: I had deliberately configured a DNS resolver on one of my VMs to allow unlimited recursive queries from the open Internet. The reason? At the time, I was behind some ISPs aggressively filtering DNS queries, and I wanted to have a DNS resolver I could trust - my own one.
Problem is, this configuration allows your DNS server to be the culprit for an interesting class of attacks, called DNS Amplification Attacks. Although I was very aware of the dangers of operating a publicly reachable open DNS resolver, I opted to hide behind the small possibility of my own VM being used as part of an attack - who scans the wide internet for open DNS resolvers, right?
Well, apparently somebody did. Fortunately, this somebody was an automated service scanning for open DNS resolvers rather than an actual attacker, but unfortunately it was also a misconfigured one: Upon discovery of an open DNS resolver, it didn't just notify their owners so they could alleviate the issue, but it also kept on hammering the said DNS resolver with thousands of DNS requests per minute, all of them destined for a couple of DNS domains - facilitating a DNS amplification attack, rather than preventing one!
My VM host provider is the more than excellent linode.com. So, what they did in this case to alleviate the issue my VM was causing - again, not only to them, but to another ISPs and another ISPs customers? Did they shut down my network connectivity, or my VM at large, as it would be easier - and safer - for them to do so?
No.
Instead, they opened a ticket with me. Supplying all the technical information needed so I could fix the issue myself.
To make a bad situation worse, my linode-registered e-mail is one that sometimes I don't read too often - especially when travelling around. And at the period in question, I was travelling around. So their ticket went unnoticed for a total of 5 days.
Did they shut me down after my inability of quickly respond to the issue, as they would be more than right to?
No.
They kept on sending me emails, until I finally noticed one of them.
Upon reading their ticket, it was just a matter of minutes to alleviate the problem. I turned off the recursive resolver on my bind configuration, and I supplied them with the list of IP addresses being responsible for the vast majority of the rogue DNS requests, as well as the domains being attacked. I also offered to contact the attacked ISP's technical contact, whose e-mail complaint linode had CC:ed to the ticket, to let them know I took care of the situation, and ask for forgiveness :)
Why, you would be inclined to ask, am I publicly exposing my mistakes like so? Only for one reason: To again, and again, and again praise linode.com, whose *EXCELLENT* technical customer support has been saving me from issues time after time, all these years I've been their customer. And let's not forget - they even give you free upgrades every time they expand - some capacity goes to accommodate for growth, and some capacity goes to free upgrades for their existing customers.
So, if you're a linode customer, be sure to read your registered linode e-mail often so you get notice of any issues, and be sure to read their blog often, so you get noticed of any offers and upgrades being offered to you.
And if you're not a linode customer, go on and be one right now. I can assure you that it will be one of the best decisions you'll ever make.
You have innumerable lots of customers, the vast majority of them having just a small - or, at worse, none at all - technical ability and knowledge to ensure that their VMs are running smoothly, without causing any issues either to you or to the larger Internet community as a whole.
Being a good netizen requires technical knowledge that is not so widely disseminated or given attention at nowadays, where the popular trend getting momentum is "anybody can fire up an Ubuntu and be happy with not ever having to deal with UNIX internals".
Now, imagine one of your customers VMs was causing trouble - not only to you, but to another ISP, and their customers too.
What would you do?
Most of us technically inclined people, would be quick to answer: "Shut them down so you alleviate the issue, and deal with them later, or not deal with them at all - they're not going to understand technical matters". Most probably, I myself would be amongst those giving the same exact answer when faced with this question.
But I wasn't.
To make it worse, this time, I was part of the problem: I had deliberately configured a DNS resolver on one of my VMs to allow unlimited recursive queries from the open Internet. The reason? At the time, I was behind some ISPs aggressively filtering DNS queries, and I wanted to have a DNS resolver I could trust - my own one.
Problem is, this configuration allows your DNS server to be the culprit for an interesting class of attacks, called DNS Amplification Attacks. Although I was very aware of the dangers of operating a publicly reachable open DNS resolver, I opted to hide behind the small possibility of my own VM being used as part of an attack - who scans the wide internet for open DNS resolvers, right?
Well, apparently somebody did. Fortunately, this somebody was an automated service scanning for open DNS resolvers rather than an actual attacker, but unfortunately it was also a misconfigured one: Upon discovery of an open DNS resolver, it didn't just notify their owners so they could alleviate the issue, but it also kept on hammering the said DNS resolver with thousands of DNS requests per minute, all of them destined for a couple of DNS domains - facilitating a DNS amplification attack, rather than preventing one!
My VM host provider is the more than excellent linode.com. So, what they did in this case to alleviate the issue my VM was causing - again, not only to them, but to another ISPs and another ISPs customers? Did they shut down my network connectivity, or my VM at large, as it would be easier - and safer - for them to do so?
No.
Instead, they opened a ticket with me. Supplying all the technical information needed so I could fix the issue myself.
To make a bad situation worse, my linode-registered e-mail is one that sometimes I don't read too often - especially when travelling around. And at the period in question, I was travelling around. So their ticket went unnoticed for a total of 5 days.
Did they shut me down after my inability of quickly respond to the issue, as they would be more than right to?
No.
They kept on sending me emails, until I finally noticed one of them.
Upon reading their ticket, it was just a matter of minutes to alleviate the problem. I turned off the recursive resolver on my bind configuration, and I supplied them with the list of IP addresses being responsible for the vast majority of the rogue DNS requests, as well as the domains being attacked. I also offered to contact the attacked ISP's technical contact, whose e-mail complaint linode had CC:ed to the ticket, to let them know I took care of the situation, and ask for forgiveness :)
Why, you would be inclined to ask, am I publicly exposing my mistakes like so? Only for one reason: To again, and again, and again praise linode.com, whose *EXCELLENT* technical customer support has been saving me from issues time after time, all these years I've been their customer. And let's not forget - they even give you free upgrades every time they expand - some capacity goes to accommodate for growth, and some capacity goes to free upgrades for their existing customers.
So, if you're a linode customer, be sure to read your registered linode e-mail often so you get notice of any issues, and be sure to read their blog often, so you get noticed of any offers and upgrades being offered to you.
And if you're not a linode customer, go on and be one right now. I can assure you that it will be one of the best decisions you'll ever make.
1 comment:
I would definitely shut you down! :)
-gkourv
Post a Comment