Noticing that you cannot access quite a few websites? Emails bouncing? Is your site traffic lower than it should be?
Is your domain’s DNS hosted with AT&T?
As of 12:21 pm EDT (we first noticed at 11:20 EDT) AT&T is experiencing a massive DNS outage.
This morning we started getting reports from users that inbound emails were bouncing. I soon discovered that I could not resolve our MX records externally, then that I could not even ping our website.
According to the Twitterverse AT&T is having a massive DNS outage presently. Our AT&T rep said it might be a DOS attack, and there was no estimate as to when it will be fixed.
UPDATE: 12:25 PM – (Glad I said something)
No sooner did I post this than our AT&T hosted DNS started responding again.
UPDATE 12:46 PM
Looks like it was only up long enough for my test server to get a cache – the authoritative DNS is still unresponsive.
UPDATE 2:20 PM
AT&T has informed some customers (including us) that the issue is resolved. Testing our site using the free Uptrends availability tool shows that it is somewhat responsive… however inbound email is still not working reliably and http://www.mxtoolbox.com says our DNS servers are still timing out.
According to a comment on this post, this outage started at 9:30 am, and it is still causing problems for us 6 hours later.
Same issues over here in Michigan. Thousands of domains are down. Check out
http://www.gwebtools.com/ns-spy/cbru.br.ns.els-gms.att.net/1
Those are the domains that one of these nameservers is authoritative for.
The problem started before 9:30 EDT. Here is part of my chat @ 10AM with AT&T and they were already admitting to a problem, so I assume it started well before that:
We apologize for the DNS Trouble. There is a problem with the backbone and as of 8:30 CDST there is no ETR.
It affected our servers in Texas by at least 7:00 AM Central.
Does anybody know of a way to protect against this. Can some redundancy be built in the system.
One of my customers said they got an email kicked back last night (8/14/12) around 10:30PM CT so it was well before this morning.
In a totally self serving way… there are products, and we have one that handle DDOS attacks and filter them out.
That said, however, one of our customers had a DDOS attack that had 22GB (bytes) of inbound DNS traffic. This broke their provider, and their edge routers before it ever made it to the DNS firewall.
AT&T could have a similar situation that the infrastructure is overwhelmed.
It is hard to know with the limited amount of information available.