At 3:15pm EST today I got an email from our BES that it had lost its SRP connection. This happens from time to time and I ignore it as long as it reconnects within a few minutes. Today it did not. A quick test revealed that internet connectivity was alive and well, but that external DNS resolution was not. The OpenDNS servers were unreachable, as was their website. Oh dear.
I switched the forwarders on our internal DNS servers back to our ISP-provided servers and we were back in business, but what does this mean for OpenDNS? I’ve been using them at the corporate level at several of our US sites for over a year with success, but today’s mishap has me wondering if I will continue to do so in the future. I await more information from OpenDNS and hope they have a good explanation as well as a plan to keep this from happening again in the future.
Update: A rather snide comment (below) from David Ulevitch of OpenDNS suggests that this was a Verizon issue rather than an OpenDNS issue. Given the general lack of buzz on the internet, I’m inclined to believe it, but shouldn’t this type of issue warrant some sort of communication to the public? A blog post on the OpenDNS website, or a sticky thread in the forum would go a long way toward getting the word out when problems prevent access to their services, even if said problems are not directly an OpenDNS problem.
Twitter
Where are you located? My guess: The northeast
Who is your ISP? My guess: Verizon
Hi David!
One affected site is in MA with Verizon as the LEC, but the other was in San Antonio TX with Grande Communications as the LEC. Both sites experienced a perceived total outage until we switched back to our old AT&T DNS servers.
Can you elaborate? The implication that this was a Verizon issue seems obvious, but I’m curious to learn more.
Funny enough I just switched our office to OpenDNS on Thursday! However to lend some credit to them, their service has been up for us ever since without interruption that I know of. Our ISP is Frontier (Rochester NY), which I believe also uses Global Crossing as it’s underlying backbone.
Also wanted to say how relevant your blog has been to me! I setup VMware ESX in Jan 2008 for our servers and went with an Equallogic iSCSI SAN at the same time. It seems sometimes that almost every post you make is something we’ve been doing ourselves! Almost kind of freaky, but in an amazing/comforting way to see someone going through similar issues, and taking the same choices forward as I have. I have a common on the iSCSI performance issues you were having, but I’ll ask it over in that post.
Back on topic: Websense who we use for filtering websites blocked Google (big oops) earlier this week, and also has made NO mention of the issue, although it was fixed in about 1/2 an hour. The only thing close I could find was a direct reply to a tweet someone sent asking about it to their official websense twitter account. Unfortunately I agree that most companies do a pretty poor job in reporting outages and status. OpenDNS does at least have a pretty detailed status page, which is more than I can say for most… Although in this instance, I wonder how difficult it is to report on outages from other networks outside your control… although I suppose a quick note saying “hey you Verizon customers, it’s not us, it’s Verizon” would have been nice…
Shawn, we use Websense too but I didn’t get any calls about the Google issue. When do you run your database download? Ours is like 2am daily so maybe we update too infrequently to have been bitten by that one.
Our websense is still v6 which seems to have been the only version effected after further review of the incident with them. We also have the realtime Security updates, so it’s updated with small changes to the security webpage categories every 5 minutes, in addition to the nightly full download.
Yep it makes sense that we didn’t get that problem with Google – we only ge nightly updates and we’re on v7.
about three days ago, i stumble upon same problem, i lost connectivty to internet.
however, after several self test i found that:
1. i can’t resolve the IP address.
2. tracert using server name, failed, but tracert using the IP address works (such as openDNS server).
3. tracert without option [-d] doesn’t reveal hops names.
4. i can ping any server using their names, but they ping-able with their IP addresses!
with that i suspect it was DNS server problem, as i change DNS server into ISP assigned, i back online.
i haven’t suspected openDNS yet, but i need to confirm this:
is it possible for ISP to block DNS request if such request was not directed to their own DNS server ??
(my ISP refuse to answer this)
i’m in indonesia using Telkomsel ISP.
Hi Joey,
Yes I believe it is possible that an ISP could block DNS requests like that. Recently there was a conflict between Comcast, a major US ISP, and OpenDNS that intermittently had a similar result to what you’re describing. Comcast is in the process of intercepting unanswered DNS requests and redirecting them to a Comcast-owned page. OpenDNS behaves in a similar manner, so the two were butting heads and leaving many customers with major connectivity problems.