Internet Non-Service
I don’t know how I should describe the nonsense I pay Virgin (“Liberty Global”) good money for. It’s supposed to be an Internet Service Provider, but it falls well short of that far too often, and sometimes for extended periods. Back in the summer I was stranded without service for several weeks.
This morning (or, more precisely, yesterday morning) I found myself unable to read my mail. I also couldn’t ssh to the server. Lynx could get the front page, but only after a long delay. This looked exactly like something that happened last week, when only after rebooting (from the rackspace console) and calling rackspace support did I realise the problem was with Virgin, and traceroute was hanging on a Virgin machine after just a few hops from here[1]. Using my EE 4G connection, all was well.
Today as last week I could see the server was fine, as I could access it from an apache.org machine, but anything from home just timed out. I let that pass, and again used the EE connection to read mail. But after a full day of downtime I thought I’d check a little more. This time traceroute gives me an entirely different destination: 62.252.172.241, which is a machine owned by Virgin! A simple DNS lookup tells me the same. So this time it’s a DNS cockup.
Except …
If it’s a DNS cockup, how come I can still browse my website (at least using Lynx, which doesn’t time out first)? There must be a HTTP proxy – with valid DNS – on 62.252.172.241. Smells like deliberate sabotage! And how come this didn’t appear to affect other sites I’ve been to today? For example, $work email (c/o gmail), or this blog @wordpress?
Probing further, this time (unlike last week) I can route to the server by IP address. So it’s definitely just DNS.
WTF is going on? I think it’s time to drop this sick joke of a non-ISP. Maybe get a second 4G connection from another provider for a bit of redundancy: that connection seems good most of the time, but wifi to the 4G modem is totally flakey so I have to use it via USB, which is a poor second-best.
[1] Yeah, of course any geek should have tested that before going to rackspace. In my defence, I was flat out in bed with a nasty lurgy and in no fit state to browse the web, let alone fix a problem on it.
niq's soapbox 
You googled the IP address, found they do this all the time, and it might be “web safe”.
You switch your DNS to another provider, ideally one over a VPN to a provider who doesn’t break your Internet, or second best recurse off the root DNS locally and enable DNSSEC (assuming the root zone isn’t hacked today – sigh). Flush DNS caches, restart browsers etc…
You then call a reputable ISP like AAISP or Entanet reseller, and switch ASAP. My former colleague, and major technical wizard, really impressed with his new AAISP connection which also is 50% faster than the prior ADSL due to them knowing how to tune it.
Even if you stop Virgin messing with DNS, they will mess with routing if the IWF tell them an IP is bad. You can’t stop them doing that one, other than voting with your feet for an ISP that deliberately and actively avoid trying to mess with your traffic.
Always set up web servers with HTTPS unless there is a really good reason not to, always use HSTS, this gives you, and your users, a first stab that at least you are talking to the right box. And that neither the recursive DNS or the routing has been intercepted (except perhaps for meta data gathering). I’ve no idea who can issue fake certificates, but in practice it is generally not a problem (aside from Iran and some similar cases – Google Inc knows), and we’ll have pinning sorted by the end of 2015 (really and truly sorted – trust me, I’m too geeky to lie).