When a Perfect Storm meets a World Class Team

"This article is a great example of Inspire, UKFast and a M3 Networks working together under very difficult circumstances to deliver an exceptional level of service to a shared client.

I've often said that Great customer service and support is not easy. World Class customer service and support takes years of experience, continuous personal development and a willingness, nigh desperation to help."

David Dwyer, Director

If property is about location, location, location, then the internet is about address, address, address.

When companies think about names, they think brand. They think: what’s meaningful and memorable for our customers? They might also think about a funky url… But they almost never think about internet protocols and how precarious the house of cards is stacked.

So it’s not surprising that when we got a call from one client, asking why their site was down on Hogmanay Sunday morning, their first thought wasn’t about network address translation (NAT). It is the reason why the internet doesn’t appear to have run out of IP addresses – yet. To most web users, that alarming prospect seemed to have gone the way of the Millennium Bug, if it had ever registered with them at all.

A little history
You see, back in 2011, fears surfaced that the internet would soon run out of available addresses for everyone who needed a unique IP. Not surprising really, as that year did mark the allocation of the last blocks of IPv4 addresses, so the clock was already ticking.

An IPv4 address is the 12-digit numeric we’re all familiar with, XXX.XXX.XXX.XXX. It makes it possible for us to connect our devices to the web.

Every device needs its own numerical IP address, because each time we want to send data through the web, a data packet containing the IP addresses of the sender and recipient devices transfers from one computer to another. Without the IP addresses, computers wouldn’t be able to communicate. Simple as!

Back to the future
When we checked things for our client, we quickly identified the root of their problem… Our client had their domain name registered through 1&1, 1&1 had taken it upon themselves to roll out updates to their domain name management platform in the UK, to transition to IPv6. This is the latest version of IP address, and has room for 2^128 different addresses — that is 340,282,366,920,938,000,000,000,000,000,000,000,000 to be exact. Room for everyone!

On the surface it’s great that IPv6 provides capacity for the internet to keep growing, and it’s been around for several years already. Only NAT had largely allowed people to work around the problem and keep their old IPv4 addresses – up to now. So, the benefit of IPv6 has passed most web users by, but it’s essential.

The problem on Hogmanay
The problem here isn’t that 1&1 were transitioning to an updated DNS platform but that:

1. ISP providers Sky and BT had enabled IPv6 on their routers through recent software updates (Ed: only those people who’d had the update installed would see this).
2. 1&1 chose to do this upgrade without telling anyone, at the height of the holiday season
3. Within the new updated DNS platform 1&1 had arbitrarily defaulted this field to 21e (the IPv6 address for 1&1’s own landing page), meaning that all traffic able to see and visit a website with an IPv6 address would not see the client website but instead see 1&1’s own  landing page, not good, in fact terrible.
4. To compound this the 1&1 DNS platform did not allow users to edit this new AAAA Record to what the correct address should be, another failing, but this time a User Interface/User eXperience one. Hence site owners or M3 Networks could not walk back the change to restore service, if something went wrong, which in this case it did.
5. The clients server also needed to be changed to handle IPv6 addresses (this was planned for April 2018).
6. The server needed to be allocated it’s own IPv6 address.

The combination of a) to f) plus many devices being IPv6 enabled meant a perfect storm.

Some good news though was that the client’s domain name was managed from inside their M3 Networks companies account, whom we have a very good relationship with.

Some tech talk
You see, the issue is that many routers and servers still don't support IPv6, making a connection impossible between a device with an IPv6 address and a router or server that only supports IPv4. Because 1&1 acted without notifying customers, or hosts, or ISP’s many of their new IPv6 addresses defaulted from their intended to client site, and instead redirected users to the 1&1 landing page.

The fix
Being Hogmanay, a call to 1&1 Support was routed to their Philippines call centre, who reported that the site was working. (But ISP’s in the Philippines hadn’t yet applied the IPv6 update, so of course the site worked normally for them!).

1&1 support refused to view the website via UK based Proxy Server tool which would show them the issue, “computer says no” being the order of the day.

To inflame the situation further they advised that 1&1 Tech Support would be available to address the Support Ticket on Wednesday 3^rd January, ie 4 days later.

By this time we’d been able to reach the client’s M3 Networks team who immediately took ownership to get 1&1 to play ball. However the changes once applied could still take up to 48hrs to take effect.

As such having identified the problem in the UK, we chose to progress 2 solutions in tandem:

1. M3 Networks would work towards getting the 1&1 DNS platform amended back to what it was before.
2. Inspire would
   1. Work closely with UKFast, the client’s hosting partner to:
      1. Have the server IPv6 enabled
      2. Create and allocate an IPv6 address to this server
   2. Provide the IPv6 address to M3 Networks to update the DNS platform

So our client was back up and running the same day.

Are you confident that your matrix-based support team (across 3 different companies, Hosting, Web, IT Support) would work so closely together to get the best solution on a Hogmanay Sunday morning?

This issue isn’t going to go away. More work is being done every day to transition to IPv6, and your site could face a similar problem at any point.

What we need
To get these transitions finished, but with minimal interruption, domain name providers ­– GoDaddy, Reg-123, 1&1 etc – need to schedule their transitions in phases, and forewarn affected clients in each phase so that they can prepare.

You can act now. Check if your current host’s server supports IPv6. If not, ask how soon they will be – or switch host to one who does asap.

If domain registering companies are planning to change their own DNS management tools, then they need to warn customers, who can check their site still functions as expected at the appropriate time. Of course, it’s always good to also ensure the user interface for the IP switch allows a delete or step-back option, albeit temporarily.

If we all work together, the transitions can all be relatively painless.

In fact if you're interested to see who else is out there right now with this problem just visit this link, Websites Who Have The Same Problem Right Now

Developer SOS
Working as we do, with hundreds of clients, analysing their site performance in real time, means we have unique insight into changes like transitioning to IPv6. Our Developer SOS service is available to anyone, even if your site wasn’t originally developed by Inspire. So, if you’re site is down, broken or appears to have been hacked, just get in touch now.


Call 01738 700 006. We’re here to help.