When it rains…

The old say “when it rains it pours” has been very applicable to Wikka the last week or so…

We were just recovering from the shock of about a day downtime as a result of our DNS problems (about a day outage 11-12 November, see DNS problem solved: wikkawiki.org is back!) and traffic was picking up again but not quite back to its old level even on the 13th, due to the common delay in worldwide DNS propagation (a change can take 72 hours to reach all DNS servers in the world)…

And then another disaster struck. While throughout the DNS problems we were still reachable from some locations (due to the propagation delay), as of the 14th we suddenly were not reachable at all. Not even via our IP address. Our host (through an obviously very junior tech support person) wasn’t very helpful in initially diagnosing the problem as a domain expiration citing from obviously still outdated data instead of going to the authoritative DNS server to check. Finally we got the real diagnosis: another domain hosted on the same IP address had been hacked, and now had a page used for that most abominable of Internet scams: phishing. Our host’s “upstream” (the service provider they get their connectivity from) had - after receiving an abuse report about it - null-routed the IP address.

Surely that was the right action to take. But we were victims in that we were not only not able to right the wrong, but our sites were as cut off from the world as the hacked domain was. And our host wasn’t exactly forthcoming with information.

In the midst of the panic (what do you do when you’re helpless?) we took some emergency measures: we started by grabbing all installed Wikkas (the main domain and the subdomains) and the corresponding databases, which luckily we could still access via an administrative login. A temporary location was found to put up all this data. During the scare of the DNS problems we had also already registered a few extra domains. The plan is to set up a system where - should disaster strike again - we can have a mirror up and running with a recent version of the databases, practically at the drop of a hat.

Throughout the disaster we took turns in feeling depressed, or very angry, and while we feverishly worked on setting up an alternative, Olivier kept us sane by stolidly continuing to work on the new WikkaEdit. Like the last bit of stability in the midst of an earthquake.

Luckily the offending phishing site has now been removed and our connectivity restored. Phew! All in all we were completely unreachable for about 109 hours: more than 4.5 days.

The access graph below - around 12:30 on the server, and excluding bots - shows the effect of it all: greatly reduced traffic throughout the DNS problems, though it never completely stopped and was picking up again when we were null-routed on the 14th.

Now visitors are clearly finding us again, and the search engine bots are catching up. We’ll have to watch closely to see if we get back to normal levels though.

Meanwhile we continue to work on our backup plan - we were not ready yet, it’s a somewhat complicated operation - so we will be able to quickly migrate to another server should anything happen to our main domain, or our connectivity on our main host. We will also set up an independent “information page” where you can go for information should anything seem amiss with our site.

We hope we’ll never need our backup plan!

Many thanks to our members who quickly alerted us of the problems, and to all of you for your patience. We’re still alive and kicking!