Wikka Developer Blog

The old say “when it rains it pours” has been very applicable to Wikka the last week or so…

We were just recovering from the shock of about a day downtime as a result of our DNS problems (about a day outage 11-12 November, see DNS problem solved: wikkawiki.org is back!) and traffic was picking up again but not quite back to its old level even on the 13th, due to the common delay in worldwide DNS propagation (a change can take 72 hours to reach all DNS servers in the world)…

And then another disaster struck. While throughout the DNS problems we were still reachable from some locations (due to the propagation delay), as of the 14th we suddenly were not reachable at all. Not even via our IP address. Our host (through an obviously very junior tech support person) wasn’t very helpful in initially diagnosing the problem as a domain expiration citing from obviously still outdated data instead of going to the authoritative DNS server to check. Finally we got the real diagnosis: another domain hosted on the same IP address had been hacked, and now had a page used for that most abominable of Internet scams: phishing. Our host’s “upstream” (the service provider they get their connectivity from) had – after receiving an abuse report about it – null-routed the IP address.

Surely that was the right action to take. But we were victims in that we were not only not able to right the wrong, but our sites were as cut off from the world as the hacked domain was. And our host wasn’t exactly forthcoming with information.

In the midst of the panic (what do you do when you’re helpless?) we took some emergency measures: we started by grabbing all installed Wikkas (the main domain and the subdomains) and the corresponding databases, which luckily we could still access via an administrative login. A temporary location was found to put up all this data. During the scare of the DNS problems we had also already registered a few extra domains. The plan is to set up a system where – should disaster strike again – we can have a mirror up and running with a recent version of the databases, practically at the drop of a hat.

Throughout the disaster we took turns in feeling depressed, or very angry, and while we feverishly worked on setting up an alternative, Olivier kept us sane by stolidly continuing to work on the new WikkaEdit. Like the last bit of stability in the midst of an earthquake.

Luckily the offending phishing site has now been removed and our connectivity restored. Phew! All in all we were completely unreachable for about 109 hours: more than 4.5 days.

The access graph below – around 12:30 on the server, and excluding bots – shows the effect of it all: greatly reduced traffic throughout the DNS problems, though it never completely stopped and was picking up again when we were null-routed on the 14th.

Now visitors are clearly finding us again, and the search engine bots are catching up. We’ll have to watch closely to see if we get back to normal levels though.

Meanwhile we continue to work on our backup plan – we were not ready yet, it’s a somewhat complicated operation – so we will be able to quickly migrate to another server should anything happen to our main domain, or our connectivity on our main host. We will also set up an independent “information page” where you can go for information should anything seem amiss with our site.

We hope we’ll never need our backup plan!

Many thanks to our members who quickly alerted us of the problems, and to all of you for your patience. We’re still alive and kicking!

We’ve been experiencing some annoying downtime since Sunday due to a problem with our DNS servers. As I write this wikkawiki.org and all its subdomains work as usual. If you still can’t access the WikkaWiki website, this may be due to some minor delay in the propagation of the DNS configuration, so make sure you check back in a few hours time. Many thanks to all the users who promptly got in touch to report the problem, Wikka and its development team are alive and well again

Usually one doesn’t go out of the way to thank someone who causes damage to other people’s stuff, but I felt it would be an appropriate gesture in this case. As some of you might have noticed, the main WikkaWiki site was again the target of a scripted attack in which automated user registrations were used to vandalize various pages on the site. While the damage was only temporary (all pages were fully restored, and the perl script used for the restoration is available upon request), the attack prompted us to re-evaluate our priorities given the limited amount of development resources available (namely, the time we volunteer to keep Wikka secure and feature-rich).

As a result of this introspection, the decision has been made to incorporate many of the security features currently in “beta” status on the main Wikka site into the main development trunk, and to create release “branches” that will allow us to continue to focus on providing new functionality while still providing the latest in security measures.

What does all this mean to someone who just wants to have a secure Wikka site? Since we are on the verge of releasing 1.1.7, you will have a choice of either being able to download the feature-rich 1.1.7 version (when it’s available), or the more secure, but less feature-rich, “trunk” version. After 1.1.7 is released, those new features will be merged with the security features in the “trunk” for a subsequent release.

Why would someone want to (temporarily) opt for more features and less security? Wikis running on an intranet probably aren’t prime targets for scripted attacks. Those sites that have restricted and/or disabled user registrations are likewise less vulnerable. For these types of sites, upgrading to 1.1.7 makes perfect sense.

We’ll be posting more details on these changes in the near future. I just wanted to take a few moments to let everyone know what we are doing in order to support both security and functionality upgrades without having to sacrifice one for the other.

And to take a few moments to thank our most recent vandal in giving us that “push” we needed to evaluate our development process and priorities.

–Brian