My World of Warcraft guild is rated by AI as 99.8% resilient

I asked an AI to rate the uptime of our World of Warcraft guild’s infrastructure. It came back with 99.8%: about 15 to 20 hours of downtime a year. That’s better than plenty of small businesses manage, for a group of people who meet up twice a week to fight a dragon.

This post explains how we got there. It covers what the game is, what a guild actually does, what our website does for us, and every layer of redundancy behind that number. It finishes with what it would take to reach 99.99%, and why we’re not going to bother.

Twenty years of Azeroth

World of Warcraft is an online role-playing game from Blizzard Entertainment. It launched in North America in November 2004 and in Europe in February 2005. Millions of people play in the same persistent world at once. You create a character, pick one of two factions (the Alliance or the Horde), level up, and take on harder and harder content.

The game has had a long run of expansions since then, each adding new zones, levels and raids. Over the years Blizzard has also brought back earlier versions of the game for players who miss how it used to be.

WoW Forever is the newest of these: fresh servers built around the original game and its original nine classes. Our guild, Dying Embers, is a Horde guild on the European Forever servers, which launch on Wednesday 4 November. Everyone starts from level 1 on the same day, so how quickly a guild gets organised really matters.

What a guild is, and why it needs organising

A guild is a club inside the game. It has a name, a shared chat channel, ranks from the guild master down to new recruits, and a shared bank of gold and items. You can play WoW alone, but its hardest content is built for groups, and a guild is how those groups form and stay together.

The centrepiece is the raid: 20 to 40 players taking on a dungeon of bosses together, at a set time each week. Molten Core and Onyxia’s Lair are the first ones on Forever. A raid night needs a lot to go right:

• The right people. A balanced line-up of tanks, healers and damage dealers, signed up in advance, with spares on the bench.
• A plan for each boss. Who tanks what, who heals whom, who stands where.
• Fair loot. Each boss drops a few items for 40 people. Most serious guilds use a loot council, where officers decide who benefits most, based on who needs it, attendance and what they’ve already received.
• Supplies. Potions and consumables from the guild bank, given out to the people who need them.
• New blood. A steady flow of applicants to replace players who move on.

Every first kill, written down: our Molten Core progress page.

All of that is run by a handful of volunteer officers, usually after work. Without some tooling it becomes a mess of spreadsheets, Discord pings and “who’s got the boss notes?” That’s why we built our own website.

Our website: the guild’s control room

The site at dyingembers.eu ties the website, Discord and the game together, so each job is done in one place and shows up everywhere else.

The roster, read from the game: mains and alts, ranks, specs and attendance. (The screenshots in this post are from our test copy of the site, filled with made-up members.)

The guild bank: what’s in stock, what’s running low, and who’s been generous.

Crafting: who can make what. Ask for something and the crafters who know it get a Discord message.

Recruitment: what we’re looking for, and the application form behind it.

The addon is the clever bit. Only officers need the companion app. Their addons pass the site’s data to every other member’s game through the guild’s own in-game messages, so ordinary members get it without installing anything extra.

The layers behind 99.8%

Redundancy here comes in layers. Each one catches what the layer below it misses, and the last one lets the guild carry on raiding even when the website is down.

Where it lives

Our VM runs on AS48070, the network of DSM (GB) Limited, trading as MyHostingSpace. The network is built on its own infrastructure across three UK sites: its own Tier III datacentre in Peterborough (home to the network operations centre), a point of presence at Telehouse Docklands North in London, and one at Teledata Delta House in Manchester. It peers at LINX LON1, LINX LON2, LONAP and LINX Manchester with over 800 peers, so traffic to the site has plenty of routes in. More at as48070.net.

Six layers

1. The datacentre. The servers sit in DSM Sibson, a Tier III facility in Peterborough. It has dual power feeds, both backed by UPS, and N+1 generators with at least four weeks of fuel stored on site and refuelling contracts in place. There’s even a 200 kW solar farm on site. A power cut or a broken uplink is handled before we ever notice.
2. A three-node Proxmox cluster. The website runs in a virtual machine on a cluster of three physical servers. If the machine it’s running on dies, the cluster starts it again on one of the other two.
3. A self-healing Docker stack. Inside the VM, everything runs as containers: the Caddy web server, the site itself, its PostgreSQL database and Plausible for visitor stats. A container that crashes is restarted automatically. Caddy renews our Let’s Encrypt certificates by itself, well before they expire.
4. Roll-backs, backups and monitoring. Each update is a versioned release installed with one command. A bad release is fixed by installing the previous one. Backups go off site to Backblaze, so a lost database doesn’t depend on anything in our cluster. And the site is monitored externally from four locations across Europe, so we hear about an outage within minutes, not when someone tries to sign up for a raid.
5. The game keeps its own copy. Every member’s addon saves the latest raids, sign-ups, news and bank requests on their own PC. Officers’ addons pass fresh copies around the guild in game. The loot council runs entirely inside the game. Bank reads and loot results are saved in game and uploaded whenever the site is back.
6. Two places to talk. Raids live or die on voice chat, so we don’t rely on one service for it. Discord is where the guild talks day to day, and we run a backup TeamSpeak server on the guild’s own VM, in the same HA cluster as the website. The two fail independently: if Discord has an outage, TeamSpeak is on our hardware; if our VM is down, Discord isn’t. If Discord goes down mid-raid, everyone hops over to TeamSpeak and the pull carries on.

Failure scenarios

Add it all up and the hardware alone would manage better than 99.95%. The rest of the 0.2% comes from the rows that need a person. Our updates restart the site for a minute or two at a time. And although the monitoring tells us about a problem straight away, anything a person has to fix at 3am waits until that person wakes up.

The part that actually matters: the people. The hall of fame keeps the guild’s firsts and milestones.

What 99.99% would take, and why we won’t

99.99% means less than an hour of downtime a year. Getting there would mean:

• Two copies of the site running at once, behind a load balancer, so updates and crashes never take it offline. Not worth it: it doubles the moving parts to save a minute or two per update. Updates can just happen outside raid times.
• A replicated database with automatic failover, so losing the database server costs nothing. Not worth it: replication is the most fragile thing in this list to run, and run badly it causes more outages than it prevents.
• A second datacentre, so a whole site going dark doesn’t matter. Not worth it: our datacentre already has redundant power and network, and doubling the hosting bill to cover a very rare event makes no sense for a hobby.
• A second public IP and DNS failover, so traffic can take another route in. Not worth it: DNS changes take minutes to spread anyway, which eats most of the gain.
• Someone on call around the clock, so the monitoring’s alerts are acted on in minutes, day or night. Not worth it: we’re volunteers playing a game. Nobody should get paged at 3am because the raid sign-up page is slow.

The honest answer is that the website isn’t the guild’s weakest point. The game already carries on without it, and guilds don’t end because a server went down. They end when the officers burn out or the raid team falls apart. We already have the cheap wins: external monitoring and off-site backups. The one that matters more than any extra nine is a second officer who knows how it all works. The rest of our effort goes into making Wednesday’s launch a good one.