2.34. The Lemmings Are Marching...

Descendants of Draenor defeats Rage Winterchill,
Hyjal Summit

Rage Winterchill

My heart was racing. I frantically scanned the landscape for anything that could kill me, but wasn't certain exactly what to look for. Twenty-four players were in the instance with me and seventeen of them were dead. My eyes darted down to the damage meters. I took a deep breath, trying to remain calm, keeping my DoT rotation as maximized as possible. Being unprepared was unsettling. Our attention had been a laser for the last five weeks, burning a focused beam of light squarely through Kael'thas Sunstrider and his council. The encounter's nightmarish complexity of tactics and coordination dominated our thoughts and invaded our dreams. With all of our energy poured into his defeat, we spent little time on anything beyond his lifeless body...yet he lay dead at our feet only an hour earlier. In any other case, this would be cause for celebration and an early night off.

Blain was uninterested in ending the raid early.

We still had a long road ahead. Two full tiers of content were now crushed under our boots, leaving us to tread the final path through tier 6. And there was a deadline to keep.

The enormous skeletal necromancer continued to lose health, while Blain kept the raid calm -- kept me calm. Panic was not an option, regardless of your level of preparedness. Be calm. Focused. Do your job. Stop calling things out. Winterchill's health melted away while I hopped out of Death and Decay, my PvP trinket granting me the ability to break out of his ice-block. A few moments later, the lich burst apart in a flash of light.

We had one-shot Rage Winterchill, the first boss of tier 6 content. The last time we had one-shot a boss on the first pull was Broodlord Lashlayer, third boss of Blackwing Lair. Vanilla. Two years earlier. I was beginning to think that the new raid mentality was sinking in. The guild's expectations of fun were starting to align with ours. Constant, consistent success. And for the first time in months, I felt like Illidan was once again squarely in our sights. Our confidence remained intact.

---

This surge of confidence catapulted me back into examining the issues that plagued the guild. Long had I been a slave to the roster, with its constant fluctuations and myriad of faces giving me varying degrees of commitment. Step one was already behind me: set expectations clearly and concisely, so there was no room to misinterpret who we needed each night in raids. In order to shift the power the roster had over me, I needed to grow the pool further, riding on the steam of our recent completion of tier 5. If I could find away to recruit more efficiently, more effectively -- I might be able to put a stop the never-ending excuse factory:

"I don't want to raid anymore."

"Sorry, I'm grounded."

"I don't want to bring my rogue, I want to change my main to a shaman."

"Oh, I thought the raid was cancelled?"

"Man, Nip/Tuck is on!"

It amazed me how a guild like Elitist Jerks could pull it off for an entire expansion (or longer) with a fixed roster -- how do you compel a player to show any sense of loyalty and dedication to the guild? I remained baffled, spending far too much time in-game trying to keep the roster from falling apart. Our new schedule demanded that we raid every Friday and Sunday without excuse. I needed to ensure that Descendants of Draenor held good on that promise. So recruitment continued. I spent hours sifting through useless Battle.net forum posts, cluttered with other guilds begging for new players; leaders in the same boat I was in. Trying to find a single player looking for a new guild was almost entirely a wasted effort, as the reams of forum topics were nothing but officers cast into the same leaky boat as I.

An item from the game Unweb,
written by Hanzo

Gimme The Cache

I needed to come up with a new recruitment strategy, something that would save me hours of sifting through Battle.net forums looking for "that one priest" or "that one druid" that could potentially close the door on the excuse factory for good. I had been toying with the idea of writing a parser in CFML that would scan the Battle.net recruitment forums for users, trying to pick out only what we wanted: players looking for a guild. This automated process could save me hours of recruitment time as it tossed aside the miscellaneous posts of other guilds sharing my pain. But there was a concern with my approach.

Fetching web pages, parsing their HTML, and performing very specific business logic on the content seemed too expensive for CFML to handle. I attempted on-the-fly parsing once before in a web-based game I called "Unweb". Inspired by Diablo II, Unweb let you type in a URL which was converted into a "monster" that you would go on to slay for items and gold. It was a pretty fun exercise to explore application development and allowed me to grow as a programmer.

In practice, however, my architecture was short-sighted. Unlike a typical website driven by static pages of unchanging content, Unweb needed to create content dynamically on each and every page click. To further complicate this, a key step in the process -- the act of reaching out to a specific URL and parsing its content -- was what we in the industry refer to as a blocking call: a part of program execution that must pause and wait for a completion flag before continuing. This pause varies in time because the data is somewhere else, and can only run as fast as the slowest result.

Confused? Let me give you an example of how horrible this particular decision would've played out if it were in WoW. Pretend for a moment that all of the character data for your toon doesn't sit natively in the WoW game client, but rather exists in its entirety in only one place: the WoW armory on the Battle.net website. So while you are playing the game and decide you want to open your character panel to review your stats, the entire game locks up the minute you press the "C" button, as the game client pauses to fetch info from the website. Can you imagine how frustrating this would be if it were the case in real life? Constant pauses and lock-ups, just to look at your character info?

This is what was happening behind the scenes in Unweb, each and every time someone tried to play the game.

The strain of many hundreds of concurrent users playing Unweb caused the webserver to hang while blocking calls waited for URLs to return website results. Alas, this design was the oversight of an amateur developer attempting a large-scale game which didn't lend itself well to HTTP. My lack of experience afforded me no strategies to mitigate its poor performance. The flawed design of Unweb essentially turned what was supposed to be a fun web-based game into a inadvertent load testing tool; the more players that piled on, the quicker the web server buckled.

With the Battle.net forum parser idea in the back of my head, but lacking a real strategy to solve the performance issue, I turned to the one person I knew that could offer me new insight: Ater.

---

"Cache the results", he said to me, biting into a sandwich. Luggage was piled up behind his kitchen table, as he prepared for his upcoming move to Illinois.

"What?" I replied. His simplicity often overwhelmed me.

"You're thinking about it the wrong way." Ater elaborated, "You're approaching this like you've got a search interface, and when you enter in a URL, it's going to go out to Blizzard's forums and parse them live. Right when you click the submit button."

I never told Ater about Unweb, but In that one sentence, he successfully described Unweb's architecture, thereby pointing out the flaw in my original design.

"The search interface will hit a cache, not Blizzard's live forums. You schedule a task for the parser and it'll run in the background, on some fixed interval. Y'know, say every thirty minutes. It'll scan the forums on its own, collect up the data, dump it in the cache...a local database on your server. Then your search interface queries the cache. Simple."

Everything was just so damn simple to Ater. Simple, and one-hundred percent correct. In his succinct explanation, he re-designed my idea and solved the problems I experienced with Unweb. Now, it was not just a CFML-based forum parser, it was a fully-fledged spidering engine.

A screenshot of WoW Lemmings, a guild
recruitment tool, written by Hanzo

Rebuild Your Guild

I went to work immediately, implementing Ater's suggestions. That entire weekend was spent coming up with the initial architecture of the spider. It would run in three passes. The first pass would scan Blizzard's guild recruitment forums, looking for posts, slurp them up, and dump them into a database. On the second pass, the engine would follow the URLs to those forum posts from the database, extracting the bodies of the actual post itself. This is where the meat was: where a player would talk about themselves, what server they were on, what class they played, and where they hoped to find a new guild they could call home. It was the third and final pass where the spider would analyze the content in the forum post, scoring it on such things as: Can I determine a class (ie. Priest, Hunter, Rogue), a server (ie. US-Deathwing), a server type (ie. PvP, PvE). Additionally, was there an armory link provided? Could I use it to process the player's gear? The more information that was provided, the better the score. This scored database of parsed data became what Ater called the "cache" -- a collection of pruned posts comprising only players looking for guilds and nothing else.

I slapped on a web-based interface to allow users to search the cache, and released the new website into the wild, watching the cache grow. Every thirty minutes, another set of players would show up:

"Priest LF new Guild"

"4/5 Hyjal Resto Shaman LFG"

"Tankadin looking for raiding guild"

It was working. Not only was it incredibly cool, but I actually started seeing people I wanted to contact immediately...just while testing the app! The days of wading through hundreds of forum posts were fast becoming a distant memory. Now all the players needing a guild would come directly to me, through a simple search interface. Towards my web site they headed, single file, awaiting their fate. They hoped to be saved, to be plucked from their death march and placed into a new raiding home. It was in that instant that I came up with the name of my new guild recruitment tool:

WoW Lemmings.

A place where guild leaders and recruitment officers could go to find the masses of lost players seeking a new home. A place to rebuild your guild.

On February 13th, 2008, after briefly discussing the tool with Gurgthock, I announced WoW Lemmings on the Elitist Jerks forums, hoping to solicit feedback. Initial response was extremely positive; I received numerous suggestions and tweaks that I used to improve the app further, including the ability to scan other forums outside of Battle.net: MMO-Champion, Wowhead...anywhere recruitment forums resided. WoW Lemmings served (and continues to serve) me, as well as countless other guild leaders, shaving significant time off our recruitment hours. Thanks to this tool, I was able to once again get a handle on recruitment without it dominating all my waking hours. Slowly, eventually, I grew the roster back to a healthy pool of progression raiders (and fillers). And like so many times before, I again had Ater to thank. His simple insight into changing my approach toward the app's architecture turned it from a doomed load-testing tool, to one of the most stable and self-reliant systems I've ever assembled.