WoW: Is Rolling the Dice a Good Way to Select Beta Testers?

I dub thee beta testerLately I’ve started wondering about the validity of the selection process for the beta test of Blizzard’s Wrath of the Lich King. In the Burning Crusade Blizzard admitted that players were selected randomly. In the new expansion players have been given the chance to opt in. Still, it seems to me that getting into a beta should be based on more then just relying on capricious Lady Luck to choose your testers. You really have to wonder how serious Blizzard is about their MMO if they are not basing the selection process on some form of transparent and reasonable criteria.

Here’s what we know so far about how Blizzard selects their beta testers:

We generally pull from many places so that we have a good variety of people in the beta. Some of it is random and some are winners from contests, or specific groups that we want to pull from. We don’t have a specific schedule for you on who we will pull in or when to share with you, but we’ll do what we can to keep people as informed as possible.

This seems purposely arbitrary. It’s designed to make Blizzard look like they have some kind of criteria but vague enough to cloud the expectations of a potential beta applicant. I’m sure that Blizzard has already contacted their “pet” guilds out there in the community and offered them beta slots. Most other MMO’s have devs that have ties with uber guilds that secretly test content for them. MMO companies love this as they don’t have to pay these semi-pro testers. One of the main reasons these guilds test content on test servers so they can go back and enjoy an intel advantage on their player servers — the very same edge that Blizzard employees have when they play WoW.

While beta tests may be somewhat dubious as to their value in MMO’s, as a game developer I can tell you that selection of candidates for focus testing in the games we developed was done with great care. Only gamers that fit a certain demographic that corresponded to our target audience were selected to play-test our games. These testers all had to meet certain requirements regarding how much time they spent playing video games each week. Evaluating the feedback gleaned from play-tests is an art and science all its own. Of course beta tests in MMO’s are a completely different animal but still…there should be some criteria that are met by the people you allow to test your game.

Here’s a few criteria that I would like to see Blizzard at least consider when granting beta slots:

Loyalty

One of Blizzard’s current methods of inviting beta testers is to reward attendee’s at various official events with guaranteed beta slots. I’m not opposed to that as people who make the effort to attend those functions are extremely loyal as they have spent a lot of money in traveling and lodging. People like that are very passionate about WoW and are likely to be better beta testers then the players who don’t attend. Why not take this idea of loyalty a bit further?

Another way of rewarding loyalty would be to give beta slots to people who have been faithfully subscribing to WoW since its release back in 2004. I’ve calculated that someone who’s been subscribing since the launch would have spent approximately $700 US funds when you combine the cost of both software packages and monthly subscriptions. That’s a substantial investment that deserves some form of acknowledgment and respect from Blizzard.

Then there are the people that have purchased both collectors editions. Add to that people with multiple accounts. Surely loyal subscribers should have preference for getting a chance to get in the beta then just dumb luck?

Character and Diligence

Another type of player that should get preferential treatment for beta slots are the players demonstrate good character by helping to police their community; specifically those vigilant players that go out of their way to report chat, naming violations and other infractions. These people have a positive impact on the quality of the play experience because as we all know Blizzard does not proactively enforce their own codes of conduct — players must do this and request a GM’s help. Without players that spend their precious leisure time filling out petitions offending players would be able to keep violating the rules with impunity.

How hard would it be for Blizzard to track these valuable members of the community by seeing how many petitions they have submitted and reward them with a beta slot for their unpaid service? Players that care enough about the WoW community probably would care enough to actually submit bug reports.

Proven Aptitude

Other beta testers should be admitted based on the strength of a well-written application. Being a good beta tester takes time and commitment. You need to be able to submit detailed bug reports. You need to be able to commit to testing for X amount of  hours per week. People that actually spend the time to thoughtfully articulate why they would be a good beta tester are more likely to be good beta testers. What’s the point of allowing people into the beta test that haven’t at least demonstrated they have the maturity or inclination or time to actually “test” the game? Giving someone a beta slot just because they “opted in” at a website seems like a very shallow criterion for determining someone’s fitness to actually test the game.

Experience and Expertise

Another good asset for a beta tester is experience. Players who’ve beta tested other MMO’s and games and previous releases of WoW have valuable experience that could benefit Blizzard. Also people who’ve raided before are obviously better suited to testing raid content then people who have not. This goes for every aspect of WoW including PVP, crafting, questing, etc.

Players that have taken the time and trouble to actually enter in bugs at the official WoW bug forum should be rewarded for their efforts. At least they have a demonstrated a past willingness to report bugs unlike the average drooling WoW fanboy.

Conclusion

For the most part, recent beta tests have been used by MMO companies as a way to promote interest in their game and I believe that Blizzard is no exception here. It seems they are using the beta test (and the leaks) as a way of generating just enough buzz to keep bored players subscribing. If they were truly serious about testing their game they would show more care about who they are selecting to be part of the beta.

Dangling the carrot of offering beta test slots to players who go above and beyond the call of duty to help other players would also be a good way to improve the community. In a MMO that is so achievement driven, it’s unfortunate that Blizzard hasn’t created volunteer program that rewards the players among us that are good Samaritans. Beta slots cost nothing and would be a great way to motivate players to do good by helping other players.

While I believe that there may be some benefit to inviting a certain small percentage of your beta testing pool based on a lottery, I think Blizzard would stand to gain much more from having a selection process based on some kind of sensible criteria. Why they are purposely missing a great opportunity to bring in qualified testers is beyond me. Just as a company wouldn’t hire unqualified people off the street to work at a company, you also shouldn’t bring in unqualified and undeserving people in to beta test your MMO.

-Wolfshead

4 thoughts on “WoW: Is Rolling the Dice a Good Way to Select Beta Testers?

  1. What you say is correct if the purpose of the beta testers is to really find bugs and help fine tune content. I get the feeling that those types of people are the ones that are included in the alpha– pet guilds, friends and family, staff members and QA, etc.

    It seems to me that the beta test is more of a “taste test” where they are really looking for overall trends among the populace. Are people having fun? How fast are they leveling? What content are they ignoring? A lot of this is probably done with built in metrics, so that the amount of feedback they’re getting directly from non alpha testers is limited.

    If you want meaningful input on the beta process, you need to work for Blizzard or be close enough friends with Blizzard staff that you get into the F&F alpha.

  2. Sorry, wanted to add a true story from The Burning Crusade beta: During the last stage (the hype/polish stage) a lot of testers spent many hours laboriously testing each and every quest, giving detailed feedback via the built in feedback tool, and so on, only to find out after the fact that the feedback tool had been disabled for months and none of that effort was actually being tabulated or read. Whatever Blizzard wanted from these late stage testers, it wasn’t the sort of detailed feedback you’d expect for bug squashing and balance.

    As a result, a lot of very blatant bugs went live despite being generally known about for 2-3 months before release.

  3. I was lucky enough to be part of the friends and family alpha back before the release of the WoW vanilla due to a connection I had with a Blizzard employee. I believe back then they were truly reading the bug/feedback command (I still miss having access to that command — I think it would be very useful even now). By the time WoW got to beta it was more polished then any existing MMO to date.

    Also I recall during the beta phase the uproar over the implementation of “rest” experience. Blizzard definitely learned their lesson there due to the massive player outrage. I doubt they would have caught that problem if there we no beta testing phase.

    Thanks Iconic for sharing that information on the Burning Crusade fiasco with regard to beta feedback. I hope they don’t make that same mistake with Lich King. I understand the WoW devs are arrogant these days but having all that beta feedback go into a black hole of nothingness it shameful.

    I’d love to get my hands on the kind of metrics that Blizzard uses to measure player activity. I’m sure the stats would be pretty shocking to say the least. I think Blizzard has their development system fine honed to the point they probably don’t even need true beta testers anymore. It’s more of a PR and marketing tool now.

  4. I don’t believe invites are random.

    From time to time the WoW clients collects your configuration setup and sends it to Blizzard. They have those configurations tied to accounts with everything you described (character stats). Blizzard is testing hardware configurations too.

    Character and Diligence – They know from gear equiped. Hk’s arena rating’s, reps your diligence.

    Proven Aptitude – They just call for more players when the bug replies fall off. I’ve seen that in other Blizzard betas.

    Experience and Expertise – See character and diligence.

    There is “some” randomness for randomness sake. But I believe beta invites are very targeted.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>