Testing the Untested: World of Warcraft Needs Help!

As you should know, I play World of Warcraft. It's been a great game for several years. Blizzard is making lots of money off of the game, and they are using that to put new content into the game regularly.

There have been a lot of changes to the game since I started playing it. Watching these changes carefully has given a lot of circumstantial evidence to the idea that World of Warcraft is primarily tested by a large quality assurance staff.

This post is a sequel to Testing the Untested. However, it will focus on one example. I think it's interesting to look at what the effects of an inadequate testing program have on a major software project like World of Warcraft.

I'll also talk a bit about what they might have to do to fix their problem. I suspect that roughly the same problems exist on any large project that has the problems that World of Warcraft has.

Blizzard's QA Staff: Are They Relevant?

This interview is interesting. There are apparently 135 "developers" working on World of Warcraft.

The Blizzard guy that they're interviewing makes an interesting distinction between developers and non-developers. It seems clear that a "developer" is a more important person to him than non-developers.

I may be reading too much into this, but that's what this sort of analysis is all about. Please read it yourself below the picture of Nova hanging out in front of an alien cave.

Interestingly, artists working on cinematics count as developers to this guy. However, QA staff does not count.

I'm biased by my work here at PC-Doctor. We hire some incredibly talented QA folks. They have a large role in the development of new and existing products, and developers tend to have a lot of respect for them.

We don't call them developers, either. We also don't sneer at non-developers and pretend they don't count in the real employee count.

I'm going to interpret that interview as a statement that Blizzard doesn't believe that QA staff are as important as artists, programmers, and designers. If Blizzard doesn't give them much responsibility, then they are probably correct.

Anyway, if the QA staff isn't given the respect they need to be relevant, then the programmers are the only ones left who can produce automated tests. In fact, it looks as though QA is the only visible source of tests for World of Warcraft. This might contribute to their perceived irrelevance. If they're spending their time doing things that could have been automated, then it may be hard to gain much respect.

I was pretty disappointed to see the lack of respect in that interview. It looks as though Blizzard's QA staff does a great job with the new content. Unfortunately, it's not possible for them to revisit old content. This is the fundamental problem with relying on a staff to do your testing. It costs a lot to run tests, and so you'll end up running them less.

Do They Have Functional Testing?

Blizzard has stated several times in the past that they're unwilling to change dusty old content that people don't run frequently. They've said that this is because the risk of screwing something else up is too great.

A statement like that is pretty much the same as saying that they don't have enough testing for the old content.

Actually, I expected this section to be a bit longer since the conclusion is so important to the rest of this post. However, if you've got a Blizzard employee who says exactly what you're hoping to prove about their project, then you don't really have to do much more!

I do wish I could find some of the other times that this has been said, but having it said once is sufficient for this article.

Adding Tests After the Fact

Here, I'm going to talk about how Blizzard should be adding tests. It's mostly interesting because the story almost exactly the same for any company that doesn't have a large set of automated tests for their software.

How should Blizzard go about creating tests? This is has already been the subject of another post. In fact, I'm going to say many of the same things.

The first thing to worry about is whether or not the corporate culture supports testing. If it doesn't, then this is the most serious problem facing someone trying to add tests. Testing has to be thought about be all of the developers. It really has to be a part of the normal operation of the programmers. It has to be a part of their culture.

World of Warcraft has been in development for almost ten years now. If they still don't have an extensive set of automated tests for the game, then they clearly don't understand what they're missing.

It's pretty hard to imagine how someone might convince them that testing is important if they haven't seen it already. The biggest advantage of automated testing is that you can make changes with some confidence that nothing was broken. However, you don't get to that point until you have relatively thorough tests.

Developing a thorough set of tests for a game as large and old as World of Warcraft would be an enormous undertaking. Therefore, some advantage would have to be found for incrementally adding tests. If tests can be created that verify parts of the game that are difficult to test with a QA staff, then these would be easy to convince people to add.

The easiest example that I can think of is a test to ensure that the floors don't have holes in them. Whenever Blizzard releases new content, there seem to be places that people can fall through the floor into a location that they're not supposed to get to. I have no idea where this problem come from, but it sounds as though it should be covered by an automated test.

Adding this sort of test allows developers to slowly add real value to the automated test infrastructure. As long as there is value in each step taken, it is easy to convince people that the work is valuable. Eventually you can hope that you'll end up with enough tests that you'll

Another, riskier approach could also be taken. Class balance would be extremely difficult to verify, but a test for it would be extremely useful and visible.

There are a large number of different character classes in World of Warcraft. Each class has different capabilities, but those capabilities are supposed to be equally useful under certain circumstances. Getting this correct is extremely important to players, and getting it correct is extremely difficult as well.

Blizzard's players and staff spend a lot of time thinking about it, and it gets tweaked over and over again. If testing this could be partially automated, then they could speed up the process. Customers and developers would both enjoy this a lot.

It's not clear that it's even possible to automate this. A few things can be analyzed easily in a simple spreadsheet. More complicated aspects of balance would require some extremely sophisticated analysis.

However, Blizzard has some really big supercomputers^*. If it were valuable enough to them, they could run some fairly sophisticated tests. I can imagine some partially automated tests that could analyze even arena class balance. Input from the QA staff could be used to speed up the tests considerably.

If this worked, then it would go a long way towards convincing the rest of Blizzard to try other problems. Again, this approach would be significantly riskier. If the project failed, it might set back automated tests even further.

World of Warcraft Isn't Alone

World of Warcraft is a huge project that clearly suffers from a lack of automated tests. Everything I've said here is specific to that game, but it comes from my experience on other, smaller projects with the same problem.

A lot of projects have exactly the same problem, and solving them requires a lot of the same tools.


* Actually, we don't know this. However, The9 gets most of their revenue from World of Warcraft and runs the Chinese server clusters for Blizzard. They also have 12 of China's fastest publicly benchmarked supercomputers. It seems safe to assume that Blizzard themselves also have similar servers. While none of those are dedicated to testing, it seems likely that they've got some extra CPUs around that could be used.

Testing the Untested

Test driven development is the cool new way to write software. TDD revolves around writing the tests for the software before or during the development of the software. If you do anything like that, then you'll end up with well tested software.

One of the fantastic things about well tested software is that you can change it confidently. Rapid change is the mantra of a large number of the newer software development methods.

I'd like to talk about the other end of the spectrum, though.

What happens if you've got a collection of code that doesn't have a good collection of tests? What happens if the code was written without thinking about testability?

It can become extremely hard to add tests to code like this after the fact.

I carefully phrased the problem so that you don't have to admit to writing the code yourself. I've written code like this, though. I'm guilty.

I've also recovered successfully from poorly tested code. What does it take to make this recovery?

Unit tests

Unit tests drive individual functions within a specific module. A good set of unit tests will help improve your API's reliability.

If you wrote the module with testing in mind, then you'd have a small executable that would load the module and run the tests. This could be done fairly quickly, so you could incorporate the testing into your build process.

If you didn't write the module to be tested, then there's a chance that the module depends heavily on the application's setup and shutdown routines. I hope that's not the case, because, if it is, you'll either have to refactor a lot of the setup code or you'll have to run your unit tests within the application.

The latter option means that, in order to run your unit tests, you've got to startup the entire application and then go into a mode where tests are run. Assuming the application startup time is significant, this will slow down unit testing and make it happen less often. Avoid this if it's possible.

If you can create a light weight process that can load the module to be tested, then you'll only be limited by the amount of time you can spend on tests.

Well... That's not quite true, actually.

In theory, you can write some unit tests and incorporate those tests into your build process, but you originally wrote that module in a corporate culture that didn't require unit tests.

I'll talk about that problem at the end of this post. It's important.

How Many Unit Tests Are Needed?

How many tests should be added right away? This is going to depend on the project's goals. You won't be able to get to the point where you can make changes with confidence until you have a mostly complete set of tests. Getting a complete set of tests will be a lot of work, however. The good thing is that you'll probably fix a bunch of bugs while you create the tests.

If you don't trust your code and you have an infinite amount of time, I'd recommend going all out and creating a complete set of unit tests. This will fix a bunch of problems with the code, and it will force you to understand the code a lot better. In the end, you'll have a lot more confidence in the code.

Of course, time constraints are likely to prevent you from spending that much time writing new unit tests. What's a more realistic amount of tests to add?

If you're adding tests late in development, it's probably because there's a problem that needs to be solved. If the problems come from a small number of modules, then these would be a great place to start.

Have a small number of modules with tests is still extremely useful. It will increase your confidence in those modules, and it will allow you to change those modules more easily.

System Testing

System tests use the entire application or almost the entire application to test a variety of use cases. Generally, these are black box tests. This means that you don't worry about what code you're testing. Instead, you test from the user's point of view. A system test will answer questions about what a user can and can't do.

System tests are often easier to insert into an application that wasn't designed for testing.

There are several ways you can create these.

First, you can use a tool that drives your application through it's UI. There are a variety of tools out there. They can click on buttons and menu items., I'm not familiar with any of them, so I won't make any recommendations.

The tools that I've seen strongly couple your tests to your user interface. This might be good if you want to test the details of your UI, but generally you'd rather have tests that are easy to maintain.

If you have a set of stable keyboard shortcuts, then you could use a tool like AutoIt or the Win32 keybd_event function to drive the program. I'd prefer this over something that looks for controls on the screen and sends mouse clicks to them, but I may be too conservative here.

A further improvement of this technique can be done by using a macro capability that's already built into the application.

This bypasses the user interface. You might consider that a problem, but a macro language is likely to be a lot more stable than the layout of menus and dialog boxes, so the tests themselves are likely to be significantly easier to maintain.

Besides sending in inputs, you'll also have to verify the program's output. Saved files, the clipboard, or scanning log outputs can all be used effectively.

System tests are significantly more complex to setup than unit tests. Furthermore, they take a lot longer to run.

Because they take so long to run, it is unlikely that you'll ever convince your team to run these as part of their build process. Instead, you're likely to have a separate machine or set of machines setup somewhere to run the tests. That's what we do here at PC-Doctor for unit tests.

It's easy to make system tests take so long to run that they become unwieldy. Try to focus your tests on individual use cases. Keep in mind the length of time that you'd like your tests to take, and keep that budget in mind.

Of course, you can always add more machines to your testing farm, but this might not happen until you convince your workplace that they're useful.

A Culture of Testing

A culture that thinks that unit tests are a waste of time isn't going to want to add tests to their build process. They aren't going to want you to spend the time to build the tests, either.

Changing corporate culture is a complex topic that will have to be in a different post. It's hard. I don't recommend it.

Instead, I'd hope that the project that you're adding testing needs it badly. In that case, the benefits of testing should be large enough that you may start to get people to notice.

Don't expect everyone to suddenly run all of your unit tests when they compile your modules. Instead, use your work to start a discussion about it.

Examples

I'd also love to give some examples. One project that I worked on many years ago went through the whole process. World of Warcraft desperately needs to go through the process. (It annoys me how few tests they've got.) I'd like to talk about both.

I'd also like to keep this post relatively short, however. I'll wait until part 2 to discuss specific examples.

Interface Design: World of Warcraft vs. Excel

Blizzard's World of Warcraft (WoW) game is extremely successful. For those who haven't been paying attention, it's a game where you coinhabit a fantasy world with thousands of other players. There are over 9 million people who play the game enough to pay Blizzard a subscription to play. The game is successful, and, in part, it's because of the interface. There are some lessons in Blizzard's interface that can be directly applied to conventional desktop or web applications.

WoW is a game that's designed to be appealing to both the casual user who doesn't play much and a smaller but more dedicated group of people who put a lot of effort into getting really good at the game. Microsoft Excel is another great example of a user interface that has to appeal to both casual and expert users, but it's a lot more fun to talk about video games than spreadsheets.

In terms of appealing to both casual and expert users, WoW is far better than any other application that I've ever used. In part this is because Blizzard is trying to convert as many players as possible from casual to expert. For many people, it's not possible to remain interested in a game for a long time without becoming an expert. Because of the subscription business model of WoW, it is critical for Blizzard to keep players interested for a long time.

In this blog, I'm going to look carefully at how Blizzard does this conversion. Microsoft Excel makes a similar attempt, but it is fundamentally different as well. Excel must be usable for both my mother who wants to make a shopping list and my wife who wants to model a wastewater treatment plant design. Microsoft does not need to convert my mother into someone who understands Excel as well as my wife. Microsoft will make the same amount of money from my mom and my wife, so there's no need to convert between the two types of users. WoW makes a lot more money off of someone like me who plays regularly than someone who plays for a bit, gets confused about how to do something, and then gives up out of boredom or frustration.

If you look carefully at WoW, a huge amount of effort has gone into slowly teaching a user about how the world works and how the avatar that the user is playing works. In fact, making the game easy to discover and learn has received an enormous amount of effort from Blizzard.

I'd like to think that authors of non-gaming applications can learn from this. Becoming an expert in an application is extremely satisfying. It's really fun to use Excel if you're comfortable using exactly the right features you need to accomplish a task. It's also really fun to play WoW after you've become enough of an expert to accomplish a difficult task in the game.

There are three things that an expert WoW user knows how to do. First, they know what their avatar can do in the world. Second, they have to understand the interface that Blizzard has provided for the game. Finally, they have to understand the "physics" of the world that their avatar is interacting with; they have to understand how the world reacts to what their avatar does.

Starting out in the world

When you start out playing the game, you don't have much to worry about. There are about 6 spells, abilities, or items that a starting avatar can use. All of the interface options are turned off, so the interface itself is almost empty. Furthermore, there's nothing that might attack you anywhere near you, so much of the complexities of the world are eliminated. Everything is extremely simple.

It's really easy to see what you're supposed to do next, too. There's a person standing nearby with a giant yellow exclamation point over their head. Later, you'll learn what this means. When you're starting out, it's something that draws your attention. You'll try to interact with that person, and this will give you your first quest. You have rapidly learned a bunch of new things about the interface at this point.

The first quest gives you something to do, and something to learn about. Frequently, it just involves walking a ways over to someone else. This is really easy, and it's impossible to be discouraged by this quest. Furthermore, you'll learn how to move around a bit.

Meanwhile, Blizzard is putting a bunch of little "tips" about your avatar and the world on your screen. If you want to, you can read these, but the design holds up without these. Only a few of these are really required to figure out what's going on.

Like Excel, WoW makes it easy to get by without knowing much. In Excel, you can type a list of grocery items without knowing anything about optimizers. In WoW, you can play with the 6 abilities that you're given at the beginning. It won't help to cast Healing Touch on yourself, but you can see what it does almost immediately. The casual user portion of both interfaces is easily discoverable.

Advancing through the world

As you progress through the game, Blizzard will try to teach you about different parts of the world, different abilities of your avatar, and different parts of the interface.

The interface can be played with right away. There are option screens that allow you to turn various features on or off. When starting out, they are all off. This is a lot like Excel hiding the less popular menu items. (But it's not annoying because the interface is changed directly by you. There are no surprises about moving menu items in WoW.) Furthermore, there are some buttons that are tied to actions that aren't needed for a while. Blizzard doesn't do a lot to make sure you know about some of these features, but you'll learn quickly about some of the more useful ones from other people you meet in game. Blizzard also has some information on them on their web site for those who are actively trying to learn about the interface. Some of the features that are only needed by experts are explained only on third party web sites.

This is a great model to use for some non-game applications. Focus on the necessary stuff. Make features that only experts need available, but don't try to get everyone to use them. Put your energy into making the casual features easy and the expert features possible.

The physics of the world becomes apparent only slowly. Blizzard has designed it so that only experts need to understand most of the physics. If you don't want to think and optimize what you're doing, it's perfectly fine to have the wrong impression of what's going on. You'll progress at a slower rate, and you might not be able to do some of the things in the game that are designed for experts, but you will be able to do a lot.

This is also a great model. If things are happening in the background in your application, make sure that the users don't have to know exactly what's going on. When I'm using Excel, I shouldn't have to know what its rules are for when it re-evaluates each cell. It should just work automatically 99% of the time. Experts might have to know, and they can look it up somewhere, but all of the programmer's energy should be devoted to making sure you don't have to know about it rather than explaining how it works.

The abilities of the avatar are slowly learned as you progress. This makes them almost trivial to teach. Every so often you'll kill enough monsters that you'll be able to learn a small number of new spells. Some might simply be more powerful versions of a spell you already knew. Sometimes, you'll learn a completely new spell. At these points, it's easy to convince a user that they should play with the new abilities they've been given.

I don't think Microsoft could hope to do this well with Excel. If Microsoft tried to force users to make a shopping list before they were allowed to use the SUM() function, you'd have a lot of bloggers like myself ridiculing them. The ability to teach a user in this way may be unique to video games.

Experts only

Blizzard has put a lot of time into creating content that experts can explore. I have to admit that I haven't explored any of the content that they've created that requires 25 people working as a team. However, I think I've got a reasonably good understanding of the behavior of monsters and the abilities of my avatar.

There are things in the game that are extremely complex, possibly even dangerous, and Blizzard doesn't do much to explain them. Users are expected to figure it out, look it up online, or not use it at all.

This is a good thing.

An expert user has to be able to think they are an expert. It's no fun being completely confused about a game after playing for days on end. There have to be things in there that are difficult to use or do correctly in order to convince people that they know what they're doing. They shouldn't be impossible to figure out, but a challenge that is overcome is a lot of fun.

Lots of people say that they are experts at Excel on their resumes. A very, very small number of people understand how to use all of the features of Excel. (Many of them are probably QA employees at Microsoft.) Users like to think that they're experts. To the developer, it really doesn't matter if they are experts or not. Both Blizzard and Microsoft make exactly the same amount of money when their so-called expert runs to the store to buy the next expansion pack or upgrade. They don't care if they really are experts.

Let's look at an example of how you can make someone feel like an expert in WoW. Last week, a small group of friends and I were playing WoW in an area that we hadn't been to before. A friend cast a spell that he almost never uses called Soulshatter. Disaster struck immediately, and we all died.

Soulshatter is a spell that makes you feel like an expert after you understand it enough to use it safely. The source of in-game info for the spell says this:

Soulshatter
8% of base Health
Instant
5 min cooldown
Reagents: Soul Shard
Reduces threat by 50% for all enemies within 50 yards.

Most of this is nonsense for someone who isn't already at least familiar with the game, but some of it is actually nonsense to a lot of people who think that they're experts!

First of all, the spell costs some small amount of health to cast, and it also takes a "Soul Shard" to cast. A Soul Shard is an item that requires some time and energy to create, so this cost will discourage people from trying it when they aren't really interested in what the spell does. This is good, because the spell shouldn't be cast without careful thought.

What does it do? It doesn't behave at all like what the tooltip says, even if you understand what threat is, what a cooldown is, and what the "instant" might mean. It causes all monsters that are even remotely near you to be interested in attacking you! If they're already attacking someone, then they'll become less interested in attacking you (that's the point of the spell), but if they're pretty far away and haven't noticed your avatar yet, they're going to come running over and attack you (that's why we died).

I don't know about you, but that's not at all what I would have guessed after reading the description. Clearly, this is a spell that's designed only for experts. I read a description of the spell before we played, so I felt like an expert when I explained it to my friend. Is it really sufficiently complicated that I deserved to feel like an expert? Absolutely not. That's not the point. I felt good making my friend feel bad, but now we both understand it. Next time, he'll feel good about using it safely.

Notice that the link I used is not to a Blizzard owned web site. This is yet another great lesson for application developers. Online communities can and will help you out with expert-only features. If you're not writing for a game with 9 million people playing it, then you may have to create your own, but if you do, then people will have a place to go to find out the obscure parts of your program that you shouldn't be spending much time perfecting.

Experts can also be entertained in ways that don't impact casual players at all. For example, It is possible to get an item that makes you completely immune to any damage for 10 seconds. If used at exactly the right time, this will wow and amaze your friends and enemies. If used very carefully, an expert might even be able to do something useful with it, but it's mostly an item that's good for a laugh. Getting the item is mildly painful, but it's available even to fairly inexperienced players. This is a "feature" of the game that can benefit an expert and keep them entertained for an hour or so, but it also helps out casual players. Again, if someone uses this successfully, they will feel like an expert.

Many other examples exist in the game. I really enjoy discovering how Blizzard has made the game both easy and fun to discover. Not all of the work they've done is applicable to desktop and web applications, but it's frequently fun to try to imagine how it might be applicable.

This originally appeared on PC-Doctor's blog.