PC-Doctor is trying to start a series of internal training talks. I'm going to give the first one next week.
It looks as though there's a lot of interest from everyone on this project. Management loves the morale boost and the training that people get. Developers are excited to learn something new. QA seems excited, too!
Here's the draft of my talk:
It's probably worth another blog to discuss the dangers of posting raw slides where the whole world can see them. It's probably also worth talking about why I don't waste my time making everything look perfect.
For now, it's exciting that publishing is as easy as copying and pasting some HTML from Google Docs! :)
Wednesday, August 20, 2008
Tuesday, August 5, 2008
Testing the Untested: World of Warcraft Needs Help!
As you should know, I play World of Warcraft. It's been a great game for several years. Blizzard is making lots of money off of the game, and they are using that to put new content into the game regularly.
There have been a lot of changes to the game since I started playing it. Watching these changes carefully has given a lot of circumstantial evidence to the idea that World of Warcraft is primarily tested by a large quality assurance staff.
This post is a sequel to Testing the Untested. However, it will focus on one example. I think it's interesting to look at what the effects of an inadequate testing program have on a major software project like World of Warcraft.
I'll also talk a bit about what they might have to do to fix their problem. I suspect that roughly the same problems exist on any large project that has the problems that World of Warcraft has.
This interview is interesting. There are apparently 135 "developers" working on World of Warcraft.
The Blizzard guy that they're interviewing makes an interesting distinction between developers and non-developers. It seems clear that a "developer" is a more important person to him than non-developers.
I may be reading too much into this, but that's what this sort of analysis is all about. Please read it yourself below the picture of Nova hanging out in front of an alien cave.
Interestingly, artists working on cinematics count as developers to this guy. However, QA staff does not count.
I'm biased by my work here at PC-Doctor. We hire some incredibly talented QA folks. They have a large role in the development of new and existing products, and developers tend to have a lot of respect for them.
We don't call them developers, either. We also don't sneer at non-developers and pretend they don't count in the real employee count.
I'm going to interpret that interview as a statement that Blizzard doesn't believe that QA staff are as important as artists, programmers, and designers. If Blizzard doesn't give them much responsibility, then they are probably correct.
Anyway, if the QA staff isn't given the respect they need to be relevant, then the programmers are the only ones left who can produce automated tests. In fact, it looks as though QA is the only visible source of tests for World of Warcraft. This might contribute to their perceived irrelevance. If they're spending their time doing things that could have been automated, then it may be hard to gain much respect.
I was pretty disappointed to see the lack of respect in that interview. It looks as though Blizzard's QA staff does a great job with the new content. Unfortunately, it's not possible for them to revisit old content. This is the fundamental problem with relying on a staff to do your testing. It costs a lot to run tests, and so you'll end up running them less.
Blizzard has stated several times in the past that they're unwilling to change dusty old content that people don't run frequently. They've said that this is because the risk of screwing something else up is too great.
A statement like that is pretty much the same as saying that they don't have enough testing for the old content.
Actually, I expected this section to be a bit longer since the conclusion is so important to the rest of this post. However, if you've got a Blizzard employee who says exactly what you're hoping to prove about their project, then you don't really have to do much more!
I do wish I could find some of the other times that this has been said, but having it said once is sufficient for this article.
Here, I'm going to talk about how Blizzard should be adding tests. It's mostly interesting because the story almost exactly the same for any company that doesn't have a large set of automated tests for their software.
How should Blizzard go about creating tests? This is has already been the subject of another post. In fact, I'm going to say many of the same things.
The first thing to worry about is whether or not the corporate culture supports testing. If it doesn't, then this is the most serious problem facing someone trying to add tests. Testing has to be thought about be all of the developers. It really has to be a part of the normal operation of the programmers. It has to be a part of their culture.
World of Warcraft has been in development for almost ten years now. If they still don't have an extensive set of automated tests for the game, then they clearly don't understand what they're missing.
It's pretty hard to imagine how someone might convince them that testing is important if they haven't seen it already. The biggest advantage of automated testing is that you can make changes with some confidence that nothing was broken. However, you don't get to that point until you have relatively thorough tests.
Developing a thorough set of tests for a game as large and old as World of Warcraft would be an enormous undertaking. Therefore, some advantage would have to be found for incrementally adding tests. If tests can be created that verify parts of the game that are difficult to test with a QA staff, then these would be easy to convince people to add.
The easiest example that I can think of is a test to ensure that the floors don't have holes in them. Whenever Blizzard releases new content, there seem to be places that people can fall through the floor into a location that they're not supposed to get to. I have no idea where this problem come from, but it sounds as though it should be covered by an automated test.
Adding this sort of test allows developers to slowly add real value to the automated test infrastructure. As long as there is value in each step taken, it is easy to convince people that the work is valuable. Eventually you can hope that you'll end up with enough tests that you'll
Another, riskier approach could also be taken. Class balance would be extremely difficult to verify, but a test for it would be extremely useful and visible.
There are a large number of different character classes in World of Warcraft. Each class has different capabilities, but those capabilities are supposed to be equally useful under certain circumstances. Getting this correct is extremely important to players, and getting it correct is extremely difficult as well.
Blizzard's players and staff spend a lot of time thinking about it, and it gets tweaked over and over again. If testing this could be partially automated, then they could speed up the process. Customers and developers would both enjoy this a lot.
It's not clear that it's even possible to automate this. A few things can be analyzed easily in a simple spreadsheet. More complicated aspects of balance would require some extremely sophisticated analysis.
However, Blizzard has some really big supercomputers*. If it were valuable enough to them, they could run some fairly sophisticated tests. I can imagine some partially automated tests that could analyze even arena class balance. Input from the QA staff could be used to speed up the tests considerably.
If this worked, then it would go a long way towards convincing the rest of Blizzard to try other problems. Again, this approach would be significantly riskier. If the project failed, it might set back automated tests even further.
World of Warcraft is a huge project that clearly suffers from a lack of automated tests. Everything I've said here is specific to that game, but it comes from my experience on other, smaller projects with the same problem.
A lot of projects have exactly the same problem, and solving them requires a lot of the same tools.
* Actually, we don't know this. However, The9 gets most of their revenue from World of Warcraft and runs the Chinese server clusters for Blizzard. They also have 12 of China's fastest publicly benchmarked supercomputers. It seems safe to assume that Blizzard themselves also have similar servers. While none of those are dedicated to testing, it seems likely that they've got some extra CPUs around that could be used.
There have been a lot of changes to the game since I started playing it. Watching these changes carefully has given a lot of circumstantial evidence to the idea that World of Warcraft is primarily tested by a large quality assurance staff.
This post is a sequel to Testing the Untested. However, it will focus on one example. I think it's interesting to look at what the effects of an inadequate testing program have on a major software project like World of Warcraft.
I'll also talk a bit about what they might have to do to fix their problem. I suspect that roughly the same problems exist on any large project that has the problems that World of Warcraft has.
Blizzard's QA Staff: Are They Relevant?
This interview is interesting. There are apparently 135 "developers" working on World of Warcraft.
The Blizzard guy that they're interviewing makes an interesting distinction between developers and non-developers. It seems clear that a "developer" is a more important person to him than non-developers.
I may be reading too much into this, but that's what this sort of analysis is all about. Please read it yourself below the picture of Nova hanging out in front of an alien cave.
Interestingly, artists working on cinematics count as developers to this guy. However, QA staff does not count.
I'm biased by my work here at PC-Doctor. We hire some incredibly talented QA folks. They have a large role in the development of new and existing products, and developers tend to have a lot of respect for them.
We don't call them developers, either. We also don't sneer at non-developers and pretend they don't count in the real employee count.
I'm going to interpret that interview as a statement that Blizzard doesn't believe that QA staff are as important as artists, programmers, and designers. If Blizzard doesn't give them much responsibility, then they are probably correct.
Anyway, if the QA staff isn't given the respect they need to be relevant, then the programmers are the only ones left who can produce automated tests. In fact, it looks as though QA is the only visible source of tests for World of Warcraft. This might contribute to their perceived irrelevance. If they're spending their time doing things that could have been automated, then it may be hard to gain much respect.
I was pretty disappointed to see the lack of respect in that interview. It looks as though Blizzard's QA staff does a great job with the new content. Unfortunately, it's not possible for them to revisit old content. This is the fundamental problem with relying on a staff to do your testing. It costs a lot to run tests, and so you'll end up running them less.
Do They Have Functional Testing?
Blizzard has stated several times in the past that they're unwilling to change dusty old content that people don't run frequently. They've said that this is because the risk of screwing something else up is too great.
A statement like that is pretty much the same as saying that they don't have enough testing for the old content.
Actually, I expected this section to be a bit longer since the conclusion is so important to the rest of this post. However, if you've got a Blizzard employee who says exactly what you're hoping to prove about their project, then you don't really have to do much more!
I do wish I could find some of the other times that this has been said, but having it said once is sufficient for this article.
Adding Tests After the Fact
Here, I'm going to talk about how Blizzard should be adding tests. It's mostly interesting because the story almost exactly the same for any company that doesn't have a large set of automated tests for their software.
How should Blizzard go about creating tests? This is has already been the subject of another post. In fact, I'm going to say many of the same things.
The first thing to worry about is whether or not the corporate culture supports testing. If it doesn't, then this is the most serious problem facing someone trying to add tests. Testing has to be thought about be all of the developers. It really has to be a part of the normal operation of the programmers. It has to be a part of their culture.
World of Warcraft has been in development for almost ten years now. If they still don't have an extensive set of automated tests for the game, then they clearly don't understand what they're missing.
It's pretty hard to imagine how someone might convince them that testing is important if they haven't seen it already. The biggest advantage of automated testing is that you can make changes with some confidence that nothing was broken. However, you don't get to that point until you have relatively thorough tests.
Developing a thorough set of tests for a game as large and old as World of Warcraft would be an enormous undertaking. Therefore, some advantage would have to be found for incrementally adding tests. If tests can be created that verify parts of the game that are difficult to test with a QA staff, then these would be easy to convince people to add.
The easiest example that I can think of is a test to ensure that the floors don't have holes in them. Whenever Blizzard releases new content, there seem to be places that people can fall through the floor into a location that they're not supposed to get to. I have no idea where this problem come from, but it sounds as though it should be covered by an automated test.
Adding this sort of test allows developers to slowly add real value to the automated test infrastructure. As long as there is value in each step taken, it is easy to convince people that the work is valuable. Eventually you can hope that you'll end up with enough tests that you'll
Another, riskier approach could also be taken. Class balance would be extremely difficult to verify, but a test for it would be extremely useful and visible.
There are a large number of different character classes in World of Warcraft. Each class has different capabilities, but those capabilities are supposed to be equally useful under certain circumstances. Getting this correct is extremely important to players, and getting it correct is extremely difficult as well.
Blizzard's players and staff spend a lot of time thinking about it, and it gets tweaked over and over again. If testing this could be partially automated, then they could speed up the process. Customers and developers would both enjoy this a lot.
It's not clear that it's even possible to automate this. A few things can be analyzed easily in a simple spreadsheet. More complicated aspects of balance would require some extremely sophisticated analysis.
However, Blizzard has some really big supercomputers*. If it were valuable enough to them, they could run some fairly sophisticated tests. I can imagine some partially automated tests that could analyze even arena class balance. Input from the QA staff could be used to speed up the tests considerably.
If this worked, then it would go a long way towards convincing the rest of Blizzard to try other problems. Again, this approach would be significantly riskier. If the project failed, it might set back automated tests even further.
World of Warcraft Isn't Alone
World of Warcraft is a huge project that clearly suffers from a lack of automated tests. Everything I've said here is specific to that game, but it comes from my experience on other, smaller projects with the same problem.
A lot of projects have exactly the same problem, and solving them requires a lot of the same tools.
* Actually, we don't know this. However, The9 gets most of their revenue from World of Warcraft and runs the Chinese server clusters for Blizzard. They also have 12 of China's fastest publicly benchmarked supercomputers. It seems safe to assume that Blizzard themselves also have similar servers. While none of those are dedicated to testing, it seems likely that they've got some extra CPUs around that could be used.
Subscribe to:
Posts (Atom)