ALL BLOG POSTS AND COMMENTS COPYRIGHT (C) 2003-2016 VOX DAY. ALL RIGHTS RESERVED. REPRODUCTION WITHOUT WRITTEN PERMISSION IS EXPRESSLY PROHIBITED.

Saturday, February 14, 2015

Design and playtesting

Ken Burnside has a very useful, and timely, piece on game development, as opposed to game design. I'm much better on the design side than the production side, so it's very useful to be given this sort of reminder of the necessity of playtesting:
The middle 20% of the work is sending a draft of the game out to playtesters, and processing feedback. This is where explanatory diagrams are drawn and the first in-text-flow examples get written.  And rewritten.  And re-done. And re-re-done.  The back half of this 20% is taking the feedback from playtesters…most of whom don’t document everything they did to solve a problem.  Or will send you heated emails because the game blew up on them after they played it for two hours, and now their friends don’t want to touch it ever again.  This is the part where the developer feels “picked on” a bit.  Just remember:

Everyone who ever told you your game sucked, but told you what it was about it that sucked, just helped you make it better. You, as a developer, need to figure out how to resolve this issue, and you need to figure out how to differentiate between “The game sucked…” and “The game isn’t one I’m interested in.”

The worst kind of playtesters are the silent ones.  I put playtest material up on the Ad Astra Games Patreon specifically to weed out the silent playtesters.  These are the guys who download the game, and maybe skim it once, and otherwise let it sit on their hard drives.  I would much rather have playtesters tell me the game sucked than download it, decide it sucked, and never tell me so. :)

You will want two separate rounds of playtesting in an ideal situation – and you really want to get playtesters who don’t know the author of the game if possible; they’ll come in with things they know from knowing the designer, rather than hit the game up from scratch.  The second group of playtesters gets a draft that incorporates any feedback the first group gave you, and ideally doesn’t have any overlap with the first group.

If you have the time, you want to take any feedback from the second group, incorporate it into the draft and put it in front of the first group and see if the two different revision passes shake out any other “Oh, that’s what that means…” moments. This 20% of the work can take up most of the time.
Most technology companies are SHOCKINGLY bad at use-testing; game companies, for all it may seem that they don't do much playtesting, are actually much better than the norm. I was amazed when I found out that in a company of over 150 people, precisely ONE person actually used the product that was the bread-and-butter of the company's market.

Even a program as hoary and well-used as Adobe Reader occasionally shows strange signs of insufficient testing. I prefer to look at files at View/Zoom/Fit Height, but for some reason my documents were opening at 100 percent, which meant that I could see about one-third of the very high resolution images I was reviewing. I went into Preferences, found Page Display, and in it, the Zoom selections, where my options were Fit Width, Fit Visible... and Fit Page. Where is Fit Height?

Now, I'm not an idiot. I correctly guessed that Fit Page, which is NOT an option under View/Zoom, was the functional equivalent of Fit Height. But how is it possible that a program that approximately 11 hundred billion people have used still has basic inconsistencies like this? It's not like Adobe doesn't have the personnel to deal with this sort of thing.

Anyhow, I'm hoping to avoid as much of this problem as possible. One of the things we'll be announcing this spring is our first miniatures game, which will also be called First Sword; it is a fantasy version of the 1977 Avalon Hill game Gladiator, only with a streamlined card-based combat system. (This may or may not be mildly revolutionary in the We the People/Hannibal sense, regardless, it's not a common mechanic.)

I'm preparing a VASSAL module to test the system, so if you happen to be familiar with either VASSAL and Gladiator (or preferably, both), and you're interested in helping me test it, send me an email with PLAYTEST in the subject. I'll probably have the combat mechanic ready for testing in about two weeks; that, the campaign rules are the only elements that really require heavy testing of the miniatures game. The electronic combat management game, on the other hand, will require a bigger group of playtesters, but we're not ready for that yet.

Labels:

22 Comments:

Anonymous DavidKathome February 14, 2015 12:00 PM  

My company has a large suite of input designs for regression testing the software and we still have tons of bug to fix after every major release. Testing is less glamorous and more boring so it often doesn't get the priority it needs.

I saved that article just in case I ever develop my own game.

Anonymous NorthernHamlet February 14, 2015 12:00 PM  

Could someone link me to a good explanatory diagram for video game user-testing? My Google skills revealed little. Thank you.

Anonymous ZeroFill February 14, 2015 12:15 PM  

I'm working on a hex-based space strategy war game. Like a boardgame from the 80s on hex paper, but with some Europa Universalis empire management. More wargame than 4X Master of Orion. Using standard military symbol counters during the prototype and will offer that as a display open when complete.

Unity3D is a God-send. The asset store is speeding things along. I'm so used to working on business software that many of my first games were pure c++ / c# and openGL. What a nightmare for a solo developer that isn't smart like John Carmack.

I'm not going to be shy about asking for brutally honest feedback from hardcore wargame players. I could gives a shit about casual gamers.

Blogger rycamor February 14, 2015 12:41 PM  

Most technology companies are SHOCKINGLY bad at use-testing

I blame that only partly on management. A lot of the blame goes to the arrogance of the developers, who think their users are such idiots that it is pointless to involve them in the process.

Also, software companies tend to use junior developers as testers, which is another bad idea, because they are cut from the same cloth as the developers. At the one dev project I ran, I insisted that the company needed to choose testers from among the other employees and rotate them in for a few hours a week. I set up a simple web-based feedback system and suddenly we started making real usability progress.

Anonymous Jack Amok February 14, 2015 12:45 PM  

at this point of the project, you’re going to see nothing but the warts on it, and everything – including cleaning the cat-box – is going to be more fun.

Man isn't that the truth. If you've been serious about making a design good and stuck with it long enough to get it close to being shippable, your initial excitement about the design, the subject, the execution, all the things about it you thought made it worth doing, that's all ebbed and you're neck deep in all the problems, starting to think the whole thing was a horrible mistake.

Power through it a little farther and it starts to smell nice again, but that last valley is a deep one.

Anonymous Jack Amok February 14, 2015 12:52 PM  

Unity3D is a God-send. The asset store is speeding things along. I'm so used to working on business software that many of my first games were pure c++ / c# and openGL.

Unity3d's the best thing in the world for the first week you use it, then the worst piece of crap ever made for the 2nd week, and if you stick with it past that, it becomes something okay for solo work, incredibly easy to do some things and frustratingly difficult to others.

And I am still working on that Cordova/Phonegap tutorial, but it got back-burnered as my son shifted to a different project. But I do still plan to get it done at some point.

Anonymous cent February 14, 2015 12:58 PM  

the webapps are getting better with more user experience based user interface using rapid wireframe tooles, but old traditional software tend to have more inconsistencies
I've been working with my 12 year old on Meteor based webapps. they are easy to develop and tons libraries. pretty good if you want to develop web applications.

Blogger Vox February 14, 2015 1:15 PM  

I'm working on a hex-based space strategy war game. Like a boardgame from the 80s on hex paper, but with some Europa Universalis empire management. More wargame than 4X Master of Orion. Using standard military symbol counters during the prototype and will offer that as a display open when complete.

Keep me in the loop. Sounds interesting.

Blogger David-2 February 14, 2015 1:28 PM  

This is why the highest quality programs (measured as fewest bugs, not best user interface or highest user satisfaction) are those for which the developers "eat their own dogfood".

For example: Source control systems, development environments, debuggers, email systems, etc. In a team that is developing one of those things the developers in the team use their own daily builds (if the team is run right). The developer needs to use those things every day and if it breaks no work gets done. So bugs get fixed.

Contrast: Document editors like Word, any random applet in the operating system like "Firewall Settings" or printing from the browser. Even the developer who is working on a word processor can go for days or weeks without having to write or edit a document. Even the developer who is working on the browser can go for days or weeks without printing a thing. Bugs accumulate because they aren't causing pain to the developer.

The solution, therefore, is obvious. For a game company, what you really want to do is provide a plugin for the game that connects to your source control system. To get latest version of a source file you've got to find a particular monster and kill it. Can't merge with the latest changes until you get to level 15. Can't check in new code unless you have 25,000 points and 95+ health. Then prevent any use of any alternative interface to your SCM and you're dogfooding your way to a bug-free game!

Anonymous Jack Amok February 14, 2015 1:59 PM  

This is why the highest quality programs (measured as fewest bugs, not best user interface or highest user satisfaction) are those for which the developers "eat their own dogfood".

For example: Source control systems...


Lucky for you that you qualified that as "fewest bugs" and specifically excluded "best user interface" because source control systems are almost universally terrible in that regard. And as far as "fewest bugs" goes, I don't agree that it's becuase of dogfooding. It's because the people who build source control systems understand the importance of never chowdering a file, the same way as developers building medical device controls systems understand it (I've worked on both in my career).

In the design triangle of price-performance-safety, source control developers will choose safety or else go out of business becuase nobody will tolerate a source control program that loses files even occasionally. A game company will usually tend towards performance first, since nobody wants to play World of Tanks at 10fps.

But this post is really about quality of design, and I'm in rycamor's camp on that with the arrogance of the designers. Well, sometimes it's not arrogance so much as ambition. Ken's article mentioned

. So many game writers hate copy-and-pasting rules text, and end up writing rules that say the same thing in subtly different ways.

Many software designers, especially those taking over an existing product, don't think they will get any credit for preserving the existing design (often they are right), and insist on changing it so that it's unquestionably theirs, and they are disinclined to hear any objections.

Case in point: Windows 8. Microsoft dogfoods it's own products extensively. There were many, many complaints from members of the team - often rather senior members - about the idiocy of removing the Start menu. Steven Sinofsky, who was the head of Windows at the time (since fired), made it very clear the decison was made and he wasn't interested in any feedback on it.

Blogger S1AL February 14, 2015 2:27 PM  

Right now I'm wishing I still used vassal.

Anonymous JP February 14, 2015 3:08 PM  

I develop lots and lots of web and intranet applications. I wish I could put into words how frustrating it is when you tell your client, "look, here's what we've done this sprint, please go through it and tell us what you think", then you don't hear anything from them, or they tell you "it looks good". Then two months later they come back to you with issues, bugs and revisions and it's very obvious they never went anywhere near the app this entire time. The best part is that they'll get mad at you because you didn't put a gun to their head and force them to do the acceptance testing. Or you ask someone to test an aspect of the application and when they encounter a problem, the only feedback is "The application doesn't work. I got a server error".

It's things like this that made me question the use of user testing. Though it would probably be different if we had the time and resources to put lots of people into a room with coffee and snacks and asked them to use the software. But as a tiny startup, we've found that user testing is largely useless. We have to make decisions for them. Thankfully we seem to have gotten good at predicting what users want and how they want it.

Anonymous Anubis February 14, 2015 3:22 PM  

"Everyone who ever told you your game sucked, but told you what it was about it that sucked, just helped you make it better."

This is the same white privilege as Edison failing at making a light bulb 200 times before getting it right.

Anonymous ZeroFill February 14, 2015 4:04 PM  

Unity3d's the best thing in the world for the first week you use it, then the worst piece of crap ever made for the 2nd week, and if you stick with it past that, it becomes something okay for solo work, incredibly easy to do some things and frustratingly difficult to others.

I tried Unity a few years ago and got frustrated with it in a week or so. But after working with different libraries and game engines since, giving Unity3d 4.5+ a second look has been refreshing. And the new UI recently introduced is really what I was looking for - strategy games are UI heavy.

The Mono compiler needs to be updated desperately though, but since I'm not even close to being able to deliver a triple-A title I can get by with the script performance. I've got thousands of lines of C# procedural galaxy/solar system code written and importing it hasn't been a big hurdle, even without the Pro license.

And I am still working on that Cordova/Phonegap tutorial, but it got back-burnered as my son shifted to a different project. But I do still plan to get it done at some point.

I have two boys under the age of 3, and until my youngest starts sleeping better I am finding the time to get a lot done frustratingly small. My oldest just wants to wrestle all the time... best part is that its great exercise.

Blogger rycamor February 14, 2015 6:14 PM  

JP February 14, 2015 3:08 PM
I develop lots and lots of web and intranet applications. I wish I could put into words how frustrating it is when you tell your client, "look, here's what we've done this sprint, please go through it and tell us what you think", then you don't hear anything from them, or they tell you "it looks good". Then two months later they come back to you with issues, bugs and revisions and it's very obvious they never went anywhere near the app this entire time. The best part is that they'll get mad at you because you didn't put a gun to their head and force them to do the acceptance testing. Or you ask someone to test an aspect of the application and when they encounter a problem, the only feedback is "The application doesn't work. I got a server error".

It's things like this that made me question the use of user testing. Though it would probably be different if we had the time and resources to put lots of people into a room with coffee and snacks and asked them to use the software. But as a tiny startup, we've found that user testing is largely useless. We have to make decisions for them. Thankfully we seem to have gotten good at predicting what users want and how they want it.


You see, you have a perfect opportunity to force things to go the right way, given that it's a web application.
1. You insist in the original contract that at each handoff stage, the company agrees up front to devote X number of employees or employee hours to review. They will always agree with that in principal, but now that you have it in writing they can't wiggle out of responsibility>
2. Give them a detailed step-by-step review plan: a) create new user account, b) create new order with n items, etc...
3. TRACK whether they actually went to each step. You have a log. Use it.
4. Require them to fill out an extremely simple web form confirming whether each step passed, or what the problem was. Present it as part of the step-by-step plan, not something they do later.
5. If they don't follow through, tell them all development stops (it's in the contract) until they do. It is part of the dependency path of the project, and any delay here pushes back the whole delivery date, not just the next immediate milestone.

Stick to your guns.

Anonymous Aeoli Pera February 14, 2015 6:47 PM  

Adobe sucks. Get Foxit.

Anonymous CorkyAgain February 14, 2015 7:58 PM  

If you're on a Linux desktop, ditch Adobe Reader and get mupdf.

Blogger Captain Atom February 14, 2015 9:01 PM  

By card based combat, are you going with something like the innovative Combat Commander: Europe by Chad Jensen? It sort of an ASL light with commands played from the hand ala the Command and Colors system. It also uses the main draw deck as the dice/CRT. Very good system.

The solo game Fields of Fire also uses the card deck for for the dice/CRT, but also uses cards drawn from a deck to "build" the map as you go. It is probably the most accurate simulation of small unit combat around, but is complex and the rulebook is very hard to follow.

Blogger Kirk Parker February 15, 2015 1:46 AM  

"Steven Sinofski ... (since fired)"

Good start! But... is he living under a bridge somewhere?


JP,

"It's things like this that made me question the use of user testing. "

Are you local to these clients? Can you show up in person and manage a bit of the user-testing process?

Anonymous Jack Amok February 15, 2015 3:33 AM  

Good start! But... is he living under a bridge somewhere?

Not unless his yacht is moored beneath one.

Actually, he never struck me as the kind of guy who'd have a yacht.

Anonymous Peter Garstig February 15, 2015 7:13 AM  

It's really the downfall of most enterprises in the software business. I was part of a successfull niche software that was ridden to the ground by a a team of CEO/Product Manager/Marketing/Sales who decided on every little feauture request, bug and emhancement without ever actually starting, much less using, the software; that gradually became unusable. They even insisted, upon being proposed that devs demonstrate teh software to them, that screenshots being sent to them is enough.
Best people left and company shrinked to 3 people (down from 54).

Blogger GK Chesterton February 15, 2015 11:05 AM  

For the reasons stated in the OP, that is I'm UNfamiliar with Gladiator I'd like to be in your second round.

Post a Comment

Rules of the blog
Please do not comment as "Anonymous". Comments by "Anonymous" will be spammed.

<< Home

Newer Posts Older Posts