On the Efficacy of Design “Truth Teams”

At a SXSW panel last week, Matthew Armstrong of Gearbox talked about how video game testing should really be. Griffin McElroy at Vox wrote what appears to be the “definitive” article on the panel, and you should check it out here if you are interested in reading about whole thing.

The tl;dr is that Gearbox created a “Truth Team” that figured out what gamers really want. They tested the game extensively, and instead of taking the information that the gamers gave them as fact-based critique, they assumed that it was emotionally fueled; the things that the gamers said were, most of the time, the things they felt rather than the things they thought.

They came out with three rules of game design that I think can be useful for developers, especially indie devs, and they are:

1. Testers try to speak in fact, but they speak in emotion.
2. Testers do not like features they do not understand.
3. Testers have expectations that may not match the developers.

Of course, we should replace “testers” with “gamers” for a more broad application, but those things are really the same thing.

The first rule makes a lot of sense to me, but I understand that it might seem strange to developers. The gaming world seems to have only recently discovered that emotional connection to characters, plots, and environments is a veritable well of design opportunity. People like to become emotionally invested in games, and they like hooks for that. McGonigal really latches on to this for her project in Reality is Broken–people want fiero, that grand feeling of accomplishment. It makes sense to me that the critique that testers give to devs is based around their ability to reach that feeling. People want to feel good, and if they don’t feel good, their entire reaction to a game is tanked. Our desire to couch that in language probably has to do with the contemporary pathologization of emotions–for things to be serious, they have to be serious facts; we’ve never shaken that strange Victorianism.

A great example of this is my current playthrough of Hard Reset. I rarely feel rewarded or appreciated as a player, and that really makes me dislike the game, and I couch that dislike in a really smart critical framework. At the end of the day, though, I can step back and realize that the game does exactly what it wants to do, and that I’m just not into 1990s-style shooters. I can see how being able to parse those two things would be really beneficial for a dev.

The second and third rules really feel like obvious things. No one likes to feel lost in a game. I have not played a number of “classic” games because they assume a kind of bootstrap knowledge and effort that just destroys my ability to play the game. I am not the smartest or best player of games at the best of times, and I really can’t do it when the game essentially drops me in a completely foreign environment.

The third rule in particular seems to make a lot of sense to me, and it is related to the first rule. People develop emotional reactions to games that the developers do not. The current debacle with the Mass Effect franchise is a direct result of that. Developers, at the end of the day, never experienced the illusion of Shepard and her universe. It was all wireframes, beautiful paintings of future skyscrapers, and grand written narratives about sacrifice and life. The players never really got any of that; they lived the life of Shepard, experienced the minutiae, and never saw the underbelly. What I am saying is that artistic intent, which is really what “developers expectations” means, is nothing compared to the affective linkages, or the “expectations,” that a gamer forms with a game.

The important thing about this whole thing comes toward the end of the article. McElroy writes:

Armstrong and Puri were careful to explain that the Truth Team doesn’t just ignore focus tester feedback. Rather, it analyzes the intention behind that feedback for the most effective solution.

And Randy Pitchford, the CEO at Gearbox, commented on the Truth Team in 2009 in a Gamasutra article:

We start with things we think work, and we actually created a group in October of last year called the Truth Team at Gearbox. The Truth Team. And The Truth Team’s mandate is to tell us the truth. Where are we at? What do gamers, what do customers — what do real customers, not developers, not even journalists. What do actual customers think right now about where we are at?

With my reservations about a team of developers that figure out the “truth” of feedback aside, I think this really shows that games are growing up a little as an industry. The movie making industry has never gotten this smart about focus groups, and many directors still have to lament that kind of testing in commentary tracks. In a world where lots of companies respond to criticism of their games with “you are playing wrong,” it is really great to see a company that is concerned with learning what it means to design better games.

This entry was posted in Video Games and tagged , , , . Bookmark the permalink.

2 Responses to On the Efficacy of Design “Truth Teams”

  1. bonmotsandblood says:

    With regards to conclusion 3: do you think a comparison could be made to authors of novels? (When it comes to character that seems a better correlative than film.) Mass Effect is a very particular example, but these characters and themes must be incredibly real to the directors and producers–and I imagine the coders at times feel like comic book illustrators: it’s a lot of time to spend with characters, even if–as in games–it’s not all narrative or narrative construction. I’d hate to see any more formulaic approach to character writing in the industry. (not that I draw that conclusion from the 3rd rule.)
    -Matt
    (sorry for any sloppy iPhone typing)

    • kunzelman says:

      Sorry for the late reply, the blog has been on auto for a few days while I was out of town.

      I fight back and forth with myself on this point. Sometimes I think that creators have strong affective bonds with their creations, and sometimes I think they see them as tools or instruments to make the fiction work.

Comments are closed.