At both playtests we put up two computers playing the game, each accompanied by an extra computer with a google survey on it and an observer from the group taking notes on what and how the player was doing. Anna(game design) had written a quite extensive observation sheet listing all the things to look out for in the player. Somehow though we all just started writing down our thoughts, observations and informal interviews with players on the back of the sheet instead. At the next meeting we compiled all this data together with the survey results and discussed what to do about it.
The common thread was always instructions. Players often didn’t know what to do, missed what instructions and cues were there, or too often didn’t realize there was a HUD at all. There were also parts of the game that felt unsatisfying, most notably the harpoon being clunky and boring. We also of course received a slew of bug reports.
Bugs were generally ironed out whenever convenient unless it was a major issue. Small game means easy bugs for the most part, so me and Matthias mostly just added bugfixes as we went along. This unfortunately meant some issues we knew about getting into playtest builds that we just hadn’t got around to fixing yet, that therefore got reported again. Bugs were the always the least interesting part of feedback, however.
As I mentioned, instructions came up again and again both sessions, and even at the final release we had people not understanding what they were supposed to do. This happened despite us trying to expand the UI and make it bolder both times, and even included a tutorial level. Partly I believe we were to timid and unwilling to let over real estate to the UI, but I suspect the game design itself was lacking as well(more on this in the next blog post).
There is more to understanding than UI of course, and a major source of confusion was inconsistency in the art, such as the bar showing your powerlevel being a different color from the crystals you picked up to refill it, or the scene transition and the powerup effect looking the same way. Easily fixed, once you’ve been made aware of it, but hard to noticed when for you the art just replaced the placeholder art that you’d grown accustomed to. Perhaps all art should have gone through a similar “playtesting” of its own, disconnected from the game.