Getting the right design and the design right: Testing many is better than one

We present a study comparing ‘usability testing’ of a single interface design versus three functionally equivalent but stylistically distinct designs. We show that when presented with a single design, users give significantly higher ratings and less negative feedback, compared to when presented with the same design in a group of three alternatives. Our results suggest that by presenting users with alternative design solutions, subjective ratings will be more accurate and one will obtain more and stronger criticisms. Contrary to our expectations, our results also suggest that usability testing in itself, even when multiple designs are presented, is not an effective vehicle for soliciting constructive suggestions about how to improve the design from end users. It is a means to identify problems, not provide solutions.