I never used to be an advocate for testing multiple models. In fact, when I was a young software designer, I hated it, because I assumed my model was always the way to go. Most Web and software designers work this way. Maybe it’s something baked into designer DNA.
But “dueling design concepts” can wreak havoc on the development process and rupture team spirit. You know the scenario: Sharon (or her team) has one approach; Jim (or his team) has another. Emotions are high. They’re starting to lash out at each other over this. Who’s right? How do you resolve it?
Design shouldn’t be a personality contest—or a guessing game. By incorporating multiple model testing in your application development process, you end up with a better product at the end of the day, and you remove much of the emotion from the decision-making process.
Depending on the nature of the design issue, you might want to consider one or more of the following multiple model testing techniques. The key in using any of these is to establish the right metrics for success before the test.
Explore User Reactions with Remote Usability Testing
SITUATION: Various members of your group propose radically divergent user interfaces to a new set of functionalities. It would be great to know what real users thought—without footing the bill for a full-scale lab test, or completely programming either version.
SOLUTION: Develop all three prototypes and test via remote Web usability services. New Web-based services, such as TechSmith’s UserVue, eliminate the high costs of lab-based testing, and allow development teams to get rapid feedback on user interface concepts. A user can see and interact with one or more of the concepts online, and talk to you (or a hired moderator) about their reactions. Observers from the design, development or product team can observe the sessions from their desktops.
Advantages
While seemingly the “poor cousin” to live usability testing, remote usability testing services are a good choice for many organizations for the following reasons:
They are less expensive overall. There are no expensive focus group facilities, and recruiting costs are extremely minimal with telephone/Web-based research.
They are time-efficient and convenient. The moderator, participants and observers don’t have to leave their desks, and sessions are usually shorter.
The voice-only channel provides a neutral environment for participants. Many find they are less inhibited as a result, and feedback can be more honest than with in-person lab testing. Honest feedback is what you’re looking for.
Downside
Remote services and software setups are never foolproof. Some bumps, false starts and occasional haywire connections almost inevitably happen in remote testing. It’s part of the territory.
When to employ it: Remote Web testing is an excellent gauge for “going in” decisions on major new releases or new functionality.
Evolve Feature Sets with A-B Testing
SITUATION: Your website or Web-based application is currently running with some known problems in performance in a critical transactional flow. A high percentage of the users who start the transaction are not completing it. Your team has a few distinctly different ideas on how to solve the problem.
SOLUTION: Test the different approaches through A-B testing. Partition the live site with different versions of the flow running, then redirect equal numbers of site traffic to each model, as well as to the control.
If you have sufficient site traffic, you can test all your models at once, then see if any of them do better against your current model. Or, you can test two approaches at a time against the control, to see if you get a performance boost in any one model. Two weeks is a sufficient time period for A-B testing on a live site or application.
Advantages
Numbers talk. It’s much easier to accept a different design model when there is a quantifiable boost to key metrics (e.g., completion rate on the flow).
Downside
As with most Web-based analytics, you are just tracking patterns or success rates, not hearing what users think. Why they do what they do will remain a mystery or be open to interpretation.
You have to be prepared for having the control occasionally win in A-B testing on a live site or application. The solution is to either work up more models, or proceed to another form of testing (see below).
When to employ it: When the need to boost the numbers on a focused flow is critical and you have multiple design concepts already on the table.
Improve Page Metrics with Multivariate Testing
SITUATION: Not enough users are clicking through from your site’s landing pages. You need to boost the numbers.
SOLUTION: You may want to try multivariate testing. Companies such as Offermatica (San Francisco, Calif.) offer Web-based multivariate testing software that can take the guesswork out of fine-tuning specific pages.
Multivariate testing software can be used to rapidly prototype, test and fine-tune new ideas and improve overall performance of any page. Major sites have used it to fine-tune the design of their home pages, for example.
The way it works is this: different “recipes” are established for basic design approaches, on top of which specific design elements such as screen prompts or buttons or graphics can be varied. A statistical model tests a subset of all possible combinations; then an algorithm determines how all combinations would perform. The result is an optimized combination of elements, based on the subset performance.
Advantages
The “plug and play” analysis is very compelling. Multivariate software tools do the number crunching for you and come up with a precise design recommendation.
Downside
It’s not labor-free to embark on multivariate testing. Your team will still have to supply the specific design elements, such as prompts, buttons or graphical elements.
You need to have a fairly large site with a constant, ongoing need for new page fine-tuning to justify the ongoing service fees.
When to employ it: When new pages are continually introduced on your site or application, and need fine-tuning (e.g., landing pages, new content pages).
Summary
Testing multiple models can seem like the slow boat to China for those who like to “run it up the flagpole and see who salutes,” but all of the methodologies discussed here are critical to success in today’s competitive environment. There is no substitute for knowing more, and by testing multiple models you learn not only what works, but what doesn’t work, which creates a base of knowledge for your development team.
Remote Web testing can help you weed out user interface design concepts that don’t appeal to your user base. A-B testing can help you pick a design approach that will improve your metrics for a particular flow. And multivariate testing can help you turn a page that doesn’t work into one that does.
Just as importantly, by using these tools and methodologies, you establish a neutral playing field for your design and development team. By incorporating such techniques into the development process on an ongoing basis, design team members come to accept the fact that design testing is part of the process—and that what works, stays (and what doesn’t work, goes away). In the end, it’s a win-win for everybody.
Meryl Enerson is president and founder of Enervision Media, a user-centered research and design consultancy. Enervision has assisted software companies as well as Fortune 1000 corporations in evaluating and improving the usability of their websites and applications. She can be reached at meryl@enervisionmedia.com.
