There are several different methods available to test users and usability. There are two dimensions to usability: 1) The location of the participants and 2) the participation level of the moderator. These can be combined into a quadrant to balance the pros and cons of each dimension.
Warning: Use each method as the project requires and time is available. Any user testing is better than no user testing so don’t use this guide as a reason to not do any testing. If there are constraints that restrict use of one tool or another, just test any way you can.
LOCATION OF PARTICIPANT: Local vs. Remote
Local testing is testing where real people are physically present in the same room. The moderator and the users are sitting together. Usually the screen is shown via a projector to keep the watchers from looking over the user’s shoulder.
- Mobile testing is easier, technologically.
- Body language and reactions can be better recorded.
- Lower no-show rate.
- Honoraria for participants is high.
- Bad for short tests.
- Set up time.
- Travel expected.
Remote testing is when the moderator and watchers are located physically in a different place from the user. Usually a screen sharing or recording program is used by the participant/user to allow remote people to watch the screen.
- Honoraria not needed or much cheaper.
- No room requirements.
- No travel needed.
- Mobile testing is difficult, due to available tools.
- Higher no-show rate.
PRESENCE OF MODERATOR: Moderated vs. Unmoderated
Moderated testing is where a live person is guiding the participant through the test. Observations are made real time and discussions are possible. Usually these tests have 5-7 participants.
- Usability trends are quickly identified.
- Interviews allow for flexible testing.
- Better for testing general usability.
- Great for longer tests.
- May have vague end states (Good for multi-situational testing)
- Watchers/Moderators can skew the results easily.
- One test at a time.
- Longer to set up (eventually) .
Unmoderated testing is where a computer guides the participant through the test. Observations are made after the fact using data points including: survey results, pass/fail rates, time to completion, and click maps. Usually these tests have 50+ participants.
- Large data sets allow for better trending
- More accurate data sets
- Harder to skew the results
- Simultaneous tests possible
- Quicker to set up. (eventually)
- Great for short tests.
- Able to recruit users in the middle of a task.
- Inflexible testing does not allow for quick changes
- Rough mock-ups are a waste of time to test with large numbers.
- Must have well-defined end states. (Participants are in control of moving forward)
- Mobile Device
- User Zoom
Local moderated testing is great for identifying general trends and overall usability problems. This method should be used near the beginning of a project or anytime a major change needs to be made to an application. Remote unmoderated testing is great for identifying patterns of behavior and for identifying the better of two possibilities. This method should be used to gather trend information for active/live applications and websites. It is also very useful for quickly testing and recruiting for nearly completed prototypes or screenshots.