Ask Us: How does a rigorous “think aloud” protocol work in qualitative testing?

Recently, we received a question asking us about how a rigorous “think-aloud” protocol works in qualitative one-on-one testing. Here is our answer:

In qualitative, one-on-one testing, we often use a “think aloud” protocol to gain more insight into how the user thinks when using a document or website. In a think aloud, the participant is given a scenario and then reads through the document talking out loud about his or her reactions. Think alouds allow us to hear the consumers’ inner dialogue as they interact with the document for the first time. To ensure that they are comfortable with this task, we have them practice on a neutral document, such as a restaurant menu. As the participant thinks aloud with the neutral document, we can give positive reinforcement about the comments. In addition, the scenario the participant works with frames a task, such as “After reading this, we’ll ask you what action you would take, if any.” This last addition to the scenario ensures that the participant reads with more focus, rather than merely providing a commentary on whatever comes to mind.

Once the session begins, the participant provides feedback based on what he or she notices, but without structured questions on the part of the moderator. Instead, we ask the participant to think aloud – telling us everything she is noticing, thinking, or reacting to. We want to capture participant’s immediate thoughts and reactions before the participant can “learn” from the testing situation and from the moderator asking questions. The think aloud provides us with a good sense of what the participant notices, where information is misinterpreted, and what the participant has trouble understanding. In addition, we learn what next steps the participant would take.

We couple the dialogue to an intense observation of the behaviors as the participant reads. We watch for subtle changes of expression that allow us to probe on whether something was confusing or problematic to the participant so that we can follow up with questions. It is often during this part of an interview that we learn things we never expected and that we might not have asked about. Throughout the think aloud, our note-takers create a visual map of the order in which the participant looks at (or ignores) the information in the document to inform our data collection.

In the analysis, we look at what the participant said, what he didn’t say, our observations, and the visual map to determine what is working within the document and what needs revision.

If you have other questions, please let us know!

Tagged , , , ,