Crowdsourcing Usability Testing VS UX Lab Usability Testing


UX Lab of UT Austin

Background: Usability testing has become more and more important for user experience (UX) research and user interface design. However, conducting usability testing is expensive and time-consuming. For traditional usability testing such as lab testing, researchers have not only to spend time recruiting participants and host lab testing, but also have to provide equipment and testing rooms. Therefore, usability researchers are seeking an efficient way to conduct usability testing. Crowdsourcing is an emergent way for usability testing.

Research summary: The project examined the usability of two popular health information systems (WebMD and MedlinePlus) to explore users’ perception of the two websites as well as the advantages and shortcomings of each websites’ design. I tested the two websites through two ways: user experience lab and Crowdsourcing platform-AMT(Amazon Mechanical Turk).

Task design: I asked participants to perform three tasks on each website and answer questions related to pros and cons for each website. At the end of the test, I asked participants to choose one of the two websites to perform the fourth task.

Data analysis: I analyzed the onscreen activities which were recorded by Morae software, and did quantitative analysis for time spent on tasks, error-free task completion, satisfaction of the answers, overall perception of each website, the website chosen for the fourth task etc.. We also did qualitative analysis for the descriptive data. See the final paper here.

Inspiration: This project inspired me to propose an internal usability small task platform inside VISA. The platform could, on the one hand, utilize the global VISA employees to help decision makers gain better design based on the crowdsourcing ideas, on the other hand, offer much fun to the employees and strength the communication cross the teams. My boss and his boss loved this proposal and they initiated this platform project when I was there, and kept building it after I left VISA.