Usability Testing – Part 1

So, I came across a coupon for a free usability test on the website UserTesting.com.  For those not familiar with the site, it basically offers crowd-sourced usability testing allowing website developers to get valuable feedback on how a typical user might experience their site.  Since I didn’t pay anything for the service, I thought I’d give back a bit by writing up my feedback as a blog post.  This topic will be divided into two posts; my experience with UserTesting.com (Part 1), and the feedback I received on my own site from one of their usability testers (Part 2.)

Overall, I thought this site was fantastic.  It’s an idea that I’m quite surprised no one has really nailed before (or if they have, it just isn’t something I’ve run across yet.)  The premise behind the service is quite simple.  You logon, you provide a scenario you want to get feedback on, and one of their testers will accept the project and visit your site.

The “tester” will be using special software that records their screen and allows them to speak into a microphone.  This allows the user to walk through the scenario and follow the instructions you provide as they articulate verbally their comments and feedback.  The scenario I provided was very simple: Create an account, dig up a couple of recipes, add those recipes to the calendar, and then create a shopping list for the week.  UserTesting.com also allows you to filter potential candidates by country (United States, Canada and the U.K.), gender, age range, household income, or computer experience (average web user or expert.)  I decided to look for females between age 35 and 45 with average computer expertise, as I felt this most closely matches my typical user.  I also mentioned that I’d prefer someone who likes to cook, just because I figured that person would more likely “connect” with the site.

You can also create a list of questions for them to answer in written form after they finish with your site.  There’s some common questions filled in automatically for you, but you can change these if you want to ask anything specific.  I decided to just leave the questions as-is, since I thought they were all good.

When you finish creating your scenario, you decide how many users you want to test your site (anywhere from 1 to 50), and the cost is $39 per user.  Since my coupon was good for a single user test, that’s what I chose.

After I posted my scenario, it only took a few minutes for someone to bite.  I spent the next 20 or 30 minutes eagerly refreshing the page as the test was running and after quite some time, the status changed to “User is uploading video”.  The video upload took quite some time (probably almost half an hour), but at last I had my results.

The first thing I did was watch the video, however this proved to be difficult.  It seems that whatever CDN that UserTesting.com is hosting their videos on is pretty slow, as I would only get a few seconds of video at a time before it would pause and have to buffer more.  I got the same issue when trying to watch their promotional video on their homepage.  I decided to just pause the video and wait for the entire thing to download while I grabbed something to eat.  When I came back, the video was completely in the buffer and displayed fine.

You’re also able to download the video to your hard disk as an MP4, MOV or WMV file, however it takes around an hour for the service to “convert” the video into your desired format.  Luckily, they’ll email you when it’s ready to be downloaded.

One useful thing about the online video viewer is you can annotate specific sections of the video if you want to jot down notes or mark certain spots in the video.

When you’re done watching the video, you can rate the user (1-5 stars) and provide any feedback as well.

I can see UserTesting.com as being a great service, but I find it a bit expensive for what it offers.  If I wanted to hire 10 people to provide feedback on my site (these are average Internet users, mind you, not UX experts) then I’d be paying $390 dollars.  I would be better off inviting ten potential users over to my house, providing free food, and doing an in-person focus group in my living room to get their feedback.  This is actually an idea I’m considering for the next iteration of the site.

Though the site appears to be an initial release, they did a great job at providing an MVP (minimum viable product), since it pretty much has only the features you really need and nothing else.  The UI is a little clunky, but I didn’t run across any major bugs or limitations.  I would have liked to see the “Status” screen update automatically to provide a real-time description of the testing process.  Instead, I had to manually refresh the page every few minutes.  Being given more control over what types of testers I want to attract might also be useful.

I think for a larger company with money to burn, the pricing is a steal for this sort of UI feedback from average, everyday people.  It’s very obvious they screen their testers to make sure they’re not complete idiots, unlike services such as vWorker and Mechanical Turk who seem to just let anyone in.  From what I understand, the tester will get $10 out of the $39 fee, which isn’t too bad for about 15-20 minutes of work.

In my next blog post, I’ll be going over the results of the test and what I learned about my site.  The feedback was definitely eye-opening!  Stay tuned!