C:\> KitchenPC.exe -search -maxtime 30 -avoid eggs

Before launching a new version of a website, especially one that radically changes the entire purpose of the site, it’s usually a good idea to go test it out on a few strangers.  I’ve found asking friends and family usually results in less than honest feedback, as those are the types who are verbally throttled by their pesky respect for your hard work.  Also, the people we get along with are usually the people who think like us too.  That’s no good.

A while back, I gave the site UserTesting.com a go and was very pleased with the results.  Though they’re a bit pricy, I found the data I were able to ascertain from the experience worth their weight in gold.  So back I went for another round, this time going all out and purchasing not one but ten testers to give the new KitchenPC a whirl.

Turns out, my new design is a complete disaster!

The Good

Overall, people have a good first impression of the site.  They understand it allows them to find recipes (no one mentioned meal planning, which means my pivot is a success,) and organize the results they find.  Almost everyone understood what a “menu” was before even trying out the feature, and several people understood “What Can I Make?” just by the description alone.  Very few people had a clue what “Pre-Made Meal Plans” were, but most admitted they were intrigued by the idea and wanted to click on it.

I also got several compliments on the overall look and feel of the site.  People said it was simple and straight-forward.  One tester even signed up with her real email address, claiming she’d be back.

People seem to connect with the value proposition of KitchenPC.  They like the idea of finding recipes and organizing them into menus. They like the idea of finding out what they can make with ingredients they have on hand.  They like the idea of saving time and money, and believe KitchenPC can help them do that.  This means if KitchenPC can deliver on that promise, there’s a customer base out there waiting.

The Bad

Everything else.  The entire search interface was a swing and a miss for strike one.  As the title of this post suggests, sometimes I think I’d be better off re-writing the entire thing as a command line tool.

Nine out of ten testers were completely and utterly confused simply by trying to search for recipes, which is somewhat important to really nail being a recipe search engine and all.  Though all were eventually able to get recipes to come up, very few were able to successfully narrow down their search to desserts without eggs that could be made in 30 minutes or less.

New search interface, no submit button!

The huge culprit here is the lack of a Submit button on the search interface.  I decided to be clever and implement an interface that allows users to narrow down results on the fly, however users just don’t understand this.  Not just some users, all users.   At least half the testers mistook the “Meal” filter (All, Breakfast, Lunch, Dinner, Dessert) as a submit button.  Thus, they would type in something like “Chocolate cake” and click “Dessert“.  In their mind, this means “Find me all chocolate cakes in the dessert category.”  They would then type in something else in the keyword textbox, and then click on “Dessert” again.  Unfortunately, this does absolutely nothing on KitchenPC as these are filter buttons that can be toggled, and not submit buttons.

Five out of ten users tried to type in multiple ingredients in either the “Include” or “Exclude” text boxes as well.  For the most part, they would just ignore the auto-complete that would pop up.  At least two users would type in an ingredient such as “eggs” and then click elsewhere on the screen rather than pressing enter or clicking on an auto-complete option.  A definite trend I noticed is that very few people expect the Enter key to do anything on a website.  Most people will look for something to click on once they finish typing.  Only a few users were able to successfully narrow down recipes to those that don’t include eggs, and almost all of them were looking for something to click on to update the results.

Impossible to see checkboxes

Once users found results, they were tasked to add a few recipes to a menu.  Nine out of ten of them completely ignored the checkboxes next to each recipe, and instead used the popup recipe viewer to create a new menu.  Adding a new menu was also confusing, as several weren’t sure the creation of the menu would also add the desired recipe to the menu as well.  This was because the recipe viewer still shows “Add To Menu” even though the recipe is already in one menu.  However, KitchenPC allows a recipe to exist in more than one menu, so in my mind the verbiage is still correct.

Next, users were asked to narrow down recipes to only those that could be made in 30 minutes or less.  Only a couple users found the “More Options” button and thought to actually click it.  Some finally ran into it by chance after fumbling around for a while.  The majority of users tried to sort the results by time, and were completely confused by the sorting experience.  This is mostly because I allow users to sort by “Total Time“, but I don’t actually display “Total Time” in the result list anywhere (I only display prep time and cook time).  Users were frustrated that they were expected to add up the two numbers in their head to find recipes that met this criteria.  One user was confused by the concept of toggling between ascending and descending sort order, since the UI provides no feedback on which direction the recipes are currently sorted in.  She was frustrated that each time she clicked on the sort link, the results would change seemingly at random.

Next, users were asked what they could make with “12 eggs, a dozen bananas and asparagus.”  This was one of the biggest disasters of all.  The majority of users never thought to click on the “What Can I Make?” circle at the top, even though several had already mentioned earlier they were intrigued by the feature.  Most users simply typed in the list of ingredients and amounts into the “Ingredients to Include” filter, and of course got absolutely nothing helpful.  Several users gave up on the task, never once getting to the meal planner page.  Those unlucky enough to get past the first step were greeted by the usability catastrophe that awaited them.

On the “What Can I Make” tab, most users attempted to type in all the ingredients at once.  I’ve purposely designed KitchenPC to accept any input here without complaint, since the natural language parsing that goes on behind the scenes is so new and I was afraid of frustrating users when I didn’t understand the input.  I figured I’d just parse what I could and log everything else so I could fix it behind the scenes.  For this reason, if a user does type in “12 eggs, a dozen bananas and asparagus” on one line, the meal planner will still run and take none of those ingredients into consideration, yet yield no errors either.  I think this made people think the feature was just broken or useless.  A few users did figure out you had to enter the ingredients one at a time, and did get some valid results back.

Only one tester seemed to understand that the ingredients listed on the right were perhaps the aggregate of all the ingredients in the selected recipes, which is the entire point of the feature (to minimize total ingredients.)  Obviously, I need to find a good way to educate users on the power of this feature, as no other site really does this.

Users were also asked to add some of the results to the queue.  Half the users admitted they had no idea what the queue was, but several made some correct assumptions about the feature or at least admitted they were interested in it.  I think even one user mentioned “Netflix”, which is why I chose the word “queue” in the first place.

Again, users did not use the checkboxes to select recipes to add to the queue, but added them one at a time using the recipe viewer.  Unfortunately, the recipe viewer uses the caption “Plan To Make” instead of “Add To Queue“, which caused a lot of confusion.  Several users then noticed the “Add selected to queue” button near the recipe list, and then finally noticed the checkboxes (usually grumbling that the checkboxes were difficult to see.)

On the queue page, users had to remove an item from their queue.  They easily found the red X to remove the item, but most users were confounded by what came next.  You see, the queue was designed to allow users to temporarily store recipes they were interested in before organizing those recipes into a menu.  This is probably why the recipe viewer calls it “Plan to Make.”  The intended design is for users to try out the recipe, then “de-queue” it and possibly add it to an official menu if they liked it.  For this reason, I nag the user to rate the recipe when they remove it from the queue, since only around 2% of users decide to rate recipes on their own.

De-Queue popup

Unfortunately, the appearance of a rating dialog only leads to mass confusion.  Though a few users just ignored it completely and pressed the “Ok” button without selecting a rating, 3 different testers simply had no idea what to do.  Most would just hit Cancel thinking they clicked on the wrong thing.  One user even opened the dialog 4 of 5 times before figuring out the rating was simply optional.

The last thing users were asked to do was go to the menu page.  Fortunately, people understand menus and really like the idea.  On this page, they were asked to move a recipe from one menu to another.  8 out of 10 users immediately tried to drag and drop a recipe from one menu to another, however at least a couple were confused because they “missed” the drop area and then assumed there was some other way to do it.  To one user, it never occurred to her to try to drag a recipe (even though she was actively hovering over a tooltip that says “You can drag this recipe to another menu”) so her instinct was to click on the recipe and select “Add to Menu” again.  In theory, this should work well (though the recipe would be added to the new menu, not moved from the old menu.)  However, since the page requires a refresh before the newly added recipes would show up, she figured the feature wasn’t working as she saw no indication the recipe was moved.  This also caused some bugs to surface with “out of date” menus in the UI, causing a few embarrassing script errors as well.

9 out of 10 users were immediately able to rename a menu simply by clicking on the title, and the remaining user eventually figured it out after trying to right click on things.  All users were able to print a menu too, however there seems to be a bug on Firefox that prevents the print dialog from actually coming up.

Overall, it seems I have a lot of work to do.  So, what needs to change?

First off, the search page needs a “Submit” button.  There at least needs to be a “Go” button next to the keyword text box so users have something to click on.  The meal filters need to be labeled as such to avoid confusion.  The ingredient inclusion and exclusion text boxes need to make it clear to enter one ingredient at a time, and the interface needs to be more forgiving when users tab away without selecting an item or pressing enter.  I’m actually thinking of just removing the auto-complete mechanism all together, since it has always just felt clunky.

Sorting needs to make more sense and show which direction results are being sorted in.  Either “Total Time” should be removed from the sorting options, or I should display the total time for each recipe in the list.

There needs to be visual cues when search results are updated, especially for those users lacking the sufficient screen real estate to see both the search interface and result list on the screen at once.

The checkboxes need to be made more obvious, or I need to use standard (albiet ugly) HTML check boxes instead.  My original design used standard check boxes, but I absolutely hated how they looked.  The fact that 100% of web users are trained to recognize a standard checkbox might outweigh my fetish for visual appeal.

When a new menu is created, a popup needs to explicitly state how many recipes were added to the menu, and the recipe viewer needs to visually indicate if the displayed recipe is part of one or more menus.

The queue page needs a full description of what the queue is, and the Queue top menu link needs a tooltip such as “Click here to learn about the queue!”

The “What Can I Make” feature needs to clarify that only a single ingredient should be entered at a time, and provide a submit button rather than assuming users will hit enter.  Perhaps there should be some subtle UI indication if an ingredient is not understood.

When removing a recipe from the queue, I should either not ask the user to rate the recipe, or make it abundantly clear that the rating is optional, or provide a special button that says “I did not make this!” or something.

Drag and drop between menus needs to work better, and users should be able to drop anywhere in the menu, not just on the title bar.  Perhaps once the user starts dragging (which almost everyone intuitively does), the UI can change and guide them visually in some way.

The Surprising

Several things really surprised me about the feedback I got.  First was how dependent the average user is on the mouse.  When people are done typing, they want to click on things.  They do not assume they can press enter.  Power users and programmers are the ones that like keyboard shortcuts for things.

I was also very surprised at the sheer number of people who understand drag and drop.  Even more surprising was the reaction they had after drag and drop worked.  They would usually shout “Wow!  I didn’t expect that would work!” or something similar.  Which is odd, since why would your first instinct be to try something that you don’t think will work.

What I Will Do Next Time

Unfortunately, the feedback shows that I’m nowhere close to having this site ready to launch any time in the near future.  This also means I will need to do a second round of usability testing after fixing the major issues that were uncovered.

I think a bit of the confusion was caused by the script itself.  The instructions for the testers might have been a bit too vague.  For example, I asked them what they could make with “12 eggs”, but I didn’t tell them to click on “What Can I Make?”  This meant the user knew it could be done, but had to figure out how to actually do it.  A typical user might not try typing in “12 eggs” into the “Ingredients to Include” textbox on their own.  I felt it was an expensive way to learn this mistake as well.  Several testers never even got to the “What Can I Make?” tab so I lost a lot of potential feedback on it.  I think it might have been smart to “test out” the script on a friend first (for free) to make sure I was communicating each step clearly.

So, with that said I guess there’s nothing left to do but get back to work!  Though these results were quite discouraging, they did show definitive patterns.  If each user fumbled on completely different tasks, that would be one thing.  However, there are clear parts of the interface that every single person failed miserably on, and these things absolutely have to be fixed.  I’m definitely glad I know about these things now rather than launching the site to huge fanfare, only to wonder why no one came back.