Skip to content

Mechanical Turkey

May 23, 2012

These days, much of the time that I would like to devote to working on my web crawler has been usurped by randomizations with pre-made meal plans.  I fret that I’m woefully behind on my goal of having ten pre-made meal plans (one for each category) by launch; though I have been moderately successful in getting some friends, as well as  some in the professional realm, to assist with this project.  One thing I have not been able to get much assistance with is recipe photography.

It’s very important to me to have ten really good, well tested and well thought-out meal plans for launch.  I want to show off what the site can do with these plans.  I want to individually advertise each of these plans on Facebook and Google.  I want whatever press I can muster up for launch to eat these plans up, and keep users coming back for more.

Obviously, one of the keys to good recipe content is good recipe pictures.  Unfortunately, most of the content I’m getting from nutritionists and dieticians is in the form of a Word document or PDF, and usually sans pictures.  I’ve had a little luck getting some friends to try some of the recipes out and snap some shots, however this approach would result in me losing friends faster than achieving my goals.

Yesterday, I decided to try a new approach.  Yes, once again, Amazon’s Mechanical Turk service.

Those who frequently peruse my ramblings might be asking, “But Mike?  Didn’t you try out Mechanical Turk before and declare it a scam-ridden wasteland of automated succubots, ensuring any user dumb enough to use it would surely spend more time cleaning up the mess it created than any time savings yielded?”

Well, yes I probably did say something similar to that.  But this is different!  Let me tell you why!

Protip: Ignore Amazon when they tell you Turk is designed for short, quick repetitive tasks.

 

When I tried Mechanical Turk before, I was using it for ingredient matching.  For example, I’d upload a list of ingredient usages found across thousands of recipes.  The goal would be to match something like “4 cups of shredded cheddar cheese” to the number 4, the unit “cups” and the “shredded” form of “cheddar cheese”.  This was somewhat of a crowd sourced version of the natural language parsing technology I have since developed.

This approach failed because the results would be tainted by scammers that are hoping to make a quick buck.  I’m quite certain some of them are running automated scripts to just fill out random answers to any Turk HIT they can find.  This caused issues such as over 1,000 HITs getting matched with Captain Crunch cereal.  This created a mess that was nearly as difficult to clean up as the BP oil spill.

Turk experts would chime in here saying building a reliable team of workers is something that doesn’t just happen overnight.  I should have tests to pass first, approve each worker individually, assign each HIT to multiple workers to ensure a consensus, blah blah blah.  Unfortunately, I had time for none of that so I’ve instead declared Mechanical Turk to be a bad fit for that particular task.  It turned out writing a natural language parser was a lot more geeky anyway.

So, I gave this another shot with recipe pictures.  Who wouldn’t want to cook a recipe and take a picture of it for money, right?  I decided to start out with a meal plan being developed by Say I Do Nutrition Services in Seattle.  This plan has seven recipes, and I already had pictures for two of them.  I created five Mechanical Turk HITs that basically said, “Cook this recipe, take a picture of it, make sure it’s not a crappy picture or I’ll reject it, and don’t just steal a picture of something similar online because I’ll know.”

I gave each worker 12 hours to complete the job after they accepted the HIT, allowing people to make it for their families for dinner last night.  Oh, and I also restricted the work force to only those in the U.S.  I’m not racist or xenophobic or anything, I just think most of the scams and bot-thingies are going on over seas, perhaps funded by fake Nigerian princes.  For this job, I require nice southern housewives trying to feed their two-and-a-half kids.

I also paid $20 a HIT, which I think should more than cover all the ingredients involved.

The Results

NOT Beef Stew

NOT Beef Stew

Within about 20 minutes, I got one HIT completed for a beef stew recipe.  Considering that recipe takes four hours to cook, I knew it was a fake.  Sure enough, attached to the HIT was a link to a random picture on the Internet of a sandwich.  Mmm, sandwich.  But no, this was not what I was hoping for.  Within a few clicks, I was able to reject the HIT, sending it back to the available HIT pool for someone else, as well as block the user so they could bother me no more with their sandwich-based trickery.  Only worrying about five HITs was far easier to manage, so this slight detour was hardly annoying and perhaps slightly entertaining.

One thing that Mechanical Turk lacks is the ability to see how many workers have accepted a HIT.  In fact, you get no feedback at all until they actually submit the results.  For this reason, nothing happened for about five hours.

Broiled Salmon

Broiled Salmon

Then, all evening, I started getting various pictures submitted of each of the five recipes.  A lot of the submissions contained a dozen or more pictures, from various angles and lighting.  Several were very artistic, such as the recipe for wine stew included a glass of red wine next to it. A few included comments saying how much they enjoyed the recipe, and that this was the best job they’ve ever found on Mechanical Turk.  I had absolutely no problems with any of the photos submitted, and I added all five workers to a custom group so I can use them again later.

Bonus!

Another highlight of this approach was free advertising.  Not only did I see a huge spike in website traffic right after I posted the HITs, I also got a few unsolicited emails, such as this one:

I wouldn’t be able to make the meal though I would love to. I just wanted to let you know that I am impressed with the recipe site and will be using it in the future! Thank you!

Several of the workers also said they loved the site and would be coming back as well.  This makes sense, of course, as the posted job seeks the exact same target demographic as my website itself does; people who can and love to cook.  Not only did I get photos for all five recipes, I also got a few new users for the site!

Though this approach is a bit expensive, at $100 bucks for 5 recipe photos, it might be the best way to get ten meal plans up and ready for launch.  I don’t think going much lower than $20 per recipe would yield the quality I expect, plus I think it’s fair to at least cover the cost of the ingredients.

With that said, I think I’ll try to cook as many recipes as I can myself, and continue to pester friends and family to help out as well, but knowing I can quickly get photos of any leftover recipes I have within a day definitely lifts away a lot of the stress associated with building this content.

Flank Steak

Flank Steak

Wine Stew

Wine Stew

Salmon Stir Fry

Salmon Stir Fry

Tofu Stir Fry

Tofu Stir Fry

Advertisements

From → Business

7 Comments
  1. Hi! We were talking about your blog post over at Turkernation and wanted to invite you to join the Turkernation.com forum if you decide to run HITs ever again. One way our best requesters keep scammer results out is putting qualifications on their HITs, preferably some form of test that you can use to easily screen out obvious bots and idiots. Granted, it doesn’t keep EVERYONE out but it does reduce it. Also up your requirements (say 1000 HITs approved with 99% acceptance rate). Again, this does not eliminate ALL scammers but it significantly reduces the number!

    Sincerely, Just another turker

    • Hah, I saw that! I will most likely be posting more cooking HITs, but I don’t know when yet. And to Sleepstar, I absolutely paid everyone for their photos!

  2. If you send me a list of recipes you want cooked, I’d be willing to negotiate a price with you on ones that I have at least some of the ingredients on-hand. Save you money, give me new things to cook, we both win :-)

    • Thanks Wendi, but I think the advantage of MTurk is the ability to have multiple workers in parallel. If I need 10 recipe photos quickly, I can have 10 people work on them and get the results in a few hours. I doubt I’d be able to get one person to cook 10 recipes in a day, no matter how much I paid!

  3. Bob permalink

    If your going to respond to comments left at Turker Nation, why don’t you register as a requester and post your comments there. I doubt if Sleepstar is going to see this one. :)

  4. inkeeper77 permalink

    Thank you for this article! I am a mturk user, college student, and a bit of a photographer. I never saw your HITS, but I saw this- and I will be using your site , as I have recently moved out of the dorms and into the real world, where I actually have to cook for myself in order to survive healthily on a decent budget..

    I agree with the above comments, but I have a bit of a suggestion. Not every good worker has 1000+ approved hits, I am currently sitting at 98, since I am new, but I have always been interested in the more invloved less repetitive tasks, such as writing tasks. Though I have such a small history, I would be more than willing to take a test for you to prove my worth!

    In my opinion, a great test would be for you to post a recipe or two with somewhat cheap or common ingredients, and have the hit run exactly the same as your paid hits. Then, anyone who does a good enough job on that could be added to your workforce, ready to absorb more and more work as your business grows, with minimal effort on your part. – You could probably even do this outside of MTurk, taking email based trial runs (applications) that include our turker ID’s. (Or it might be easier on mturk itself, I do not know how the requester interfaces look)

    (And yes, I am saying all of this because I selfishly want in! It would fit my lifestyle fantastically, especially since I am trying to learn good recipes anyway. If you like my idea, an email with information on when the HIT was going to be posted would be great)

    Thank you for your time!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: