Skip to content

Mobile Intentions

It is with great pleasure that I officially announce that I am now planning to begin the process of thinking about the initial development of KitchenPC mobile and tablet apps.  In other words, the phrase “I need a mobile story… but now what?” has popped into my head recently.

The very first launch of KitchenPC included an extremely basic, and poorly thought out mobile solution (which was reachable at m.kitchenpc.com).  Thinking I needed some sort of mobile solution prompted me to craft this web-based turd in the matter of a single weekend, and it ended up being used by absolutely no one.  Seriously, I think it got about 0.000001 visitors per day.  Though this mobile site was optimized for smaller screens, allowed basic scenarios like recipe search, shopping list management, and basic calendar functions, it was a flop.  It suffered from a very common problem.  Mobile sites are often just watered down versions of a real website, since most developers fail to look past any differences between the desktop and mobile beyond mere screen size.

Surely, a better solution needs to be devised that takes into account what sorts of culinary tasks people are actually doing on their phones.  Surely, creating a mobile app that’s a straight port of the KitchenPC website would be an exercise in futility.  Surely, I should stop calling you Shirley.

Brainstorming

Last night, I grabbed my dry erase markers and tapped them against my knuckles as I stared pensively at the massive 8×4′ whiteboard I have on my office wall.  I started thinking about the types of features this awesome new mobile app would need.  As I started writing down some ideas, it became clear after about ten minutes that I was repeating the exact same dumb mistake that I made before.  I was enumerating all the KitchenPC features in my head and writing down the ones that I thought could potentially be useful on a phone; a scaled back version of the website.  These included things like recipe search, adding recipes to menus, managing saved menus, adding recipes to the queue, creating a shopping list based on a menu, and more.  I was also trying to add extra bells and whistles, such as voice search by ingredient, and sharing recipes between devices using NFC.

It became increasingly clear that I was looking at months of development time, and all to develop a product that would be a mobile port of what I already have today on the web.  I really needed to take a step back and figure out what people are doing on phones.  What’s the minimum viable product to create on a phone that will target these key scenarios?  I was writing a version 1.0 product, not “KitchenPC 2.0 Now With Smart Phone Support”.

Take a Step Back

So, really now, what is the most basic mobile recipe app I can create?  One that will insult users over the features it’s blatantly missing?  When you hear users complaining that your app is missing features, only then do you know you’re on the right track; it means people are using it, they want to be using it, and reacting to these requests becomes consumer driven, not engineer driven.

Well, I need the ability to log on to KitchenPC, which means using an existing account, creating a new account, or logging on with Facebook credentials.  Yes, log on is good.  What else?

Shopping Lists?

One of the scenarios that miserably failed during the beta was a persistent shopping list.  Turns out, no one wanted to manage their weekly shopping list on a desktop computer.  Most people just write this on paper, or throw it together right before they leave for the store.  Plus, the people who were using this feature complained that adding specific ingredients and amounts was somewhat of a hassle, as the database schema was normalized just as recipe ingredient data is.  So, the current incarnation of KitchenPC has a “What You’ll Need To Buy” list whenever you’re looking at multiple recipes, however no real shopping list management feature.

Perhaps a mobile device is different though.  People take their smart phones to the grocery store with them, and there are plenty of successful shopping list apps for every mobile platform.  Using the KitchenPC back-end, I could easily build a very powerful shopping list app.  This would be flexible, allowing NLP ingredient entry, storing arbitrary non-food items (such as “paper towels”,) and let the user cross off items after they were purchased.  It could also use push-updates to sync shopping list changes instantly between devices that were logged on to the same account.  This is a great example of a feature that’s designed specifically for mobile scenarios, rather than a straight port of a website feature.

My Menus?

One of the most powerful features of KitchenPC is the ability to organize recipes you find into various menus.  This allows for basic meal planning (since you can create a menu each week, or a menu for a certain gathering or event) and also just lets you save your favorite recipes, grouped any way you wish.  However, does this correspond to any mobile scenarios?

For meal planning scenarios, most definitely.  Users who plan out recipes to make during the week need to be able to scroll through them while at the store and easily see what they’ll need to buy.  I’ve also more than once found myself somewhat peckish while wandering the food aisles and wanted to look up the ingredients for a specific recipe.  Allowing users to access their saved menus on their smart phone is definitely a primary goal.

Recipe Search?

All of my initial specs for KitchenPC Mobile have included a basic recipe search.  KitchenPC is a recipe search engine, of course you’d need to be able to… like.. search for recipes on the mobile version, right?

Right?

However, recipe search seems to infect any spec I write, like a virus, slowly spreading into more and more features.  Once you have recipe search, you need a mobile interface to define various search criteria.  Keywords, meal types, ingredients to include and exclude.  Then, users would start wanting some of the more advanced search options like time limits, ratings, pictures, dietary restrictions, etc.  Then you’d need a results list, the ability to add results to menus, which turns into full menu management features (creating menus, managing the recipes within those menus, and removing recipes from menus.)  Suddenly, you’re back to creating a mobile port of the entire website.

So, do people really need recipe search in the mobile app?  I found myself leaning towards no.  However, then one thing struck me.  Cutting this feature creates a dependency between the KitchenPC website and the mobile app.  The app is all of a sudden useless for first time users, as well as users who rarely use desktop computers.  Without the ability to find recipes in the database, users can’t create menus.  It would be useful as a shopping list app, if they wanted to type in everything manually, but there’s already myriad apps for this.  Do I want KitchenPC Mobile to basically be an extension of the website?  Probably not.

Obviously, I need an extremely simplified search feature, even if it just allows keyword search and nothing else.

Priorities

In the end, I decided to create a list of “must-haves” for the mobile site, which would enable the key scenarios I wanted to deliver.  Then, create a list of “bonus features” or “nice-to-haves” that would extend those scenarios, or perhaps be on the road map for future versions.

Must-Haves:

  • Ability to log on, create an account, and log off (Duh)
  • Powerful and flexible shopping list feature
  • At least a read-only view of saved menus, with perhaps a sample menu for new users
  • Ingredient aggregation within menus, allowing users to see what they’ll need to buy at the store to make the recipes within a menu
  • Basic recipe search (keywords only) and a results list that lets you save a recipe to a menu, or create a new menu

Nice-To-Haves:

  • More powerful recipe searches, more on parity with the website
  • More control over saved menus; moving recipes between menus, removing recipes from menus, creating new menus and renaming menus
  • Queue feature (adding any recipe to the queue, viewing the queue and de-queueing recipes)
  • Selecting individual recipes from search results or menus and seeing a real time tally of ingredients you’d need to buy

Next Step, Wireframes

The next step in the process will be defining the overall flow of this app.  This includes mocking up wire frames, defining each screen, and figuring out how the features relate to each other.  Since I know the basic scenarios, I know to make those tasks easy to accomplish using the fewest number of button clicks.

Hopefully, I can resist the urge to overly complicate the design.  I’d really like to be able to get something together within a month or so, and get it out there for users to try out.  Getting a tablet version out (which of course targets different scenarios, like in-kitchen use) is also important, and can hopefully be available for all the new tablets sold during the holiday season.

Comments?  Wishlist features?  Just want to talk about how you use your smart phone or tablet?  Let me know in the comments below, or shoot me an email!

 

A Better Way To Deploy (Technical Debt Part 5/3)

During the last month, I’ve been working on paying off some of the technical debt that has been collecting interest over the course of KitchenPC’s development. Most of these have been stability changes, or just things that, when fixed, remove roadblocks to a smoother development process. I’d like to go over these in a three part blog series that, like the Hitchhikers Guide to the Galaxy trilogy, will consist of five parts.

Part 5/3: A Better Way To Deploy

During the beta, I was more focused on code and features than boring process and tools.  One of the things I never got around to was any sort of modern deployment system to push new changes out to production.  This wasn’t a huge deal, since I only had a single web server and didn’t have enough traffic to really care if I had to bring down the site for a couple minutes.

The original technique for pushing out changes is rather embarrassing, but I’ll share it with you anyway.  I’d go out on a limb and say there are plenty of entrepreneurs who do the exact same thing.

First, I ran MSBuild (I had previously used nAnt but switched to MSBuild for a variety of reasons) to build all the individual KitchenPC projects.  The MSBuild script would also crunch the CSS and JavaScript files using the YUI Compressor.  Everything would be copied out to a directory called /Build which had a subdirectory for binaries, a subdirectory for site content, and a subdirectory for the KitchenPC Queue service which also runs on the web server.  Building the entire site would take around 20 seconds.

Next, I’d zip up everything.  If I was making a lot of changes to the site, I’d just zip up the entire Build directory.  However, sometimes I was just making some script changes or binary changes, and would just zip up the files that were changed.  Then, I’d Remote Desktop (we need a better transitive verb for that) into the production web server, and transfer my files over.  Originally, I put them up on my own private web server running off my home network and just use wget to download the file, but later on I started using FTP instead and manually uploaded the files to the web server.

From the desktop, I’d unzip the package and copy out the files, usually running an iisreset when everything was in place.

So, in other words, everything was manual.  It would usually take five or ten minutes to get everything deployed.

After launch, I finally had time to think about this process and devise a better solution for pushing changes out.  Ideally, I wanted to do this without having to Remote Desktop into production web servers, and also be able to push changes out with a single command line tool.  A bonus would be the ability to not have to take the site down, lose any traffic, or reset IIS when there were no binary changes being deployed.

On the Microsoft technology stack, WebDeploy is the cutting edge solution for doing all of the above, and more.

WebDeploy is a protocol that’s able to compare a set of files on a source and destination, and synchronize those files automatically.  It can be used for synchronizing files between two web servers (making the provisioning of more web servers extremely efficient) or synchronizing files between a development or staging environment and a production web server.  It could also be used to create backups of a web server and its configuration by dumping an entire IIS application to a ZIP package, or a raw directory.  It’s far superior to FTP, as it’s smart enough to compare the timestamps on files to figure out exactly what has changed between a source and destination, and only transfer that content over the wire. WebDeploy is used on Windows Azure, as well as the Microsoft WebMatrix platform, and also integrated with recent versions of Visual Studio. Basically, WebDeploy is where it’s at.

WebDeploy runs as an IIS extension, and can be installed easily using the Microsoft Web Platform Installer.  When installed on a machine without IIS (such as a development box,) only the client side tools (such as MSPublish.exe) are installed so deploying changes from the command line to a remote IIS box can be accomplished.  Also, Visual Studio 2010 and above will install MSPublish.exe automatically, since the Publish feature in Visual Studio calls MSPublish.exe under the covers when publishing a site using WebDeploy.

As it would not make a very good blog post to walk you through setting up WebDeploy on your web servers, I’ll refer you to this page which will walk you through the steps.

If you’re running Visual Studio 2010 or 2012, setting up Web Deploy on your web server is really all you need to do.  Once you configure WebDeploy on a site, it will export a settings file which can be used directly in Visual Studio.  However, since I’m a command line freak (plus, I was still running VS2008 at the time,) I decided to integrate WebDeploy into my MSBuild scripts instead, and then be able to build and deploy changes in a single command.

To use MSDeploy from within an MSBuild file, you’ll first want to setup some properties within your project.  These properties should be defined directly under the <Project> tag so they’ll be global to all tasks.  I define a few properties to hold various publishing settings, and support the ability to publish to both a staging and production web server:

  <PropertyGroup>
    <ArchiveDir>$(MSBuildProjectDirectory)\$(BuildDir)</ArchiveDir>
    <AppHost>KitchenPC</AppHost>
    <UserName>publish_acct</UserName>
    <Password>secret</Password>
  </PropertyGroup>

  <Choose>
    <When Condition="'$(Configuration)'=='Staging'">
      <PropertyGroup>
        <PublishServer>https://staging</PublishServer>
      </PropertyGroup>
    </When>
    <When Condition="'$(Configuration)'=='Release'">
      <PropertyGroup>
        <PublishServer>https://www.kitchenpc.com</PublishServer>
      </PropertyGroup>
    </When>
  </Choose>

Let’s go through each of these properties:

ArchiveDir: For me, this is simply where I build my site to.  The directory structure needs to be identical to the way MSPublish exports your site, so you can run:

msdeploy -verb:sync -source:appHostConfig=”Default Web Site”,computerName=server -dest:archiveDir=c:\Test

To dump your site to a temporary directory on your hard drive to see.  Replace Default Web Site with the name of your site in IIS, and replace server with the computer name or IP address of the web server.

For me, it puts all the files in /Content/c_C/Website but this might be different for you.  It’s also possible MSDeploy has the ability to define exactly how the folder layout should look, but I’m not yet enough of an expert to know.  Also note that Visual Studio will do all this packaging for you.

AppHost: This is the name of your application in IIS.  This is probably Default Web Site if you haven’t mucked with the default IIS settings, but for me this is KitchenPC.

UserName: This is the username MSPublish will try to connect to your web server with.  This might be Administrator if you haven’t granted anyone else access, but it’s advisable to setup a user account just for publishing.

Password: This is the password for the username above.

It may or may not be necessary to hard code all these publishing settings into your MSBuild script.  Visual Studio is able to store these settings in a settings file, but from what I can tell, those settings are extracted and passed directly into MSDeploy.exe when you publish.

I also have a conditional property called PublishServer which changes depending on the build configuration.  If I’m building the Staging configuration, I’ll publish to my local staging server.  If I’m building in Release mode, I’ll deploy to production.

Next, you’ll need to define a new Target for your deploy task:

  <Target Name="Publish" DependsOnTargets="Build">
    <Message Importance="High" Text="Deploying to $(PublishServer)" />

    <Exec
      WorkingDirectory="C:\Program Files\IIS\Microsoft Web Deploy V3\"
      Command="msdeploy.exe -verb:sync -source:archiveDir=$(ArchiveDir) -dest:appHostConfig=$(AppHost),computerName=$(PublishServer):8172/msdeploy.axd,authType=Basic,userName=$(UserName),password='$(Password)' -allowUntrusted" />
  </Target>

Notice this target depends on the Build task, which compiles the site with the proper configuration, crunches all the CSS and JavaScript files, and moves everything to the correct output directory using the right folder hierarchy.

It then simply executes the msdeploy.exe command with the appropriate command line arguments. Note if you’re using a self-signed SSL certificate on your server (which the Web Platform Installer will create by default), you’ll have to use the -allowUntrusted command line parameter as well. Hopefully, with a bit of tweaking, this will get you started on your own solution!

What About Web.config files?

Glad you asked!  Another very cool thing about Visual Studio 2010 and above are web.config transforms.  In the past, I just had separate files for various web.config files (such as prod.web.config and staging.web.config), and my MSBuild script would copy the correct one to the build output directory.  When I zipped up the site, the production web.config would be included.

The correct way to do this is using a transformation, which is basically an XSLT file that contains instructions on transforming your default web.config file to a file suitable for a specific environment.  Visual Studio takes care of all of this for you if you’re using a Web Application Project, allowing you to right click on your web.config file and add a new configuration.

However, the actual code for doing these transforms is neatly packaged up in managed libraries, and can be accessed as an MSBuild task.  Since, again, I’m a command line freak this is the approach I took.

The first step is to reference the Microsoft.Web.Publishing.Tasks.dll assembly in your MSBuild script.  This can be done by adding the following line directly under the <Project> node:

  <UsingTask TaskName="TransformXml"
    AssemblyFile="C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>

I just hard coded in the path, but you could easily make this an environment variable, or copy the DLL to a more appropriate directory.

Once this assembly is referenced, you can start using build tasks defined within that library. This is ridiculously easy:

    <TransformXml
      Source="KPCServer/web.config"
      Transform="KPCServer/web.$(Configuration).config"
      Destination="$(DeployDir)/web.config" />

Basically, we take the default web.config checked into your source tree, run the appropriate transformation against it (such as web.staging.config or web.release.config) and output the file to the correct output directory; in my case, the deployment directory that MSPublish picks up.

I won’t get into the details about building these transformations, since there’s tons of documentation online.

Once this task is added into your build target, then we’ve nearly got a complete build system in place.  Running the deploy target will first build your site, copy and transform your web.config file over, and run MSDeploy.exe to publish your site (copying only the files that have changed, of course!)

There’s one more thing we want to do.  Minimize CSS and JavaScript files.

Crunch Time!

Previously, I used YUI Compressor, a Java utility developed by Yahoo! to minimize JavaScript and CSS files.  It’s fantastic and pretty much the standard in the industry.  I would just execute this utility directly from an MSBuild script using an Exec task and call it good.

However, this time I decided to try out the .NET port of YUI Compressor, which is called YUICompressor.net.  The author claims the algorithm is identical, and the results should be nearly byte-for-byte the same.  It can be easily installed with NuGet and it also has MSBuild tasks included.

One drawback is the documentation is wildly out of date!  I had to spend a lot of time in ildasm digging through the assembly to figure out what was going on.  Hopefully, I can provide some better documentation here for people who are attempting to integrate YUI Compressor .NET into their MSBuild scripts.

The first step is to reference the build tasks.  There’s two of them, one for JavaScript minification and one for CSS files:

  <UsingTask
    TaskName="JavaScriptCompressorTask"
    AssemblyFile="packages\YUICompressor.NET.MSBuild.2.1.1.0\lib\NET20\Yahoo.Yui.Compressor.Build.MsBuild.dll" />
  <UsingTask
    TaskName="CssCompressorTask"
    AssemblyFile="packages\YUICompressor.NET.MSBuild.2.1.1.0\lib\NET20\Yahoo.Yui.Compressor.Build.MsBuild.dll" />

These directories happen to be where NuGet installed the binaries, but you can probably put them wherever.

I then made two separate MSBuild tasks, one for JavaScript and one for CSS.  We’ll go through the JavaScript task first:

  <Target Name="CrunchJs">
    <!-- Select all kpc*.js files -->
    <CreateItem Include="$(DeployDir)/scripts/kpc*.js">
      <Output TaskParameter="Include" ItemName="JsCrunch" />
    </CreateItem>

    <!-- Copy them to .debug files since JavaScriptCompressorTask cannot read and write to the same file -->
    <Copy SourceFiles="@(JsCrunch)" DestinationFiles="@(JsCrunch->'%(Identity).debug')" />

    <!-- Compress the .debug files, and delete them afterwards -->
    <JavaScriptCompressorTask
      SourceFiles="%(JsCrunch.Identity).debug"
      ObfuscateJavaScript="True"
      EncodingType="Default"
      LineBreakPosition="-1"
      LoggingType="None"
      DeleteSourceFiles="Yes"
      OutputFile="$(DeployDir)/scripts/%(Filename)%(Extension)"
      />
  </Target>

First, we select all the files we want to crunch. In my case, this is anything in the output directory that matches kpc*.js (as I give all my script files a kpc prefix). Next, we copy each file to a new file with .debug appended to the end. For example, we copy kpccore.js to kpccore.js.debug and kpchome.js to kpchome.js.debug. We do this because YUI Compressor .NET is not capable of using the same filename for both the input and output files, and I want to simply compress all the files in my build directory.

Lastly, we run the JavaScriptCompressorTask task. Note YUI Compressor .NET can only handle a single output file. While it can handle an enumeration of items for the SourceFiles parameter, it can only output them as a single file. For example, I can pass in kpccore.js, kpchome.js, kpcmenu.js, and all my other files as the SourceFiles parameter. However, it will not output separate crunched files. It will output everything to a combined file, such as kpcall.js. I don’t really like this, since I don’t need to include kpchome.js on any page besides the home page, or kpcqueue.js on any page besides the queue page.

For this reason, I use MSBuild batching to call the JavaScriptCompressorTask task multiple times, one for each item in JsCrunch. This is indicated using the percent sign prefix, %(JsCrunch) rather than @(JsCrunch), which would send in all the files as a single input parameter.

So, I run the JavaScriptCompressorTask on each file within the JsCrunch item set, appending the .debug prefix to the end since the input file can’t be the same as the output file. I also add the DeleteSourceFiles parameter to delete the temporary .debug files after I’m done.

The CSS task looks very similar:

  <Target Name="CrunchCss">
    <!-- Select all *.css files -->
    <CreateItem Include="$(DeployDir)/styles/*.css">
      <Output TaskParameter="Include" ItemName="CssCrunch" />
    </CreateItem>

    <!-- Copy them to .debug files since CssCompressorTask cannot read and write to the same file -->
    <Copy SourceFiles="@(CssCrunch)" DestinationFiles="@(CssCrunch->'%(Identity).debug')" />

    <!-- Compress the .debug files, and delete them afterwards -->
    <CssCompressorTask
      SourceFiles="%(CssCrunch.Identity).debug"
      EncodingType="Default"
      LineBreakPosition="-1"
      LoggingType="None"
      DeleteSourceFiles="Yes"
      OutputFile="$(DeployDir)/styles/%(Filename)%(Extension)"
      />
  </Target>

I’ll let you figure that one out, since it’s basically a copy of the JavaScript task, only using the CssCompressorTask task instead.

The last step is running this target after I build. I add the following lines to the end of my Build target:

    <!-- Crunch Files -->
    <Message Importance="High" Text="Crunching JavaScript/CSS Files..." />
    <CallTarget Targets="CrunchJS;CrunchCSS" />

This basically instructs the Build target to call both the CrunchJS and CrunchCSS targets after it finishes, minimizing the JavaScript and CSS files that have been copied to the build output directory.

Wrapping Everything Up in a Bow:

Since I use PowerShell for my build environment, I’ve also created a new PowerShell function to build and deploy the site to either staging or production:

function Publish
{
   if ($args.Count -gt 0)
   {
     switch ($args[0].ToUpper())
     {
        "STAGE"   { msbuild .\build.xml /t:Publish /p:Configuration=Staging /verbosity:minimal }
        "RELEASE" { msbuild .\build.xml /t:Publish /p:Configuration=Release /verbosity:minimal }
     }
   }
   else
   {
      Write-Host "Please specify either STAGE or RELEASE build.`n"
   }
}

Now, I can simply type Publish Stage or Publish Release at the command prompt to run the Publish target in build.xml.  This will build everything in the appropriate configuration, transform the correct web.config and copy that to the deployment directory, minify everything using YUI Compressor .NET, and then publish that whole package to the correct server using MSDeploy.exe.  Magic!

Hopefully some of this has inspired you to take a look at your current deployment strategy and think about some improvements.  What I have is far from perfect, but it was a solid weekend of work and far superior to what I had before.  Combined with the fact that my app starts nearly immediately now, and the fact that only modified files are copied out, I can now deploy new versions of the site extremely quickly without losing any traffic.  Requests that are coming in while I deploy would probably just see a couple second lag while the new binaries are loaded and the app pool recycles.

Out with the old, in with the new (Technical Debt Part 4/3)

During the last month, I’ve been working on paying off some of the technical debt that has been collecting interest over the course of KitchenPC’s development. Most of these have been stability changes, or just things that, when fixed, remove roadblocks to a smoother development process. I’d like to go over these in a three part blog series that, like the Hitchhikers Guide to the Galaxy trilogy, will consist of five parts.

Part 4/3: Out with the old, in with the new

I’ve been using Visual Studio 2008 since the very beginning of KitchenPC.  It’s fast, it’s stable, it works.  When Visual Studio 2010 came along, I used it briefly but just couldn’t convince myself to make the switch.

Visual Studio 2010 suffered through a similar development cycle as Project Server 2007, the application I worked on at Microsoft for a solid chunk of my career there.  Previously, Project Server was written as an ASP website (Yes, ASP, no not ASP.NET) and had a very mature code base mostly built in JScript (Yes, JScript, no not VBScript.)  We decided to rewrite the entire server in managed code, while at the same time, cramming as many new bells and whistles into the product as we possible could.  We decided to do all of this during the Office 2007 time frame, a two-year release cycle where most products weren’t changing anything huge, and certainly not being rewritten from scratch.

The result?  A giant disaster.  The product was unstable, buggy, most innovative features ended up getting cut anyway, and the new features that did go in were poorly thought out and implemented.  On the plus side, we had a brand new, more modern code base.  Albeit, that code base took a lot of shortcuts, ensuring that down the road, backwards compatibility would come at the cost of being stuck with a poorly laid out set of APIs.  The amount of technical debt accrued during that release would make China shudder.

Visual Studio 2010 went through a similar release.  The product was rewritten as a WPF app using managed code.  They brought in two architects, both of whom came up with wildly unrealistic estimates for what could be done.  They spent years on this redesign, getting nowhere.  The result was a release that was unstable, buggy, slow, and really didn’t do anything new besides its support for the .NET 4.0 framework.  When I tried VS2010, I ran into nothing but problems.  Sometimes it would act like the shift button was stuck down, and highlight text when I moved the arrow keys.  Often times I’d be editing an ASPX file, and the entire IDE would just spontaneously vanish.  It was noticeably slower, and some larger files would just grind the entire thing to a halt, forcing me to do edits in Notepad2 instead.  I never quite did figure out how to debug both .NET 4 server code and JavaScript code at the same time; I would have to detach the debugger from the web server and reattach it to Internet Explorer, wasting all sorts of time.  In the end, this product was a huge step backwards from the 2008 release, which simply worked.

However, years past and we’ve now moved on to the .NET 4.5 framework.  A lot of the open source projects I use now require at least .NET 4.0 to even compile.  I find myself really wanting to take advantage of some of the new language features, and plus when I start doing Windows Phone or Surface development, I’m going to have to upgrade anyway.  So I figured I would give Visual Studio 2012 a shot.

Wow, am I impressed with this release.

This IDE is beautiful, it works, it’s slick and it’s fast.  You can tell they really took the time to pay off some of their own technical debt.  It no longer seems clunky, it no longer crashes, and the debugger works flawlessly.  It has all sorts of great new features, like the ability to pin a pane to any monitor, better code searching, and improved JavaScript support.  I’ve also been taking advantage of new build features, which I’ll go into more detail in the next post.

Even more impressive was the fact that I could open my VS2008 Solution file and it was upgraded on the fly to target the .NET 4.5 framework, and everything compiled successfully on the very first try.  What I thought was going to be an evening of work turned out to be a few minutes.

I also took the time to install Windows 8 on my dev box, which I’m equally happy with.  It’s also a fast and stable environment that looks nice and is easy to use.  The weird new metro-style start menu is a bit weird, but it actually works fairly well when you’re using two monitors.  You’ll only get the start menu on one screen, and you can click on the second monitor for it to go away.  Your shortcuts in the launch bar are duplicated on both monitors, so you can use those to quickly launch applications if that’s what you’re used to.  I’m also getting fairly comfortable with the Win+Q key to find and launch programs, which works almost as well as Spotlight on OS/X.  I also really enjoy the panoramic themes, which are collections of beautiful images that span multiple monitors.

I think upgrading to Windows 8, .NET 4.5 and Visual Studio 2012 sets me up for a much better development environment for the KitchenPC web site, as well as mobile and tablet apps in the near future.  Needless to say, if there’s anyone reading from either the Windows 8 team or the Visual Studio 2012 team; fantastic work, and keep it up!

Static files? Not on my server! (Technical Debt Part 3/3)

During the last month, I’ve been working on paying off some of the technical debt that has been collecting interest over the course of KitchenPC’s development. Most of these have been stability changes, or just things that, when fixed, remove roadblocks to a smoother development process. I’d like to go over these in a three part blog series that, like the Hitchhikers Guide to the Galaxy trilogy, will consist of five parts.

Part 3/3: Static files? Not on my server!

After implementing my new session management system, one of the main problems was that I was stuck with really big cookies – somewhere over 700 bites, err bytes.  That’s a lot of data to be sent on each request, including requests to static script files, images, CSS files, fonts, and other web resources that have no need for the session data stored in each cookie.  Now, after I posted the last blog I ended up writing my own custom serialization code for the session data, which got the cookie size down from 700+ bytes to 77 bytes (well, depending on the size of your user name,) so that’s a huge win.  However, I still wanted to implement a system that would not transmit cookies over the wire on each request.  For this reason (and others,) I decided to move all my static files over to a CDN.

A CDN, according to Wikipedia, is a “large distributed system of servers deployed in multiple data centers in the Internet.”  More specifically, it’s a network of servers that can easily store static files for you so that you don’t have to house them on your own web servers, which are busy with dynamic content, database queries, finding out what you can do with 47 ounces of asparagus, etc.  This network is usually built using servers across dozens of locations.  There are two huge advantages of this geographical diversity.  First, if one node goes down due to a wind storm, bird strike, or someone tripping over a power cord (they told me they were going to tape down those cords!), then your content can be served up from elsewhere.  Second, requests can automatically be routed to data centers near-by, which will result in faster load times.  In other words, someone in Hong Kong visiting KitchenPC will download graphics and script files from servers near Hong Kong, and someone in Seattle visiting KitchenPC will download those same graphics and script files from servers near Seattle.  You get the idea.

Finding the Right CDN For You

There are tons of different CDN companies out there.  However, not all of them directly deal with end users.  For example, EdgeCast is a huge provider, but their minimum package is a thousand gigs of data for around $300 a month.  Way more than a small website needs.  For this reason, there are many smaller companies that simply buy up bandwidth at wholesale prices from the big guys, and resell it in smaller chunks to people like me.  The advantage of going through one of these companies is that you have no minimum to buy; your monthly bill can literally be like 88 cents.

Rackspace CloudFiles

The first CDN I checked out was Rackspace, whose solutions is called CloudFiles.  They’re built on Akamai CDN, and were desirable because I already host my web servers through Rackspace, and having less services billing me each month is always a good thing.  I have been using CloudFiles already for storing recipe images on.  During the beta version, if you uploaded either a profile picture or a recipe image, that file would go straight to CloudFiles and be served from there.  However, when I checked into CloudFiles for hosting my site’s static content on, I was hugely disappointed.

First, they don’t support origin-pulls.  An origin-pull, in CDN vernacular, is the ability to have the CDN download the file from your server the first time it’s requested, then cache it from that point on.  In other words, if someone requests http://cdn.kitchenpc.com/images/logo.png, and that particular node in the CDN doesn’t yet have that file, it will download the file from its origin, which is http://www.kitchenpc.com/images/logo.png.  This greatly simplifies things, as you don’t need to push out all your content ahead of time, and push it out again whenever things change.  Using Rackspace, I’d have to setup a new publishing process that copies all my files out to CloudFiles using their proprietary API (they can’t even be bothered to support something standard, such as FTP) – which means I have to either copy everything (very slow), or keep track of exactly what files were changed since I last published.  No thanks.

With a CDN supporting origin-pulls, I can simply deploy my changes out to my web server (which I already have scripts to do) and then call a single API which invalidates the CDN cache, causing the CDN to pull the modified files from my server once again.  For me, this is a huge win.

CloudFiles also doesn’t support folder hierarchies.  I find this completely ridiculous, especially considering how I have a “RecipeImages” folder with 60,000 files in it.  Their web interface basically chokes, and graphical front-ends to CloudFiles, such as CyberDuck, also are too slow to be usable.  I have various folders on my web server, such as /scripts, /styles, /images, etc to organize my content.  I’m absolutely not about to redo all my HTML to put every file in the same directory, especially since I use various plugins, such as TinyMCE, that require static resources to be laid out in specific locations.

CloudFiles actually allows you to hack around this limitation, as it supports the forward-slash character in an object name, but then you’d be writing some sort of publishing mechanism that flattens out your folder hierarchy, comes up with all the right object names, and deploys files using their proprietary REST API.  I had time for none of this.  Hopefully some day RackSpace will catch up with the rest of the world and provide a real CDN solution, but for now, they were out of the running.

GoGrid

Next, I decided to give GoGrid a try.  They came recommended to me, and were quite high up on a lot of the performance comparisons I looked at.  I mentioned earlier that EdgeCast has various resellers.  GoGrid is one of these.

GoGrid not only sells CDN services, but they provide complete cloud hosting (as well as dedicated hosting) as well.  Unfortunately, you have to sign up for their cloud hosting account before you can provision a CDN account, which must be done by emailing customer support and waiting an hour.  Signing up for this account proved difficult.  For starters, my credit card was denied due to a vague error.  After calling the bank, I was told the error was on GoGrid’s side.  Luckily, their customer support was incredibly helpful and stayed in chat with me for quite some time.  I was eventually told I should try a debit card, and I could switch over to a credit card once the account was setup.  However, at this time, their servers started misbehaving.  I would start typing in my billing information, and after about 10 seconds, the page would refresh and tell me the session had expired.  I tried on two different computers and three different web browsers, all with the same issue.  The customer support rep had no idea what the problem was.  Eventually, I connected to a remote server on RackSpace’s network and for some reason that computer worked.  It took me about two hours just to sign up for GoGrid.  The customer service rep was great, and even gave me a $100 credit on my account.

When I had the account setup, it didn’t take me too long to figure out how to setup the CDN.  Their interface is not the greatest, and changes (such as defining a CNAME) take a while to propagate, but it’s definitely usable.  Very quickly, I found a huge limitation in their design.

Basically, a CNAME (such as cdn.kitchenpc.com) can only point to a single root origin, such as http://www.kitchenpc.com/images.  I didn’t want to have to define cdn-images.kitchenpc.com, cdn-scripts.kitchenpc.com, etc.  Now, I can point cdn.kitchenpc.com to http://www.kitchenpc.com directly, however then any resource on my server can be pulled through the CDN; including dynamic content such as the home page.  Now, probably this doesn’t really matter that much.  In fact, it doesn’t at all.  However I’m just OCD about how my servers are setup, and decided I didn’t like it.  I did try emailing customer support, and they were completely ignorant as to what I was trying to do.  I decided it would take more time to explain it to them than it was worth, so I decided I’d had enough of GoGrid.

Amazon CloudFront

The last CDN I tried was Amazon CloudFront.  Amazon is, of course, a huge player in the startup world, hosting countless sites.  They have a huge CDN spanning the globe, and very competitive prices.  One nice thing about Amazon is I already do a lot of business with them and already have an account, so they already had all my billing information and everything.  Setting up a CloudFront account took about 30 seconds.

Their web interface is also superior and very easy to use.  Defining a CNAME was a snap, every change I made was instant, and everything just worked on the first try.  Amazon also has the same limitation; a CNAME can only point to a single root resource on the origin.  However, Amazon provides a feature called Behaviors.  A Behavior can define exactly what URLs are allowed and not allowed.  Using this feature, I was able to define behaviors to allow only requests to /scripts/*, /images/* and /styles/*.  Problem solved.

Amazon does have a bit of a downside.  Their pricing is quite convoluted compared to GoGrid, who just charges a flat rate per gigabyte.  Amazon has various prices depending on where you want data cached, how many times you call their APIs, how many times you invalidate the cache (though your first 1,000 are free), etc.  I came to the conclusion that though their pricing was not as straight forward, I just wasn’t dealing with enough data to care.  I expect my usage to be under $5/mon.

So…

So in the end, I ended up with Amazon CloudFront and am quite happy with them.  Modifying my site HTML was also quite easy.  Since I use XML templates for all my pages, I was able to add some code into the pre-processor to rewrite HTML such as:

<img cdn.src=”/images/logo.png” />

to:

<img src=”http://cdn.kitchenpc.com/images/logo.png&#8221; />

And then be able to define the CDN URL prefix within the web.config file.  This lets me run my site locally on my dev box without hitting the CDN, and use the CDN in the production environment.  I then had to change a bunch of HTML, however this was pretty easy with Visual Studio’s “Find and Replace” tools.  Within about an hour, I was completely up and running on Amazon CloudFront, with the site running quite nicely.

For startups, I’d definitely recommend using a CDN.  It’s very cheap (the bandwidth is probably cheaper than whatever you’re paying the server hosting company), provides fail-over and redundancy, takes stress off your web servers, and speeds up a lot of requests in other countries.  Also, since the domain name is different, you won’t be sending cookies over every HTTP request, which can speed up things as well.  Win-win.

Mmmm cookies (Technical Debt Part 2/3)

During the last month, I’ve been working on paying off some of the technical debt that has been collecting interest over the course of KitchenPC’s development.  Most of these have been stability changes, or just things that, when fixed, remove roadblocks to a smoother development process.  I’d like to go over these in a three part blog series that, like the Hitchhikers Guide to the Galaxy trilogy, will consist of five parts.

Part 2/3: Mmmm cookies

The authorization and session management code in KitchenPC is among the oldest and most cobweb infested in the entire code base.  The design is fairly straight-forward, and designed to be completely stateless so that any HTTP request could be handled by any server at any time.  Here’s how it works; I keep track of each user using three UUIDs in the database.  The first UUID is the user’s public ID.  This ID can be shared with anyone, and was actually used in several URLs on the old version of the site, which allowed users to subscribe to other users and see their public profile.  This is also the primary key for the Users table in the database.  The second UUID is called the Security ID.  This was also stored in the database, but never surfaced in any URL or HTML.  It wouldn’t be possible to see the Security ID of any other user besides yourself.  The third UUID is called the Session ID.  This UUID changes every time you log on to the site using the Logon form or the Facebook connect mechanism.

After the user is authenticated, all three of these UUIDs are joined into a single byte array and then Base64 encoded into a cookie.  This auth token is passed into any web service call as a way of representing the currently logged in user, and those three UUIDs are then de-serialized and validated against the database.  In order to hack the database, you’d need to know not only the user’s ID and security ID, but also their current valid session ID which changes on every logon.  It may not be industry grade security, but it’s fairly secure and works well.

The one problem with this approach is sessions must be maintained in the database.  To make matters worse, my design only supported a single session per user, as there was just one Session UUID column in the users table.  When users would logon with another browser, then that browser would have their currently valid session and the first browser would then show a “Session expired” error message.  This was sometimes not trapped, and weird “Unknown Error” messages would popup.  Most people just use a single computer at a time, so this wasn’t a huge deal.  However, the design just wasn’t going to cut it for mobile and tablet apps, when I want multiple users on multiple devices to use the same KitchenPC account.  A redesign was in order.

One approach would be to extend the current design.  I could create a sessions table, where a single user could have zero or more sessions.  Each time a user logged on, they would create a new session and have a valid cookie pointing to that session UUID.  I believe systems such as GMail work this way, as they allow you to list your currently active sessions.  The main drawback, in my opinion, was that it becomes messy to clean up after stale sessions.  Ultimately, you’d need some sort of cleanup process that ran every so often and deleted abandoned sessions.  You’d also have to keep timestamps on each session to keep it active, which would mean database updates on every web service call.  This could lead to scalability problems later on.

Instead, I took an approach modeled after the built in ASP.NET FormsAuthentication class, which runs as an HttpModule to validate each request.  Unlike my current implementation, the server does not keep track of valid sessions at all.  Everything needed to validate the authenticity of the session is stored within the cookie itself.  For example, a cookie might contain the string “Mike” as well as a UUID representing my user ID.  Upon each request, the server would say, “Yup that looks like a KitchenPC user account to me” and the request would be serviced under the credentials of that user account.

One major problem with this design.  What if the cookie is forged?  In other words, what if you changed the cookie to “Bob” and found out Bob’s user Id.

To prevent this, cookies are encrypted with a private key only known on the server.  It would now be difficult, if not impossible, to purposely change the cookie to represent another valid user.  Luckily, the .NET Framework has all the tools necessary to encrypt your cookies.  My new implementation uses a Triple DEA block cipher, which is a pretty good algorithm for this sort of encryption.  First, you’d want to serialize your session data to a byte array, which can be done manually or using the BinaryFormatter class.  We’ll call that byte array rawBytes.  Next, you’d encrypt the data to a new byte array:

TripleDES des = TripleDES.Create();
des.Key = key;
des.IV = iv;

MemoryStream encryptionStream = new MemoryStream();
CryptoStream encrypt = new CryptoStream(encryptionStream, des.CreateEncryptor(), CryptoStreamMode.Write);

encrypt.Write(rawBytes, 0, rawBytes.Length);
encrypt.FlushFinalBlock();
encrypt.Close();
byte[] encBytes = encryptionStream.ToArray();

In the code above, we create a new instance of the TripleDES class. The Key and IV properties are set to a byte array which represent your private key. These values need to be the same when you decrypt the cookie, so it would be wise to store them in a configuration file somewhere.

We then create a CryptoStream, and write the rawBytes array containing your unencrypted session information. We can then call encryptionStream.ToArray() to get the encrypted bytes out. You’d then serialize this out (perhaps as a Base64 string) to a cookie.

There’s one small problem with this. Though it would be nearly impossible to forge a valid cookie, one could still tamper with the cookie bytes and cause who knows what havoc. For example, if my encrypted bytes were [1, 2, 3, 4, 5] and I changed that to [1, 2, 3, 4, 6], this would still decrypt, just not to what I had originally encryted. This is because encryption algorithms don’t self-validate. It’s not like in the movies where if you crack the key, you get a big “Access Granted!” flashing message on the screen. This is actually good, as you don’t actually know when or if you’ve successfully decrypted the message; you just know that some bytes went in, some bytes came out.

However, we want to make our sessions tamper resistant. If someone changes even a single byte in the cookie, I want to just toss that cookie out and ignore it. To do this, you’d use a validation hash to store a hash of the unencrypted bytes. SHA-256 is a great cryptographic hash function for that purpose.

HMACSHA256 hmac = new HMACSHA256(valKey);
byte[] hash = hmac.ComputeHash(rawBytes);

valKey is an array of bytes, usually 64 bytes long, that would also be kept securely on your server. So long as you used the same valKey each time, the bytes you pass in to ComputeHash would hash the same each time.

So, now we’ve encrypted rawBytes to byte array called encBytes, as well as stored a hash of the original data stored in a byte array called hash. We’d now want to combine them into a single byte array as such:

byte[] ret = encBytes.Concat&lt;byte&gt;(hash).ToArray();

You now have a single byte array that you can serialize to a Base64 string and store in a cookie. Users would pretty much have no way to change the data without generating a new hash, which they could not do since they don’t have your hash key (valKey).

To decrypt the cookie, you’d do the same thing in reverse:

private static byte[] DecryptCookie(byte[] encBytes)
{
   TripleDES des = TripleDES.Create();
   des.Key = key;
   des.IV = iv;

   HMACSHA256 hmac = new HMACSHA256(valKey);
   int valSize = hmac.HashSize / 8;
   int msgLength = encBytes.Length - valSize;
   byte[] message = new byte[msgLength];
   byte[] valBytes = new byte[valSize];
   Buffer.BlockCopy(encBytes, 0, message, 0, msgLength);
   Buffer.BlockCopy(encBytes, msgLength, valBytes, 0, valSize);

   MemoryStream decryptionStreamBacking = new MemoryStream();
   CryptoStream decrypt = new CryptoStream(decryptionStreamBacking, des.CreateDecryptor(), CryptoStreamMode.Write);
   decrypt.Write(message, 0, msgLength);
   decrypt.FlushFinalBlock();
   decrypt.Close();

   byte[] decMessage = decryptionStreamBacking.ToArray();

   //Verify key matches
   byte[] hash = hmac.ComputeHash(decMessage);
   if (valBytes.SequenceEqual(hash))
   {
      return decMessage;
   }

   throw new SecurityException(&quot;Auth Cookie appears to have been tampered with!&quot;);
}

First, we split apart the byte array into its two components; the decrypted bytes and the 64 byte hash.

Once again we setup a TripleDES instance, only this time we decrypt the data to the stream. You’ll notice I use hmac.HashSize / 8 instead of hard coding in 64, since technically you can use any length key you want and I wanted my code to be flexible. I then have an array called message, which contains the TripleDES encrypted session data, and valBytes, which contain the validation bytes from the hash. I then decrypt message to a byte array called decMessage, and then compute the hash on that unencryted message. If it matches valBytes, then we know the key is legit.

Why Re-invent the Wheel?

Internally, this is very similar to what FormsAuthentication does.  So why did I create my own implementation?  First off, because I wanted to.  I actually enjoy implementing this sort of thing.  But mainly because I didn’t find the built-in support very flexible.  The ASP.NET mechanism only allows you to store a username as a string, and no other information on the user.  I wanted to be able to store the user’s UUID, as well as their name and perhaps a few other pieces of information.  I wanted to serialize this as my own datatype, allowing me to extract user data without having to hit the database for more information.  It also made the transition between my old code and the new code a lot easier, without the need to re-factor a lot of code.  The old cookie could be deserialized to a data type that was used in various methods, constructors, etc in the code base.  I wanted the new code to work in a similar way.

After doing quite a bit of testing with the new auth code, I must say I’m quite pleased.  Sessions last forever, and users don’t get random “Session Expired” popups when they logged on with another computer.  The code is also a lot faster.  I don’t need to check the authenticity of a cookie in the database each page load or web method call.  I can decrypt the data and validate the hash to make sure the information within the cookie is valid.  I can now be logged on to KitchenPC on all my computers and not worry about a thing!

The one drawback of this approach is these cookies are fairly long.  Mine is about 700 bytes or so.  This means another 700 bytes is passed in to each HTTP request, which can hurt performance.  This can be optimized by minimizing the data you’re storing in the encrypted package, and also using shorter (albeit less secure) keys and hashes.  I’m also thinking I could do my own serialization rather than using the BinaryFormatter class, which embeds things such as type data.

In the end, I implemented all this code as an IHttpModule which will run on each request, setting the value of HttpContext.Current.User if the cookie is valid, which is a fairly good approach.  You would then just add this class in the <httpModules> section of your web.config.  I was planning on sharing the entire code, but I’m afraid this post is just getting too long.  I’d be happy to share it with anyone who asks though!  Just let me know.  That’s it for now!

Optimizing the Application Loading Process (Technical Debt Part 1/3)

During the last month, I’ve been working on paying off some of the technical debt that has been collecting interest over the course of KitchenPC’s development.  Most of these have been stability changes, or just things that, when fixed, remove roadblocks to a smoother development process.  I’d like to go over these in a three part blog series that, like the Hitchhikers Guide to the Galaxy trilogy, will consist of five parts.

Part 1/3: Optimizing the Application Loading Process

KitchenPC is implemented as a web application that runs within an IIS worker process.  Everything from “What Can I Make” to recipe searches to NLP queries is run within this context, and nothing much is handled out of proc.  There is one exception, email queuing, which runs as a Windows service.  However, pretty much everything else happens under IIS.  For this reason, there is a rather lengthy load time when the app initially boots up, as it has to load every ingredient and recipe from the database, compile the data, build in-memory graphs, build search trees, and initialize other data on the heap which makes the user experience lightning fast later on.

During the beta, this was pretty fast since I only had about 10,000 recipes and no NLP dictionaries.  App startup time was about 2-3 seconds.  However, I now possess a collection of recipes nearly six times larger, plus a ton of grammatical data to make natural language queries possible.  The application start time, even on a fast computer, is now somewhere around 10-15 seconds.

This presented two problems.  First, I’m impatient.  Working long hours on KitchenPC requires many, many rebuilds.  Each time I fire up the application in Visual Studio, I had to twiddle my thumbs while all the data was loaded into memory.  Often times, I was changing things that had nothing to do with this data, and didn’t even need to use these features.  Second, there were many bugs where if requests came in while the app was loading, there’d be random dictionary collisions and other problems that would cause exceptions to be thrown, and sometimes the app initialization to fail.  This was mostly due to poor code that didn’t lock objects and make them thread safe.  As site traffic picked up, this created a lot of friction deploying new changes into production.  My goal was to quickly be able to deploy changes, and not have a single HTTP request error out.

Multi-threaded Application Loading

My solution to this problem was to make the Application_Start code as simple as possible, and simply spawn a new thread to handle the heavy lifting.  So now, the application start looks something like this:

public class Global : System.Web.HttpApplication
{
   void Application_Start(object sender, EventArgs e)
   {
      Thread thread = new Thread(LoadData);
      thread.Start();
   }

   private void LoadData()
   {
      // I can haz data
   }
}

The first time a user goes to the site, the LoadData method is called on a new thread.  The user will immediately see the home page, even though KitchenPC is actually not fully initialized.  Of course, this means that certain functionality on the site will not be available for about 10-15 seconds, which could be a problem if the user immediately uses something like “Ingredients to Exclude” (which relies on NLP), selects a recipe within the search results (which relies on recipe aggregation code), or clicks on “What Can I Make?” (which relies on the modeling engine) – All of these features depend on data that’s loaded into static memory on initialization.

To solve this, I put write locks on this data as it’s being loaded.  The LoadData() method, cute as the lolcatz reference is, actually looks more like this:

private static ReaderWriterLockSlim _cacheLock = new ReaderWriterLockSlim();

public static void LoadData()
{
   _cacheLock.EnterWriteLock();
   try
   {
      // Load data from the database
   }
   finally
   {
      _cacheLock.ExitWriteLock();
   }
}

Then, any code that needs to reference any of this data enters a read lock:

public static string ReadData(Guid key)
{
   _cacheLock.EnterReadLock();
   try
   {
      // Lookup key in data and return value
   }
   finally
   {
      _cacheLock.ExitReadLock();
   }
}

From the user’s point of a view, if they were to use one of these features while the app was still loading, they would get a ten second pause or so until the initialization code finished and the data were fully available.  Remember, this would only be a problem for the first few seconds after the IIS app pool was reset (such as new binaries were sent out to the server, or a configuration change was made), which usually happens at night when traffic is low.

Another advantage of this approach is I can also reload data without reloading the entire app.  There’s also administrative web services that can call LoadData() again, re-acquiring that write lock, to refresh the data in memory.  This is done when we change NLP data, push new recipes into the site, or change certain ingredient metadata.  Basically, certain site features can temporarily be halted or taken offline as data is being refreshed.

This makes making minor site config changes or site updates much smoother, and everything is now done in a much more thread-safe manner without weird random exceptions cropping up all over the place.

Not all components support this!

One gotcha with this design is not all ASP.NET components support being run outside the local thread scope.  For example, anything that relies on HttpContext.Current having a value will throw a NullReferenceException when being run in a newly spawned thread, as HttpContext.Current is actually a dictionary keyed by a thread ID.  If you’re no longer running in the thread that started the HTTP request, this value will all of a sudden be null.

Castle ActiveRecord, which is the ORM that KitchenPC is built on top of, uses the HttpContext as a key to figure out which database connection and transaction to use.

Luckily, ActiveRecord ships with a class called HybridWebThreadScopeInfo which will solve this problem.  This IWebThreadScopeInfo implementation will detect if HttpContext.Current is null, in which case it will initialize a new database context just as if it were a new request.

If you take a look at the code, you’ll see:

HttpContext current = HttpContext.Current;

if (current == null)
{
   if (stack == null)
   {
      stack = new Stack();
   }

   return stack;
}

Where-as the normal WebThreadScopeInfo class will throw an exception if current is null.

You can easily configure this in web.config with one line:

<activerecord isWeb="true" threadinfotype="Castle.ActiveRecord.Framework.Scopes.HybridWebThreadScopeInfo, Castle.ActiveRecord.Web">

That’s all there is to it!

Hope you’ve enjoyed this little technical insight into the inner workings of KitchenPC, and hopefully some of these techniques will help you come up with a better design for your app.  Stay tuned for part 2, which will be equally technical and nerdy!

To Mispell Is Only Human

Since launch, I’ve been logging natural language ingredient queries within the KitchenPC UI.  There are two places where the NLP engine is exposed; the search by ingredient mechanism, which allows users to include or exclude specific ingredients in their searches, and the What Can I Make? feature, which allows users to enter an ingredient with an optional amount.

During development, my initial design was to simply ignore anything I didn’t understand.  I would silently log it so I could improve the NLP vocabulary later, but I didn’t want to bother the user with silly errors messages.  Unfortunately, this totally bombed during usability testing.  Users typed in all sorts of randomness, and then couldn’t complete the tasks as they weren’t sure if the feature was working or not.  Ultimately, I decided to popup a message when I encountered something I didn’t understand, with a link to a video showing the user how to properly use the feature (those videos are quite popular, by the way!)

After a few weeks of use, I’d like to share the list of failed NLP queries with you, which will hopefully shed some light on the way users interact with the site.  This shows how incredibly difficult it is to teach computers to understand user intentions based on human language.  If nothing else, you’ll probably chuckle over a few of the queries.

beans
garbonzo beans
chick peas
1 pound of chicken
1 pd of chicken
capsicum
capsicum
cucmber
brussel sprouts
brussel sprout
brussel sprout
palak
obergine
Pasta
noodles
roast beef
beef
cooked pork
cooked pork
left over pork
left over cooked pork
left over cooked pork
cooked pork
boiled green beans
small beans
moong
pulled pork
pasta
a pound of pasta
lake trout
lake trout
wheatberries
annisette
grilled pork tenderloin
pork tenderloin cooked
water chestnuts
Tuna fish
Canned tuna
2 heads of celery
dairy
brussel sprouts
broccoli, pasta, olive oil
pasta
pasta
noodles
Mozarella
Bred
Beef
Grou d
beef
beef
meat
beef
red meat
pile of lettuce
16 ounce can of kidney beans
14.5 ounces reduced sodium chicken stock
asparagas
asparagas
linseed oil
Linseed oil
meat loaf
noodles
chicken, lentils
pureed jalapenos
"jethro tull" and "new haven" AND collection:etree
mac nuts butter eggs flour hamburger
eggs flour
eggs flour butter hamburger
meyer lemon
meyer lemon
meyer lemon
greek salad
beans
garbenzo beans
garbenzo beans
chick peas
melbourne cricket ground
brioche
brioche
brioche
noodles
noodles
noodles
noodles
sun dries tomatoes
pasta
sun dried tomatos
makarel
tomatos
tomatos
Canned tomatoe
wheal
chocolote
chocolate
chocolate
chocolate
chocolate
pasta
pasta
full milk
fruit
fruit
flouer
bakeing soda
bakeing soda
peppers
peppers
chiles
cooked rice
cooked rice
cooked rice
cooked rice
cooked rice
CARAMEL
CARMEL
CHOCOLATE
8 OZ COOKED CHICKEN
COOKED CHICKEN
COOKED HAM
Bambi rice chile pepper cheese
Bamba rice
cannelini beans
Risotto
French radish
meatloaf
chicken fingers
meatloaf
meatloaf
meatloaf
meat loaf
meatloaf
squash
meatloaf
cherry
cherri3es
jello, marshmallows, fruit cocktail  c
fruit cocktail, marshmallows
soja
beef
beef
Tao sauce
General tao
mayonaise
pound of ground beef
Red curly kale
tomatos
brats
brats
brats
bratwarts
portabello
malt
malt
crackers
monosodium
agave
molassas
Potatoe
potaoes
wasabi
wasabi
gluten free
peacon
maladextren
chocolate
aged cheese
chedder
honey chicken
honey grilled chicken
pasta
noodles
a bag of noodles
a bag of rice
ground turken
meat
avacodo
avacodo
top round steak
top round steak
top round
round rump roast
sirloin tip roast
potatos
calliflour
califlour
culiflour
pasta
Mozarella
Mozarela
grilled chicken
cornstart
corn surup
fresh spinanch
aparagus
fetta cheese
fetta
glurten
corn starcfh
fetta
chicken corn potatoes black beans
cooked chicken
cooked chicken
cooked chicken
1 cooked chicken
thai basel
blachan
Noodles
Pasta
Pasta
pasta
chicken, pasta, scallions
meatloaf
brussel sprout greens
greens
chili
lamb chop
fried tofu
lamb chop
Zuchini
1 LB. GROUND BEEF
sweet peppers
sweet peppers
sweet peppers
peppers
braising greens, parsnips
braising greens
tomatoe
leftover pork
beans
lentils,beans, bacon, peas
pound of split peas
ice cream
hot dog
pasta
parsley root
squash
squash
squash
squash
kiwitcha
farro
garbanzo
pumplkin
lunch meat, apples, lettuce, tomato, chinese noodles
pasta
pasta
parmesaen
squash
squash
sguash
meatloaf
1 instant noodle
meat ball
meat
butter/oil
margerine
margerin
canola
canola
chocolate
chocolate
chicked
ganulated sugar
brocoli
candy corn
meat
bell papper
mushroom soup
can of cream of mushroom soup
chicken potatoes
squash
tom
meat
pasta
noodles
noodles
noodles
black eyed peas
black eyed pea
blackeyed peas
caper
chili peeppers
chili pepper
cornstartch
corn startch
crackers
cracker
soy beans
soy bean
splendda
tartar sauce
veggie burgeer
Ricotta,spaghetti,gark
Ricotta,spaghetti,ragu
Chicken, pasta, garlic, tomato paste
canned salmon
canned salmon
canned salmon
tin of salmon
can of salmon
salmon tin
mexican corn
canned corn
salsa verde
black eyed peas
refried black  beans
sliced chiles
canned jalapenos
chipotle peppers canned
beef
frozen scallops
frozen scallops
Bowie pasta,ricotta, frox
Rotisserie chicken
rotisserie chicken
Leftover chicken
chicken piccata
fish filets
whole tomatoes
tomatoes whole
tomatoes canned
refried black beans
chicken corden blu
coocke chicken
cooked chicken
dry pasta
milk cheese
tortalini
pasta
frozen vegetables
can of chili
a can of chili
chili
noodles
pasta
minced beef
pasta
pasta
left over turkey
pasta
roast
pasta
crab
seafood
shellfish
fish
refrigerated biscuits
wheate
beans
red peper
bell peper
brocolli
cooked pork chops
pumpkin 2 cups
beans
beans
beans
potatos

This list can be divided into the following categories:

Misspellings:

Entries such as “garbenzo beans” instead of “garbanzo beans” and “tomatos” instead of “tomatoes” are difficult to fix.  I would either have to list common misspellings of these ingredients in the database as ingredient synonyms, or I would have to implement some sort of on-the-fly spelling checking algorithm to actually correct potentially misspelled words.  The former might be a good solution in the short term, especially for very common misspellings, or correct spellings in other dialects of English, however this won’t be of much use to people who are simply really bad spellers.  You know who you are.

Integrating NLP with standard spell checking algorithms is an approach, but both complicates and slows down already extremely complex code.  Plus, it would be no easy task and difficult to prioritize very high with all the things I have on my plate.

Generic Ingredients

Queries such as “chocolate”, “pasta”, “ice cream” and “squash” are very generic.  There’s many types of species of squash in the database, and from a culinary point of view, they can’t be treated as the same ingredient.  However, it’s obvious that users are hoping to include or exclude all varieties of squash from their queries.  I could easily see someone wanting to figure out what to do with a bunch of left over ice cream by searching for recipes that use any type of ice cream.  Unfortunately, the current KitchenPC database design doesn’t support this concept.

This is a limitation that has been on my mind for quite some time, especially for grouping ingredients that could pose issues with certain allergies (such as nuts.)  It could also be used for substitutions on the fly, such as “2% milk” and “1%” milk are quite often interchangeable between recipes.

I think the solution is to introduce a concept of an abstract ingredient into KitchenPC.  This ingredient would help relate similar ingredients together, and could be tied into NLP, searching, etc.  However, this is most likely a huge undertaking and would affect almost every aspect of the code and database.  In fact, when I think about it for more than ten seconds, I start to shudder at all the stuff I’d have to re-write.

People Still Don’t Read The Instructions

There are still queries such as “lentils,beans, bacon, peas” and “mac nuts butter eggs flour hamburger” even though the instructions explicitly say to type in one ingredient at at time.  The help video even shows this very clearly.  Apparently, no matter what you do, there will always be some users who just ignore this and want it to work in a certain way.

There is the possibility of either supporting multiple ingredients at once, or at least detecting when this is done and providing a clearer error message – not that these people are likely to read error messages.

Hopefully this is something that doesn’t happen a lot, and can be ignored as user error.

What The???

Lastly, there are a few queries that just make you scratch your head.  For example, “left over pork”.  Who cares if it’s left over or not?  Just type in pork!

Then of course there’s:

"jethro tull" and "new haven" AND collection:etree

I won’t even ask.

Fixing Valid Queries

It’s also obvious there’s a bunch of queries that should work but don’t.  I’m spending some time this week going through these and fixing as many as possible, or at least the most common ones.  Hopefully, over time, the interface will get harder and harder to fool.  However, that’s going to take a while.

Pretty Please, Check Out My Site!

It’s been over two weeks since I launched the new version of KitchenPC, and already over 2,000 people have checked it out.  Yes, this is going to be another one of those numbers posts.  You’ve been warned.

I thought I’d share some of the techniques I’ve been experimenting with to generate some traffic, as it seems to be a complete guessing game as far as what works and what has little effect.  As a control, I’d like to mention the average person (since the 10/1 launch) spends 4 minutes and 1 second on the site, and I have an average bounce rate of 39% (this means 39% of visitors leave the site after only visiting a single page.)  So, if you’re launching a new website, what should you do to attract visitors?  Is it worth spending money?  Where can you get the most bang for your buck?

Facebook Ads?

One easy approach, or so I thought, would be to just spend a few hundred bucks on Facebook ads.  What can be simpler than just paying people to visit your site, right?  Well, turns out it really isn’t as easy as it sounds.  I decided to create a single ad targeted at people with the interest “Cooking” on their profile.  If you like to cook, you’ll probably like my site.

On launch day, I wanted to get about 500 people to the site so I figured I’d bid about a buck and set the daily spending limit at $500/day for one day.  Turns out, such a strategy does absolutely nothing.  First, it took over a day to even get the ad approved from Facebook.  Second, when the ad was approved, traffic slowly just trickled in at under 10 clicks per day.  Most likely, the limiting factor in this result was the bid price.  I’m actually entirely unsure how Facebook’s magic algorithm works (I’m assuming it’s a very well-kept secret,) but I would guess the higher the bid price, the more people will see your ad.  I will go out on a limb and assume you’d have to make some insane bid price to get any real amount of traffic on a single ad.  I suppose the other technique would be to create dozens and dozens of ads under a single campaign, targeting various demographics, interests, age ranges, etc.  However, even doing all that, it seems trying to get 500 users onto a site in a day with Facebook ads would be a tall order.

So, does this technique even work?  Let’s look at the numbers.  I ended up lowering my budget to just $200 (total), and running the campaign for 2 weeks, ending yesterday.  Out of the 260,000 people the ad reached, just 128 people clicked on the link.  Oddly enough, I can only account for 116 of them using my own logs.  This means I paid $1.56 per visitor using Facebook.

This Facebook ad linked to a special URL that I could track using Clicky, which lets me compare this traffic against normal site traffic.  Clicky tells me that there were 109 unique visitors who used this entry point into the site (which means many people were clicking my ad more than once?) and spent an average time of 3 minutes and 24 seconds on the site; 15% less than the average visitor!  What’s even more depressing is their bounce rate was 51.4%, or 31% higher than average.  Only 11 of these visitors created a user account, two watched the intro video, and only a couple used any sort of real site feature.  Overall, it seems $1.56 for each of these users is a bit steep.

I’m well aware that there are whole companies who can optimize these sorts of ad campaigns, but in all honesty I can’t really say I’d recommend using Facebook ads to drive traffic into your site unless you have huge truckloads of money to burn, and a team of people working on these campaigns and monitoring and optimizing traffic.  Plus, you should probably know your site is actually something people want before spending any real amount.

Just Ask Some Press People

About a week before launch, I decided to send a link out to the test version of the site to various technology blogs.  These included sites such as AllThingsD, GeekWire, Sprouter, TechCrunch, ReadWriteWeb, GigaOm, LifeHacker, and PandoDaily.  I also emailed a few food bloggers, though most have very strict guidelines about promoting a product (i.e., most won’t do it.)

Out of this list, I only heard back from GeekWire, who agreed to run a Startup Spotlight segment on the site on launch date, which was fantastic!  The Startup Spotlight article drove 150 people to the site, who spent an average of 5 minutes and 47 seconds on the site with a bounce rate of just 11.9%.  This is already a huge improvement over Facebook ad traffic, not to mention it was completely free.  Though many of these people used various site features, only 3 new user accounts were created; perhaps GeekWire readers are interested in checking out new technology, but aren’t really my target demographic.  I think it’s safe to say emailing a few blogs is a better way to generate site traffic than paying a couple hundred bucks to Facebook.

One quick side note on Facebook.  I also played around with their “Promoted Posts” feature, which allow you to pay either $5 or $10 to promote a post.  This appears to be a complete scam.  You’ll get a bunch of new Likes, however, most of these Likes were from obviously fake Facebook accounts.  Some of them had pornographic materials, or were obvious marketing scams (mostly overseas.)  I don’t think I got any new real Facebook followers by promoting a post.

Press Release?

The first time I launched KitchenPC back in 2010, I did a press release which was pretty fun and also generated some great leads.  The story got picked up on a major food blog (generating 4,000 visitors in a single day!) and also created a lot of buzz on Twitter.  It’s really difficult to quantify the success of a press release, but I’m of the opinion it’s a good use of your time and money.

I decided to issue another press release for this launch, which went out the same day the new site went online.

With press releases, you’ll get dozens of sites that simply re-publish anything that comes over the wire.  If you’re a wanna-be news site, it’s cheaper than writing your own articles.  However, I got very little traffic from most of these.  The one exception seems to be a newspaper, The Sacramento Bee.  They published the press release word for word (which now appears to be gone), and drove 167 people to the site.  These users spent an average of 5 minutes and 33 seconds on the site with a bounce rate of 12.6%.  8 of them signed up, and a pretty good chunk used many of the major site features.

This article, plus the various other republications definitely make a press release a pretty cost effective way of generating site traffic, and judging from last time, the effects of this press release could pay dividends for many months.

Spam, I mean Email

I also once again used MailChimp to send out an update to the thousands of existing user accounts.  Once again, this got my account locked due to an excessive number of people reporting the mail as spam (I think their cut off is above 0.5%).  I’ve really found no way to prevent this from happening, as no matter what, some people don’t appreciate being emailed.

I’m somewhat kicking myself for not tracking email clicks with a track-able URL like I did with Facebook, as I’d love to have metrics on this.  However, I can see 65 visitors from Yahoo! Mail and 8 visitors from Live mail.  I also got a few email replies directly.  These traffic sources had similar stats than the press release traffic, which was of course far better than Facebook Ads.

My Advice?

So, my advice?  Try to muster up some press.  Write a good press release, get it out there.  Write bloggers related to your market.  If you’ve built up an email list through a beta release or a landing page, email them.  Don’t waste your time with online ads; they don’t work and are not cost effective.

I think it goes without saying the primary goal for launch is not to generate massive amounts of traffic.  The goal is to get a small amount of traffic (perhaps a few thousand people) and see if that traffic sticks.  See if they’re signing up, see if the features you worked so hard on are being used, and see if they keep coming back for more.  From what I’ve seen so far, it’s obvious this version of KitchenPC is a huge improvement over the last one.  My daily traffic is now over double what it was at the end of the beta, so things are definitely heading in the right direction.  At some point, you’ll generate something that really clicks and people will start sharing it with their friends, bloggers will start writing about it on their own, and you’ll build up a loyal user base organically.  At least, that’s what I’m hoping!

 

A Brand New Day

It’s been nearly two years since I launched the initial beta version of KitchenPC.com, a site that attempted to bring online meal planning to the masses.  It’s been over a year since I decided to pivot, and re-invent the site as a powerful recipe search engine.  During that year, I’ve developed natural language parsing technology to understand the inaccurate human expressions of recipes…  web crawling technology to import hundreds of thousands of recipes from major recipe sites, and categorization code to make sense of it all.  Not to mention rewriting the entire site UI from scratch with a cleaner, simpler to use interface.  It’s been a busy year.

So, as you can imagine, it’s an enormous relief to announce the new site is now live!

I invite all of you to check out this brand new iteration of KitchenPC, and explore the powerful new search features, simple to use menu tools and delightful pre-built meal plans.  Better yet, tell your friends!  Everyone knows someone who loves to cook, is looking to eat better, or is just starting out.  Share some of the recipes or meal plans on Facebook, Twitter or Pinterest.  Bloggers, feel free to share your opinions as well.  Let’s get the word out!

Lastly, I appreciate everyone who has shown support and interest in this project over the last year.  I’ve made many new connections, from foodies, other entrepreneurs, and even developers looking to incorporate KitchenPC technology into their own products.  I also expect the next year to be even more packed with KitchenPC innovations; from mobile to tablet apps, new features and innovation, and lots of new content for the site.  Anything you’d like to see?  Just send me a note, or leave a comment below!

That’s it for now!

What’s For Dinner? SALT!

This evening, I’ve been toying around with some SQL queries to model exactly how the “common recipe” search should behave.  This allows me to play with limits and values before I commit anything to code.

In reality, these efforts have exposed some of the culinary cruft that exists within the lower ranks of the major recipe websites.

First, I decided to rank each ingredient in the database with a commonality ranking.  I’ve decided that this ranking would be the total uses in the database divided by the total uses of the most used ingredient.  For example, salt is the most used ingredient in the database, with 28,016 recipes using it.  Salt would have a commonality of 1, since 28,016 / 28,016 = 1.  Granulated sugar is the second most common ingredient, with 25,389 uses.  It’s ranking would be 25389 / 28016.0, or  0.90623.  I created a view to generate these rankings as such:


CREATE VIEW IngredientCommonality AS
 WITH Ing AS (
 SELECT
 ShoppingIngredients.DisplayName,
 ShoppingIngredients.IngredientId,
 (SELECT Count(1) AS Count FROM RecipeIngredients WHERE RecipeIngredients.IngredientId = ShoppingIngredients.IngredientId) AS Uses
 FROM ShoppingIngredients
 )
 SELECT Ing.DisplayName, Ing.IngredientId, Ing.Uses, (Ing.Uses / (MAX(Ing.Uses::float) OVER ())) as Commonality FROM Ing;

Next, I decided to see the top 100 recipes in the database, sorted by how common their average ingredient was:


SELECT
 RecipeId,
 Title,
 (select AVG(Commonality) from RecipeIngredients inner join IngredientCommonality USING (IngredientId) where RecipeId=r.RecipeId) as AvgCommonality
FROM Recipes R ORDER BY AvgCommonality DESC LIMIT 100;

According to KitchenPC, the most common recipe on the entire Internet was crawled from AllRecipes.com, and is simply called “…”.  It appears to be some sort of test recipe, and its only ingredient is salt.  It has a perfect average commonality rating of 1.  You can check out this brilliant piece of culinary genius for yourself here.  By the way, it has a prep time of only 10 minutes, so it’s also an excellent choice for the busy single parent!

Also found was this gem on Food.com.  It’s called “Spa Cream” and consists of milk… and apparently more milk.

Ideally, I’d like to design this feature in such a way where the more common recipes appear at the top of the results, so users will see recipes they can most likely make at the top.  However, the fact that this exposes a lot of these junk recipes is a downside, to say the least.  One idea would be to filter out recipes that only call for a single ingredient, or devise other criteria to help avoid these less than useful matches.  On the bright side, it’s somewhat comforting that other large recipe sites (with huge budgets) also have less than perfect recipe databases.  Now it’s the job of KitchenPC to clean up the Internet!