A Better Way To Deploy (Technical Debt Part 5/3)

During the last month, I’ve been working on paying off some of the technical debt that has been collecting interest over the course of KitchenPC’s development. Most of these have been stability changes, or just things that, when fixed, remove roadblocks to a smoother development process. I’d like to go over these in a three part blog series that, like the Hitchhikers Guide to the Galaxy trilogy, will consist of five parts.

Part 5/3: A Better Way To Deploy

During the beta, I was more focused on code and features than boring process and tools.  One of the things I never got around to was any sort of modern deployment system to push new changes out to production.  This wasn’t a huge deal, since I only had a single web server and didn’t have enough traffic to really care if I had to bring down the site for a couple minutes.

The original technique for pushing out changes is rather embarrassing, but I’ll share it with you anyway.  I’d go out on a limb and say there are plenty of entrepreneurs who do the exact same thing.

First, I ran MSBuild (I had previously used nAnt but switched to MSBuild for a variety of reasons) to build all the individual KitchenPC projects.  The MSBuild script would also crunch the CSS and JavaScript files using the YUI Compressor.  Everything would be copied out to a directory called /Build which had a subdirectory for binaries, a subdirectory for site content, and a subdirectory for the KitchenPC Queue service which also runs on the web server.  Building the entire site would take around 20 seconds.

Next, I’d zip up everything.  If I was making a lot of changes to the site, I’d just zip up the entire Build directory.  However, sometimes I was just making some script changes or binary changes, and would just zip up the files that were changed.  Then, I’d Remote Desktop (we need a better transitive verb for that) into the production web server, and transfer my files over.  Originally, I put them up on my own private web server running off my home network and just use wget to download the file, but later on I started using FTP instead and manually uploaded the files to the web server.

From the desktop, I’d unzip the package and copy out the files, usually running an iisreset when everything was in place.

So, in other words, everything was manual.  It would usually take five or ten minutes to get everything deployed.

After launch, I finally had time to think about this process and devise a better solution for pushing changes out.  Ideally, I wanted to do this without having to Remote Desktop into production web servers, and also be able to push changes out with a single command line tool.  A bonus would be the ability to not have to take the site down, lose any traffic, or reset IIS when there were no binary changes being deployed.

On the Microsoft technology stack, WebDeploy is the cutting edge solution for doing all of the above, and more.

WebDeploy is a protocol that’s able to compare a set of files on a source and destination, and synchronize those files automatically.  It can be used for synchronizing files between two web servers (making the provisioning of more web servers extremely efficient) or synchronizing files between a development or staging environment and a production web server.  It could also be used to create backups of a web server and its configuration by dumping an entire IIS application to a ZIP package, or a raw directory.  It’s far superior to FTP, as it’s smart enough to compare the timestamps on files to figure out exactly what has changed between a source and destination, and only transfer that content over the wire. WebDeploy is used on Windows Azure, as well as the Microsoft WebMatrix platform, and also integrated with recent versions of Visual Studio. Basically, WebDeploy is where it’s at.

WebDeploy runs as an IIS extension, and can be installed easily using the Microsoft Web Platform Installer.  When installed on a machine without IIS (such as a development box,) only the client side tools (such as MSPublish.exe) are installed so deploying changes from the command line to a remote IIS box can be accomplished.  Also, Visual Studio 2010 and above will install MSPublish.exe automatically, since the Publish feature in Visual Studio calls MSPublish.exe under the covers when publishing a site using WebDeploy.

As it would not make a very good blog post to walk you through setting up WebDeploy on your web servers, I’ll refer you to this page which will walk you through the steps.

If you’re running Visual Studio 2010 or 2012, setting up Web Deploy on your web server is really all you need to do.  Once you configure WebDeploy on a site, it will export a settings file which can be used directly in Visual Studio.  However, since I’m a command line freak (plus, I was still running VS2008 at the time,) I decided to integrate WebDeploy into my MSBuild scripts instead, and then be able to build and deploy changes in a single command.

To use MSDeploy from within an MSBuild file, you’ll first want to setup some properties within your project.  These properties should be defined directly under the <Project> tag so they’ll be global to all tasks.  I define a few properties to hold various publishing settings, and support the ability to publish to both a staging and production web server:

  <PropertyGroup>
    <ArchiveDir>$(MSBuildProjectDirectory)\$(BuildDir)</ArchiveDir>
    <AppHost>KitchenPC</AppHost>
    <UserName>publish_acct</UserName>
    <Password>secret</Password>
  </PropertyGroup>

  <Choose>
    <When Condition="'$(Configuration)'=='Staging'">
      <PropertyGroup>
        <PublishServer>https://staging</PublishServer>
      </PropertyGroup>
    </When>
    <When Condition="'$(Configuration)'=='Release'">
      <PropertyGroup>
        <PublishServer>https://www.kitchenpc.com</PublishServer>
      </PropertyGroup>
    </When>
  </Choose>

Let’s go through each of these properties:

ArchiveDir: For me, this is simply where I build my site to.  The directory structure needs to be identical to the way MSPublish exports your site, so you can run:

msdeploy -verb:sync -source:appHostConfig=”Default Web Site”,computerName=server -dest:archiveDir=c:\Test

To dump your site to a temporary directory on your hard drive to see.  Replace Default Web Site with the name of your site in IIS, and replace server with the computer name or IP address of the web server.

For me, it puts all the files in /Content/c_C/Website but this might be different for you.  It’s also possible MSDeploy has the ability to define exactly how the folder layout should look, but I’m not yet enough of an expert to know.  Also note that Visual Studio will do all this packaging for you.

AppHost: This is the name of your application in IIS.  This is probably Default Web Site if you haven’t mucked with the default IIS settings, but for me this is KitchenPC.

UserName: This is the username MSPublish will try to connect to your web server with.  This might be Administrator if you haven’t granted anyone else access, but it’s advisable to setup a user account just for publishing.

Password: This is the password for the username above.

It may or may not be necessary to hard code all these publishing settings into your MSBuild script.  Visual Studio is able to store these settings in a settings file, but from what I can tell, those settings are extracted and passed directly into MSDeploy.exe when you publish.

I also have a conditional property called PublishServer which changes depending on the build configuration.  If I’m building the Staging configuration, I’ll publish to my local staging server.  If I’m building in Release mode, I’ll deploy to production.

Next, you’ll need to define a new Target for your deploy task:

  <Target Name="Publish" DependsOnTargets="Build">
    <Message Importance="High" Text="Deploying to $(PublishServer)" />

    <Exec
      WorkingDirectory="C:\Program Files\IIS\Microsoft Web Deploy V3\"
      Command="msdeploy.exe -verb:sync -source:archiveDir=$(ArchiveDir) -dest:appHostConfig=$(AppHost),computerName=$(PublishServer):8172/msdeploy.axd,authType=Basic,userName=$(UserName),password='$(Password)' -allowUntrusted" />
  </Target>

Notice this target depends on the Build task, which compiles the site with the proper configuration, crunches all the CSS and JavaScript files, and moves everything to the correct output directory using the right folder hierarchy.

It then simply executes the msdeploy.exe command with the appropriate command line arguments. Note if you’re using a self-signed SSL certificate on your server (which the Web Platform Installer will create by default), you’ll have to use the -allowUntrusted command line parameter as well. Hopefully, with a bit of tweaking, this will get you started on your own solution!

What About Web.config files?

Glad you asked!  Another very cool thing about Visual Studio 2010 and above are web.config transforms.  In the past, I just had separate files for various web.config files (such as prod.web.config and staging.web.config), and my MSBuild script would copy the correct one to the build output directory.  When I zipped up the site, the production web.config would be included.

The correct way to do this is using a transformation, which is basically an XSLT file that contains instructions on transforming your default web.config file to a file suitable for a specific environment.  Visual Studio takes care of all of this for you if you’re using a Web Application Project, allowing you to right click on your web.config file and add a new configuration.

However, the actual code for doing these transforms is neatly packaged up in managed libraries, and can be accessed as an MSBuild task.  Since, again, I’m a command line freak this is the approach I took.

The first step is to reference the Microsoft.Web.Publishing.Tasks.dll assembly in your MSBuild script.  This can be done by adding the following line directly under the <Project> node:

  <UsingTask TaskName="TransformXml"
    AssemblyFile="C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>

I just hard coded in the path, but you could easily make this an environment variable, or copy the DLL to a more appropriate directory.

Once this assembly is referenced, you can start using build tasks defined within that library. This is ridiculously easy:

    <TransformXml
      Source="KPCServer/web.config"
      Transform="KPCServer/web.$(Configuration).config"
      Destination="$(DeployDir)/web.config" />

Basically, we take the default web.config checked into your source tree, run the appropriate transformation against it (such as web.staging.config or web.release.config) and output the file to the correct output directory; in my case, the deployment directory that MSPublish picks up.

I won’t get into the details about building these transformations, since there’s tons of documentation online.

Once this task is added into your build target, then we’ve nearly got a complete build system in place.  Running the deploy target will first build your site, copy and transform your web.config file over, and run MSDeploy.exe to publish your site (copying only the files that have changed, of course!)

There’s one more thing we want to do.  Minimize CSS and JavaScript files.

Crunch Time!

Previously, I used YUI Compressor, a Java utility developed by Yahoo! to minimize JavaScript and CSS files.  It’s fantastic and pretty much the standard in the industry.  I would just execute this utility directly from an MSBuild script using an Exec task and call it good.

However, this time I decided to try out the .NET port of YUI Compressor, which is called YUICompressor.net.  The author claims the algorithm is identical, and the results should be nearly byte-for-byte the same.  It can be easily installed with NuGet and it also has MSBuild tasks included.

One drawback is the documentation is wildly out of date!  I had to spend a lot of time in ildasm digging through the assembly to figure out what was going on.  Hopefully, I can provide some better documentation here for people who are attempting to integrate YUI Compressor .NET into their MSBuild scripts.

The first step is to reference the build tasks.  There’s two of them, one for JavaScript minification and one for CSS files:

  <UsingTask
    TaskName="JavaScriptCompressorTask"
    AssemblyFile="packages\YUICompressor.NET.MSBuild.2.1.1.0\lib\NET20\Yahoo.Yui.Compressor.Build.MsBuild.dll" />
  <UsingTask
    TaskName="CssCompressorTask"
    AssemblyFile="packages\YUICompressor.NET.MSBuild.2.1.1.0\lib\NET20\Yahoo.Yui.Compressor.Build.MsBuild.dll" />

These directories happen to be where NuGet installed the binaries, but you can probably put them wherever.

I then made two separate MSBuild tasks, one for JavaScript and one for CSS.  We’ll go through the JavaScript task first:

  <Target Name="CrunchJs">
    <!-- Select all kpc*.js files -->
    <CreateItem Include="$(DeployDir)/scripts/kpc*.js">
      <Output TaskParameter="Include" ItemName="JsCrunch" />
    </CreateItem>

    <!-- Copy them to .debug files since JavaScriptCompressorTask cannot read and write to the same file -->
    <Copy SourceFiles="@(JsCrunch)" DestinationFiles="@(JsCrunch->'%(Identity).debug')" />

    <!-- Compress the .debug files, and delete them afterwards -->
    <JavaScriptCompressorTask
      SourceFiles="%(JsCrunch.Identity).debug"
      ObfuscateJavaScript="True"
      EncodingType="Default"
      LineBreakPosition="-1"
      LoggingType="None"
      DeleteSourceFiles="Yes"
      OutputFile="$(DeployDir)/scripts/%(Filename)%(Extension)"
      />
  </Target>

First, we select all the files we want to crunch. In my case, this is anything in the output directory that matches kpc*.js (as I give all my script files a kpc prefix). Next, we copy each file to a new file with .debug appended to the end. For example, we copy kpccore.js to kpccore.js.debug and kpchome.js to kpchome.js.debug. We do this because YUI Compressor .NET is not capable of using the same filename for both the input and output files, and I want to simply compress all the files in my build directory.

Lastly, we run the JavaScriptCompressorTask task. Note YUI Compressor .NET can only handle a single output file. While it can handle an enumeration of items for the SourceFiles parameter, it can only output them as a single file. For example, I can pass in kpccore.js, kpchome.js, kpcmenu.js, and all my other files as the SourceFiles parameter. However, it will not output separate crunched files. It will output everything to a combined file, such as kpcall.js. I don’t really like this, since I don’t need to include kpchome.js on any page besides the home page, or kpcqueue.js on any page besides the queue page.

For this reason, I use MSBuild batching to call the JavaScriptCompressorTask task multiple times, one for each item in JsCrunch. This is indicated using the percent sign prefix, %(JsCrunch) rather than @(JsCrunch), which would send in all the files as a single input parameter.

So, I run the JavaScriptCompressorTask on each file within the JsCrunch item set, appending the .debug prefix to the end since the input file can’t be the same as the output file. I also add the DeleteSourceFiles parameter to delete the temporary .debug files after I’m done.

The CSS task looks very similar:

  <Target Name="CrunchCss">
    <!-- Select all *.css files -->
    <CreateItem Include="$(DeployDir)/styles/*.css">
      <Output TaskParameter="Include" ItemName="CssCrunch" />
    </CreateItem>

    <!-- Copy them to .debug files since CssCompressorTask cannot read and write to the same file -->
    <Copy SourceFiles="@(CssCrunch)" DestinationFiles="@(CssCrunch->'%(Identity).debug')" />

    <!-- Compress the .debug files, and delete them afterwards -->
    <CssCompressorTask
      SourceFiles="%(CssCrunch.Identity).debug"
      EncodingType="Default"
      LineBreakPosition="-1"
      LoggingType="None"
      DeleteSourceFiles="Yes"
      OutputFile="$(DeployDir)/styles/%(Filename)%(Extension)"
      />
  </Target>

I’ll let you figure that one out, since it’s basically a copy of the JavaScript task, only using the CssCompressorTask task instead.

The last step is running this target after I build. I add the following lines to the end of my Build target:

    <!-- Crunch Files -->
    <Message Importance="High" Text="Crunching JavaScript/CSS Files..." />
    <CallTarget Targets="CrunchJS;CrunchCSS" />

This basically instructs the Build target to call both the CrunchJS and CrunchCSS targets after it finishes, minimizing the JavaScript and CSS files that have been copied to the build output directory.

Wrapping Everything Up in a Bow:

Since I use PowerShell for my build environment, I’ve also created a new PowerShell function to build and deploy the site to either staging or production:

function Publish
{
   if ($args.Count -gt 0)
   {
     switch ($args[0].ToUpper())
     {
        "STAGE"   { msbuild .\build.xml /t:Publish /p:Configuration=Staging /verbosity:minimal }
        "RELEASE" { msbuild .\build.xml /t:Publish /p:Configuration=Release /verbosity:minimal }
     }
   }
   else
   {
      Write-Host "Please specify either STAGE or RELEASE build.`n"
   }
}

Now, I can simply type Publish Stage or Publish Release at the command prompt to run the Publish target in build.xml.  This will build everything in the appropriate configuration, transform the correct web.config and copy that to the deployment directory, minify everything using YUI Compressor .NET, and then publish that whole package to the correct server using MSDeploy.exe.  Magic!

Hopefully some of this has inspired you to take a look at your current deployment strategy and think about some improvements.  What I have is far from perfect, but it was a solid weekend of work and far superior to what I had before.  Combined with the fact that my app starts nearly immediately now, and the fact that only modified files are copied out, I can now deploy new versions of the site extremely quickly without losing any traffic.  Requests that are coming in while I deploy would probably just see a couple second lag while the new binaries are loaded and the app pool recycles.