Rise of the TwitterBot

Last night I found myself in the mood for a little late night coding, and decided to tackle a project that I’ve had in mind for a while now; setup a Twitter feed that KitchenPC would automatically post new recipes to.  I’ve seen some other sites do this, and figured that even if no one follows it’ll at least be an excuse to go learn Twitter’s API and do something new.

I first decided to create a new Twitter account exclusively for this purpose.  This feed would only contain KitchenPC recipe links as not to pollute the existing @KitchenPC Twitter account which advertises company news and updates.  This new feed could potentially tweet a hundred or more new recipes a day, so one would have to be somewhat crazy to follow this feed in the first place.  The last thing I’d want to do is have people un-follow @KitchenPC because it became too “spammy.”  With that idea in mind, I created @KPCRecipes and started my research.

The obvious design for my TwitterBot was to add code into the KitchenPC Queue, which is a Windows Service that sits around all day emailing people when stuff happens.  If you’ve ever received an email from KitchenPC (notifications, password resets, etc), that was the queue in action.  It basically watches for files dropped to a certain directory and processes them one at a time.  For example, if someone posts a new recipe on KitchenPC, an XML file is created on the file system.  The KPC Queue will then open the file, deserialize the data within the file and see that it contains information about a new recipe post.  It then loads the recipe data from the database, along with any users who subscribe to that event, and emails them.  This way, the website itself is not being blocked while it emails potentially dozens of people.  In the future, I could also scale this out by adding more KPC Queue instances or creating a read-only copy of the database for queue processing.  This queue design also provides fault tolerance.  If there’s an error sending emails or the SMTP server is down, the file is skipped and tried again later.  Even if the queue process crashes, files will just start piling up and will be processed when the queue comes back online; no emails would be lost.

The KPC Queue provides the perfect place for the TwitterBot code to live as well; whenever a “recipe post” event is processed, I can call up the TwitterBot and handle the tweeting out of band, as not to slow down the website at all.  Perfect!

Turns out, there is absolutely no good information online about how to create a TwitterBot, or an automated program that posts Twitter status updates to a hard coded user account.  Twitter’s API documentation is decent, however most of it assumes you’re writing a web based application where the user logs on to their Twitter account and provides your application access.  They also briefly cover how to write a desktop application where the user can provide your application rights to their Twitter account.  Every blog and tutorial I found on the Twitter API was also geared around one of these scenarios.

Those of you even slightly familiar with Twitter’s APIs will know that Twitter is built around OAuth, a protocol for transferring user credentials without providing a username and password, as users might not want your app to have this.  The problem I found with OAuth is tokens are exchanged via various HTTP redirects, which are usually encapsulated in a web based interface.  While it’s true that Twitter used to allow credentials to be expressed using HTTP basic auth, they’ve blocked that ability as of last year and forced all apps to use OAuth.  I did not want my TwitterBot to have to pop up a web browser for me to logon to my KPCRecipes Twitter account; I wanted the account credentials to be hard coded into the program and require zero user interaction.  After all, the queue runs as an NT Service and doesn’t even require a user to be logged on to the system.  Digging up information on this design was rather difficult.  Sounds like the perfect opportunity to write a how-to blog!

Step 1 – Register your application on Twitter

The first thing you’ll need to do to get Twitter’s API to even talk to you is register your application on Twitter.  It doesn’t matter that I’ll be the only one in the world using this application, it still needs to be registered with Twitter.  This gives you an application key which you can use when calling APIs.  You can register your application by going to http://twitter.com/apps/new and filling out the form.  The form is fairly easy to follow, and I chose “Client” for my Application Type for reasons I’ll get into below.  When you submit the form, you’ll get a Consumer Key and a Consumer Secret.  This information is specific to your application and will never change, unless you go in and reset it.  You’ll need to hard code this information into your app, and from this point on I’ll refer to these values as CONSUMER_KEY and CONSUMER_SECRET.

Step 2 – Dig up an API wrapper for your language of choice

There’s several Twitter Libraries written in .NET – I checked out TweetSharp and Twitterizer and decided I liked Twitterizer a bit better.  One thing I liked about Twitterizer is you do pretty much everything by calling static utility methods, so everything is very stateless.  Once you download Twitterizer, just copy the two DLLs (Newtonsoft.Json.dll and Twitterizer2.dll) into your project directory and reference them.

The Twitterizer “getting started” tutorial not only provided a great outline with code samples to get started, it also provided a very simple explanation of how OAuth works.

Step 3 – Authorize your application using your Twitter account

Ok so now we have a Twitter account, in my case KPCRecipes, and our application we registered in step 1.  We want this application to be able to post status updates on our Twitter account, so the Twitter account has to authorize the application to do so.  This process must be done over the web; I suppose you could automate this, it’s just HTTP traffic after all, but luckily it’s something you only need to do once.  I wrote a simple little program to do this for me, which looks like this:

OAuthTokenResponse authorizationTokens = OAuthUtility.GetRequestToken(CONSUMER_KEY, CONSUMER_SECRET, "oob");

string url = String.Format("http://twitter.com/oauth/authorize?oauth_token={0}", authorizationTokens.Token);
Console.WriteLine("Go to:\n\n{0}\n\nLogon as @KPCRecipes and enter the pin number below:\n\n", url);
string pin = Console.ReadLine();

OAuthTokenResponse accessTokens = OAuthUtility.GetAccessToken(CONSUMER_KEY, CONSUMER_SECRET,    authorizationTokens.Token, pin);

Console.WriteLine("Here are your access tokens:\n\nScreenName: {0}\nToken: {1}\nTokenSecret: {2}\nUserId: {3}\n\n", accessTokens.ScreenName, accessTokens.Token, accessTokens.TokenSecret, accessTokens.UserId.ToString());

Let’s walk through this code.  First, we use GetRequestToken to ask Twitter for an authorization code for our app, using the app’s CONSUMER_KEY and CONSUMER_SECRET.  The third parameter is a callback URL which Twitter will redirect to after the user approves the app.  Normally, Twitter would redirect to that callback URL and pass along the access tokens that we seek.  In our case, we need to get these access tokens programmatically.  We thus pass in the string “oob” to instruct Twitter to give us a PIN code to access these tokens later.  Yes, kinda weird but just go with it.

Next, we spit out a URL for the user to go to and approve our app.  We could spawn a web browser here, but I was lazy and decided to just write out the URL for the user to copy and paste into their browser.  When you go to this URL, you’ll see the name of your app and are asked if you want to allow it to have access to your Twitter account (be sure you’re logged in using the correct Twitter account!)  Once you accept it, you’ll see a PIN code.  Paste that PIN code into the app, and this value will now be stored in the string pin.

So, at this point of the code, my Twitter app has access to my KPCRecipes Twitter account and can post status updates programmatically.

However, I still need the access tokens to programmatically access that Twitter account in the future.  These access tokens are sort of like a username and password, but can only be used by this one app and only to access this one Twitter account.  Anyone else who tried to use them would need to know my app’s CONSUMER_KEY and CONSUMER_SECRET.

The next part of the code calls GetAccessToken, passes in the authorization token we had originally, along with the pin that Twitter gave us.  This call returns all the information we need to access the KPCRecipes account using our Twitter app.  The good news is this information will never change, so I can just hard code this into the KPC Queue.

The two pieces of information you’ll need are accessTokens.Token and accessTokens.TokenSecret.  We will call these ACCESS_TOKEN and ACCESS_SECRET.  I added these two values to the queue’s .config file so I can refer to them at runtime whenever I need them, and it’s of course much less hacky than actually hard coding them into the compiled app.

Step 3 – Go Tweet something!

So, now we can throw this program away.  It’s completely not needed unless for some reason you have to re-approve the app again or there’s a zombie apocalypse.  We can now use all this information to programmatically update our Twitter status.  The first thing to do is build an OAuth token which we can send to Twitter to tell it which app we are and what Twitter account we want to access.  You can do that with the following code:

OAuthTokens tokens = new OAuthTokens();

tokens.ConsumerKey = CONSUMER_KEY;
tokens.ConsumerSecret = CONSUMER_SECRET;
tokens.AccessToken = ACCESS_TOKEN;
tokens.AccessTokenSecret = ACCESS_SECRET;

This is pretty straight forward.  An OAuth token has the CONSUMER_KEY and CONSUMER_SECRET to tell Twitter which app we are, as well as the ACCESS_TOKEN and ACCESS_SECRET to tell it which user we want to authenticate as.  Using this tokens object, you can now call any Twitter API you want!  Let’s call the TwitterStatus.Update method:

TwitterResponse tweetResponse = TwitterStatus.Update(tokens, "Yummy Cake Recipe :: http://www.kitchenpc.com/Recipes/YummyCake.html");

Console.WriteLine("Result: {0}", tweetResponse.Result.ToString());

All we do here is call the static Update() method of TwitterStatus, pass in our tokens, and pass in a string with our new status.  The resulting TwitterResponse object will contain information about whether the post was successful or contain any error information.  You can now go to your Twitter timeline and see the update immediately.  Easy enough!

In Summary

To create a TwitterBot, you first create a Twitter account and register a Twitter application.  You then allow that application access to your account, and store the access tokens within your TwitterBot’s configuration.  This is the exact technique the KitchenPC TwitterBot uses and the best way I’ve found to do the job.

More Cool Things

I wasn’t done there!  Since recipe names and permalinks can make for rather long Tweets (perhaps over the 140 character limit), I decided to use bit.ly to shorten the URLs.  Twitter will now automatically shorten URLs you post via their web interface, but URLs posted via the API did not get automatically shortened when I tried (though, perhaps they do if the message is too long?)  Either way, I thought it would be fun to tie into bit.ly’s API and have them automatically shortened.  This was pretty darned easy to do.

First, you’ll need a bit.ly account.  You can get one at http://bit.ly/a/sign_up and filling out the form.  Next, go to http://bit.ly/a/your_api_key and get your API key.  Store this key in your .config file as well!  I’ll refer to this key as BITLY_APIKEY.

Your can access bit.ly’s URL shortening services using simple REST commands.  There might be some nifty API wrapper somewhere, but I decided to just use the WebRequest class to interface with bit.ly directly.

string recipeLink = "http://www.kitchenpc.com/Recipes/YummyCake.html";
string url = String.Format("http://api.bit.ly/v3/shorten?login=kitchenpc&apiKey={0}&longUrl={1}&format=txt",
BITLY_APIKEY, HttpUtility.UrlEncode(recipeLink));

WebRequest webRequest = WebRequest.Create(url);
WebResponse webResponse = webRequest.GetResponse();
Stream responseStream = webResponse.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
string shortUrl = reader.ReadToEnd();

This code creates an HTTP GET request to api.bit.ly and reads the response into a string called shortUrl.  The URL contains a login (which is the username you use to login to bit.ly), your bit.ly API key, the long URL (safely escaped, or bit.ly will error out) you want to shorten, and the format you want to read the results in (you can also get the result in JSON or XML if you’d like.)

The response will simply contain something like http://bit.ly/x12345.

This code can be combined with the Twitter code to first shorten URLs so they nicely fit within a Twitter status update.  It’s possible that the Twitter APIs might provide this functionality using their own URL shortener, but if they do, this doesn’t appear to be exposed through Twitterizer.

So there you have it

My TwitterBot has been online for about 24 hours now and has been happily posting new recipes to the @KPCRecipes Twitter feed.  What started out as an excuse to learn something new has actually been met with some success.  I’ve done almost no advertising (just a quick post on @KitchenPC and the KitchenPC Facebook page) but already got a handful of followers, including even a couple restaurants!  Apparently, there’s people who love recipe feeds and it provides yet another way for users to interact with my site.  I’ll get traffic from people clicking on the links and through retweets, and it creates a new face to my site.  Another advantage of using bit.ly to shorten to URLs is bit.ly provides me some data and graphs about clicks to each of these URLs, so I can see how popular these tweets actually are.

Thanks for reading!