Having run my toy web service wishsys for a few months, I thought it would be nice to setup a continuous delivery/deployment pipeline for it, so that I could develop a new feature and push it into production as quickly as possible (if it passes all tests). I thought writing a small article about this would be nice as well, as I found few resources for doing CD with haskell.

The wishsys code is written in haskell and uses the yesod web framework. Update: I noticed a comment on reddit mentioning keter, which is essentially what I should have used instead of debian packages since I’m using yesod. This guide should still be relevant regarding CI though.

Choosing CI system

After doing a little research, I ended up with two alternatives to use for CI.

• Jenkins
• Travis

I have some experience with Jenkins at work, and since its very generic, I thought it would be easy to build a haskell project with it. Since the wishsys code is hosted at github, I was tempted to try travis, since it integrates well with github. I decided to go with Jenkins instead, mainly because I had a VPS to run it, and I didn’t need the great Heroku support, as I would be deploying it to the same VPS. There is a good introduction of CI for Haskell and Yesod at yesodweb.

Git and branches

Having chosen which CI-system to use, I needed to structure the git repository in a way that fits my workflow. Since I wanted to have some control over what changes that are pushed out to production, I created a stable branch in the repository, which is the branch from which the production binaries are compiled. All development goes into other branches. For CI to have any meaning, all of these branches should be built all the time so I can be sure that my tests are stable and that they pass. The plan was to create a build job for every branch that builds and tests that particular branch.

Setting up Jenkins

Setting up Jenkins is very easy on Debian, as you just need to add the jenkins debian repo and install packages. Having done that before, I just started on creating a new project for wishsys. Jenkins has a lot of plugins, and since this is a github project, I installed the github plugin for jenkins as well.

Having only the stable branch, I created one job called wishsys-build-stable. To build haskell, I added an entry under “Execute shell”, which runs the following commands when building:

$CABAL sandbox init
$CABAL --enable-tests install

$CABAL is parameterized to /var/lib/jenkins/.cabal/bin/cabal since I needed a newer cabal version to get the sandbox functionality. I also configured the job to trigger for every commit, though I will probably change it to build as often as possible to ensure there are no unstable tests.

Creating a debian package

The next step was to create debian packages of the software, so I could easily install it in my VPS (manually or automatically). Since this was unfamiliar territory, it took some time and reading to grasp the package layout, but I found an intro guide for creating debian packages that helped me. In addition, I added a makefile that invokes all of the cabal commands to build my project and to do what the debian package tools expect.

Creating debian packages in jenkins

I then started looking for a jenkins plugin to help me build the debian package, and found the following plugins:

• debian-package-builder
• jenkins-debian-glue

debian-package-builder seemed to only support automatic version manipulation for subversion repositories. Since wishsys uses git, I went for jenkins-debian-glue. Creating a debian package is complicated in itself, and I initially spent some time doing a lot of what jenkins-debian-glue tries to do automatically (automatically creating a tarball of the repo and running git-import-orig and git-buildpackage).

I used this guide to setup the jenkins jobs. I ended up with a wishsys-debian-source job for building the source package, and a wishsys-debian-binaries job for creating binary packages.

The jobs are run in the following order: wishsys-build-stable -> wishsys-debian-source -> wishsys-debian-binaries

The wishsys-build-stable job is run for each commit, and reuses the git checkout between builds to reduce build times. The wishsys-debian-source job simply creates a tarball of the git repo, and forwards the resulting tarball to the wishsys-debian-binaries job, which does a full clean build of the software before creating the binary package itself.

Summary

Setting up a CD pipeline is a great way to get your features tested and into production in minimal time. Though this setup is somewhat debian specific, the generic pattern should be reusable. In the future, I would like to avoid building the project twice (once in wishsys-build-stable, and once in wishsys-debian-binaries), but the build times are currently not an issue. Another improvement would be to get hunit and quickcheck test reports displayed in Jenkins.

The set of files necessary for debian build are available at github.

Today, I launched my wishsys service, which is just a simple service for creating wish lists with separate access for owners and guests. The original use case was my own wedding, so I created an even simpler version for that using snap. Snap worked great, but I had some hassle building the authentication mechanisms properly.

After the wedding, one of the guests wanted to use the same system for their wedding, so I thought I might as well create a more generic wish list service. This time, I went with yesod, as it seemed to provide more of a platform than a framework.

At Yahoo!, I work on a search platform, and there are a few things I expect from a platform. It should provide

• Storage
• Access control
• Higher level APIs for request handling
• Internalization
• Good APIs
• Good documentation
• Ease of deployment
• Test framework

Yesod did not let me down. It provides a real book, and not just a bunch of outdated wiki pages. Its solution for storage is excellent. Persistent allows me to write the definition of my data structures in a single place, and automatically generate a database schema and haskell types. I chose to use postgresql as my persistent backend, and by using the scaffold code, getting it working was trivial. Creating request handlers was so easy, I won’t even tell you how I did it.

My biggest yesod issue was authentication, since I had somewhat special requirements where I wanted to have two users with different access levels (admin and guest). I also missed a method in the authentication system to request a user to be logged in, regardless of what authentication backend used. I ended up looking at what HashDB did internally, and just copy that (If there is a better way, please let me know).

I used the hamlet template system to write HTML with minimal haskell clutter. Forms are a pleasure to work with, because I don’t have to repeat myself. I just had to create the form in one place, and I could then use it both for generating correct HTML and easily parse the POST request.

I just followed the deployment chapter when deploying, and then the service was suddenly live. Even more important to note is the development server, which automatically compiles the app if something changes. Great for local testing!

My biggest issue with yesod was understanding compilation errors messages. But, when I got things working, yesod was a great experience. It is one of the few open source projects I’ve seen that understands what it means to be a platform, and it thinks of your needs before you realize them. Kudos!

Btw, the wishsys source code can be found on github

I have tried using Haskell to various smaller projects, such as wishsys and a game that I never got really far into making. But learning a new programming language through the means of hobby projects only work as long as the project is contained and small. For my part, most hobby projects start out with great ideas and grand designs, but end up as a mess since I am unfamiliar with the programming language.

When using a new programming language, time is spent learning the language rather than developing the project. This in turn means that I end up learning the bare minimum to get the job done. And this defeats the purpose of using a project to learn a new language. If the goal is to finish the project, you should have used something you know well and feel most productive with. If the goals is to learn a programming language, you should start out with a small project instead.

For me, project euler is a great way to learn Haskell, because it contains a lot of problems that Haskell (and functional languages in general) is the perfect tool for solving. The projects I mentioned above involves using databases, multiple threads and other scary real world stuff, but I just wanted to learn Haskell. And better yet, once you have solved a problem, chances are you can find someone with an even more elegant solution written in the same programming language you are using. A great way to learn!

This time I thought I’d share our dinner plans for the next week. We take turns creating dinner list every week, and next week is my turn!

• Monday: Tortellini with sun dried tomatoes and mozzarella
• Tuesday: Fish with avocado, ruccola salad and hot mustard
• Wednesday: Fennel soup with chicken
• Thursday: Albondigas
• Friday: Salmon with fennelrisotto
• Saturday: Breaded cod with salad and potatoes
• Sunday: Home made tomato soup

Lets hope it tastes as good as I think it does.

I have started to learn myself haskell using the book named “Real world Haskell”. I have so far only come to chapter 4, but I am already in love with some of the features:

• Its strict static type system, which makes it easy to understand what a function does. Moreover, it allows you to think through what your code is going to do as well as make the decisions of what to do for special cases up front. The following is a definition of a function which compares the length of two lists, and returns their order (==, <, >). The definition clearly states that it operates on two lists of any type, and returns a value of type Ordering. Crystal clear!

    listCmp :: [a] -> [a] -> Ordering
  

• Partially due to the above point, one can avoid unpleasant bugs later on, because you chose to postpone your decision on what to do with your input.

• Pattern matching. I came across this in the Oz programming language when I was in university, but I didn’t really understand how powerful and readable everything becomes until using it in Haskell. The following function takes a separator and a list of lists as argument, and combines the lists using the separator:

    intersperse :: a -> [ [a] ] -> [a]
    intersperse sep [] = []
    intersperse sep (x:[]) = x
    intersperse sep (x:xs) = x ++ [sep] ++ (intersperse sep xs)
  

  I love how you can just look at the patterns to see what cases is covered by the function, rather than nesting into some complex if sentence.

• Readability when using ‘where’ syntax. This is the implementation of the listCmp function:

    listCmp lhs rhs
        | lengthLhs < lengthRhs = LT
        | lengthLhs > lengthRhs = GT
        | otherwise             = EQ
      where lengthLhs = (length lhs)
            lengthRhs = (length rhs)
  

  What I like about it is that you can separate the logic performed on values from the function calls, so that when you read the code, you see the actual computation done by the function in the different cases. You can also do this with the let syntax, but I think the above reads really well.