An informal explanation of stackage-sandbox

Works on my machine, will it work on yours?

Suppose there’s a Haskell project called stackage-cli that I’d like to share with you. It builds just fine on my machine, but will it build on yours? If we have different versions of installed Haskell packages, we might run into cabal hell trying to sort things out.

To start off, let’s first agree on using ghc-7.8.4 and cabal-install-1.20.0.3. (In this blog post, I’ll assume you are able to install these somehow. Next time I might have some more to say about how to get this far.)

Next, let’s agree to use the same package versions for the dependencies. Cabal provides a handy command to help with this: cabal freeze. This creates a file called cabal.config which lists package constraints. Here’s what I got when I did a cabal freeze on the project as built on my machine: http://lpaste.net/130815.

It would be rather tedious for you to blow away your entire package database, start fresh, and install these exact versions of dependencies. Thankfully, there’s a better way.

cabal sandbox

You don’t have to blow away your whole package database just to build this project I want to share with you. You can instead create a “cabal sandbox” just for this project. Then you can install the dependencies there.

I did a cabal freeze when I was on git commit 9b68a74, so let’s check out that particular version.

$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.8.4
$ cabal --version
cabal-install version 1.20.0.3
using version 1.20.0.0 of the Cabal library 
$ git clone git@github.com:fpco/stackage-cli
$ cd stackage-cli
$ git checkout 9b68a74
$ wget -O cabal.config http://lpaste.net/raw/130815
$ cabal sandbox init
$ cabal install --only-dependencies -j && cabal build

And now we wait for the dependencies to install. Maybe take a bathroom break, grab a coffee. This took about 8 minutes on my machine. Presumably, it worked, and we have successfully avoided a trip through cabal hell. Not bad, but we can do even better.

shared sandboxes

Local cabal projects can share local sandboxes. So, if you happen to be working on different projects that have the exact same dependency versions, you can safely share one sandbox between the various projects. If they have differing dependency versions, then you’re in cabal hell territory. You might get it to work. You might have some butterflies to battle. It’s probably easier to just have different sandboxes in that case.

Let’s check out the same project and rebuild it against the same sandbox.

$ git clone git@github.com:fpco/stackage-cli
$ cd stackage-cli
$ git checkout 9b68a74
$ cp ../cabal.config ./cabal.config
$ cabal sandbox init --sandbox ../.cabal-sandbox
Writing a default package environment file to
/home/dan/stackage-cli/stackage-cli/cabal.sandbox.config
Using an existing sandbox located at /home/dan/stackage-cli/.cabal-sandbox

See how it said “using an existing sandbox”, because we told it which sandbox to reuse?

$ cabal install --only-dependencies
Resolving dependencies...
All the requested packages are already installed:
Use --reinstall if you want to reinstall anyway.
$ cabal build

No need to reinstall the dependencies. They’re all there. The build itself took under 30 seconds on my machine.

The exact same dependencies? Really?

Suppose someone else wants your help developing a different project. It seems pretty unlikely that she and I would just happen to be using the exact same dependency versions.

Suppose you’re starting up your own project. How do you pick dependency versions that will work together? Just cabal install dependencies as they arise and cross your fingers?

Stackage

FP Complete has developed a project called Stackage.org that is meant to help with these concerns. The main service provided by Stackage.org that we will make use of in this blog post is the cabal.config files it provides. Remember how we got our cabal.config via cabal freeze, which allowed me to share with you the dependency versions that worked for me? Stackage provides cabal.config files that include dependency constraints for a very large subset of Hackage. These dependency versions are known to all work together. New builds are calculated nightly, but to provide a more stable point of reference, LTS Haskell is also available.

So here’s an idea: create a shared sandbox on your machine with only LTS Haskell packages inside. Let’s create a location for lts-2.3 and install things there. We’ll make sure to use the lts-2.3 cabal.config with our project, so that only lts-2.3 packages get installed into the shared sandbox.

$ git clone git@github.com:fpco/stackage-cli
$ cd stackage-cli
$ git checkout 9b68a74
$ mkdir -p ~/.stackage/sandboxes/ghc-7.8.4/lts-2.3/
$ wget -O cabal.config http://stackage.org/snapshot/lts-2.3/cabal.config
$ cabal sandbox init --sandbox ~/.stackage/sandboxes/ghc-7.8.4/lts-2.3/
$ cabal install --only-dependencies -j

Since this is the first time anything has been installed to that sandbox, it will take the same 8 minutes as before. But only the first time. Now, any of my other projects that can be developed against lts-2.3 can share those same dependencies! No need to reinstall the same dependencies over and over into fresh sandboxes for each project.

The stackage command line interface

Installation

The stackage-cli project provides a tool that automates some of these processes for you. Move out of these sandboxed project directories and do cabal install stackage-cli to install it to your usual place. If that fails, then use the sandbox tricks I’ve described above to install it into a sandbox:

$ mkdir sandboxed-dir
$ cd sandboxed-dir
$ wget -O cabal.config http://stackage.org/lts/cabal.config
$ cabal sandbox init
$ cabal update
$ cabal install stackage-cli

You can see where the binaries got installed with cabal exec, which lets you execute commands “in the sandbox context”:

$ cabal exec which stackage
/home/dan/sandboxed-dir/.cabal-sandbox/bin/stackage

Copy the various binaries named stackage and stackage-* onto your path, or add that bin directory to your path. Let’s review the functionality provided by the stackage command-line tool.

Managing cabal.config and a pre-configured sandbox

First off, is stackage purge, which deletes your current cabal.config and prompts you to unregister everything in your sandbox. If you don’t have a sandbox configured, it will instead prompt you to unregister from your global package database.

$ stackage purge
(Sandbox) /home/dan/.stackage/sandboxes/ghc-7.8.4/lts-2.3/x86_64-linux-ghc-7.8.4-packages.conf.d
Detected 221 packages to purge from this database
Unregister 221 packages (y/n)? [default: n] 

I don’t actually want to purge that shared sandbox, so I chose the default: no. As you can see, I’ve installed a lot of packages in there that I’d rather not purge. If you have a non-shared project sandbox, you may want to purge it prior to stackage init.

Indeed, next up is stackage init. This just downloads the appropriate cabal.conig for you.

$ stackage init lts-2.3

Leave off the lts-2.3 argument and it will just download the latest LTS. You can also use stackage upgrade which is simply purge followed by init.

These three commands are not particularly aware of shared sandboxes. You can use stackage sandbox to automatically set up and use LTS-based shared sandboxes.

Managing cabal.config, cabal.sandbox.config and shared sandboxes

$ stackage sandbox delete

This command deletes both your cabal.config and your cabal.sandbox.config. It doesn’t touch your actual sandbox unless you give it an argument. There are certian sandboxes that are managed by stackage sandbox, and it can help you delete them so you don’t have to go looking to see where they are.

$ stackage sandbox delete lts-2.3

I didn’t actually run that command, though, because I don’t actually want to delete my precious lts-2.3 sandbox, because it has 221 packages that I really don’t want to bother reinstalling until the next LTS release.

$ stackage sandbox init lts-2.3

This does a couple things.

  • It downloads the lts-2.3 cabal.config
  • It creates a folder for the lts-2.3 shared sandbox, if it doesn’t already exist
  • It calls cabal sandbox init with the --sandbox argument set to the lts-2.3 shared sandbox

In other words, it readies your project to be built with your shared lts-2.3 sandbox.

You can easily do a delete followed by an init by using:

$ stackage sandbox upgrade lts-2.3

This leaves your old sandbox intact, wherever it was, but replaces your cabal.config and cabal.sandbox.config with the appropriate configurations for the lts-2.3 shared sandbox (which it also sets up if necessary). stackage sandbox upgrade is the command I recommend using most of the time.

Build it again

Remember this example from before?

$ git clone git@github.com:fpco/stackage-cli
$ cd stackage-cli
$ git checkout 9b68a74
$ mkdir -p ~/.stackage/sandboxes/ghc-7.8.4/lts-2.3/
$ wget -O cabal.config http://stackage.org/snapshot/lts-2.3/cabal.config
$ cabal sandbox init --sandbox ~/.stackage/sandboxes/ghc-7.8.4/lts-2.3/
$ cabal install --only-dependencies -j

We can replace the wget and cabal sandbox init commands with stackage sandbox upgrade, and accomplish the same thing.

$ git clone git@github.com:fpco/stackage-cli
$ cd stackage-cli
$ git checkout 9b68a74
$ stackage sandbox upgrade lts-2.3
$ cabal install --only-dependencies -j

Again, since we’ve already installed this package’s deps there, there should be nothing new to install.

Summary

Let’s recap the main ideas behind stackage sandbox upgrade.

  • LTS Haskell provides a common platform of dependency versions for developers to develop against. It stays fresh enough to remain relevant, but stable enough to provide a solid point of reference.
  • Shared sandboxes allow you to develop your various projects against the same platform. No more reinstalling all-the-things for every single project.
  • stackage sandbox upgrade helps you to easily manage your shared sandboxes based on LTS Haskell

Upgrade your sandboxes to Stackage! Or if you want to customize the workflow, then mix and match whichever commands provided by stackage that you find convenient. You can even develop your own executable as a “stackage-cli plugin”. Any executable on your path with a name that starts with “stackage-” will be treated as a plugin, and you’ll notice that stackage-init, stackage-purge, stackage-upgrade, and stackage-sandbox are all simply plugins.

How is this different than Halcyon?

Halcyon is a project focused on installation of Haskell executables. In contrast, the stackage command line tool is focused on aiding development of Haskell projects.

Future work

We still haven’t gotten rid of all of the tedium in this process yet. For example, imagine that we both remotely log in to the same machine; we could share build artifacts! But that’s unrealistic. Instead, we can agree on using the same kernel or VM; that way we can share build artifacts between machines.

That’s roughly the idea behind docker, and FP Complete is working on docker-based build tools. It’s also roughly what Halcyon does, and we want to extend this benefit to development, not just installation.

Your feedback is more than welcome on the issue tracker.

https://github.com/fpco/stackage-cli/issues

Further reading

For a more nuanced example of stackage-cli, check out the project wiki on github:

https://github.com/fpco/stackage-cli/wiki/Example

Advertisements

About Dan Burton

I love functional programming and awesome type systems, which makes Haskell my obvious language of choice.
This entry was posted in Uncategorized. Bookmark the permalink.

One Response to An informal explanation of stackage-sandbox

  1. Pierre says:

    I have just entered the Haskell world, and you article is much appreciated. Coming from javascript/node.js, I was searching for a workflow similar to npm, and quickly fell deeply in cabal hell. You illustrate very well all the pitfalls, and the tools to circumvent them.

    Many thanks for the beginner that I am.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s