Archive for category builds

Does Erlang/OTP need a new package management solution?

Since mid 2011 I’ve been thinking on and off about this question. There are some package management solutions available for Erlang/OTP already, but none of them really seem to meet my needs. I had been considering writing a new solution from the ground up, but decided to take a pause and engage with members of the open source community first. I reasoned that it’s better to build something that benefits the whole community and supports a wide range of user experiences, rather than just hack something together for my own use. Since the turn of the year, I’ve had some very constructive conversations with the Erlware developers, as well as some recent discussions on erlang-questions about this topic, with Joe Armstrong contributing to the pool of ideas. This post looks at the origin of these conversations, some of the driving forces, and concludes with a review of the direction in which the Erlware developers and I think we ought to consider going in.

Read the rest of this entry »



rebar plugins – customising your build

This is the first post in a series on customising rebar using plugins. Initially, I want to focus on how plugins work, then move on to see what kind of extensibility they can provide to developers.


There are so many caveats to this, I hardly know where to start. Here’s a short-list to begin with:

  1. What’s written here is based on my experience and could be lacking important detail, or even plain wrong!
  2. Rebar’s support for plugins may disappear at any time in the future, as they’re not a fully documented or official feature AFAIK – though this probably isn’t likely in practise, it is possible
  3. Rebar has no official API for plugins, so their use is at best undocumented, and in all cases probably completely unsupported – you will find friends on the mailing list though, I’m sure

I’ve written quite a few plugins by now, and have even submitted a few patches relating to them. It’s fair to say that I’m quite opinionated about how plugins work and how they should work, but it’s also worth remembering that I’m only a very minor contributor and my opinions are just that – my private opinions.

First things first

Currently rebar supports two kinds of extensibility mechanism: Plugin Extensions and Hooks. Hooks are a lightly documented feature, with the only real explanation being the sample rebar config file. We’re not going to cover hooks in much detail, as they are simple enough to understand and are only really applicable to simple scripting tasks that don’t require, for example, cross platform support or complex logic. Plugin extensions on the other hand, are documented (to some extend anyway), and provide a much greater degree of extensibility to developers.

Before we can talk sensibly about plugins, we need to take a look at some of the fundamentals behind rebar, especially its handling of build configuration files and processing of commands. For any given command, say foo, rebar understands the command if (and only if) one of the modules it knows about exports a function with a signature that matches:

  1. the name of the command and arity 2
  2. the name of the command prefixed with pre_, and arity 2
  3. the name of the command prefixed with post_, and arity 2

We’ll be covering how rebar knows about modules later on, but for now we’ll just assume it’s magic. For the command foo to have any meaning then, we’d need at least one module with at least one of the following signatures exported:


pre_foo(_, _) ->
	io:format("we're running just before foo!~n").

foo(_, _) ->
	io:format("we're in foo!~n").

post_foo(_, _) ->
	io:format("we're running just after foo!~n").

Another essential is how rebar handles build configuration. There are four ways that rebar handles configuration settings. Firstly, rebar loads the global config file from $HOME/.rebar/config if it actually exists, otherwise it creates an empty config set. Secondly, rebar loads configurations for any directory by either (a) examining the terms in the local file rebar.config if it exists, or (b) creating an empty config set. Thirdly, when first executing in the current directory (known as base_dir), rebar will check for a special global variable (passed as config= or alternatively -C <config-name> instead)  which overrides the name of the config file it should search in. This latter technique is only applied to the configuration in the base_dir.

The fourth approach to configuration handling is not just for initialising new configurations. As rebar executes user commands (e.g., clean, compile, eunit) in a given directory, it uses two special commands to obtain a list of directories that need to be processed before the current one, and providing the current directory is processed without error, afterwards as well. These commands, preprocess and postprocess, can be exported by any module.

When rebar executes, it builds up a list of modules that understand the current command. For each of these modules it tries to call pre and postprocess, then it traverses any pre- directories before handling the current command in the current directory. Once all the pre-processing is done, each module that exports one of the three function signatures compatible with the current command is called (for one or more of the pre_<command>/2, <command>/2 and post_<command>/2 exports) to handle the actual command. The directories returned by any postprocess calls are handled last of all.

What is vital to understand about all of this, is that as rebar traverses the file system, recursively handling any pre- directories, in each new dir it executes with a brand new rebar config set. This config set inherits the parent configuration (i.e., the config set for the parent dir) but can override certain configuration variables by providing it’s own rebar.config file. This is how dependencies and applications stored in sub-directories are handled. The salient points about this mechanism are that

  1. the only configuration file rebar notices in sub-directories is the one named rebar.config
  2. any configuration override (passed with -C for example) is ignored in sub-directories
  3. just because a local rebar.config overrides a variable/setting, this might not be applied

Point #3 is a bit scary if you’re new to rebar, but essentially it is the result of rebar’s config handling module exporting multiple config handling functions, some of which get the local (i.e., the most locally scoped) value, some a list of possible values and others the combined list of all values. Depending on which of these functions a particular command/module uses when reading the configuration, you can potentially see a number of things happen:

  1. you might see the local value (from rebar.config) get applied
  2. you might see the value from the parent config get applied (e.g., if there is no local config)
  3. you might see the local value get ignored

I strongly recommend spending some time looking at rebar’s config module if you’re planning on writing plugins (or using complex plugins written by others), as it’ll save you a lot of head scratching time if you understand this up front.

What are plugins?

As far as rebar is concerned, plugins are simply Erlang modules that it knows something about. There are essentially two ways that rebar knows about modules:

  1. From the configuration file
  2. Via the plugins section of the build configuration

Modules which are registered in the configuration file are basically part of rebar itself. Plugins on the other hand, are modules which the build configuration (somewhere in the tree) knows about via the plugins configuration element. This configuration is built up to include every branch, including the global config, so just because you’ve got no local plugins configuration, doesn’t mean plugins won’t get run in your subdirectories. In practise, this means that plugins registered up top (e.g., globally or in the base_dir configuration) will get run in all your sub-directories, including of course dependencies. Bare this in mind when using plugins, and take advantage of skip_deps, apps= and skip_apps= where necessary to avoid unexpected things happening in your sub-dirs and deps folders.

To (hopefully) make the differences between plugin extensions and built-in modules a bit clearer, we’re going to classify plugin extensions into three groups, and will hereafter refer to them simply as plugins:

  1. Internal/Built-in
  2. External/Pre-packaged
  3. Local

Let’s look at what these classifications mean in practise, and hopefully get an understanding of the terminology I’ve chosen. Internal (or built-in) modules come bundled as part of rebar itself, and as per the documentation, these are registered in the rebar application config file. The functionality exposed by these modules is available to every rebar user, so they work Out Of The Box. These plugins are the least likely to be used for extending rebar however, because in practise they require you to either (a) maintain a custom fork of rebar or (b) submit a pull request in order for your extension(s) to be accepted as part of the main source tree. It is the other two types of plugin we will be looking at in this post.

External Plugins

Pre-packaged plugins are bundled as separate Erlang/OTP libraries to be installed globally, or included in a project using rebar’s dependency handling mechanism. The latter technique is more useful, as it ensures that someone who fetches your source code to build/install it, will be able to obtain the right plugins without going outside of the project source tree.

The key thing to understand here is that the plugin must be installed somehow in order for rebar to pick it up. We’ve mentioned that rebar knows about plugins because they’re in the {plugins, ListOfPlugins} configuration element, but in practise things aren’t quite that simple. In order for a plugin to actually get executed (in response to a specific command, it’s pre/post hooks or indeed the special preprocess and postprocess commands), it needs to be on the code path! This is fine if the plugin is installed globally into the user’s erl environment (for example by putting it’s application root directory somewhere on the ERL_LIBS environment variable), but not so fine if you’re fetching it into the project’s dependencies. If the dependency is a direct one, then the preprocess handler in rebar_deps will nicely update the code path for all commands, so as long as you’re not trying to make the plugin run before rebar’s built-in modules (which is, in fact, impossible) then it’ll be on the path. This once again doesn’t always work in practise however, because the function that builds up the code path makes no attempt to deal with transitive dependencies. I keep meaning to do a pull request for this, but I’m waiting for others to get through the queue first.

Local Plugins

You probably recall that I mentioned plugins need to be on the code path in order to be executed by rebar? Well thanks to a nifty pull request from yours truly, there is in fact another way. If rebar cannot find a module on the code path matching the name given to the plugins configuration element, it will attempt to locate a source file for the module in either the base_dir or the directory indicated by the plugin_dir config element. If it finds a source file with a matching name, it attempts to compile it on the fly and load the beam code into the running emulator, thereby making the plugin available dynamically.

The aim of local plugins is to provide a mechanism for scripting complex tasks/hooks that apply only to your specific project. This is in contrast with the idea of external/pre-packaged plugins, which provide add-on re-usable features to rebar that can be used across projects.

Next time…

Next time we’ll be looking at the structure of the plugin callback functions and how to use them in practise. We’ll also be taking a whirlwind tour of some of the commonly (re)used rebar modules such as rebar_config, rebar_utils and rebar_log, as well as discussing some of the pros and cons of using plugins and what the current workarounds look like. We’ll finish with a working example of an external plugin that adds new functionality to rebar with all the source code available on github.


Leave a comment

Post compile hooks with rebar

Sometimes you want your build to do some additional work after compiling. For example, in my Erlang implementation of hamcrest, I dynamically generate the main header file after compiling to ensure that all the functions exported from the hamcrest_matchers module are imported when the header gets included and to ensure that wrapper macros for eunit are also generated for each of them.

Typically you’d just have a makefile deal with this, maybe like so:

./rebar compile; escript header_gen

But that kind of breaks other people’s builds if they want to use rebar themselves. The problem is that rebar doesn’t yet provide a mechanism for overriding the build/compile command for dependencies (at all, let alone individually) and thus if your build requires a makefile to be run then your consumers build has to incur the same kind of complexity. Fortunately there is a work-around. Rebar provides a compile_post_script hook which you can specify in your rebar.config file. You specify a shell command and rebar will run this hook immediately after compilation. Unfortunately it also runs it when the compile command is applied to all your dependencies (again this is something which isn’t controllable yet) which typically will break because your shell command is not relevant to these apps. I spent a little time trying to do something clever with erl --eval '...' but in the end that didn’t play nicely at all. I then tried to encode the whole thing in a compound shell command like if [ -f hrlgen ]; then escript hrl; fi but then discovered that rebar_utils:sh/1 (which runs the shell command) wraps the supplied command text in an exec system call (which obviously isn’t going to work for a complex if..then expression). In the end, it was a comedic reptile that came to the rescue:

"env ERL_LIBS=deps python -c \"import os; os.path.isfile('hrlgen') and os.system('escript hrlgen') or None\""}.


Using epm to overcome rebar dependency handling limitations

I love rebar as a build tool, but occasionally it doesn’t do what I want. One example that keeps cropping up is managing dependencies that require complex commands in order to build them. If the dependency you want to manage is built using autotools or requires a series of commands (configure && make for example) then rebar is going to fail when you try and run ./rebar check-deps which is annoying. Read the rest of this entry »

Leave a comment

Getting GHC/Cabal-Install working on Snow Leopard

Cribbed from

* Open /Library/Frameworks/GHC.framework/Versions/Current/usr/bin/ghc-6.10.4 in a text editor
* Insert -optc-m32 -opta-m32 -optl-m32 just before the last parameter.

The last line in that file is rather long, and should now end like -dynload wrapped -optc-m32 -opta-m32 -optl-m32 ${1+”$@”}

Don’t be tempted to just put an edited local copy of the script in ~/bin/ghc, or edit all the various GHC files in /usr/bin. There is a maze of twisty symlinks that all eventually lead to the above script, and fixing just it is far simpler. Many packages need hsc2hs. To make it work correctly you need similar hackery:

* Open /usr/bin/hsc2hs
* Insert –cflag=”-m32″ –lflag=”-m32″ before $tflag

Leave a comment

fixing git-svn woes

At work we mostly use subversion for revision control, although some of the cooler kids are using git. I find subversion a bit of a pain compared to git, so I use the git-svn wrapper to get the best of both worlds whilst at work. Since upgrading to the latest Ubuntu however, I’ve noticed that a clash between gnutls and open-ssl causes git/subversion integration to fail. A fellow git-ite pointed out a quick fix however!

$ sudo rm /usr/lib/
$ sudo ln -s /usr/lib/ /usr/lib/

Works a treat. 🙂

Leave a comment

Launch Anywhere fails on Ubuntu 9

I ran into a problem installing (wait for it…..) Borland Together 2007 (gasp!) on Ubuntu this morning. The binary installation package failed to load with an interesting stack trace:

An internal LaunchAnywhere application error has occured and this application cannot proceed. (LAX)
Stack Trace:
java.lang.IllegalArgumentException: Malformed \uxxxx encoding.
at java.util.Properties.loadConvert(Unknown Source)
at java.util.Properties.load(Unknown Source)
at com.zerog.lax.LAX.<init>(DashoA8113)
at com.zerog.lax.LAX.main(DashoA8113)

After some googling around, I discovered that Launch Anywhere often screws around with PS1, so changing this for the current shell resolves the issue:

$ export PS1=">"

And after that everything works fine (apart from having to use Together 2007 in the first place).