Consider a typical small software project - a few programmers working together on a single project for a prolonged period of time. Requests for new features are coming in from a customer every few weeks. Some of the requests are probably similar in some ways, so people will want to build reusable modules for the most common tasks. Immediately, perhaps the single biggest problem of software development comes into play - communication.

As a typical software project grows, an implicit “design” emerges from every programmer building tiny APIs only with his/her own use in mind, giving raise to problems:

• Many different APIs will get created for doing the same things. If you do not explicitly agree to have a library for a particular common task, everybody will create their own little “libraries”.
• There will be many components sharing parts of the functionality, but each having its own code for achieving it. This code could instead form a shared API and decrease the overall amount of work it would take to build those components. If you don’t explicitly try to design APIs together with normal project development, you will miss many occasions for code reuse.

The longer the project is developed, the worse the effects of the above are. In the end, the system takes more time to build then necessary, maintaining and extending it is harder and the overall design is much more complex, as various aspects of the functionality aren’t separated clearly enough.

Why is it this way? I think there are three keywords that lead to the answer:

1. Perspective - without conscious effort, one tends to have the “just get the task done” perspective on software. Programmers don’t ordinarily think of the code they write as of something that is meant for other people to use directly, but as of an mean to make the computer do a particular thing - it is both.
2. Communication - very often team members simply don’t know each others work, do not agree on common practices or on places to look for documentation and don’t build modules to be used by everyone, even for the most common little tasks.
3. Technical skills - building modules other people will want do use requires particular skills that can only be acquired by continuous practice and proper training.

I think that the true indicator of a successful programming team, is people using each others code without anyone explicitly urging anyone else to do so. This can be achieved only be everyone having the necessary skills individually and by having appropriate procedures and conventions in place for the whole team. What I want to encourage in this article, is a shift of perspective that will emphasize communication and a set of best practices to help one acquire the technical skills needed to build clean software as a member of a larger team.

Try the following: the next time you are done with a piece of work you have done in your usual way, try to take a look at it as if you were designing an API for other programmers to do similar things to the one you just implemented. Try to really get into skin of another person trying to modify your code or use it for his/her own purposes. Is it clear what the code does? Are the parts well separated? I put together a set of specific guidelines to help one design programs with this API perspective in mind:

1. Look for abstractions. This is the key point, with each given task, one has to decide if it is a specific instance of some more general problem, how much additional effort does solving the general problem take (sometimes the general one is simpler to solve than the specific one!) and if it is worth to design an API for tasks of this kind. It takes some experience, which comes from a lot of experimentation, to recognize the situations where generalization is worth it. Sometimes the whole “API” you may extract from the task at hand is a single method to format the date in a consistent way across the whole project.
2. Split things into pieces. It is surprising how underused are the simplest ways of managing complexity, like splitting modules into separate files, splitting methods into smaller ones, distributing code across classes and so on. To avoid every team member “reinventing the wheel”, the modules for common tasks have to be easy to find by just browsing the source tree. If you have your date function in a “date.c” file it is much easier to find it, than if it is in an “utils.c” file. Think of it - if you don’t see an obvious place where to look for a method you need, you will most probably assume it doesn’t exist at all and proceed with writing your own one. Every project has some files or directories that are just there to keep things that people do not know where to put, with names like “utils”, “misc” etc. A good first exercise in API design would be to take such a file and split it into well-named modules, preferably until there is no need for “utils” at all anymore.
3. Use design methods other then just direct experimentation with code. You don’t have the right perspective to do API-level design when looking at your individual classes or functions. I like text-based approaches - a short spec of the public interface of the code you added, consisting of a list of classes/methods with short descriptions might already give you some new insights. Pseudocode can be very helpful when doing lower-level design. You may try UML diagrams, informal pictures, mind-maps or anything else you think may work.
4. Document and communicate your APIs. You have to agree with other team members on a place where all of you will list and document the APIs you create - perhaps a Wiki page. Let each of the members put descriptions of their modules there. Over time, see if they use your modules or not, if they have any problems with them and so on. This is invaluable feedback on what is good about your code and what isn’t. Every week let one person (alternating between all team members) review all the code added during this week and write an e-mail summarizing any issues found.
5. Keep consistent. Remember you are programming things for other people. They did not build the API you provided them, so they will certainly make guesses about how it works. To make it easier for them to comprehend your API, you have to strive for maximum consistency to make their guesses right as often as possible. This is in the spirit of “The principle of least surprise”.
6. Clearly state contracts and signal their violations. In a similar way to 4., the pre- and post- conditions that were obvious to you during writing the code, may not be obvious to other people, so you have to state them explicitly, to make sure that your users immediately know when they are misusing your API. Don’t be afraid to define and use a few custom exception classes for every larger general module you build, it will pay-off very quickly.
7. Write tests. A module will not really be general and reusable unless it is really well tested. It might be that the API you created works well for your particular problem, but will it also work correctly for a similar one? You have to make sure that what the API promises to do for the user really holds true.
8. Provide extensions points. If you work on a single project for a longer time, you might sometimes see in advance the ways in which you or other people will want to extend something in the future. You might also have some intuition for what pieces of the program are most likely to change. It is possible to take some steps ahead of time and provide explicit extension points, for example allow the API user to define callbacks that will be used at particular moments of your API operation. In fact, if you have striven for a clear design for some functionality, it might well be that some natural extension points are already in place and if you would extend your code yourself you would use them. The point is to make them explicit and documented, so other people who do not know the merits of your design, but have to extend upon your code, know they exist.
9. Avoid over-generalization. All of the above points can be misunderstood or misapplied. There are situations where generalization does not make a system cleaner, in fact it does the opposite. Unfortunately, I do not see a clear rule here, this is again a place where you acquire the necessary intuition by experience and experimentation.

Further resources

If you listen carefully to experienced programmers, you might find traces of what I have written in what they say. The closest to saying it directly I think is Joshua Bloch in his presentation How to Design a Good API. There also two great books showcasing how to write really reusable code: Paradigms Of Artificial Intelligence Programming and C: Interfaces and implementations.

What I would consider a part of Ruby’s normal programming features are the things programmers who ship end-user applications use everyday - classes, objects, methods, loops, variables etc. The metaprogramming features are what is mostly used by people who build applications for other programmers, but normally not by the “average Joe”, so the techniques behind things like Rails, RSpec, Rake, Rack and other major Ruby frameworks and libraries. Much of the appeal of those projects lies in the elegance of the languages they make available to their users, making it easy to describe solutions to problems the given framework is targeting. The specific features I have in mind are things like metaclasses, instance_eval, class_eval, method_missing, alias_method an so on, all hidden behind an easy to use abstraction layer.

The big success of Ruby was that it enabled things like Rails to happen at all. They were languages before that could make frameworks of this kind possible, like Common Lisp or Smalltalk, but they never became mainstream and nothing similar in popularity and impact to Rails never was built around them. Ruby had some of the same capabilities of Lisp or Smalltalk, while being more accessible to the average Joe (or to the not-so-average David). It is interesting to consider how much syntax Ruby has - I am not strongly conviced about this myself, but after reading parts of Chomsky’s “On Language” it occurred to me that maybe the more syntax a programming language has, the better it maps to the parts of our brains shaped by the syntax-rich natural languages. This would perhaps explain why Lisp or Smalltalk with their minimal syntax didn’t gain that much popularity despite their huge advantages. Instead they ended up being used mostly by people that have their brains already trained at thinking about things in different ways then just with natural language, for example people well versed in mathematics or physics, or specifically trained in programming with them for a prolonged period of time.

The problem I see is that the superiority of Ruby in the domain of creating programming frameworks or DSLs happened to a great extent accidentally. Ruby was not designed with things like Rails in mind and it is clearly visible now. The metaprogramming features are very specific and very chaotic, they do not form a coherent set of generic tools, the way the “programming part” of Ruby is a coherent set of generic tools for describing computations. People building advanced libraries like Rails have to use those features in ways they were not meant for.

You can see my metaprogramming presentation for advanced examples of various Ruby oddities in this area, one simple example is the situation where you want to augument an existing method with some code around it. The way you do it in Ruby is by aliasing methods - the language feature you use has no relation to your intent. This is how it looks like:

def wrapped_method
  puts "2 + 2 = 4"
end

puts "Before augumentation"
wrapped_method

puts

alias :old_wrapped_method :wrapped_method

def wrapped_method
  puts "1 + 1 = 2"
  old_wrapped_method
  puts "3 + 3 = 6"
end

puts "After augumentation"
wrapped_method

The lack of an adequate programming “metalanguage” is what I consider the biggest flaw of Ruby, but also the most promising direction of programming languages development. It is good to look at some other languages and compare them in this respect. For example JavaScript can achieve some of the same metaprogramming capabilities of Ruby with much less language constructs - objects as hash-maps, first order functions, the ability to changes the context of a function etc. are all aplicable to a wide variety of problems without having the narrow nature of the Ruby techniques used to achieve the same effects. Alan Kay has some very interesting things to say about this, so if you are interested in those topics, you may start with listening what he has to say.

The failure to deliver browser-based applications in those domains is in my opinion caused by:

• the weakness of the request-response scheme of HTTP as a basis for a programming model - it works well for the operations on hypertext documents it was designed for, but it is a poor basis in almost all domains that do not naturally deal with documents, especially those requiring sending excessive amounts of information back and forth
• poor capabilities of the browser itself - what is being made available to the programmer is insufficient for any kind of more sophisticated development, as no access at all is given to important parts of the user environment - neither to the hardware (graphic cards, sound cards, video cameras, gaming devices) nor to the software resources (files on the client computer)

Many technologies were created to deal with the two problems. AJAX, continuations-based web servers or Comet were attempts to workaround the first issue, but they all have been built on top of the flawed model, while also suffering problems with the web browsers, without trying to directly fix them. The second problem was meant to be solved by the likes of Flash or Silverlight, which do not integrate well with the rest of the web (for example search engines), making it questionable why do they at all run inside the web browser, experiencing efficiency and stability issues. Java the platform was the closest to what I want to propose, but even through it had the necessary pieces, it wasn’t clearly steered in this direction and suffered from slow start-up times, slow and ugly GUI tool-kits and weaknesses of Java the programming language.

The author is seeing much potential in an integrated attempt to solve both of those problems with a new set of standardized technologies. We already have excellent protocols for hypertext, efficient file transfer or sending messages, why shouldn’t a protocol for sharing applications exist? Links could be put to apps://musicplayer.com/ on traditional web pages in the same way as links to e-mail addresses or file servers, leaving the browsers with what they do best - processing documents.

The proposed solution that will be described here is called OAPP, standing for Open Application Publishing Platform. The following things should be specified as a part of OAPP:

1. A virtual machine of predefined capabilities running on the client computer
2. A way of sending executable computer programs to the virtual machine from the server
3. A way of singalizing events on the virtual machine to the server
4. A programming model based on points 1, 2 and 3

Having each aplication run in a separate virtual machine creates a common denominator for all the client platforms, while giving the possibility to privilege access to the sensitive parts of the clients environment. The virtual machine should not only deliver means of executing programs, but also give abstractions (libraries) over users hardware and software resources. In fact, applications could require additional libraries that would also be delivered on demand to the client from different OAPP servers. This forms a rich and fully programmable platform developers can target. This virtual machine, together with the libraries and an interface for choosing from the available programs, is in OAPP terms an application browser.

For a practical demonstration of the benefits of such approach, the V8 virtual machine was chosen, both for the server and for the client. JavaScript is already well-known by web developers, while being a powerful programming language, in fact turning out to be surprisingly well suited for the problem. The quality of the virtual machine may decide about the success of the project and V8 with its beautiful design, very quick start-up time and some decent libraries already available seems to be a good candidate. The package Node.js which provides a modified version of V8 will be used, as it contains networking support - to run one of the examples, just use the ‘node’ executable with the name of the source file as an argument. Links to the source files can be found at the end of the article.

The OAPP communication protocol consists of packets in the form of JSON literals. For simplification of the examples packets are separated simply by three consecutive newline characters, while in practice a thin transport layer should be implemented, with the ability to store different applications under the same host name and to compress and encrypt the information being exchanged.

After a connection is established, the OAPP server sends the first packet with the initial code to the client. For example, a server delivering an application showing the start-up time looks like this:

var sys = require('sys');
var tcp = require('tcp');

function remoteClosure(fun, variables) {
  return "(" + JSON.stringify({
    proc: "(" + fun.toString() + ")",
    data: variables
  }) + ")\r\n\r\n\r\n";
}

function Application() {
  var sys = require('sys');
  sys.puts("Server was started at: " + this.date);
};

var date = new Date();

var server = tcp.createServer(function (socket) {
  socket.addListener("connect", function (conn) {
    socket.send(remoteClosure(Application, {date: date.toString()}));
  });

  socket.addListener("eof", function () {
    socket.close();
  });
});

server.listen(7000, "localhost");

The Application function is the code that will initially be sent to the client, which in this example is rather trivial. The JSON object transferred is built from two fields:

• proc stores the text of the Application function, retrieved using the toString() method of the function object
• data is any data dynamically computed on the server, in this particular case it is the date the server saved at startup

The extra parenthesis are just syntactic sugar for easier evaluation on the client side. This is how the sent JSON actually looks like:

({
  "proc": "(function ClientApp() {
              var sys = require('sys');
              sys.puts("Server started at: " + this.date);
            })",
  "data": {
    "date": "Sat Nov 21 2009 19:00:18 GMT+0100 (CET)"
  }
})

The client role is very simple - it is supposed to fetch consecutive packets and evaluate the code that is represented by them. When a valid packet arrives, the client parses the received JSON literal. By using eval on the proc field, it gets an identical Application function to the one that was defined in the code of the server, on the client side. It executes the function by using the apply method, making all the usages of this inside the function code refer to what is in the JSON literal under the data key. In this way, the function is executed in a dynamic context specified by the server. Otherwise, it would not be possible to have the entire source code of the application in one source file in normal form. I call it a remote closure, because I look at it as an analogy to the closure concept you may know from programming languages. That’s how the code for the client looks like:

var tcp = require('tcp');

var jsonBuffer = "";
var connection = tcp.createConnection(7000, "localhost");

connection.addListener("receive", function(data) {
  jsonBuffer += data;
  if(jsonBuffer.indexOf("\r\n\r\n\r\n") > -1) {
    var remoteClosure = JSON.parse(jsonBuffer);
    remoteClosure.proc = eval(remoteClosure.proc);
    remoteClosure.proc.apply(remoteClosure.data);
    connection.close();
  }
});

For the communication to be complete, a way of signalizing the events on the clients virtual machine to the server has to be established. For security reasons, it seems that it is better to use simple data objects in this direction, instead of the code evaluation scheme just presented. Each event sent will also be a JSON packet.

To clarify the code of the next example, a library was built around the concepts discussed so far:

var tcp = require('tcp');

function packet(object) {
  return "(" + JSON.stringify(object) + ")\r\n\r\n\r\n";
}

function packetBuffer() {
  var buffer = "";

  return {
    append: function(data) {
      buffer += data;
    },
    packetReady: function() {
      return buffer.indexOf("\r\n\r\n\r\n") > - 1;
    },
    fetchPacket: function () {
      var packetEnd = buffer.indexOf("\r\n\r\n\r\n");
      var packet = buffer.substring(0, packetEnd);
      buffer = buffer.substring(packetEnd + 1, buffer.length);
      return JSON.parse(packet);
    }
  };
}

function event(name, data) {
  env.connection.send(
    packet({
      name: name,
      data: data
    }));
}

function closure(fun, variables) {
  return packet({
    proc: "(" + fun.toString() + ")",
    data: variables
  });
}

function publish(application, host, port) {
  var handlers = {};
  var buffer   = packetBuffer();

  var server = tcp.createServer(function (socket) {
    socket.addListener("connect", function (conn) {
      socket.send(closure(application, {}));
    });

    socket.addListener("receive", function(data) {
      buffer.append(data);

      while(buffer.packetReady()) {
        var event   = buffer.fetchPacket();
        var handler = handlers[event.name];
        socket.send(handler(event.data));
      }
    });

    socket.addListener("eof", socket.close);
  });

  server.listen(port, host);

  return {
    addListener: function(eventName, listener) {
      handlers[eventName] = listener;
    }
  };
};

exports.packetBuffer = packetBuffer;
exports.packet       = packet;
exports.event        = event;
exports.closure      = closure;
exports.publish      = publish;

The library introduces two major new elements:

• an event function to send event objects from the client to the server
• a set of event handlers for the application server, each reacting to a particular kind of event

To illustrate how the full communication scheme works in practice, a slightly more complex program should be considered: a calculator. It is the simplest possible example of the general two-way communication scheme that is the basis for all OAPP applications:

var sys  = require('sys');
var tcp  = require('tcp');
var oapp = require('./oapp');

function Calculator() {
  function parse(input) {
    var position = 0;

    function peek()    { return input[position]; }
    function consume() { return input[(position += 1) - 1];  }

    function token() {
      var token = consume(), c;
      if(token < '0' || token > '9')
        return token;

      while((c = peek()) && c >= '0' && c <= '9')
        token += consume();
      return parseInt(token);
    }

    function binary(left, right, ops) {
      var token, a = left();
      if((token = peek()) && (ops.indexOf(token) == -1))
        return a;
      var op = consume(), b = right();
      return [op, a, b];
    }

    function term() { return binary(token, term, ['*', '/']); }
    function expr() { return binary(term, expr, ['+', '-']); }

    return expr();
  }

  process.stdio.open();
  process.stdio.addListener("data", function(input) {
    oapp.event("expressionEntered", { expr: parse(input) });
  });
};

var app = oapp.publish(Calculator, "localhost", 7000);

app.addListener("expressionEntered", function(event) {
  function compute(expr) {
    if(!(expr instanceof Array))
      return expr;

    var op = expr[0];
    var a  = compute(expr[1]);
    var b  = compute(expr[2]);

    switch(op) {
      case '+': return a + b; case '-': return a - b;
      case '*': return a * b; case '/': return a / b;
      default:  return null;
    }
  }

  return oapp.closure(function() {
    sys.puts("Result is: " + this.result);
  }, { result: compute(event.expr) });
});

The following scenario is executed when a client connects to the server:

• The Calculator application is sent to the client and installed. In JavaScript terms the application is a function, and installation is just function evaluation.
• The client application waits for a line of input from the user
• The input is parsed, forming a tree from the infix expression the user entered, for example parsing 9*8+2*3 returns ['+', ['*', 9, 8], ['*', 2, 3]]
• The resulting expression tree is sent to the server with the expressionEntered event
• The server computes the answer and sends back a closure displaying the result to the client

As one can see, the abstractions introduced make the application code look almost indistinguishable from an event-driven client side program, for example one built with a toolkit like Swing. For a library that is not even 2 kilobytes large, this is not a bad result and a big compliment for the techniques and technologies used.

The slightly extended client code looks like this:

var sys  = require('sys');
var tcp  = require('tcp');
var oapp = require('./oapp');

var buffer = oapp.packetBuffer();
var global = this;

env = {};
env.connection = tcp.createConnection(7000, "localhost");

env.connection.addListener("receive", function(data) {
  buffer.append(data);

  while(buffer.packetReady()) {
    var remoteClosure = buffer.fetchPacket();
    remoteClosure.proc = eval(remoteClosure.proc);
    remoteClosure.proc.apply(remoteClosure.data);
  }
});

The two examples considered were of course trivial, meant to illustrate the techniques used for developing OAPP programs. To check if the idea for the platform is at all bearable and sensible, one should consider how more complex kinds of applications would look and work like, especially if the virtual machine would be extended with additional capabilities, for example with a graphics library. This will perhaps be a topic of another article in the future.

I think that the set of ideas presented, if complemented with a much more mature implementation, would offer an interesting alternative to the way we develop web applications today. A real-life project based on this would of course have to deal with the topics of latencies, security or ease of debugging and be tested much more extensively. I leave the evaluation of the usefulness of such a project to the reader, at the very least I hope you have just seen an interesting usage of the JavaScript programming language.

Code of the first example (server start-up time):

server.js
client.js

Code of the second example (calculator):

oapp.js
server.js
client.js

Github repository

Proel makes one fundamental assumption - you have a set of dedicated directories for your software projects (ie. ~/code/work or ~/code/own), and each project has a single root directory in one of the projects directory, which name is also the name of the project (ie. ~/code/work/boring-app or ~/code/own/fun-app).

If you’re configuration is meeting this assumption, proel enables you to effectively open one of the projects and than to find files in the project using fuzzy matching (like in TextMate) ie. you can type SemaAn.g to open SemanticAnalysis.g. You have to have the wonderful “anything” package for that (see also here), and “fuzzy-match”, the library doing the string matching.

Proel also arguments the find-file function, so whenever you open a file “manually”, the project gets recognized. If the project directory contains the “TAGS” or “cscope.out” files, proel will also automatically visit the tags table or load the cscope database. You can also use the ‘proel-vc-in-project’ command to open up the Emacs version control status interface in projects directory (SVN and git are currently supported, but you can easily add your preffered VC) or ‘proel-grep-in-project’ to do a recursive grep somewhere in the project directory.

This is now one of the most producitivity-enhancing packages I have installed (together with anything, psvn, git.el, ediff and some other stuff I rolled on my own I will relase one day hopefully). The only thing I miss is being able to locate symbols/definitions using
anything and fuzzy matching the way locating files work, but I hope I will be able to work out that one too. Also, for other people, another way of recognizing projects could be desirable (ie. by detecting the topmost parent directory of the current file has a “Makefile” or “build.xml” like eproject).

A picture is worth a thousand words, so here you can see how it works in practice:

(Download the original, sharp video in Theora here)

And here’s a sample .emacs I use in the screencast:

(add-to-list 'load-path "/home/sztywny/.elisp")
(add-to-list 'load-path "/home/sztywny/.elisp/git")

(require 'anything)
(require 'git)
(require 'proel)
(require 'xcscope)

(tool-bar-mode nil)
(scroll-bar-mode nil)

(setq cscope-do-not-update-database t
      grep-find-template "find .  -type f  -print0 | xargs -0 -e grep  -nH -e "
      anything-sources
        '(proel-anything-projects
          proel-anything-current-project-files))

You will probably want to customize other things too, just look through the source code to get an idea what variable controls what
aspect of proels behaviour. I hope you will like it :)

proel.el

Despite the quirk with my own presentation, I really enjoyed RuPy as a participiant and I think it’s a great event. Zed Shaw had a really nice presentation, with some funny bits, but also with many things you can pick up and use weeks after the conference ended. I also liked the talk about PyPy by Carl Friedrich Bolz, even through I’m a Ruby guy - they really have some interesting ideas about compiling/interpreting dynamic languages and they made a hell of the effort to push it forward. I hope I will be able to visit Poznań again the next year and I encourage you to do the same too.

I wrote about the great fun factor of programming in Erlang before, but I didn’t quite get the reason this is the case. Now, after thinking about for a while, I think I know what Erlang does so well, that it makes programming a joy.

You see, Erlang creators criticize OOP eagerly, distancing from it as much as possible, which is a bit misleading. I believe Erlang is the quintessentional cross between object orientation and functional programming. Even through Erlang doesn’t have implict support for “objects” of any kind, it does have processes. Contrary to what the Erlang crowd says, they seem to share all the common properties of object/classes in traditional object oriented languages - they capture state, express behaviour and you can even (sort of) inherit a process from another process or compose them, if you think this is necessary to call something object-oriented, and/or if you think this has use in Erlang. You can also build reusable components with them, as you can see for example in the OTP behaviours.

This is a simple observation, but it has some strong consequences - this programming model seems to has some unique advantages. I observed that while using traditional OOP, I tend to think much more about classes, one piece at a time, thinking in terms of fields/methods. Erlang shifts the focus from classes/methods/state towards interactions, or messages crossing, between objects/processes in time. Also, you think about the whole thing at once, about all the components involved and cooperating in solving a subproblem/problem of your program. To put it in another way, you don’t think only about the internal construction of the entities involved, but you also think about how they’re playing together to solve the problem. It feels like the concurrent language was promoting more “concurrent”, cooperative, big-picture thinking. Than, when you get the picture, you can compose the the bigger things (processes) from little, stateless functions doing the things you want.

As a result of this paradigm change, it seems that it’s easier to write more modular and coherent applications. It doesn’t even have anything to do with so-called “concurrent” programs, think of it for a while. Aren’t objects in your traditional language exchanging messages all the time? Aren’t they independent “beings” exisiting concurrently in some “space”? That’s where the title reference to Smalltalk came from. Erlang only removes the limitation that only two object can talk to each other at a time (when your traditional program isn’t threaded, and I’m talking concept-wise, not implementation-wise). At the same time, the functional nature forces you to use as small amounts of state as possible, which in consequence rewards writting very small modules.

Obviously, they are also strong differencies ie. you have many things in Erlang that aren’t at all realized using processes, whereas in “true” OO languages everything is an object. Processes aren’t also used as a data type, as you have records and tuples for this. Still, through, the essential modelling is done using processes. I believe that what Erlang does with them is the best way you currently can cross a variant of object orientation with functional programming, even if concurrency isn’t a priority. It certainly seems much more reasonable than trying to fit the traditional model with classes to the functional world.

Joe Armstrong certainly knows all this, and talks about it between the lines sometimes, but I didn’t seen it discussed in depth anywhere. For some more insight into the language, you may want to see this great talk with Joe: part1, part2, and also this article on oop.

$table({cellpadding: 10},
  $tr(
    $td(
      $div({klass: 'container', id: 'main'},
        $h1('Foobar'),
        $h2('Foobar2'),
        $h3('Foobar3'),
        $h4('Foobar4'),
        $h5('Foobar5'),
        $div({id: 'layer', style: 'border: solid 3px #f00; padding: 10px;'},
          $p({klass: 'foo'},
            'foo', $strong('bar')),
          $p({klass: 'shmoo'},
            'shmoo'))))));

And you get this:

<table cellpadding="30">
  <tr>
    <td>
      <div class="container" id="main">
        <h1>Foobar</h1>
        <h2>Foobar2</h2>
        <h3>Foobar3</h3>
        <h4>Foobar4</h4>
        <h5>Foobar5</h5>
        <div id="layer" style="border: 3px solid #f00; padding: 10px;">
          <p class="foo">foo<strong>bar</strong></p>
          <p class="shmoo">shmoo</p>
        </div>
      </div>
    </td>
  </tr>
</table>

I generated the necessary data about tags / possible properties from HTML 4.01 W3C specification, so all the tags should work and invalid properties should be ignored. “Should”, because the generating was a little bit tricky and I’m not 100% sure if everything is the way it should be. The dollar sign is a bit annoying, too, but my goal was to avoid cluttering of the global namespace, especially a variable name like “i” may be quite popular… I don’t know how it works performance-wise or memory-wise either (certainly some optimizations are possible), but it seems to work under Firefox, Opera, Konqueror (Safari?) and MSIE 6/7 and it even doesn’t crash any of them in a short time, what I consider a big success. I don’t really know if this is useful at all (most people don’t seem to like balancing parenthesis or don’t know a decent editor which highlight pairs of matching parenthesis, judging by Lisp popularity), but you can grab it here. To start using it simply include the js file using <script>, nothing more. If you somehow find it fun, just leave a comment. Afterwards, feel free to do whatever you want with the code.

One more tip: keep in mind that jsTemple just returns a string, so you can for example use it with Prototype’s Template class.

Now, 900mb is a lot. A few years ago I had a pretty complete and useful Slackware setup on my old computer weighing 900mb. I choosed packages by hand, I used some advice from the Saving Space HOWTO, made ruthless decisions about what software I need and what not. Now, I have 12gb filled up on the partition with Gentoo. I wonder how much of it is garbage. I decided I don’t want it this way anymore and I don’t want to spend that much time on keeping up to date with the software, upgrading packages, fiddling with this stuff etc.

Anyway, the idea is to keep the set of the software you use as minimal and as portable as it gets - in guerilla you never know where you will wake up the next day if you wake up at all, and software certainly is one hell of a battlefield those days. The first step for me was kicking out XFCE and installing dwm, the tiling window manager I told you about in the beginning. Actually I installed a lot of tiling window managers, but this is the one I sticked with for now, as it’s so small it’s easy to learn and start using.

Dwm & dmenu are plain cool. Their compressed sources take up 27,121 bytes, yet they manage to make a better job managing windows and streamlining work than anything else I used. What makes dwm cool? Short version: tags and tiling. Basically, you define a set of tags for the things you’re doing with you computer, or for the programs you use, or whatever you will come up with. I currently have 8 major tags: “firefox”, “emacs”, “terminal”, “docs”, “chat”, “music”, “p2p” and “graphics”. Dwm lets you define which window should be tagged which tag. Now when you start it, you have a list of tags in the top bar and you can switch to a particular tag by pressing the shortcut of choice, Alt-1, Alt-2 etc. by default. You can also toggle a tag, for example to display both all the windows tagged “docs” and all the windows tagged “music”. When you have more than one window with the same tag, the display gets split, so the windows never overlap. One of the advantages of dwm is that it doesn’t have problems with floating windows - you can toggle a window between being tiled and floating, use whole workspaces containing only floating windows and finally you can resize/move windows, which isn’t always the case in titling WMs. The reason I finally took tiling window managers for a spin is that I recently found myself using a workspace consiting of fullscreen emacs on first desktop, fullscreen firefox on second desktop, maybe a maximized terminal running screen on the thrid one and everything else (being pretty much unimportant) floating around all the other workspaces. Using dwm was like capturing the pattern of usage into software and making it automatical.

A tiling window manager is like the ultimate answer to “distraction free” computing. If you look at the buzz from the MacOsX crowd about apps like WriteRoom, Backdrop, SpiritedAway etc. you can understand how a tiling wm can make sense. It just doesn’t have a cool logo and a good marketing campaign, but it realizes the same goal quite nicely.

Dmenu is 7,912 bytes and is a hell of a good application. The idea is dead simple, you give a list of options to the standard input, dmenu shows a bar on the top of the screen letting you choose an option using tab completion and substring matching and return the choosed option to the standard output. When someone says it’s like Quicksilver, you may think he’s nuts, as I first did. But hell, yes, it’s like Quicksilver made the unix way. It’s a simple tool, which power lies in the combination with other tools. The first thing I did with it and already used a hundred of times or so is selecting music to play. I bound Mod+Shift+m to the following command:

cd ~ && exec audacious "$((find downloads/ -name '*.mp3' && find music/) | dmenu)"

What id does is that it feeds dmenu with a list of loose mp3 in ~/downloads and all the music files and directories in ~/music. I can press the shortcut, type in the first few letters of the band / album / song I wanna play, press enter and audacious is going to play it. With a bit of fantasy you can do literally thousands of things this way. By using tools like xclip, which let you access the kill buffer from command line and a good scripting language or shell you can do thousands of _wicked cool_ things this way. When I realized this, the Quicksilver comparison didn’t seem that funny anymore.

Let’s see how we can achieve something more sophisticated using dmenu (or a simillar tool, the idea is important here, you can roll your own one without too much struggle). Let’s say we have an address book kept as a text file:

John Kowalsky <john@kowalsky.com>
Chuck Norris <chuck@karate.com>
Elvis Presley <the@king.com>

We would like to press a shortcut, select a person using dmenu and had a gmail “new message” page opened in firefox with the right data filled in. I never really used Quicksilver, but guessing from screencasts/articles, this is one of the common ways people use it. It’s not a big deal to implement this:

URL=$(echo -n "http://mail.google.com/mail/?view=cm&fs=1&to=" && \
  (cat < addresses | dmenu))
firefox -remote "openUrl($URL)"

You make a shell script from this, bind it to a key and you have one more thing automatized. Using tools that let you access the X11 kill you could also implement easy adding of addresses to the text-file address book etc. Your imagination is the main limit here, the other being your knowledge of the shell, your scripting language of choice etc. Also remember you can nest dmenus to create something actually more reassembling a menu, without binding every single command to a keyboard shortcut.

What I want to achieve with my setup now is as much portability as you can get - I need an environment I can set up as quickly as possible on any modern computer no matter what. If my current box explodes, I want to be able to recreate the very same configuration in one day, nothing more. Currently, recreating my Gentoo setup would take a few weeks.

The next step I took after removing the bloated desktop environment is moving some of the activites to the web. I uninstalled Sage, putting my RSS feeds into Google Reader, and removed Sylpheed in favour of GMail and setting up forwarding from my second mail account. I always was and I still am a bit sceptical about web applications, but those two things I should have moved to the web long time ago. I’m only not sure about the Google, Che Guevara certainly wouldn’t like the idea of someone reading his e-mails. From the other side, they simply have the best products right now.

Now the basic stuff I spend most of my time in everyday looks like this:

Dwm, Emacs, Urxvt+Screen+Zsh, Firefox, Gmail, Greader, Ruby/Rails, Pidgin, Audacious

You may wonder about using pidgin or audacious in a tiling window manager, but in fact it works good enough. There is also obviously a whole lot of other apps I use and still need to pick down a bit, but the main set is compact enough. Now I’m trying to find a distribution that will make setting all this up as fast as possible… The first thing I come up is a Debian meta-package with all the software I need to get things done. After creating it, I would just install debian from a minimal installation CD, set up the internet connection, and install the metapackage. I have a separate partition for /home (one of the best Linux tips is to have one), so on my main box I wouldn’t even have to copy the configuration files. I was also thinking about checking out Arch Linux and OpenSuse, two distros I haven’t seen for a long time, but rising hope to not suck.

The nice thing is that all my major applications can tightly cooperate. Dwm takes the status bar content from Zsh, which for example checks Pidgin for new recieved messages, Dmenu can drive firefox using its remote interface, Emacs can send JavaScript to Firefox using MozRepl (thanks to Phil for this one!), Zsh can send a file to emacs using emacsclient etc. They are also extensible, what makes them the great apps they are. I could get even more from this setup, if I knew more JavaScript, shell magic, regular expressions, Lisp etc. It’s a nice motivation to expand your knowledge.

As always, here go the mandatory screenshots:

The whole setup is heavily influenced by various writing by various authors, make sure to checkout following articles/webpages if you’re curious:

- Suckless, the home of dwm, dmenu, wmii. You may also want to check out the mailing list and the irc channel (I got some very helpful advice there)

- The original “Good Easy”

- Mark Pilgrim’s “Good Easy”

- “The Antidesktop” from freshmeat.net

- Various articles from Fravia searchlores.org (actually not those Linux related, this one is more about the spirit than about the technical details)

1: General overview
2: Making an RSS
3: Fighting with POP3 / SSL
4: Serving the thing through HTTP
5: Putting the pieces together
6: Conlusions

This article is my submission for the Erlang blogging contest. You can also grab it as a PDF file - in a4 format or in letter format.

1: General overview

Functional programming languages are often underestimated, as they seem to be not suited for the “real world”, being thought of as designed by scientists who spend their time mainly developing theories, not practical software. What makes Erlang stand out from the crowd is the fact it was designed by the industry and for the industry. Ericsson, where the language originated, has a telecom switch with more than 2 million lines of Erlang code running it that has a downtime of a few minutes per year - now this is reliability!

Telecom switches don’t make programmers too excited those days through, so we will try to do something more related to your everyday hacking. In this tutorial I will show you how to write a standalone daemon, that checks your email every x minutes, and makes the results available as an RSS feed. If you want to centralize all your notifications around RSS this may even be useful a bit and even if not, at least it makes a good example of some of Erlang strengths. I try to explain everything in as much detail as possible, but be warned - we won’t be writing any “hello worlds” here, we will just rapidly introduce a lot of awkward stuff as it becomes necessary, so if you don’t have a solid programming background you may have serious trouble. Either way, you will probably want to consult Programming Erlang and the free, official Erlang docs if you want to do anything serious in Erlang. Also see my previous post for an overview of general high level Erlang concepts.

Let’s start with an overview of the parts that we will need. One component will have to connect to a POP3 server once in a while, grab the list of messages (or message titles) and make a nice list out of it. We don’t want it to interfere in any way with accessing the feed, so it will be a separate process. We’ll also need a function that given a simple list of messages, returns a well formed RSS feed created from it. When we have the feed, we want to have a simple HTTP server running that will simply send the feed to anyone who will make a GET. Of course there should be a possibility to update the feed the server is sending, when new mail arrives or simply when we check it the next time. As we said before, serving the feed should be independent from checking email, so it will be another process - and as we allow multiple connections to the server at the same time (maybe you will want to show you email to the world one day, who knows?), it will spawn a new request handling process for each accepted TCP connection, so multiple requests will be handled concurrently. Thinking in terms of processes and concurrent activities in the system is the core of Erlang programming paradigm and is the essence of learning the language. The cool part is that in Erlang modeling software as multiple processes cooperating finally becomes a natural, seamless abstraction and you don’t have to worry all the time about low-level stuff like deadlocks.

2 : Making an RSS

The data structures we will use are rather basic - we represent messages as tuples and store them in a list. A tuple simply groups a bunch of variables into a single entity. Our program will use tuples to represent messages - the first field will be the “message” symbol used to emphasize the meaning of the gathered data, then goes the date of arrival, sender, subject and body. Here are examples of our messages using Erlang syntax:

{message, "12:00 01-10-1987", "Foo Bar", "Hello Bar!", "Nice to meet you"}

{message, "16:32 05-05-2006", "Whoever", "Some subject", "Whatever"}

You can get to individual tuple fields using the element function:

1> element(2, {abc, def, ghi}).
def

We want to store at least the 10 most recent messages, so we have to group them somehow. Erlang has a powerful list abstraction in the spirit of Lisp or Scheme, so it should suit our needs. A list in Erlang syntax looks like that:

[{message, "12:00 01-10-1987", "Foo Bar", "Hello Bar!", "Nice to meet you"},
 {message, "16:32 05-05-2006", "Whoever", "Some subject", "Whatever"}]

From the components we outlined, creating the feed from a supplied list of messages seems to be the simplest task to do. Just put the various message field into corresponding XML fields and wrap it with some more XML on the beginning and on the end.

-module(rss_wrap).
-compile(export_all).

-define(prologue(Date),
        "<?xml version=\"1.0\"?>
         <rss version=\"2.0\">
         <channel>
         <title>GMailRSS</title>
         <description>Your mailbox in RSS using Erlang.</description>
         <language>en-us</language>
         <docs>http://blogs.law.harvard.edu/tech/rss</docs>
         <generator>RssGmail</generator>" ++
         "<pubDate>" ++ Date ++ "</pubDate>").

-define(epilogue,
        "</channel>
         </rss>").

escape_from(From) ->
    Left = element(2, regexp:gsub(From, "<", "(")),
    Right = element(2, regexp:gsub(Left, ">", ")")),
    Right.

rss_wrap(MsgList) ->
    case MsgList of
        [] ->
            CurrTime = erlang:universaltime(),
            Date = httpd_util:rfc1123_date(CurrTime);
        List ->
            Date = element(2, lists:last(MsgList)),
    end,

    ?prologue(Date) ++ rss_wrap_list(MsgList) ++ ?epilogue.

rss_wrap_list(MsgList) ->
    WrapItem = fun(Item, Result) -> Result ++ rss_wrap_item(Item) end,
    lists:foldl(WrapItem, "",  MsgList).

rss_wrap_item({message, Date, From, Subject, Content}) ->
    "<item>" ++
      "<title>" ++ Subject ++ "</title>" ++
      "<description>" ++
        "From: " ++ escape_from(From) ++
        Content ++
      "</description>" ++
      "<pubDate>" ++ Date ++ "</pubDate>" ++
    "</item>".

The “-module” declaration gives you a namespace for the functions you will define in the file and it must be the same as the filename in which the code is stored (but without the “.erl”). “-compile(export_all)” - tells the Erlang compiler to export all function automatically, so that they become globally available. In production you would like to export each of them manually, but while you are developing something it’s convenient not too change the exports everytime you add a function.

“-define” lets you define constants, which you can later refer to with a preceding quotation mark. You can also use it to write something similar to C macros - like in the prologue definition. “++” is string concatenation, so the prologue will contain the beginning of the feed’s with the publication date embedded somewhere inside.

The main function “rss_wrap” checks if the message list is empty and either sets the date to the current time using the matching RFC’s syntax, or to the arrival date of the last message in the list. Later it returns the result - the concatenation of the XML “prologue” with the date embedded, the result of the rss_wrap_list function call and the XML “epilogue”. Note that consecutive statements in functions are separated by commas, and that each definition ends with a dot. We will cover the details of the “case” construction later on.

The rss_wrap_list function shows one of the most common used idioms in any serious programming language. What foldl does (it is also present in Lisp, and in Ruby as “inject”) is accumulating all the elements of a list into a single variable in a way defined by the user. In Erlang you first define a fun (which is simply a function with no name bound to it, that’s why we have to assign it to something or pass to a function immediately) taking two parameters, one representing the element of the list that is currently being processed and the second one being the accumulator. The fun has to return the new value of the accumulator after processing one element. We give that fun to foldl, together with the initial value of the accumulator and with the actual list. If you have trouble understanding it look at a simpler example first:

-module(test123).
-compile(export_all).

foldl_test() ->
  ProcessedList = [1, 2, 3, 4, 5],
  CollectFun = fun(ProcessedItem, Accumulator) -> Accumulator + ProcessedItem end,
  lists:foldl(CollectFun, 0, ProcessedList).

We start the foldl with the initial accumulator value of 0. The first item of the array is 1 and the fun returns the accumulator summed with the item, so here it returns 1, which becomes the new value of the accumulator. Now the fun is called for the second item, which is 2, summing it up with the accumulator we get 3, which becomes the new accumulator. And so it goes until the end of the list - finally foldl returns the sum of all elements in the array.

In rss_wrap_list we do preciesly the same thing, but we’re using a string accumulator and the rss_wrap_item function which wraps each message tuple in XML conforming to the RSS 2.0 standard. We also need to escape the “<" and ">” characeters the From header may contain, so we introduce the escape_from function doing preceisly that. We use “element” in it, because the gsub function returns a tuple with several fields, not only the string after the substitution.

One more thing: in the above examples we’re using assignment - keep in mind that in Erlang all “variables” are immutable - once you define something, you can’t change its value. This may sound bad to you, but this is also one of the reasons concurrency in Erlang is easy.

Now we have one component ready. We can save it in a file named “rss_wrap.erl” and test it in the Erlang shell:

c(rss_wrap).
rss_wrap:rss_wrap([{}, {}]).

3: Fighting with POP3 / SSL

Now time for the tough part. We need to connect to the POP3 server and create the list we will wrap in the XML later on. This is about writing practical stuff, so we will use GMail’s POP3 as a testbed. This doesn’t make things easier, as Google requires you to use ssl. Could it be so hard with Erlang excellent libraries, will it be so hard? Lets see…

First of all, if you happen to not know anything about the POP3 protocol, consult the suitable RFC. RFCs aren’t an easy read (well, sometimes they are), but hey, studying them will make you feel like a real hacker and may even turn you into one someday (if you really want it). Basically it’s a simple dialogue:

Client: [opens the socket]
Server: +OK I’m one cool pop3 server [Introduces himself, waits for authorization]
Client: USER foobar
Server: +OK Give me the password
Client: PASS whatever
Server: +OK [Gets into transaction mode]

When in transaction mode, you can send any number of commands you want, until a final QUIT. You can actually have this kind of conversation with your POP3 server using the fabulous netcat tool. Sadly, the standard netcat doesn’t support SSL, so for gmail you will have to download something more fancy like socat or ncat. Or you could write a short app for chatting with remote hosts through SSL…

Anyway, let’s see how the basic communication works. We will open a SSL socket, receive the initial POP3 greeting line, display it, close the socket and exit:

-module(pop3).
-compile(export_all).

fetch() ->
    application:start(ssl),
    {ok, SecureSock} = ssl:connect("pop.gmail.com", 995,
                                   [{mode, list}, {packet, 0}, {active, false}]),
    {ok, Response} = ssl:recv(SecureSock, 0),
    io:format("~s~n", [Response]),
    ssl:close(SecureSock).

What do those fancy statements looking like assignments with tuples mean? Well, this is pattern matching - yet another feature of many functional languages that didn’t yet made it into the mainstream. If you look at the docs for the ssl module you will see that the ssl:connect function either returns a tuple where the first element is the symbol “ok”, and the second the socket you just opened, or accordingly the “error” symbol and the cause of the error, also as symbol ie. “timeout”. If everything goes good than the leftside “ok” will get matched with the rightside “ok” doing preciesly nothing, and the leftside unbound variable “SecureSock” will get matched with the returned socket - assigning the socket to the variable. But if an error occurs, than “ok”, will be matched with “error”, the match will not succeed and you will get a runtime error from Erlang. Trust me, you really don’t want to see an Erlang runtime error - they are mostly incomprehensible without the use of the Erlang debugger, which can at least show you the line the error occurred. You can avoid seeing them with writing your code in a fault-tolerant way - we could do error handling by matching the result of the call against more than one pattern, but this article is already getting long… Also, we should have been using pattern matching instead of the “element” function before - this way we make sure the tuple is of the format we’re expecting.

Going back to pop3, the general pattern of communication will be not too surprisingly Request -> Response, so maybe we can capture a higher level abstraction… Actually, we also don’t have a guarantee that the response will come in one portion (hence one ssl:recv), so we need to pull data out of the socket until we reach a CR/LF at the end:

request(Socket, Command) ->
    io:format("Request: ~s~n", [Response]),
    ssl:send(Socket, Command ++ "\n").

retrieve_line(Socket, Data) ->
    Length = length(Data),
    ValidP = (Length > 2) andalso (string:substr(Data, Length - 1, 2) =:= "\r\n"),
    if
        ValidP =:= true ->
            Data;
        true ->
            {ok, NewData} = ssl:recv(Socket, 0),
            retrieve_line(Socket, Data ++ NewData)
    end.

response_line(Socket) ->
    Response = retrieve_line(Socket, []),
    io:format("Response: ~s~n", [Response]),
    string:substr(Response, 1, string:len(Response) - 2).

The request method simply sends whatever you want through the socket, only adding a newline so that you don’t have to type it every time.

Getting the response is a bit more complicated. Before checking the string ending for the CR/LF we need to check if the string is long enough - if the indicies in string:substr are out of string bounds we will get a run-time error from Erlang. We do it using “andalso” - if the length is less than or equal to two the second statement will not be evaluated. Afterwards we actually check for the CR/LF, and if it is found, we return the data to the caller function. Otherwise, we try to get yet more data from the socket and we call the function we’re actually in - note that nothing gets returned to the caller until the recursion ends. Also, the if statement works different a bit than we’re used to - it does pattern matching in fact, that’s why use “true” instead of “else”, because “true” always matches.

Receive_line may look inconspicuous, but it illustrates one of the important “patterns” in Erlang - the way you deal with things that change over time. Normally you would use variables for that, but as we already said they are immutable. So in Erlang you use function parameters and recursive calls to deal with that. Actually, to represent a process, one also uses something that looks like a recursive function call running for very long.

Stepping back to our program, the response_rn method strips the final newline from the response and outputs it to the standard output - it’s mainly a convenience wrapper. Notice that things in Erlang are numbered from one, not from zero like in C for example.

Now we can have the conversation we talked about:

fetch() ->
    application:start(ssl),
    {ok, SecureSock} = ssl:connect("pop.gmail.com", 995,
                                   [{mode, list}, {packet, 0}, {active, false}]),
    response(SecureSock),

    request(SecureSock, "USER foobar"),
    response(SecureSock),
    request(SecureSock, "PASS hello"),
    response(SecureSock),

    ssl:close(SecureSock).

Just remember to replace foobar with your gmail user name and hello with you password. Here comes the hardest part - we need to get the emails out of the POP3 and turn them into a nice list in the format we introduced on the very beginning. This sounds like pain to me. We will also need to take a look at another RFC, describing the format of the message returned by the server.

Generally, after authenticating we need to do a STAT. This will give us the number of messages in the mailbox. Than we need to make a RETR for each consecutive message number.

-module(pop3).
-compile(export_all).

response(Socket) ->
    {ok, Response} = ssl:recv(Socket, 0),
    io:format("Response: ~s", [Response]),
    Response.

request(Socket, Command) ->
    io:format("Request: ~s~n", [Command]),
    ssl:send(Socket, Command ++ "\n").

authenticate(Socket, Login, Pass) ->
    request(Socket, "USER " ++ Login),
    response(Socket),
    request(Socket, "PASS " ++ Pass),
    response(Socket).

do_stat(Socket) ->
    request(Socket, "STAT"),
    {ok, Tokens} = regexp:split(response(Socket), " "),
    {list_to_integer(lists:nth(2, Tokens)),
     list_to_integer(lists:nth(3, Tokens))}.

fetch() ->
    application:start(ssl),
    {ok, SecureSock} = ssl:connect("pop.gmail.com", 995,
                                   [{mode, list}, {packet, 0}, {active, false}]),
    response(SecureSock),
    authenticate(SecureSock, "foobar", "barfoo"),
    {MessageCount, _} = stat(SecureSock),
    ssl:close(SecureSock).

I extracted the authentication into a separate method, and added the “stat” function. It makes a STAT request, which in response gives you something like this:

+OK 10 12345 Anything can be here

Where 10 is the number of messages and 12345 size of all of them. So what the stat function does is to split the response on spaces, extract the two fields we may care about, and convert them to integers. Strings in Erlang are just a list of integers (unfortunately), so the function list_to_integer is also used for converting strings to integer. The function nth from the lists module simply extracts the nth element from a list.

Now we need to make a “RETR” request for every message. It takes a number as an argument, where valid numbers go from one to the number STAT gave us. So “normally” we would probably use a for loop for this, but in Erlang iteration is often done in a different manner, take a look at the following code snippet:

retrieve_messages(Socket, From, To) when From < To ->
    [retr(Socket, From)|retrieve_messages(Socket, From+1, To)];

retrieve_messages(Socket, To, To) ->
    [retr(Socket, To)].

This may look like a nonsense, but you must know that Erlang does also pattern matching on function arguments. Thus, the second definition of “retrieve_messages” will only be called when the second and third argument is exactly the same, otherwise the first definition will be used. Syntactically, notice that multiple “versions” of the same function are separated with semicolons, the only exception being the last definition, which you end with a dot. Also, if you define a “retrieve_messages” with two arguments, it will be a completely different function.

If you’ve written functional programs before, you probably already know what the code above does, otherwise it may be a little bit hard to wrap your mind around this for the first time. What the first definition says is that the result of calling it is a list built from calling retr and from calling retrieve_messages with the same arguments, except the From parameter being increased by one. “|” is the basic list building thing, like “cons” from Lisp. This obviously is recursive, so the second call to retrieve_messages may also expand futher. Let’s say we call retrieve_messages(SomeSocket, 1, 3). It will go like this:

retrieve_messages(SomeSocket, 1, 3) returns:

[retr(Socket, 1) | retrieve_messages(Socket, 2, 3)]

But retrieve_messages(Socket, 2, 3) returns:

[retr(Socket, 2) | retrieve_messages(Socket, 3, 3)]

So the first call gets futher expanded to:

[retr(Socket, 1), retr(Socket, 2) | retrieve_messages(Socket, 3, 3)]

retrieve_messages(Socket, 3, 3) matches the second definition, so
the recursion ends, because we get “[retr(Socket, 3)]” from it. So the
final expansion looks like this:

[retr(Socket, 1), retr(Socket, 2) | [retr(Socket, 3)]]

Which is evaluated simply to:

[retr(Socket, 1), retr(Socket, 2), retr(Socket, 3)]

Well, nobody said that this is going to be easy… Doing the actual RETR isn’t all that fun either. More often than not, the message will be send in portions, so our response method isn’t exactly valid anymore. As we can read in the RFC, a multiline response is terminated by the “\r\n.\r\n” string. So we can write another response function:

retrieve_multiline(Socket, Data) ->
    Length = string:len(Data),
    Ending = string:substr(Data, Length - 4, 5),
    ValidP = (Length > 5) andalso (Ending =:= "\r\n.\r\n"),
    if
        ValidP =:= true ->
            Data;
        true ->
            {ok, NewData} = ssl:recv(Socket, 0),
            retrieve_multiline(Socket, Data ++ NewData)
    end.

response_multiline(Socket) ->
    Response = retrieve_multiline(Socket, []),
    io:format("Response: ~s~n", [Response]),
    string:substr(Response, 1, string:len(Response) - 5).

But that is very repetitive. Let’s rewrite both response functions in a more generic way:

retrieve(Socket, Data, Ending) ->
    DataLen = length(Data),
    EndingLen = length(Ending),
    ValidP = (DataLen > EndingLen) andalso
             (string:substr(Data, DataLen - EndingLen + 1, EndingLen) =:= Ending),
    if
        ValidP =:= true ->
            Data;
        true ->
            {ok, NewData} = ssl:recv(Socket, 0),
            retrieve(Socket, Data ++ NewData, Ending)
    end.

response(Socket, Ending) ->
    Response = retrieve(Socket, [], Ending),
    io:format("Response: ~s~n", [Response]),
    string:substr(Response, 1, string:len(Response) - length(Ending)).

response_line(Socket) ->
    response(Socket, "\r\n").

response_multiline(Socket) ->
    response(Socket, "\r\n\r\n").

Now we can retrieve every message in the mailbox:

fetch() ->
    application:start(ssl),
    {ok, SecureSock} = ssl:connect("pop.gmail.com", 995,
                                   [{mode, list}, {packet, 0}, {active, false}]),
    response(SecureSock),
    authenticate(SecureSock, "foobar", "barfoo"),
    {MessageCount, _} = stat(SecureSock),
    if MessageCount > 0 ->
            Messages = retrieve_messages(SecureSock, 1, MessageCount),
       true ->
            io:format("No new messages~n"),
            Messages = []
    end,
    ssl:close(SecureSock),
    Messages.

The fetch function should return the list of messages at the end; I showed it at the very beginning. So the last thing we need to do is to write something that will make a tuple of the format we want out of a plain text message. When we have this function, we will map it over the Messages list and our POP3 mission will finally be finished. Parsing the message has not much to do with pop3, so we will create another module:

-module(message).
-compile(export_all).

get_header(Headers, HeaderName) ->
    FindHeader = fun(Item) -> lists:prefix(HeaderName, Item) end,
    Header = lists:nth(1, lists:filter(FindHeader, Headers)),
    element(2, regexp:sub(Header, HeaderName, "")).

parse(Message) ->
    {_, Boundary, _} = regexp:first_match(Message, "\r\n\r\n"),
    Header = string:substr(Message, 1, Boundary),
    Body = string:substr(Message, Boundary + 4, length(Message) - Boundary),
    {ok, Headers} = regexp:split(Header, "\r\n"),
    {message, get_header(Headers, "Date: "), get_header(Headers, "From: "),
     get_header(Headers, "Subject: "), Body}.

Lets start the analysis with the function parse. First we need to split the header part of the message from the body - they are separated by an empty line, or two CR/LFs if you wish. The function regexp:first_match return a tuple of the format “{match, Start, Length}”, but we only care about that the middle field - thats why we use “_”. Its simply a way to say that you know about the presence of other fields, but you don’t want them to be bound to anything. We could also say:

Boundary = element(2, regexp:first_match(Message, "\r\n\r\n"))

because the element function simply extracts the nth element of the tuple. Than we actually split the string using the found Boundary. The extracted header gets splitted one more time, so that each line (and each header field at the same time) is an element in a list. The final expression builds the tuple representing the message using the function get_header.

In get_header we use another powerful function operating on lists - filter. It takes a fun returning a boolean value as an argument and returns only those elements of the list, which when supplied to the fun give “true” as a result.

In the definition of the fun we once again take advantage of the misery of Erlang string as a list approach - thats why we can use lists:prefix to check if the string starts with the header name we supply to get_header as an argument.

We actually only care about the first occurrence of the header, so we make it a variable instead of a list using lists:nth. We don’t want to have the header name in our final tuple, so in the last step we remove it and return only the “body” of the header. Now we only have to modify the fetch function from the pop3 module to apply message:parse to each message:

fetch() ->
    application:start(ssl),
    {ok, SecureSock} = ssl:connect("pop.gmail.com", 995,
                                   [{mode, list}, {packet, 0}, {active, false}]),
    response_line(SecureSock),
    authenticate(SecureSock, "foobar", "barfoo"),
    {MessageCount, _} = do_stat(SecureSock),
    if MessageCount > 0 ->
            Messages = retrieve_messages(SecureSock, 1, MessageCount),
            MsgProc = fun(Item) -> message:parse(Item) end,
            ProcessedMessages = lists:map(MsgProc, Messages);
       true ->
            ProcessedMessages = []
    end,
    ssl:close(SecureSock),
    ProcessedMessages.

Voila - after all the struggles we have our messages ready for processing. We can call something like rss_wrap:rss_wrap(pop3:fetch()) to get a nice feed representing our mailbox content.

4: Serving the thing through HTTP

Now we need a simple web server to make our feed available in the browser. We could use Erlangs built-in inets module for doing HTTP, but since we don’t need anything fancy here we can as well implement it on our own, as it will be an excellent opportunity to introduce more language features - what we did above was all sequential stuff, now we will get to the part that makes people curious about Erlang - concurrency.

The basic skeleton of a server operating through TCP looks like this:

-module(http).
-compile(export_all).

start() ->
    spawn(http, server, []).

server() ->
    {ok, ListenSocket} = gen_tcp:listen(80, [list, {packet, 0}, {active, false}]),
    listen(ListenSocket).          

listen(ListenSocket) ->
    case gen_tcp:accept(ListenSocket) of
        {ok, Socket} -> ->
            spawn(http, retrieve, [Socket, []]),
            listen(ListenSocket);
        _ ->
            listen(ListenSocket)
    end.

retrieve(Socket, Data) ->
    case gen_tcp:recv(Socket, 0) of
        {ok, Packet} ->
            %% Do something with the retrieved data
            retrieve(Socket, Data ++ Packet);
        {error, closed} ->
            gen_tcp:close(Socket)
    end.

Several new constructions appear in this example. The most important thing here is the spawn function - it runs a function as a separate process. This means that you don’t recieve any return value immediately like when normally calling function - instead computing the function starts, but at the same time the control flow is passed back to the caller function. Spawn in the most commonly used from takes three parameters - the module name, the function name, and a list of arguments to be supplied to the function. It returns a pid - a unique identifier of the process in the Erlang system, which can be used later mainly for passing messages. Knowing that, understanding start() shouldn’t be hard - it simply starts the http server (represented by the server() function) without blocking any other activities going on in the system and returning the control flow to the caller function as soon as the server process starts.

The spawned process sets up a socket for accepting incoming connections. The socket is passed to the listen function. Listen() makes use of the case instruction, used very often in Erlang. It basically does pattern matching on the result of the expression you supply it - here it checks for a incoming connection, if there is one pending it spawns a separate process to handle the request and goes back to listening for new connections. If there are no connection attempts, it just waits. Notice another use of recursive function definitions - here they are used to make a continuously running process.

Finally, the retrieve function pulls data out of the supplied socket and responds somehow to the recognized request. It uses its second argument to accumulate the received data over time.

Now we need to extend the general scheme to use HTTP. We will simplify it as much as possible - we will only respond to one kind of request, to a “GET /”. For now, we will also respond with some hardcoded text, not with content that can be dynamically changed:

-module(http).

-compile(export_all).

-define(ok_200, "HTTP 1.1 200 OK\r\n").
-define(content_type,
        "Content-Type: text/plain\r\n").
-define(content_length(Length),
        "Content-Length: " ++ integer_to_list(Length) ++ "\r\n").

start() ->
    spawn(http, server, []).

server() ->
    {ok, ListenSocket} = gen_tcp:listen(80, [list, {packet, 0}, {active, false}]),
    listen(ListenSocket).          

listen(ListenSocket) ->
    case gen_tcp:accept(ListenSocket) of
        {ok, Socket} ->
            spawn(http, retrieve, [Socket, []]),
            listen(ListenSocket);
        _ ->
            listen(ListenSocket)
    end.

retrieve(Socket, Data) ->
    case gen_tcp:recv(Socket, 0) of
        {ok, Packet} ->
            Request = Data ++ Packet,
            process_request(Socket, Request),
            retrieve(Socket, Request);
        {error, closed} ->
            gen_tcp:close(Socket)
    end.

process_request(Socket, "GET / " ++ _) ->
    Text = "Hello, HTTP!",
    Response = ?ok_200 ++
        ?content_type ++
        ?content_length(length(Text)) ++ "\r\n" ++
        Text,
    gen_tcp:send(Socket, Response),
    gen_tcp:close(Socket).

process_request(Socket, _) ->
    0.

The function process_request does something only when the request begins with a “GET / ” - the “++” in patterns is spliting the string, not concatencating it. In case of a GET, we compose a simple response consisting of the HTTP headers and some hardcoded text. Than we send the response through the socket and close it - remember that in HTTP each request is completly independent. If the request is not a GET, we just return 0 and do nothing else with it. We can start in the Erlang shell

c(http).
http:start().

Note that Erlang lets only privileged users grab a port, so you may have to run it with sudo.

5: Putting the pieces together

Until now all the modules we developed were rather separated. Now we need to glue the pieces together, so the processes we will be running can communicate.

Communication between processes in Erlang is done mainly by using two primitives - the “!” operator and the “receive” block. “!” sends a message to a pid. A message can be quite any valid Erlang data type. We use it like this:

SomePid ! do_something
SomePid ! {this, is, a, tuple}
SomePid ! [1, 2, 3, 4, 5]

The only requirement is that the process represented by SomePid has the matching receive that will do something with the message. As many other constructions in Erlang, it uses pattern matching to recognize various kinds of messages, ie.:

receive
    {something, Data} ->
        io:format("~s~n", [Data]);
    {anotherthing, AnotherData} ->
        process(AnotherData)
end.

Before we use it in practice, lets write a separate simple example to illustrate message passing:

-module(test).
-compile(export_all).

server() ->
    receive
        Data ->
            io:format("~w~n", Data),
            server()
    end.

This tiny function listens for any kind of message. When it recieves one, it simply outputs it using standard Erlang syntax. You can now send something to it from the shell:

SomePid = spawn(test, server, []).
SomePid ! {something, foobar}.
SomePid ! [1, 2, 3, 4, 5].
SomePid ! "Whatever".

Going back to our main program, we need messages to pass the feed to the HTTP server. What we will try do is making the main function server accept two kinds of messages - one for setting the server response, and another for returning it. Actually “returning” is a tricky word here - a process wanting to get the response will have to supply his Pid and we sent the result back using messages. Look at the final version of the module:

-module(http).

-compile(export_all).

-define(ok_200, "HTTP 1.1 200 OK\r\n").
-define(content_type, "Content-Type: text/plain\r\n").
-define(content_length(Length),
        "Content-Length: " ++ integer_to_list(Length) ++ "\r\n").

control_server(Response) ->
    receive
        {set, response, NewResponse} ->
            control_server(NewResponse);
        {get, response, Pid} ->
            Pid ! {response, Response};
        _ ->
            control_server(Response)
    end.

server() ->
    ControlServerPid = spawn(http, control_server, [""]),
    register(control_server, ControlServerPid),
    spawn(http, initialize, []),
    ControlServerPid.

initialize() ->
    case gen_tcp:listen(80, [list, {packet, 0}, {active, false}]) of
        {ok, ListenSocket} ->
            listen(ListenSocket);
        _ ->
            stop
    end.

listen(ListenSocket) ->
    case gen_tcp:accept(ListenSocket) of
        {ok, Socket} ->
            spawn(http, retrieve, [Socket, []]),
            listen(ListenSocket);
        _ ->
            listen(ListenSocket)
    end.

retrieve(Socket, Request) ->
    case gen_tcp:recv(Socket, 0) of
        {ok, Packet} ->
            process_request(Socket, Request ++ Packet),
            retrieve(Socket, Request ++ Packet);
        {error, closed} ->
            gen_tcp:close(Socket)
    end.

process_request(Socket, "GET / " ++ _) ->
    control_server ! {get, response, self()},
    receive
        {response, Text} ->
            Response = ?ok_200 ++
                ?content_type ++
                ?content_length(length(Text)) ++ "\r\n" ++
                Text,
            gen_tcp:send(Socket, Response),
            gen_tcp:close(Socket)
    end;

process_request(Socket, Request) ->
    0.

First we introduce a “control server”. This process will serve as a buffer between the outside world and the HTTP server. The function starting the server (http:server()) returns the Pid of the control server to the caller process, so the user of the http module can send messages to the control server, altering the behaviour of the http server. But since we don’t have explict state in Erlang, all the “variables” are stored as function parameters. We have only one dynamic thing here, but if there was more of them, we would probably want to use a property list for storing various settings of the server. server() does also one more interesting thing - it registers the control server process, so you can refer to it by a symbolic name instead of the pid and you don’t have to pass its pid to every function.

The reason we want the control server to be available everywhere is that the only way to check a setting (or in our case the desired server response) is to send a message to it. This is exactly what we do in the process_request function, body of which also explains why we called the process a “control_server”. We send a message to it that we want to get the reponse and we also supply the pid the control server should respond to, by using the function self() returning the pid of the current process. Than we wait for the response from the control server and when it arrives we’re ready to actually send the response through the socket.

Now we can finally put the pieces together using this little beauty:

-module(gmailrss).
-compile(export_all).

main() ->
    ServerPid = http:server(),
    spawn(gmailrss, loop, [ServerPid]).

loop(ServerPid) ->
    Messages = pop3:fetch(),
    Rss = rss_wrap:rss_wrap(Messages),
    ServerPid ! {set, response, Rss},
    timer:sleep(300000),
    loop(ServerPid).

Just run the Erlang shell and call:

gmailrss:main().

Than you can fire up your RSS reader, add “http://localhost/” as the feeds adress and enjoy your email in it! The time value for the sleep function is measured in miliseconds, so it will get checked every 5 minutes.

6: Conclusions

Wow, that turned out really long and comprehensive, considering it was first meant to be a short blog post. You can grab the source code of the finished code here. We still skipped a lot of subjects ie. we totally ignored any errors that might occur, to make explaining things easier. We didn’t care too much about message encoding and stuff like that either. There is also obviously much more cool stuff in Erlang, but that’s what books and docs are for. I wonder if someone made it this far at all, but nevertheless…

Happy hacking!

[ Additional thanks for Martinez, who reviewed the article and pointed out some language mistakes ]

Now, have you ever tried to do a different thing with your left hand and a different one with your right at the same time? Ask somebody playing a piano how hard it can be without a good amount of practice. Most of the time, our brain isn’t too good at doing more than one thing at once - no wonder we have also problems conceptualizing computer programs that do multiple things simultaneously. The only thing we can do is use our intelligence and write tools that will ease the job. Here also lies the biggest problem - most of our tools for writing concurrent applications are dumb and more often they cause a lot of confusion than help us build better software. When you say “concurrent” in the presence of a programmer, he will probably have those four associations - Java, threads, deadlocks, headache.

Threads as most of us know them suck big time, even on a conceptual ground, as a model of computation, what to say about the practical realizations. It’s hard to get something non-trivial done with them, it’s hard to prove the correctness of a thread-based computer program and when later something goes wrong it’s sometimes almost impossible to find the bug and even harder to remove it. Of course, it’s possible to write correct and efficient concurrent applications like this, but it requires a lot of knowledge and skills from the programmer, which he could otherwise use to work on the actual program logic.

Actually, the problem does not only apply to threads, it’s also the paradigm of the most popular languages that makes concurrency a serious issue. It’s hard to write a parallel program when state is shared between multiple processes running at the same time and in commonly used imperative languages there is a lot to share. That’s the chance for Erlang. It’s the only relatively known language that tackles concurrency in a different way. First of all, it is declarative, so there is no state to share. The declarativeness will strike you from the very beginning - once you “assign”, or rather match a variable with a value you cannot change it in Erlang, so most of your Java/C++/Python/Ruby tricks won’t work and you will have to learn a different way of thinking about programming.

The basic unit of Erlang concurrency model is called a process. Processes are a bit like threads, but they can communicate only using messages, there is no other way they can share anything. They are also very lightweight, so you can have a whole lot of them without worrying too much about hogging resources. Processes are also a way to model real-world objects, as there is no traditional OOP in Erlang. Each process has a unique id, through, and you can easily replace one process accepting a set of messages with another one accepting the same set as a form of polymorphism, so you can think of processes also like of a substitution for classes.

As you see, even when you forget about the parallel stuff, Erlang has some interesting traits, and we didn’t touch everything yet. You will find pattern matching, higher order functions, list comprehensions and other weird stuff in the language. Besides, it has some commercial support behind it and they were some serious projects at Ericsson using it, unlike many of the other languages that are theoretically interesting in some way. The libraries are quite big and well enough documented (at least for my standards) so there is a solid base for doing something useful. By being a declarative language it is also totally different from the stuff you usually code in “software factories”, so it also seems to be fun and challenging to learn.

I tried it a month or two ago, mostly by accident, researching options for writing a backend server needed for my job. I didn’t have to write it in the end, but instead I found some nifty and surprising stuff in Erlang and got into it a bit. One more unusual thing that strike me was the runtime environment - it’s very powerful and you can do a wide variety of things using it. Its something between an interpreter, compiler, debugger etc. all driven by a single shell. The system has also the advantage Steve Yegge blogged about - it doesn’t have to reboot. You can hot swap a piece of code without problems while the rest of the system is running. When one of the processes fails, the rest can continue to operate perfectly, like you can see in the Erlang movie. Like most good languages, Erlang has also a Emacs mode bundled ;)

There is so much new stuff to discover here, that I feel a lot of fun trying to explore it - the last time I felt like this was when I started learning Ruby a few years ago. Of course, the field where you can apply it in “real world” maybe isn’t that wide, but the fun factor makes other things less important…

If you think Erlang may be interesting, check out some more resources:

The Problem With Threads (through Algorithm.au) - A in-depth paper explaining why traditional threading sucks

Erlang Homepage - with a complete reference, tutorials, programming guidelines, the interpreter etc.

Erlang In Real Time - A free, quite comprehensive book about Erlang

Thinking In Erlang [pdf] - A short (30 pages), but really nice tutorial on the basics of Erlang

Joe Armstrong’s thesis - The best introduction to the language you can get for free, if you don’t mind the formal, scientific tone

Programming Erlang - A new, in-progress book from the Pragmmatic Programmes about Erlang, you can download a beta pdf right now

Erlang For C, C++, and Java Programmers - Another short tutorial, aiming to teach Erlang quickly to people with experience in more traditional, imperative languages.

Erlide - An eclipse plugin for Erlang

Yariv’s Blog - A weblog mostly about writing web applications in Erlang

Why I choose Erlang - An interesting rant about programming languages

TrapExit - A wiki about Erlang, with lots of excellent practical articles