Elixir Native Interoperability – Ports vs. NIFs

Lately I’ve been working on a personal project of creating a wireless sensor network across my home. Elixir is a perfect fit for this project, but I quickly hit a road block: serial device access. While there are Erlang serial libraries that I could use, I wasn’t ultimately comfortable doing so, due to many forks and adding yet another layer of complexity.

The Erlang VM, which Elixir runs on top of, has a number of different mechanisms for interoperability with external programs. Two I’ll be discussing here are ports and NIFs. They are the two simplest, each with a unique set of pros and cons. For the examples, I’ll be interfacing with the operating system to read and write data to a serial device.

NIF and Port Basics

First let’s take a high-level look at two of the most basic Erlang interoperability mechanisms Elixir supports.

(There are a few others interoperability mechanism such as Erl_Interface, port drivers, and interoperability with Java, but I won’t cover here. For more information check out the Erlang Interoperability Tutorial. These are particularly useful for if you need greater integration with Elixir such as access to message passing.)

NIFs from 30,000 Feet

NIFs, or Native Implemented Functions, are the technique that we tend to think of when looking for interoperability between languages. Implemented in Erlang R13B03, they are similarly to FFI in Ruby.

NIFs allow us to load a dynamic library and bind the libraries’ native functions to an Elixir function. NIFs appear as a normal function to the Elixir code invoking it, but invoke the underlying dynamic library’s implementation of the logic.

NIF Benefits

• Fast and simple implementation
• No context switch required
• Simple debugging

NIF Drawbacks

• Not very safe
• NIF-specific C library
• Native imperative implementation can leak into functional Elixir code

The most important thing to note here is that a crash in the C library will cause a crash of the entire Erlang VM. This goes against the inherit philosophy of Erlang errors, failing fast and using supervisors to recover.

Ports from 30,000 Feet

A port is a technique to communicate with an external native process over STDIN and STDOUT. When a port is created, a connected process is created which is used to communicate via message passing to the external native process.

Port Benefits

• Safety
• Error trapping
• Flexible communication
• No external erlang/elixir specific libraries required

Port Drawback

• Awkward STDIN/STDOUT communication

Other Pieces of the Puzzle

Arduino Sketches

For easy testing of the library, I used 2 arduino sketches: one that sends random blocks of text over the serial connection, and another that’s a loopback. For more details on these sketches and working examples of all code shown here, checkout the Github repository.

Makefiles and Mix

Elixir ships with an awesome build tool called mix, which is used to create, compile, test and manage dependencies. Integrating a make file for a C or C++ library into mix has enormous value in unifying a project’s build and is super simple.

Mix creates a few unsurprising directories when it creates a project: config, lib and test. We’ll create two more for the purpose of housing our native code and it’s compiled form: c_src and priv_dir.

A basic mix file looks like this:

defmodule SerialNif.Mixfile do

  use Mix.Project

  def project do

    [app: :serial_nif,

     version: "0.0.1",

     elixir: "~> 1.0",

     deps: deps]

  end

  def application do

    [applications: [:logger]]

  end

  defp deps do

    []

  end

end

We will add a new task for compilation, configure the project to invoke that on compilation, create a new task for cleaning, and configure the aliases for clean to execute that task. This will allow us to execute

mix clean

and

mix compile

to clean and execute our Elixir code with mix and our native code with make.

defmodule SerialNif.Mixfile do
  use Mix.Project

  def project do
    [app: :serial_nif,
     version: "0.0.1",
     elixir: "~> 1.0",
     compilers: [:make, :elixir, :app], # Add the make compiler
     aliases: aliases, # Configure aliases
     deps: deps]
  end

  defp aliases do
    # Execute the usual mix clean and our Makefile clean task
    [clean: ["clean", "clean.make"]]
  end

  def application do
    [applications: [:logger]]
  end

  defp deps do
    []
  end
end

###################
# Make file Tasks #
###################

defmodule Mix.Tasks.Compile.Make do
  @shortdoc "Compiles helper in c_src"

  def run(_) do
    {result, _error_code} = System.cmd("make", [], stderr_to_stdout: true)
    Mix.shell.info result

    :ok
  end
end

defmodule Mix.Tasks.Clean.Make do
  @shortdoc "Cleans helper in c_src"

  def run(_) do
    {result, _error_code} = System.cmd("make", ['clean'], stderr_to_stdout: true)
    Mix.shell.info result

    :ok
  end
end

Elixir Serial Device Access with Ports

Next let’s take a look at how to use ports to access a C program which manipulates serial devices.

Elixir Implementation

The Elixir implementation for our ports example is pretty simple:

defmodule Serial do

  def init() do

    Process.flag(:trap_exit, true)

    port = Port.open({:spawn, "priv_dir/serial"}, [{:packet, 2}])

  end

  def open(p, device, speed) do

    Port.command(p, [1,device])

    Port.command(p, [2,"#{speed}"])

  end

  def write(p, str) do

    Port.command(p, [3, str]);

  end

end

We have an init function that spawns a new port to our compiled c program located in priv_dir. The second argument to Port.open/2 says we will be sending packets of data that are prefixed with a 2 byte length indicator. The implementation of the actual open and write functions is also minimal. We use Port.command/2 to send commands to our native program. The second argument to Port.command/2 is the packet which will be sent down to our native implementation. In the write example, we use the integer 3 to represent invoking the write function in our native implementation, and str is our string which we will be writing out to the serial device. You will notice that there isn’t a read function, this is because we are taking advantage of message passing from the port which will send data up to Elixir as it is received by our native implementation.

C Implementation

The C implementation of the Port is a little bit more complicated:

#include 

#include 

#include 

#include "erl_comm.h"

#include "serial.h"

int bytes_read;

int serial_bytes_read;

int serial_fd = 0;

char serial_buf[5];

void reset_state() {

  bytes_read = 0;

  serial_bytes_read = 0;

  strcpy(serial_buf, "");

}

void process_command(byte *buf, int bytes_read) {

  int fn = buf[0];

  if(bytes_read > 0) {

    if (fn == 1) {

      char device_name[1024];

      get_str_arg(buf, device_name, bytes_read);

      serial_fd = serial_open(device_name);

    }

    else if(fn == 2) {

      serial_speed(serial_fd, get_int_arg(buf, bytes_read));

    }

    else if(fn == 3) {

      char str[1024];

      get_str_arg(buf, str, bytes_read);

      serial_write(serial_fd, str, bytes_read-1);

    }

    else {

      fprintf(stderr, "not a valid fn %i\n", fn);

    }

  }

  else if(bytes_read < 0) {
    exit(1);
  }
}
void poll_serial_data(int serial_fd) {
  serial_bytes_read = read(serial_fd, serial_buf, 5);
  if(serial_bytes_read > 0) {

    write_cmd( (byte*) serial_buf, 5);

  }

}

int main() {

  byte buf[100];

  while (1) {

    reset_state();

    if(input_available() > 0 ) {

      bytes_read = read_cmd(buf);

      process_command(buf, bytes_read);

    }

    if(serial_fd > 0) {

      poll_serial_data(serial_fd);

    }

  }

}

Looking at the main function first, we have an infinite loop which checks to see if we have input available from elixir on STDIN. If so, then it parses the data based upon the 2 byte length indicator and executes the specific command. We also check to see if the serial device has been opened, and if it has, we poll it for any available serial data, writing it to STDOUT. You can see I am also using STDERR for error messages, it can also be used for debugging the C code from Elixir if needed.

For the sake of simplicity I abstracted away the details of parsing the command into erl_comm.h and manipulating the serial port into serial.h. If you would like to view the dirty details, you can find the full code on Github. I stuck with the example protocol of a 2 byte packet since that code was provided by the Erlang Interoperability Tutorial and fit our use case well.

Elixir Serial Device Access with NIFs

Makefile

Since we will be using an Erlang provided C library for type conversion and communication with the Erlang VM, we need to make that library known to GCC.

ERLANG_PATH = $(shell erl -eval 'io:format("~s", [lists:concat([code:root_dir(), "/erts-", erlang:system_info(version), "/include"])])' -s init stop -noshell)

CFLAGS = -g -O3 -ansi -pedantic -Wall -Wextra -I$(ERLANG_PATH)

Elixir Implementation

The Elixir implementation for our NIFs example is a bit more complicated:

Serial do

  @on_load :init

  def init() do

    :erlang.load_nif("./priv_dir/lib_elixir_serial", 0)

    :ok

  end

  def open(device, speed) do

    _open(String.to_char_list(device), speed)

  end

  def read(fd) do

    _read(fd)

  end

  def write(fd, str) do

    _write(fd, String.to_char_list(str))

  end

  def close(fd) do

    _close(fd)

  end

  def _open(device, speed) do

    "NIF library not loaded"

  end

  def _read(fd) do

    "NIF library not loaded"

  end

  def _close(fd) do

    "NIF library not loaded"

  end

  def _write(fd, str) do

    "NIF library not loaded"

  end

end

On loading of our module, we load the NIF C library we wrote which replaces _open, _read, _close and _write functions with their native implementations. I also have basic wrappers around our 4 functions for the public interface which guarantees the data is in the types we are expecting them to be in our C implementation.

C Implementation

Looking at the C code for the NIF example:

#include "erl_nif.h"

#include 

#include 

#include 

#include "serial.h"

#define MAXBUFLEN 1024

static ERL_NIF_TERM _open(ErlNifEnv* env, int arc, const ERL_NIF_TERM argv[])

{

  char path[MAXBUFLEN];

  int fd;

  int int_speed = -1;

  enif_get_string(env, argv[0], path, 1024, ERL_NIF_LATIN1);

  enif_get_int(env, argv[1], &int_speed);

  fd = serial_open(path);

  serial_speed(fd, int_speed);

  return enif_make_int(env, fd);

}

static ERL_NIF_TERM _read(ErlNifEnv* env, int arc, const ERL_NIF_TERM argv[])

{

  int fd=-1;

  char buf[1];

  ErlNifBinary r;

  int res;

  enif_get_int(env, argv[0], &fd);

  res =read(fd,buf,1);

  if (res > 0)

  {

    enif_alloc_binary(1, &r);

    strcpy(r.data, buf);

    return enif_make_binary(env, &r);

  }

  enif_alloc_binary(0, &r);

  return enif_make_binary(env, &r);

}

static ERL_NIF_TERM _write(ErlNifEnv* env, int arc, const ERL_NIF_TERM argv[])

{

  char str[MAXBUFLEN];

  int fd, size;

  enif_get_int(env, argv[0], &fd);

  size = enif_get_string(env, argv[1], str, 1024, ERL_NIF_LATIN1);

  serial_write(fd, str, size-1);

  return enif_make_int(env, size);

}

static ERL_NIF_TERM _close(ErlNifEnv* env, int arc, const ERL_NIF_TERM argv[])

{

  int fd=-1;

  enif_get_int(env, argv[0], &fd);

  close(fd);

  return enif_make_int(env, fd);

}

static ErlNifFunc nif_funcs[] =

{

  {"_open", 2, _open},

  {"_read", 1, _read},

  {"_write", 2, _write},

  {"_close", 1, _close}

};

ERL_NIF_INIT(Elixir.Serial,nif_funcs,NULL,NULL,NULL,NULL)

Starting off at the bottom, we use ERL_NIF_INIT to actually invoke the Erlang VM magic to hot swap our bare functions for their native implementations. The first argument has to match the module that we load the NIF from, prefixed with Elixir., so for our example it is Elixir.Serial. The nif_funcs array is a mapping of our functions in Elixir, their arity, and their counterparts in C.

Now that we are set up, lets take a look at the implementation of one of our functions:

static ERL_NIF_TERM _open(ErlNifEnv* env, int arc, const ERL_NIF_TERM argv[])

{

  char path[MAXBUFLEN];

  int fd;

  int int_speed = -1;

  enif_get_string(env, argv[0], path, 1024, ERL_NIF_LATIN1);

  enif_get_int(env, argv[1], &int_speed);

  fd = serial_open(path);

  serial_speed(fd, int_speed);

  return enif_make_int(env, fd);

}

We have to return something of type ERL_NIF_TERM in order to get data back to Elixir and our arguments come in a form defined by erl_nif.h. We can use the enif_get_string and enif_get_int functions to convert from ERL_NIF_TERMs to C data types. We then invoke our functions. To convert from C data types back to Elixir data types, we use the enif_make_int function to convert back to an ERL_NIF_TERM.

The erl_nif.h library has a number of other interesting functions such as enif_send which allows the native C program to send messages to an Elixir pid. We could have used message passing to send data to our Elixir process instead of using the read/1 function, but this would have required threading, which opens up a whole can of worms given the safety concerns that NIFs bring with them.