4 Comments

FFI: Foreign Function Interfaces for Fun & Industry

I love writing code in high-level languages, like Ruby. But a lot of cool and useful libraries are written in lower-level languages, like C. Alas! What is a programmer to do?

There are three common approaches to solving this conundrum:

  1. Port all or part of the library to your language of choice.
  2. Write an extension in C code to bridge the gap between the library and your language.
  3. Wrap the library using your language’s foreign function interface (FFI) support.

I used each of these three approaches over the 7-year history of developing Rubygame. I began with a combination of Ruby port and C extension, then migrated to a combination of Ruby port and FFI wrapper using Ruby-FFI. And a few years ago, fellow Atom Shawn Anderson wrote about his experience creating a Ruby-FFI wrapper for the Chipmunk game physics engine.

Based on our experience, we have to say: FFI is awesome. An FFI wrapper is much easier to write and maintain than a C extension, more portable across platforms and language implementations, and easier for users to install.

Although most of my FFI experience is with Ruby, many other languages offer some form of FFI support: Common Lisp, Haskell, Lua, Python, Perl, and Java, to name just a few. Many of the same benefits (and limitations) of FFI that I’ll be describing in this post apply just as much to other languages as they do to Ruby.

What Is FFI?

FFI refers to the ability for code written in one language (the “host language,” such as Ruby), to access and invoke functions written in another language (the “guest language,” such as C). The term “foreign” refers to the fact that the functions come from another language and environment.

Depending on the language and its FFI support, you might also be able to access global named variables, automatically convert data types between the host and guest languages, and have code in the guest language invoke functions in the host language as callbacks.

In interpreted languages like Ruby, it’s usually not possible to use a library’s compile-time features like C preprocessor macros and constants (i.e. things #define’d in the library headers). This is because the FFI accesses the library’s binary code (e.g. its .so, .dylib, or .dll) directly, without compiling any code.

However, the FFI support in some compiled languages works by compiling down to C code; in these cases, you may be able to use compile-time features. It all depends on the language and how it implements FFI.

Why Use FFI?

As I mentioned earlier, using FFI has many benefits over writing extensions in C. When I migrated Rubygame from an extension written in to a wrapper using Ruby-FFI, the most significant benefits I observed were:

  • It’s much easier to write and maintain your code, because it’s written in your language of choice and can leverage the language’s high-level features and metaprogramming to abstract the wrapper code.
  • It’s easier for users to install, because they don’t need a C compiler or the development headers for the library you are wrapping. This is especially important if you want your library to be accessible to Windows and MacOS X users, because those platforms don’t have a C compiler installed by default.
  • In the case of Ruby, FFI code is more portable to other implementations than extensions written in C. For example, code using Ruby-FFI can run just as well on MRI (the “usual” Ruby implementation), JRuby, and Rubinius with no modifications.

Whatever your language of choice, if you need to interface with a library written in C, I highly recommend trying out FFI first.

When Not To Use FFI

As amazing as FFI is, it’s not always the right tool for the job. For example, a C extension may be a better approach in any of these scenarios:

  • You need to implement your own low-level or highly-optimized code. Ideally, the functions in the C library you are wrapping will do most of the heavy lifting, but if you need to write some custom code to directly process huge arrays of numerical or binary data, you might need to write code in C or another lower-level language to get the performance you want.
  • You need to perform some delicate callbacks from the guest language into the host language. Although it’s sometimes possible (depending on the host language’s FFI support) to perform callbacks, some kinds of complex callback function signatures can be quite tricky to satisfy through FFI.
  • The library makes heavy use of compile-time or preprocessor features, such as C macros. In the case of simple macros, you may be able to reimplement its behavior as a function in your language of choice. But if the library does some serious macro-fu, you might be better off just writing a C extension.

Constants created using #define can also be a slight nuisance, since most FFI systems will not be able to see them. You can re-define the constants in your own code, but be aware that if the library changes the values of those constants, you’ll need to update your code as well. This can make supporting multiple library versions a tricky endeavor — especially if the library’s version number itself is defined only as a preprocessor constant!

Making a C Library FFI Friendly

Looking at it from another angle, here’s a checklist for making a C library FFI friendly. It’s not hard, and it drastically increases the potential user base of the library.

  1. Always, always, always export the library’s version number(s) as either global const variables (e.g. three const uint8_t variables) or as a function that returns the version number(s) in a struct. It’s fine to also #define version numbers in the library headers, but #define alone is not sufficient.
  2. Either export important constants as global const variables (perhaps in addition to using enum or #define), or publicly document their values and make a commitment to announce any future changes well in advance.
  3. Provide function equivalents of any important preprocessor macros, and/or clearly document the purpose and functionality of each important macro, so that it can be reimplemented in another language.
  4. Keep callback signatures simple, shallow, and well-documented. This is especially important if callbacks are necessary to use the library’s core functionality.
  5. For libraries written in C++, consider also offering a C-compatible interface (using extern C { ... }), at least for the core functionality of the library. When C++ libraries are compiled, method names may be “mangled” in ways that are not entirely predictable, making it tricky to wrap them with FFI.

It’s worth noting that, besides making the library FFI friendly, these are all just general good practices that make the library more stable, mature, and accessible.

Reaching Beyond C

Many languages provide a C library that allows developers to interface with that language — for example, libobjc for Objective-C, liblua for Lua, and the Java Native Interface (JNI). Such libraries are typically used within programs written in C or C++, but if you’re feeling clever — and more than a bit foolhardy — you can actually use FFI to build a bridge from Ruby into the other language!

I used this technique in Rubygame to interface with Objective-C so I could make calls into MacOS X’s Cocoa UI framework. Here’s a barebones example of using Ruby-FFI to interface with Objective-C on MacOS X:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
# Load the 'ffi' gem.
require 'ffi'
 
module ObjC
  extend FFI::Library
 
  # Load the 'libobjc' library.
  ffi_lib 'objc'
 
  # Bind the 'sel_registerName' function, which accepts a String and
  # returns an equivalent Objective-C selector (i.e. message name).
  attach_function :sel_registerName, [:string], :pointer
 
  # Bind the 'objc_msgSend' function, which sends a message to an
  # Objective-C object. It accepts a pointer to the object being sent
  # the message, a pointer to a selector, and a varargs array of
  # arguments to be sent with the message. It returns a pointer to the
  # result of sending the message.
  attach_function :objc_msgSend, [:pointer, :pointer, :varargs], :pointer
 
  # A convenience method using objc_msgSend and sel_registerName to easily
  # send Objective-C messages from Ruby.
  def self.msgSend( id, selector, *args )
    selector = sel_registerName(selector) if selector.is_a? String
    return objc_msgSend( id, selector, *args )
  end
 
  # Bind the 'objc_getClass' function, which accepts the name of an
  # Objective-C class, and returns a pointer to that class.
  attach_function :objc_getClass, [:string], :pointer
end
 
 
module Cocoa
  extend FFI::Library
 
  # Load the Cocoa framework's binary code
  ffi_lib '/System/Library/Frameworks/Cocoa.framework/Cocoa'
 
  # Needed to properly set up the Objective-C environment.
  attach_function :NSApplicationLoad, [], :bool
  NSApplicationLoad()
 
  # Accepts a Ruby String and creates an equivalent NSString instance
  # and returns a pointer to it.
  def self.String_to_NSString( string )
    nsstring_class = ObjC.objc_getClass("NSString")
    ObjC.msgSend( nsstring_class, "stringWithUTF8String:",
                  :string, string )
  end
 
  # Accepts a pointer to an NSString object, and returns the string
  # contents as a Ruby String.
  def self.NSString_to_String( nsstring_pointer )
    c_string_pointer = ObjC.msgSend( nsstring_pointer, "UTF8String" )
    if c_string_pointer.null?
      return "(NULL)"
    else
      return c_string_pointer.read_string()
    end
  end
end
 
 
# Create a new empty NSMutableArray instance
nsmutablearray_class = ObjC.objc_getClass("NSMutableArray")
array = ObjC.msgSend( nsmutablearray_class, "array" )
 
# Add two NSString objects to the array
%w( Hello World ).each do |string|
  ObjC.msgSend( array, "addObject:",
                :pointer, Cocoa.String_to_NSString(string) )
end
 
# Print the array's description (analogous to Ruby's "inspect")
description = ObjC.msgSend( array, "description" )
puts Cocoa.NSString_to_String( description )
 
# The console output when this script is run:
#
#   (
#       Hello,
#       World
#   )

As you can see, connecting Ruby ↔ C ↔ Objective-C requires a significant amount of glue code. After all, you’re dealing with three different systems for objects/data and functions/methods. But, it can be done! And you can actually make it quite manageable and easy to use by investing in some useful abstractions, as I did for the real Rubygame code.

Of course, if you want to write Ruby code that interfaces with Objective-C, it’s probably easier to use MacRuby (or RubyMotion for iOS development). Likewise, JRuby is a solid platform for interfacing with Java code. But FFI can still be useful to provide broader platform support, or just for the intellectual challenge. (My code actually allowed JRuby code to interface with Cocoa. That’s a chain of Ruby ↔ Java Virtual Machine ↔ C ↔ Objective-C!)

Wrapping Up

As you can see, FFI can be a powerful tool in Ruby and other languages. If you need to wrap a library in another language, I would encourage you to reach for FFI first, and only resort to a C extension if some special circumstance makes in a necessity. Your users will thank you, and so will your future self as you maintain your code.