Foreign Function Interface (FFI) for Erlang/OTP

Description

All the details are described in a draft EEP (Erlang Enhancement Proposal) (available formats: HTML and TXT source) (last update: Thu, 15 Nov 2007 17:55:04 +0100).

The proposal is being discussed as EEP-0007. (Note: the version hosted on erlang.org may be slightly out of date)

Implementation

The draft EEP has been implemented by patching the Erlang/OTP R11B-5 sources. Everything has been tested on GNU/Linux (i386, x86-64, powerpc, powerpc64).

You can download the changes as a single, big patch (see the diffstat) (last update: Tue, 29 Jul 2008 14:48:15 +0200).

Otherwise, you can download the changes as 14 small patches (for easier readability).

First patch series (last update: Wed, 12 Sep 2007 14:49:59 +0200):

00-ffi_call_3.diff: Adds the ffi:call/3 BIF.
01-erl_ddll_load_library_3-no-preload.diff: Adds the erl_ddll:load_library/3 BIF, without preloading support.
02-erl_ddll_load_library_3-preload.diff: Adds preloading support to the erl_ddll:load_library/3 BIF.
03-ffi_call_2.diff: Implements the ffi:call/2 BIF.

Second patch series, based on the Erlang-questions mailing list feedback (last update: Wed, 03 Oct 2007 10:34:04 +0200):

04-small-changes.diff: Some small and easy changes (ffi:call/* becomes ffi:raw_call/*, erl_ddll:load_library/2, erlang:open_port/1, void functions return 'void').
05-binary-support.diff: Adds support for passing/returning Erlang binaries to/from C functions (as ErlDrvBinary'es). Also adds some new C types (cstring, nonnull, off_t, pid_t).
06-erliovec-support.diff: Adds support for passing iolists to C functions (as ErlIOVec's).
07-preload-info.diff: Allows erl_ddll:info/2 to return information about preloaded functions.
08-high-level-api.diff: Implements the high-level, type-tagged FFI API.
09-hardcodes-and-checks.diff: Adds ffi_hardcodes.hrl, and type/range checks for type-tagged values.
10-i386-bignum-fix.diff: Fixes bignum conversions on i386.
11-smp-fixes.diff: SMP fixes (d'oh).

Third patch series, mostly based on developers feedback (last update: Tue, 29 Jul 2008 14:48:15 +0200):

12-ffi-library.diff: Moves the high-level FFI API from the kernel to the ffi library, and makes the build process warn when the FFI support could not be compiled. Also performs some minor beautification in erl_bif_ffi.c (removes a compiler warning).
13-bigendian-support.diff (last update: Mon, 03 Dec 2007 11:08:16 +0100): Adds support for bigendian architectures.
14-smp-optimizations.diff (last update: Tue, 29 Jul 2008 14:48:15 +0200): Decouples port locking and refcounting in FFI BIFs, increasing SMP performance.

The small patches are incremental (i.e. they should be applied in the given order) and self-contained (every patch leaves a working Erlang/OTP source tree, with the related enhancements).

Building instructions

You will need all the standard dependencies for compiling the Erlang/OTP sources, plus GNU Autoconf and libffi (the GCC FFI library). If you are a Debian/Ubuntu user, you can install them easily:

apt-get install autoconf libffi4-dev

To patch and build Erlang/OTP with the FFI extension:

Download the Erlang/OTP R11B-5 sources from here (either the full or minimal version), and extract the tarball;
cd in the main directory of the source tree;
Apply the patch(es) above:
```
patch -p1 < patch-filename.diff
```
Update the Erlang/OTP configuration scripts:
```
./otp_build autoconf
```
Follow the standard Erlang/OTP build procedure.

Testing the FFI extension

ffi-test.tar.gz: This is a simple, EUnit-based test suite for the FFI extension. The tarball contains a README file with instructions. (last update: Mon, 03 Dec 2007 11:08:16 +0100)

Discussion and comments

The FFI extension proposal has been sent to the Erlang-questions mailing list. You can join the discussion there, or write directly to me (see the email address at the end of the page).

A first summary of the discussion so far is available here.

News and updates

Tue, 29 Jul 2008 14:48:15 +0200: New patch (no. 14) that increases SMP performance of FFI BIFs.
Mon, 03 Dec 2007 11:08:16 +0100: Support for bigendian architectures. Thanks to Alan Louis Scheinine for his help and his GNU/Linux-based Playstation 3!
Thu, 15 Nov 2007 10:33:16 +0100: Updated EEP and patches, according to developers' feedback. (Special thanks to Serge Aleynikov!). A status report is available on the Erlang-questions mailing list.
Tue, 16 Oct 2007 16:19:39 +0200: Fixed the big FFI patch (it was corrupted by a script). Thanks to Serge Aleynikov for the report!
Wed, 03 Oct 2007 10:34:04 +0200: Bugfix for SMP locking issues: 11-smp-fixes.diff (the other patches from 00 to 10 have not been altered).
Sat, 29 Sep 2007 12:00:52 +0200: New patches and draft EEP, based on the the Erlang-questions mailing list feedback. An announcement, and a FFI discussion summary, is available here.
Thu, 13 Sep 2007 17:02:33 +0200: A summary of the discussion about the Erlang FFI proposal is available on the Erlang-questions mailing list.
Wed, 12 Sep 2007 17:12:49 +0200: The FFI proposal was accepted as a draft EEP-0007.
Wed, 12 Sep 2007 14:49:59 +0200: Bugfix. The FFI patches have been updated accordingly. If you have still applied the old ones, you can download the bugfix alone.

Copyright (C) 2007 by CRS4 - Center for Advanced Studies, Research and Development in Sardinia
Author: Alceste Scalas <alceste (at) crs4 (dot) it>