pagetop
Javablog
by Java coders, for Java coders RSS

Making JNI cross platform

May 19th, 2007 by Sam

Recently I’ve had the pleasure of using the JNI to speed up some of our numerical code. Unfortunately, I encountered some cross-platform issues before writing a line of C… the name mapping for a library called NAME to a filename goes like this in System.loadLibrary:-

  • Linux: libNAME.so
  • Windows: NAME.dll
  • Apple: libNAME.jnilib
  • SUN: libNAME.so

There is a name collision between Linux and SUN, but it gets worse… it doesn’t make any reference to the CPU architecture. This is a big deal in the modern mish-mash of 32 and 64 bit machines, but luckily there is a simple solution.

The standard solution is to place native libraries in a directory structure depending on the operating system and architecture, then specify this path to the JVM on startup. This has a big disadvantage as it doesn’t allow you to include native libraries in jar files. Time for an alternative.

It is not possible to create wrappers for the System.load* methods as underneath the hood they need to know which class called them. But it is possible to create a pre-processing method for the name of the library. I wrote a simple method that takes in NAME and converts it to:-

  • Apple (G3, G4): name-apple-ppc
  • Apple (G5): name-apple-ppc_64
  • Apple (Intel): name-apple-x86
  • Linux (i686): name-linux-x86
  • Linux (Intel/AMD 64): name-linux-x86_64
  • Linux (sparc): name-linux-sparc
  • Linux (PPC 32 bit): name-linux-ppc
  • Linux (PPC 64 bit): name-linux-ppc_64
  • Windows XP/Vista (i686): name-windows-x86
  • Windows XP/Vista (Intel/AMD 64): name-windows-x86_64
  • Sun Solaris (Blade): name-sun-sparc
  • Sun Solaris (Intel 64 bit): name-sun-x86_64

Which now allows for a flat list of library names, allowing you to create separate Bundle-NativeCode entries in your MANIFEST file and ship the native libs in the jar. Note that Apple have introduced their own file format that allows a .jnilib file to contain the binaries for all Apple platforms. The unfortunate thing for my native code is that it is Fortran and Apple don’t supply a compiler that will create multi-platform output.

Writing the JNI code itself is a completely different matter, but one that isn’t that bad. Most j* C types are equivalent to the standard ones with the exception of jboolean which is a byte and C/Fortran uses int.

Detecting operating system and architecture is actually quite simple, so I’ll just write the code here, as most forum threads are about detecting a particular system rather than asking for a more general solution.

Detecting Operating System

The operating system is contained within the system property os.name, but it isn’t in a favourable format and only by experimentation did I discover the common values. If any code is going to be making use of the operating system it is better to create an enum

public enum OSType {
    APPLE, LINUX, SUN, UNKNOWN, WINDOWS
};

and use a helper method (this is public, but in reality my class calculates this and places it in a public static final field on loading)

public static OSType calculateOS() {
    String osName = System.getProperty("os.name").toLowerCase();
    assert osName != null;
    if (osName.startsWith("mac os x")) {
        return OSType.APPLE;
    }
    if (osName.startsWith("windows")) {
        return OSType.WINDOWS;
    }
    if (osName.startsWith("linux")) {
        return OSType.LINUX;
    }
    if (osName.startsWith("sun")) {
        return OSType.SUN;
    }
    return OSType.UNKNOWN;
}

I’m probably missing a few, like AIX.

Detecting Architecture

The cpu detection is also pretty much the same, but there are a lot of inconsistencies in naming conventions. For example, a 64 bit processor can be called AMD64, X86-64, X64 or Intel 64. I’m probably missing quite a few of the alternatives here…

public enum ARCHType {
    PPC, PPC_64, SPARC, UNKNOWN, X86, X86_64
};
 
public static ARCHType calculateArch() {
    String osArch = System.getProperty("os.arch").toLowerCase();
    assert osArch != null;
    if (osArch.equals("i386")) {
        return ARCHType.X86;
    }
    if (osArch.startsWith("amd64") || osArch.startsWith("x86_64"))
        return ARCHType.X86_64;
    }
    if (osArch.equals("ppc")) {
        return ARCHType.PPC;
    }
    if (osArch.startsWith("ppc")) {
        return ARCHType.PPC_64;
    }
    if (osArch.startsWith("sparc")) {
        return ARCHType.SPARC;
    }
    return ARCHType.UNKNOWN;
}

This entry was posted by by Sam on Saturday, May 19th, 2007 at 12:18 pm, and is filed under Apple, CPU, JNI, Java, Linux, SUN, Windows. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.



6 comments on “Making JNI cross platform”

Nice post but why do you think “This has a big disadvantage as it doesn’t allow you to include native libraries in jar files.”? I dont see any BIG disadvantage. If it is a big project, you should already have an installer which copies required jars and dlls to installation directory. And for small projects you can directly give specific jars + dlls for specific os+architecture.

But you are right that naming is a problem (although not a big one).

Out of interest, what was the mathematics operation you were speeding up by calling JNI ?

I probably exaggerated by saying it’s a big disadvantage not to be able to ship native libs in the jar, but it is certainly an inconvenience. I prefer the idea of one file per version (for small to medium builds), rather than one file per version per OS per CPU, as that defeats the point of compile once run anywhere.

Incidentally, I would never encourage use of a JNI library as a complete replacement for a Java library… we always have a pure Java alternative sitting there for automatic use if the JNI version isn’t available.

Even in our build system it is also easier to have a flat list of files… a directory structure would mean having to get all the developers to setup system variables for each JNI they may be running.

Paul, it’s all matrix stuff… a Fortran library that sits atop of BLAS and LAPACK. We got a 20x speedup using the JNI! The matrix library MTJ has a JNI substitute for some of its internal calculations, but it actually has some serious bugs in its JNI code… I’ve been in contact with the author about a few other bugs, so I’ll probably collate a big patch at some point and send it over.

I don’t expect the JNI to work as fast on JVMs that do not support pinning (e.g. the IBM JVM) as it involves GB arrays being passed between Java and C… we explicitly check for that.

Hello there Sam. I notice you are using MTJ and JNI matrix operations. I fixed some bugs in MTJ, and see you have too. We need to move MTJ to a new server. Can you contact me? Thanks!

Have a look at one-jar It supports a single jar for different os.arch/os.name with native libraries.

Leave a comment

Markdown is supported.

To include code snippets in your comment, use

<pre><code># lang java
... code here ...
</code></pre>

or use 4 spaces at the start of the line instead of using code and pre tags.

Comment feed: RSS