Calling native (c/c++) libraries from Kepler via the Java Native Interface (JNI)

Author: Chad Berkley (berkley@nceas.ucsb.edu)
01/04/2005

CVS Info:
'$RCSfile: JNIHowTo.txt,v $'
'$Author: berkley $'
'$Date: 2005/02/16 17:11:18 $'
'$Revision: 1.3 $'
--------------------------------------------------------------------------------

Introduction
------------
In this document, I attempt to explain, from start to finish, how to enable
native library calls within a Kepler actor.  This process is programming 
intensive and is not intended for a user with no programming experience.  
I'll start with building the needed JNI interfaces and header files
then go through compilation and inserting the JNI calls into your actor.

Creating a link to native code via JNI is a two prong process.  The first prong
is setting up the native interface and the second prong is setting up the 
java interface.  If either of these are done incorrectly, or if the variable
and method naming across the two does not match, the JNI calls will fail.  
Pay specific attention to case and the naming conventions associated with
setting up the native interface. 

Since we are now dealing with native code which does not execute on the 
Java Virtual Machine (JVM), we also have to deal with cross-platform issues.  
I have noted where discrepencies between Win32, Linux and OS X may lie.

One further note.  This tutorial focuses on C/C++ only.  I have not done JNI
calls to other native languages, although it is possible, so you're on your
own if you are not calling a C/C++ library.

Step 1 - Preparing the Java interface
-------------------------------------
The Java part of the JNI process is by far the easiest part.  Java requires
only one class to show the JVM how to access native code.  This class
is commonly referred to as the "JniGlue".  Here's and example glue class:

==Example==
package org.ecoinformatics.seek;
class MyJniGlue
{
  /**
   * MyMethod1 glue
   */
  public native String MyMethod1(
    String in1,
    String in2
  );
  
  /**
   * MyMethod2 glue
   */
  public native void MyMethod2(
    String in1
  );

  /**
   * load the library
   */
  static
  {
    System.loadLibrary("mylibrary");
  }
}
==End Example==

This is a basic java class with a few extra keywords.  This class is in the
package org.ecoinformatics.seek.  It has two public *native* methods (MyMethod1
returns a String and takes two String arguments and MyMethod2 returns nothing
and takes one String argument) and a static section.  The native 
keyword tells the JVM that these methods are not defined in java, but 
rather have to be called natively.  The static section tells the system 
to load the native library given in the argument to loadLibrary.  
This will be the name of the dll or shared object file you will eventually 
create in the next step.

This class file would be named MyJniGlue.java and it should be put in a source
directory org/ecoinformatics/seek to match its package name.

Step 2 - Preparing the native code
----------------------------------
Step 2 Section 1 - Creating the header file
-------------------------------------------
The first thing you need to do is create a JNI specific header (.h) file for 
your C code.  You can have java do this for you via the javah utility.  I
like to do it manually so I know what's going on.  Here's an example:

==Example==
#include <jni.h>

#ifndef _Included_org_ecoinformatics_seek_MyJniGlue
#define _Included_org_ecoinformatics_seek_MyJniGlue
#ifdef __cplusplus
extern "C" {
#endif

/*
 * Class:     org_ecoinformatics_seek_MyJniGlue
 * Method:    MyMethod1
 */
JNIEXPORT jstring JNICALL Java_org_ecoinformatics_seek_MyJniGlue_MyMethod1
(JNIEnv *env, jobject obj, jstring in1, jstring in2);

/*
 * Class:     org_ecoinformatics_seek_MyJniGlue
 * Method:    MyMethod2
 */
JNIEXPORT void JNICALL Java_org_ecoinformatics_seek_MyJniGlue_MyMethod2
(JNIEnv *env, jobject obj, jstring in1);

#ifdef __cplusplus
}
#endif
#endif
==End Example==

This header file would be saved to a file named 
org_ecoinformatics_seek_MyJniGlue.h.  The name of the file with the underscores
comes from the package name of your java JniGlue class.  The header file
*MUST* be named like this and should be put in the same directory as the
C/C++ code that you are going to compile.  You should also note the similar
naming convention of the method declarations.  The names *MUST* match up
with the naming scheme from you JniGlue file.  The method names are in the form
Java_<package_name_seperated_with_underscores>_<className>_<methodName>.

Inside the method signatures themselves, the first two arguments are required.
JNIEnv *env and jobject obj are both used by the JNI interface and cannot
be changed.  Add your custom arguments after these (these arguments should match
what you declared in the JniGlue class).  Note the use the 
'j' types instead of normal type names.  Here is a list of 'j' types:

Type      'J' Type       Description
----      --------       -----------
boolean   jboolean       Unsigned 8-bit integer C/C++ type
byte      jbyte          Signed 8-bit integer C/C++ type
char      jchar          Unsigned 16-bit integer C/C++ type
short     jshort         Signed 16-bit integer C/C++ type
int       jint           Signed 32-bit integer C/C++ type
long      jlong          Signed 64-bit integer C/C++ type
float     jfloat         32-bit floating point C/C++ type
double    jdouble        64-bit floating point C/C++ type
Object    jobject        Any Java object, or a type that doesn’t have a corresponding native type
Class     jclass         Class objects
String    jstring        String objects
Object[]  jobjectArray   Array of objects of any type
boolean[] jbooleanArray	  Array of booleans
byte[]    jbyteArray     Array of bytes
char[]    jcharArray     Array of chars
short[]   jshortArray    Array of shorts
int[]     jintArray      Array of ints
long[]    jlongArray     Array of longs
float[]   jfloatArray    Array of floats
double[]  jdoubleArray   Array of doubles


Step 2 Section 2 - Creating the C/C++ file
------------------------------------------
Creating (or altering) your C/C++ code is pretty straight forward as long as
you get the naming schemes correctly.  Any deviation from your naming schemes
in the JniGlue or the header file will cause the jni links not to work at
runtime.  Here's some example C code:

==Example==
//include the jni formatted header file and the jni.h file
#include "org_ecoinformatics_seek_MyJniGlue.h"
#include <jni.h>

//JNI function call for GDALTranslate
JNIEXPORT jstring JNICALL Java_org_ecoinformatics_seek_MyJniGlue_MyMethod1
(JNIEnv *env, jobject obj, jstring in1, jstring in2)
{
  jboolean iscopy;
  //translate the j type args to c types
  char * in1c = (char*) env->GetStringUTFChars(in1, &iscopy);
  char * in2c = (char*) env->GetStringUTFChars(in2, &iscopy);

  char * args[] = {"gdal_translate", "-ot", ot, "-of", of, iFilename, oFilename};
  int argv = 7;

  char* returnval = doSomething(in1c, in2c);
  //translate the return val to a j type
  jstring returnval_jstring = env->NewStringUTF(returnval);
  return returnval_jstring;
}

//JNI function call for GDALWarp
JNIEXPORT jstring JNICALL Java_org_ecoinformatics_seek_MyJniGlue_MyMethod2
(JNIEnv *env, jobject obj, jstring in1)
{
  jboolean iscopy;
  //translate the j type argument to a char*
  char * ip = (char*) env->GetStringUTFChars(in1, &iscopy);
  //no return value so just do something then return void.
  doSomething(ip);
}
==End Example==

The two #include lines in this example must be present.  The first includes
the header file that we created in Step 2 Section 1 and the second includes
the jni.h file that should be part of your java distrobution (it usually 
is in the $JAVA_HOME/include directory).

The funcions must start with the keyword JNIEXPORT followed by the return type
followed by the keyword JNICALL followed by the full name of the method (note
the naming convention) followed by the arguments.  The body of the function
is normal C/C++ code except that you must convert the j type to a C type to use
it.  This is done with the env variable.  More information on the conversion
methods can be found here:
http://java.sun.com/docs/books/tutorial/native1.1/implementing/index.html

You can see in this example that both MyMethod1 and MyMethod2 are implemented.
The names are kept consistent with the naming we used in the header and the
JniGlue files.  A common way of segmenting the code is to use the jni
functions as wrappers (i.e. just make one function call inside your jni 
function) for other native functions or methods, thus keeping them logically 
seperated in the code.  

Now that we have all this code written, what do we do with it?  Let's compile...

Step 3 - Compiling the native code
----------------------------------
In order for java to be able to run native code, it must be compiled into a 
dynamic library (dynamic link library (dll) on windows, shared object (so) 
on linux, dylib on Mac OSX).  This dynamic library then needs to be put into
the library path so that the system can find it at runtime in order to load
its methods and members.  

The first thing we have to do is compile the code to object files.  This is 
where things get platform specific so I'll divide it up into the different
platforms.  Note you should have your .c or .cpp file in the same directory
as your .h file.

Linux
-----
Building JNI libraries on linux is by far the easiest of the 3 OS's that I talk 
about here.  I use gcc on linux.  It's free, it's standard and it's fast.  
This should compile your code:

cd into the directory where your cpp and h files are and issue this command:
gcc -c -I /usr/local/include MyLibrary.cpp

You can specify more -I (include) directories if you have .h files located
elsewhere that you need to include.  The output of this command should be
MyLibrary.o.  Next we need to build the shared object library.  Issue the 
command: 

gcc -L /usr/local/lib --shared -o libmylibrary.so *.o

Note that you can add more -L directories if you want it to search more than
one place for external resources.  Also, if you want gcc to load a specific
lib name, use the flag -l<libname>.  For example, if you issue this command: 

gcc -L /usr/local/lib --shared -o libmylibrary.so *.o -lgdal -lgeos

gcc will search /usr/local/lib for the libraries libgdal.so and libgeos.so.  
Note that when you specify the library name with -l, you don't include the
word 'lib' before them.  You should now have a libmylibrary.so file.  If you 
don't, read the compiler errors and make sure all your names match up
right.  If it can't find libraries or header files, you might need to
specify more -I or -L directories.

Skip ahead to Step 4 if you don't want to compile on OSX or Windows.

Mac OSX
-------
Mac OSX is based on FreeBSD with Apple exensions so the shared library process
is a bit different than linux but with great similarities.  You must have the 
Apple developer kit that includes the gcc compiler for this to work.  You 
also need to have Mac Terminal or another shell emulator installed.  

To get started, open a terminal and change to the directory where your cpp and
h files are.  To compile the code to an object file, issue the following
command:

gcc -c -I /usr/local/include -I /System/Library/Frameframework/Versions/1.4.2/Headers MyLibrary.cpp

Note the second -I argument is telling gcc where the jni.h file is.  On the 
Mac, Java goes into some really weird directory structures.  The headers
can usually be found in the Headers directory inside the version of the 
JVM that you are using. 

After this command successfully executes, you should have a .o file.  Next you 
need to compile the dynamic library.  Issue this command: 

gcc -Wall -pipe -arch ppc -o libmylib.dylib *.o -bundle -lstdc++

This should create your dylib file.  If the compile or link fails look at the 
errors and try to trace the problem.  If it is not finding an external function 
or library, you may have to add a -L argument and specify the path to the
library.

Skip ahead to Step 4 if you don't want to compile on Windows.

Windows
-------
Windows is the trickiest of all the OS's to build on.  While technically, you 
should be able to use Cygwin and the gcc compiler built into it to build
jni dll's, I've never gotten it to work to my satisfaction.  For this reason,
I use Microsoft Visual C++ (version 6 or greater).  It's a beast of a compiler
to use, but once you figure out the small idiosyncracies, it does work well.
If you don't have MS VC++, I would encourage you to try to use Cygwin (which
is free) and loosely follow the linux instructions above.

When you start MSVC++, it'll ask you which kind of project you want to create.
Choose Win32 Dynamic-link library then tell it to create an empty project.  
Go to the Project menu and add both your source (c/cpp) and header (.h) files
to the project.  If you go to the file view on the pain on the left, you should
be able to see the files you added.  Next, the compiler and linker need to be
told where to find the jni.h file and any other libraries or headers you're
compiling against or linking to.  There are several places where you provide
this information.  Since every project is different, I can't tell you exactly
where to put your path information.  The places to look are as follows:

*Project/Settings Menu.  The Link tab contains the paths to libraries that you 
want to link against.  You can also set many other project specific settings
here.  
*Tools/Options Menu.  The Directories tab contains the search path for header
files.  Add any directories holding header files (including the directory 
housing your jni.h file) here.  

When you think you have all of your search paths set correctly, click the build
all button and check the error log at the bottom of the screen.  If it builds
successfully, you should get a dll file put inside the Debug directory inside
your project directory.  If you get errors it's probably because it can't
find header files or libraries to link against.  Go back and check your paths.

Step 4 - Writing your actor and compiling the java code
-------------------------------------------------------
Now that we have all of these pieces, we need to put them together and actually
call the JNI code from within an actor.  Here is a very basic actor that
calls the jni code:

==Example==
package org.ecoinformatics.seek;

import ptolemy.data.StringToken;
import ptolemy.data.IntToken;
import ptolemy.data.type.BaseType;
import ptolemy.kernel.CompositeEntity;
import ptolemy.actor.TypedAtomicActor;
import ptolemy.actor.TypedIOPort;
import ptolemy.data.expr.Parameter;
import ptolemy.data.expr.StringParameter;
import ptolemy.kernel.Port;

public class MyActor extends TypedAtomicActor
{
  //port declarations
  //input ports
  public TypedIOPort in1 = new TypedIOPort(this, "in1",
    true, false);
  public TypedIOPort in2 = new TypedIOPort(this, "in2",
    true, false);
  //output ports
  public TypedIOPort out1 = new TypedIOPort(this, "out1",
    false, true);
  
  /**
   * Constructor
   */
  public GDALTranslateActor(CompositeEntity container, String name)
    throws NameDuplicationException, IllegalActionException
  {
    super(container, name);
    //set the types of the ports.
    in1.setTypeEquals(BaseType.STRING);
    in2.setTypeEquals(BaseType.STRING);
    out1.setTypeEquals(BaseType.STRING);
  }

  /**
   * Initialize
   */
  public void initialize()
    throws IllegalActionException
  {
  }

  /**
   * Prefire
   */
  public boolean prefire()
    throws IllegalActionException
  {
    return super.prefire();
  }

  /**
   * Fire.  This is what gets called when the actor actually gets executed.
   */
  public void fire()
    throws IllegalActionException
  {
    super.fire();
    //get the inputs from the ports
    StringToken in1Token = (StringToken)in1.get(0);
    StringToken in2Token = (StringToken)in2.get(0);
    
    //instantiate the jniglue object
    MyJniGlue glue = new MyJniGlue();
    //call MyMethod1 from the glue object
    String results = glue.MyMethod1(outputTypeStr, outputFormatStr,
      inputFilenameStr, outFile);
    //call MyMethod2 from the glue object
    glue.MyMethod2(in1);
    
    //send out the output of MyMethod1 on the output port.
    out1.broadcast(new StringToken(results));
  }
}
==End Example==

In the fire() method, The MyJniGlue object is instantiated into a variable
called glue.  Your native methods that were defined in the JniGlue class
can be called from the glue variable.  

This actor should be put in the same package (and hence, directory) as the 
MyJniGlue.java file (though it's not required).  Once it's there, you can 
compile all of the java code together.  Kepler uses the ant build system.
You need to install ant before you can build kepler.  It is freely available
from http://jakarata.apache.org/ant.  Once you have an installed and 
configured, change to the root of the kepler directory and run 
'ant compile-dev'.  This should compile the entire kepler project as well
as your code.  

Before we can run kepler, we need to put your native library in a place
where java can find it.  In kepler, we have designated the kepler/lib
directory for this purpose (among others).  Copy your dll/dylib/so file into
the kepler/lib directory.

Adding your actor to the actor library
---------------------------------------

Next we need to add your actor to the actor library so that it is accessible
via the vergil user interface.  To do this, you need to edit two kepler
configuration files.  In the future, this will be handled by a graphical
user interface.  The two files are both in the directory 
kepler/configs/ptolemy/configs/kepler.  The first one is 
basicKeplerActorLibrary.xml.  Open this file and scroll to the very bottom.
After the last </entity> tag but before the last </group> tag, make an entry
that looks like this:

<entity name="MyActor" class="org.ecoinformatics.seek.MyActor">
  <property name="entityId" class="org.ecoinformatics.seek.util.NamedObjId"
            value="urn:lsid:ecoinformatics.org:kepler.274.1"/>
</entity>

In the value field of the entityId property, you need to change the number
so that it is unique.  For instance, if the last ID is 
urn:lsid:ecoinformatics.org:kepler.273.1, make yours 
urn:lsid:ecoinformatics.org:kepler.274.1.  Save this file.

The next file you have to edit is in the same directory and is called 
annotations.owl.  Open this file and scroll to the bottom.  After the last
</asc:Annotation> tag but before the </rdf:RDF> tag make an entry that looks
like this : 

<asc:Annotation rdf:ID="A332.1"> <!--CHANGE THE ID HERE (i.e. A332.1 to A333.1)-->
  <asc:lsid>urn:lsid:ecoinformatics.org:SMS:A332:1</asc:lsid> <!--CHANGE THE ID HERE (i.e. A332.1 to A333.1)-->
  <asc:annotates>
    <asc:ItemTag>
      <rdf:type rdf:resource="&ont;InternalInputActor"/>
      <asc:taggedItem>
        <asc:Actor>
          <!--MAKE THE ID HERE MATCH THE ID FROM THE basicKeplerActorLibrary.xml FILE-->
          <asc:lsid>urn:lsid:ecoinformatics.org:kepler.274.1</asc:lsid> 
       </asc:Actor>
      </asc:taggedItem>
    </asc:ItemTag>
  </asc:annotates>
</asc:Annotation>

Once again, make the IDs unique where I have noted in the text above.  Save
the file.  Note that this annotation is going to put your actor in the 
"Internal Inputs" category.  You can change this later by changing the 
rdf:resource attribute.

Now it's time to see if all of this works.  CD to the root of the kepler
directory and issue the command 'ant run-dev'.  Kepler should recompile itself
and start to run.  Once kepler comes up, go to the file menu and choose
new/graph editor.  You should be able to search for your actor by name
and drag it to the canvas.  Give it the appropriate inputs and add a director
then run the workflow.  If everything works well, your native code should 
execute and you should get whatever output you were expecting.  If you 
get an unsatisfied link exception, or if kepler crashes entirely, there is
a problem with one of your jni files.  Go back and make sure all of the 
method names are spelled correctly (including case) and make sure all your
paths are correct.  You may need to recompile your native code or your java
code or both.  

If you need any further help, contact kepler-dev@ecoinformatics.org or
berkley@nceas.ucsb.edu.






























