Getting RocksDB working on Raspberry PI (Unsatisfied linker error when trying to run Kafka Streams)

 If you are here its probably because you've tried to get RocksDB working on a Raspberry PI and had the following exception:

Exception in thread "main-broker-b066f428-2e48-4d73-91cd-aab782bd9c4c-StreamThread-1" java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni7453541812184957798.so: /tmp/librocksdbjni7453541812184957798.so: cannot open shared object file: No such file or directory (Possible cause: can't load IA 32 .so on a ARM platform)

at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)

at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2452)

The reason for this is that your rocksdbjni jar file doesn't include the required shared library (.so file) for the ARM platform, it doesn't include x86 for Linux, a Windows DLL and a PPC(!) library but not for Linux so you are going to have to roll your own.

Step 1 - Doing the apt-gets

You have to do the build ON the Raspberry PI as its C code that needs to be compiled and the instructions for rocks aren't overly helpful as they assume a couple of things are already installed.

The following appear to be required:

sudo apt-get install cmake

sudo apt-get install vagrant

sudo apt-get update --fix-missing

sudo apt-get install vagrant

The vagrant part appears to be not actually required but I had issues when it wasn't installed so it is probably a dependency of the apt-get pulls in.

Step 2 - getting the code for RocksDB

Now you are going to have to download the code for RocksDB, I hope you've got a fairly large SD card as this will take up around a gig of space all told

Make a directory under your home directory (I call mine 'dev') and change into the directory (cd dev).

git clone https://github.com/facebook/rocksdb.git rocksdb

Then change directory into rocksdb.  If you are a Java developer you are possibly about to discover for the first time a tool that is why Java people created ant and maven because they REALLY hated how it worked...

Step 3 - running make

Then you want to run 

make rocksdbjavastaticrelease

Which will take a while but produce you a nice JNI jar file for your Raspberry PI that you can add to your classpath and you will be away.... the jar file will be under the "target" directory under rocksdb, so if you are on the default install with user "pi" it will be 

/home/pi/dev/rocksdb/java/target/rocksdbjni-6.12.0-linux32.jar

Copy that file into your classpath and rocks DB should now work fine

Step 4 - if running a Kafka Streams application

I'm sure there is a logical reason for this and that I could have created another work around, but having run the original and had the linker error when it accessed RocksDB there was a new error when running Kafka Streams:

Exception in thread "main-broker-00a490b4-b50a-4cc4-aded-d6324ad0f291-StreamThread-1" org.apache.kafka.streams.errors.ProcessorStateException: Error opening store reading-topic-STATE-STORE-0000000000 at location /tmp/kafka-streams/main-broker/0_0/rocksdb/reading-topic-STATE-STORE-0000000000

at org.apache.kafka.streams.state.internals.RocksDBTimestampedStore.openRocksDB(RocksDBTimestampedStore.java:87)

at org.apache.kafka.streams.state.internals.RocksDBStore.openDB(RocksDBStore.java:188)

at org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:224)

at org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)

at org.apache.kafka.streams.state.internals.ChangeLoggingKeyValueBytesStore.init(ChangeLoggingKeyValueBytesStore.java:42)

This is weird as it had never successfully run (as it couldn't create the DB), but for some reason there is state somewhere (I assume in the broker) that still exists for the app id, I assume because that runs before the linker.  So the fix there is to change "application.id" in the kafka configuration file to be something new and it should all run fine.