Monday, June 12, 2017

JavaOptions to pass to Spark driver and Snappy native library to pass to Java program

/usr/hdp

JavaOptions to pass to Spark driver:  Here is a Hortonworks example:

JavaOptions=-Dhdp.version=2.4.2.0-258 -Dspark.driver.extraJavaOptions=-Dhdp.version=2.4.2.0-258

Snappy native library to pass to Java program.  Here is an example for CDH5:

1. Copy /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hadoop/lib/native/* to your java program machine and put them to any directory such as /opt/native
2. Run your Java program with following Java options:
JavaOptions=-Djava.library.path=/opt/native