Thursday, June 28, 2018

Inotify

›
Start from CDH 5.4 and Hadoop 2.6. Only superuser can do it. No filtering. org.apache.hadoop.hdfs.inotify.Event.* org.apache.hadoop.h...
Monday, April 30, 2018

HCatalog: Find Hive table name from file path

›
I believe I have talked about this logic multiple times.  But still be asked, so decide to document it so I will only need to send a link to...
Friday, March 30, 2018

Different locations of map reduce logs

›
HDP: sudo -u hdfs hadoop fs -ls /app-logs/hdfs/logs (7 days) CDH5: sudo -u hdfs hadoop fs -ls /tmp/logs/hdfs/logs/ (7 ...
Thursday, March 29, 2018

SPARK_HOME, etc.

›
export SPARK_HOME==/usr/hdp/2.6.3.0-235/spark2 JavaOptions=-Dhdp.version=2.6.3.0-235 -Dspark.driver.extraJavaOptions=-Dhdp.version=2.6.3...
Thursday, March 22, 2018

Additional steps for Ranger/Kerberos enabled Hadoop

›
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_security/content/ch06s01s01s01.html Add values for the following properties i...
Friday, February 2, 2018

Increase map reduce timeout value and JVM heap size, or set different log level for map reduce job

›
Update the mapred-site.xml on client machine only: For increasing heap size:   <property>     <name>mapreduce.map.java.opts...
Wednesday, January 3, 2018

Workaround to the S3 colon issue on Hadoop MapReduce

›
When S3 file name or path contains colon (:), you will get following exception when you submit your map reduce job: 2018-01-03/00:43:52.63...