Friday, February 2, 2018

Increase map reduce timeout value and JVM heap size, or set different log level for map reduce job

›
Update the mapred-site.xml on client machine only: For increasing heap size:   <property>     <name>mapreduce.map.java.opts...
Wednesday, January 3, 2018

Workaround to the S3 colon issue on Hadoop MapReduce

›
When S3 file name or path contains colon (:), you will get following exception when you submit your map reduce job: 2018-01-03/00:43:52.63...
Tuesday, November 7, 2017

Command to use jconsole to monitor java program remotely

›
java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=18745 -Dcom.sun.ma...
Tuesday, August 1, 2017

log4j trick when creating ConsoleAppender from Java code

›
Right way to do it: /** * Dynamically initialize a log4j console appender so we can log info to stdout * with specified level and p...
Monday, June 12, 2017

JavaOptions to pass to Spark driver and Snappy native library to pass to Java program

›
/usr/hdp JavaOptions to pass to Spark driver:  Here is a Hortonworks example: JavaOptions=-Dhdp.version=2.4.2.0-258 -Dspark.driver.extra...
Wednesday, May 10, 2017

Search anything from a bugzilla bug given a range of bug numbers

›
[root@centos my-data-set]# for ((i=13420;i<13500;i++)); do echo $i >> /tmp/search_result.txt; curl http://192.168.5.105/bugzilla/sh...
Tuesday, May 2, 2017

A simple way to clone 10,000 files to cluster

›
# Generate 10,000 files from one seed and put them into 100 subdirectories [hdfs@cdh1 tmp]$ for((i=1;i<=100;i++));do mkdir -p 10000fil...