Tuesday, August 1, 2017

log4j trick when creating ConsoleAppender from Java code

›
Right way to do it: /** * Dynamically initialize a log4j console appender so we can log info to stdout * with specified level and p...
Monday, June 12, 2017

JavaOptions to pass to Spark driver and Snappy native library to pass to Java program

›
/usr/hdp JavaOptions to pass to Spark driver:  Here is a Hortonworks example: JavaOptions=-Dhdp.version=2.4.2.0-258 -Dspark.driver.extra...
Wednesday, May 10, 2017

Search anything from a bugzilla bug given a range of bug numbers

›
[root@centos my-data-set]# for ((i=13420;i<13500;i++)); do echo $i >> /tmp/search_result.txt; curl http://192.168.5.105/bugzilla/sh...
Tuesday, May 2, 2017

A simple way to clone 10,000 files to cluster

›
# Generate 10,000 files from one seed and put them into 100 subdirectories [hdfs@cdh1 tmp]$ for((i=1;i<=100;i++));do mkdir -p 10000fil...
Tuesday, April 25, 2017

Prepare Python machine learning environment on Centos 6.6 to train data

›
# major points: 1. Has to use Python 2.7, not 2.6.  But Centos 6.6 uses Python 2.6 for OS so upgrading to 2.7 is not a solution.  Need to in...
Tuesday, March 21, 2017

Mapreduce logs have weird behavior on HDP 2.3 Tez

›
When launching map reduce on Tez, we will not see our logs from HDP UI.  Click ‘History’ and got nothing although we can see Hadoop system ...
Monday, March 6, 2017

Additional jar files when running Spark under Hadoop YARN mode (CDH 5.10.0 with Scala 2.10 and Spark 1.6.0)

›
lrwxrwxrwx 1 root root     91 Feb 23 15:02 spark-core_2.10-1.6.0-cdh5.10.0.jar -> /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/ja...