Tuesday, November 7, 2017

Command to use jconsole to monitor java program remotely

›
java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=18745 -Dcom.sun.ma...
Tuesday, August 1, 2017

log4j trick when creating ConsoleAppender from Java code

›
Right way to do it: /** * Dynamically initialize a log4j console appender so we can log info to stdout * with specified level and p...
Monday, June 12, 2017

JavaOptions to pass to Spark driver and Snappy native library to pass to Java program

›
/usr/hdp JavaOptions to pass to Spark driver:  Here is a Hortonworks example: JavaOptions=-Dhdp.version=2.4.2.0-258 -Dspark.driver.extra...
Wednesday, May 10, 2017

Search anything from a bugzilla bug given a range of bug numbers

›
[root@centos my-data-set]# for ((i=13420;i<13500;i++)); do echo $i >> /tmp/search_result.txt; curl http://192.168.5.105/bugzilla/sh...
Tuesday, May 2, 2017

A simple way to clone 10,000 files to cluster

›
# Generate 10,000 files from one seed and put them into 100 subdirectories [hdfs@cdh1 tmp]$ for((i=1;i<=100;i++));do mkdir -p 10000fil...
Tuesday, April 25, 2017

Prepare Python machine learning environment on Centos 6.6 to train data

›
# major points: 1. Has to use Python 2.7, not 2.6.  But Centos 6.6 uses Python 2.6 for OS so upgrading to 2.7 is not a solution.  Need to in...
Tuesday, March 21, 2017

Mapreduce logs have weird behavior on HDP 2.3 Tez

›
When launching map reduce on Tez, we will not see our logs from HDP UI.  Click ‘History’ and got nothing although we can see Hadoop system ...