Monday, June 12, 2017

JavaOptions to pass to Spark driver and Snappy native library to pass to Java program

›
/usr/hdp JavaOptions to pass to Spark driver:  Here is a Hortonworks example: JavaOptions=-Dhdp.version=2.4.2.0-258 -Dspark.driver.extra...
Wednesday, May 10, 2017

Search anything from a bugzilla bug given a range of bug numbers

›
[root@centos my-data-set]# for ((i=13420;i<13500;i++)); do echo $i >> /tmp/search_result.txt; curl http://192.168.5.105/bugzilla/sh...
Tuesday, May 2, 2017

A simple way to clone 10,000 files to cluster

›
# Generate 10,000 files from one seed and put them into 100 subdirectories [hdfs@cdh1 tmp]$ for((i=1;i<=100;i++));do mkdir -p 10000fil...
Tuesday, April 25, 2017

Prepare Python machine learning environment on Centos 6.6 to train data

›
# major points: 1. Has to use Python 2.7, not 2.6.  But Centos 6.6 uses Python 2.6 for OS so upgrading to 2.7 is not a solution.  Need to in...
Tuesday, March 21, 2017

Mapreduce logs have weird behavior on HDP 2.3 Tez

›
When launching map reduce on Tez, we will not see our logs from HDP UI.  Click ‘History’ and got nothing although we can see Hadoop system ...
Monday, March 6, 2017

Additional jar files when running Spark under Hadoop YARN mode (CDH 5.10.0 with Scala 2.10 and Spark 1.6.0)

›
lrwxrwxrwx 1 root root     91 Feb 23 15:02 spark-core_2.10-1.6.0-cdh5.10.0.jar -> /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/ja...
Thursday, February 16, 2017

My co-worker Shawn's blogs for Hadoop machine setup and installation

›
http://wp.huangshiyang.com/env-set-up-centos