Tuesday, March 21, 2017

Mapreduce logs have weird behavior on HDP 2.3 Tez

›
When launching map reduce on Tez, we will not see our logs from HDP UI.  Click ‘History’ and got nothing although we can see Hadoop system ...
Monday, March 6, 2017

Additional jar files when running Spark under Hadoop YARN mode (CDH 5.10.0 with Scala 2.10 and Spark 1.6.0)

›
lrwxrwxrwx 1 root root     91 Feb 23 15:02 spark-core_2.10-1.6.0-cdh5.10.0.jar -> /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/ja...
Thursday, February 16, 2017

My co-worker Shawn's blogs for Hadoop machine setup and installation

›
http://wp.huangshiyang.com/env-set-up-centos

Solving protobuf version conflicting in your Java applications

›
Due to the bad design of protobuf, we faced the version conflicting from time to time. Finally, we figured out a way to solve this issue usi...
Tuesday, February 14, 2017

Hadoop Proxy user - Superusers Acting On Behalf Of Other Users

›
https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Superusers.html
Wednesday, February 8, 2017

High performance regex open source packages

›
Java library to check for multiple regexp with a single deterministic automaton. Just a wrapper around dk.brics.automaton: https://github...
Monday, February 6, 2017

My first scala from Spark system: word count

›
[root@john1 john]# cd /var/lib/hadoop-hdfs [root@john1 hadoop-hdfs]# sudo -u hdfs spark-shell --master yarn --deploy-mode client Setting d...