Search anything from a bugzilla bug given a range of bug numbers

[root@centos my-data-set]# for ((i=13420;i<13500;i++)); do echo $i >> /tmp/search_result.txt; curl http://192.168.5.105/bugzilla/show_bug.cgi?id=$i | grep "passed to DE constructor may be null" >> /tmp/search_result.txt; done;

A simple way to clone 10,000 files to cluster

# Generate 10,000 files from one seed and put them into 100 subdirectories

[hdfs@cdh1 tmp]$ for((i=1;i<=100;i++));do mkdir -p 10000files/F$i; for((j=1;j<=100;j++));do echo $i-$j; cp 1000line.csv 10000files/F$i/$i-$j.csv;done;done;

# Move them to cluster.  One sub-directory at one time.

[hdfs@cdh1 tmp]$ for((i=1;i<=100;i++));do echo $i; hadoop fs -mkdir /JohnZ/10000files/F$i; hadoop fs -copyFromLocal 10000files/F$i/* /JohnZ/10000files/F$i/.;done;

Prepare Python machine learning environment on Centos 6.6 to train data

# major points: 1. Has to use Python 2.7, not 2.6.  But Centos 6.6 uses Python 2.6 for OS so upgrading to 2.7 is not a solution.  Need to install Python 2.7 in addition to 2.6.  2. Use setuptool to install pip and use pip to install rest.

# Download dependency files
yum groupinstall "Development tools"
yum -y install gcc gcc-c++ numpy python-devel scipy
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel

# Compile and install Python 2.7.13
wget https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz
tar xzf Python-2.7.13.tgz
cd Python-2.7.13
./configure
# make altinstall is used to prevent replacing the default python binary file /usr/bin/python.
make altinstall

# Download setuptools using wget:
wget --no-check-certificate https://pypi.python.org/packages/source/s/setuptools/setuptools-1.4.2.tar.gz
# Extract the files from the archive:
tar -xvf setuptools-1.4.2.tar.gz
# Enter the extracted directory:
cd setuptools-1.4.2

# Install setuptools using the Python we've installed (2.7.6)
# python2.7 setup.py install
/opt/python-2.7.13/Python-2.7.13/python ./setup.py install

# install pip
curl https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py | python2.7 -

or (following works for me)

[root@centos python-2.7.13]# /opt/python-2.7.13/Python-2.7.13/python ./setuptools/setuptools-1.4.2/easy_install.py pip

# install numpy
[root@centos python-2.7.13]# /opt/python-2.7.13/Python-2.7.13/python -m pip install numpy

# Install SciPy
[root@centos python-2.7.13]# /opt/python-2.7.13/Python-2.7.13/python -m pip install scipy

# Install Scikit
[root@centos python-2.7.13]# /opt/python-2.7.13/Python-2.7.13/python -m pip install scikit-learn

# Install nltk
[root@centos python-2.7.13]# /opt/python-2.7.13/Python-2.7.13/python -m pip install nltk

# Download nltk data (will be stored under /root/nltk_data)
[root@centos SVM]# /opt/python-2.7.13/Python-2.7.13/python -m nltk.downloader all

Mapreduce logs have weird behavior on HDP 2.3 Tez

When launching map reduce on Tez, we will not see our logs from HDP UI.  Click ‘History’ and got nothing although we can see Hadoop system logs.

But if you use following command, you will get map reduce logs from system stdout:

sudo -u hdfs yarn logs -applicationId application_1490133530166_0002

But the format is modified:

2017-03-21 15:04:24,708 [ERROR] [TezChild] |common.FindAndExitMapRunner|: caught throwable when run mapper: java.lang.UnsupportedOperationException: Input only available on map

The ‘source’ is changed to ‘TezChild’. 

Our package name is truncated to only have last part so our Java class name is not full name anymore. On this example, “com.xxx.hadoop.common.FindAndExitMapRunner” is changed to “common.FindAndExitMapRunner”

          To be compare with normal log (i.e. without Tez), here is what we should have from map reduce log (you see full package name and class name):

      2017-03-20 14:29:54,778 INFO [main] com.xxx.hadoop.common.ColumnMap: Columnar Mapper

Important! Important! Important! ---->
To review the log, you have to use exact user who launched such application: sudo -u hdfs"
Otherwise, you will see following error:
"Log aggregation has not completed or is not enabled."

Additional jar files when running Spark under Hadoop YARN mode (CDH 5.10.0 with Scala 2.10 and Spark 1.6.0)

lrwxrwxrwx 1 root root     91 Feb 23 15:02 spark-core_2.10-1.6.0-cdh5.10.0.jar -> /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/jars/spark-core_2.10-1.6.0-cdh5.10.0.jar
lrwxrwxrwx 1 root root     80 Feb 23 15:15 scala-library-2.10.6.jar -> /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/jars/scala-library-2.10.6.jar
lrwxrwxrwx 1 root root     37 Mar  6 14:06 commons-lang3-3.3.2.jar -> ../../../jars/commons-lang3-3.3.2.jar
-rw-r--r-- 1 root root 185676 Mar  6 14:11 typesafe-config-2.10.1.jar
lrwxrwxrwx 1 root root     55 Mar  6 14:26 akka-actor_2.10-2.2.3-shaded-protobuf.jar -> ../../../jars/akka-actor_2.10-2.2.3-shaded-protobuf.jar
lrwxrwxrwx 1 root root     56 Mar  6 14:28 akka-remote_2.10-2.2.3-shaded-protobuf.jar -> ../../../jars/akka-remote_2.10-2.2.3-shaded-protobuf.jar
lrwxrwxrwx 1 root root     55 Mar  6 14:29 akka-slf4j_2.10-2.2.3-shaded-protobuf.jar -> ../../../jars/akka-slf4j_2.10-2.2.3-shaded-protobuf.jar
lrwxrwxrwx 1 root root     70 Mar  6 14:40 spark-assembly-1.6.0-cdh5.10.0-hadoop2.6.0-cdh5.10.0.jar -> ../../../jars/spark-assembly-1.6.0-cdh5.10.0-hadoop2.6.0-cdh5.10.0.jar
[root@john2 lib]# pwd
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop-yarn/lib

My co-worker Shawn's blogs for Hadoop machine setup and installation

http://wp.huangshiyang.com/env-set-up-centos

Solving protobuf version conflicting in your Java applications

Due to the bad design of protobuf, we faced the version conflicting from time to time. Finally, we figured out a way to solve this issue using a routine way.

Assume:

1. Using Maven, you have found all dependency jar files.
2. These jar files contains different versions of protobuf so your application fails to work.  For example, MapMaker.keyEquivalence method is not found.
3. You have Java de-compiler installed.

Solution:

1. Rename Maven repository (for example, rename .m2\repository to .m2\repository_backup)
2. Re-generate Maven repository (mvn test or mvn package on pom.xml file)
3. Under Windows Explorer, go to .m2\repository folder and type '.jar' from search box (top right corner).
4. Now, you get all jar files under the same window.
5. Copy all and paste them into a single folder (i.e. c:\my_jars).
6. Move these jar files into Linux machine since we will need some Linux scripting there.
7. Assume all of these jar files are under your Linux machine now under subdirectory /tmp/myJars
8. Do following scripting(assume you have Java JDK installed so you have jar somewhere under your Linux system; otherwise, install JDK first):

[root@john1 myJars]# for x in `ls`; do jar -tvf $x | grep com.google.common.collect.MapMaker; done

9. Above will find all jar files which contains class MapMaker.
10. Under my case, it found following two jar files:

google-collections-1.0.jar
  1479 Wed Dec 30 10:59:50 PST 2009 com/google/common/collect/MapMaker$1.class
  1756 Wed Dec 30 10:59:50 PST 2009 com/google/common/collect/MapMaker$ComputationExceptionReference.class
  2041 Wed Dec 30 10:59:50 PST 2009 com/google/common/collect/MapMaker$LinkedSoftEntry.class
  2055 Wed Dec 30 10:59:50 PST 2009 com/google/common/collect/MapMaker$LinkedStrongEntry.class

guava-14.0.1.jar

   227 Thu Mar 14 19:56:56 PDT 2013 com/google/common/collect/MapMaker$1.class

  2329 Thu Mar 14 19:56:56 PDT 2013 com/google/common/collect/MapMaker$ComputingMapAdapter.class

  2545 Thu Mar 14 19:56:56 PDT 2013 com/google/common/collect/MapMaker$NullComputingConcurrentMap.class

  4134 Thu Mar 14 19:56:56 PDT 2013 com/google/common/collect/MapMaker$NullConcurrentMap.class

11. Use Java decompiler to search 'keyEquivalence' and you will find only guava-14.0.1.jar contains it.
12. Remove google-collections-1.0.jar from your classpath.  Now, there is no complain about method MapMaker.keyEquivalence is not found or similar errors.