Mapreduce logs have weird behavior on HDP 2.3 Tez

When launching map reduce on Tez, we will not see our logs from HDP UI.  Click ‘History’ and got nothing although we can see Hadoop system logs.

But if you use following command, you will get map reduce logs from system stdout:

sudo -u hdfs yarn logs -applicationId application_1490133530166_0002

But the format is modified:

2017-03-21 15:04:24,708 [ERROR] [TezChild] |common.FindAndExitMapRunner|: caught throwable when run mapper: java.lang.UnsupportedOperationException: Input only available on map

The ‘source’ is changed to ‘TezChild’. 

Our package name is truncated to only have last part so our Java class name is not full name anymore. On this example, “com.xxx.hadoop.common.FindAndExitMapRunner” is changed to “common.FindAndExitMapRunner”

          To be compare with normal log (i.e. without Tez), here is what we should have from map reduce log (you see full package name and class name):

      2017-03-20 14:29:54,778 INFO [main] com.xxx.hadoop.common.ColumnMap: Columnar Mapper

Important! Important! Important! ---->
To review the log, you have to use exact user who launched such application: sudo -u hdfs"
Otherwise, you will see following error:
"Log aggregation has not completed or is not enabled."

Additional jar files when running Spark under Hadoop YARN mode (CDH 5.10.0 with Scala 2.10 and Spark 1.6.0)

lrwxrwxrwx 1 root root     91 Feb 23 15:02 spark-core_2.10-1.6.0-cdh5.10.0.jar -> /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/jars/spark-core_2.10-1.6.0-cdh5.10.0.jar
lrwxrwxrwx 1 root root     80 Feb 23 15:15 scala-library-2.10.6.jar -> /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/jars/scala-library-2.10.6.jar
lrwxrwxrwx 1 root root     37 Mar  6 14:06 commons-lang3-3.3.2.jar -> ../../../jars/commons-lang3-3.3.2.jar
-rw-r--r-- 1 root root 185676 Mar  6 14:11 typesafe-config-2.10.1.jar
lrwxrwxrwx 1 root root     55 Mar  6 14:26 akka-actor_2.10-2.2.3-shaded-protobuf.jar -> ../../../jars/akka-actor_2.10-2.2.3-shaded-protobuf.jar
lrwxrwxrwx 1 root root     56 Mar  6 14:28 akka-remote_2.10-2.2.3-shaded-protobuf.jar -> ../../../jars/akka-remote_2.10-2.2.3-shaded-protobuf.jar
lrwxrwxrwx 1 root root     55 Mar  6 14:29 akka-slf4j_2.10-2.2.3-shaded-protobuf.jar -> ../../../jars/akka-slf4j_2.10-2.2.3-shaded-protobuf.jar
lrwxrwxrwx 1 root root     70 Mar  6 14:40 spark-assembly-1.6.0-cdh5.10.0-hadoop2.6.0-cdh5.10.0.jar -> ../../../jars/spark-assembly-1.6.0-cdh5.10.0-hadoop2.6.0-cdh5.10.0.jar
[root@john2 lib]# pwd
/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/hadoop-yarn/lib

My co-worker Shawn's blogs for Hadoop machine setup and installation

http://wp.huangshiyang.com/env-set-up-centos

Solving protobuf version conflicting in your Java applications

Due to the bad design of protobuf, we faced the version conflicting from time to time. Finally, we figured out a way to solve this issue using a routine way.

Assume:

1. Using Maven, you have found all dependency jar files.
2. These jar files contains different versions of protobuf so your application fails to work.  For example, MapMaker.keyEquivalence method is not found.
3. You have Java de-compiler installed.

Solution:

1. Rename Maven repository (for example, rename .m2\repository to .m2\repository_backup)
2. Re-generate Maven repository (mvn test or mvn package on pom.xml file)
3. Under Windows Explorer, go to .m2\repository folder and type '.jar' from search box (top right corner).
4. Now, you get all jar files under the same window.
5. Copy all and paste them into a single folder (i.e. c:\my_jars).
6. Move these jar files into Linux machine since we will need some Linux scripting there.
7. Assume all of these jar files are under your Linux machine now under subdirectory /tmp/myJars
8. Do following scripting(assume you have Java JDK installed so you have jar somewhere under your Linux system; otherwise, install JDK first):

[root@john1 myJars]# for x in `ls`; do jar -tvf $x | grep com.google.common.collect.MapMaker; done

9. Above will find all jar files which contains class MapMaker.
10. Under my case, it found following two jar files:

google-collections-1.0.jar
  1479 Wed Dec 30 10:59:50 PST 2009 com/google/common/collect/MapMaker$1.class
  1756 Wed Dec 30 10:59:50 PST 2009 com/google/common/collect/MapMaker$ComputationExceptionReference.class
  2041 Wed Dec 30 10:59:50 PST 2009 com/google/common/collect/MapMaker$LinkedSoftEntry.class
  2055 Wed Dec 30 10:59:50 PST 2009 com/google/common/collect/MapMaker$LinkedStrongEntry.class

guava-14.0.1.jar

   227 Thu Mar 14 19:56:56 PDT 2013 com/google/common/collect/MapMaker$1.class

  2329 Thu Mar 14 19:56:56 PDT 2013 com/google/common/collect/MapMaker$ComputingMapAdapter.class

  2545 Thu Mar 14 19:56:56 PDT 2013 com/google/common/collect/MapMaker$NullComputingConcurrentMap.class

  4134 Thu Mar 14 19:56:56 PDT 2013 com/google/common/collect/MapMaker$NullConcurrentMap.class

11. Use Java decompiler to search 'keyEquivalence' and you will find only guava-14.0.1.jar contains it.
12. Remove google-collections-1.0.jar from your classpath.  Now, there is no complain about method MapMaker.keyEquivalence is not found or similar errors.

Hadoop Proxy user - Superusers Acting On Behalf Of Other Users

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Superusers.html

High performance regex open source packages

Java library to check for multiple regexp with a single deterministic automaton. Just a wrapper around dk.brics.automaton:

https://github.com/fulmicoton/multiregexp


RE2/J: linear time regular expression matching in Java:

https://github.com/google/re2j

My first scala from Spark system: word count

[root@john1 john]# cd /var/lib/hadoop-hdfs
[root@john1 hadoop-hdfs]# sudo -u hdfs spark-shell --master yarn --deploy-mode client
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_67)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc (master = yarn-client, app id = application_1486166793688_0005).
17/02/06 16:13:53 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.1.0
17/02/06 16:13:54 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
SQL context available as sqlContext.

scala> val logFile = "hdfs://john1.dg:8020/johnz/text/test.txt"
logFile: String = hdfs://john1.dg:8020/johnz/text/test.txt

scala> val file = sc.textFile(logFile)
file: org.apache.spark.rdd.RDD[String] = hdfs://john1.dg:8020/johnz/text/test.txt MapPartitionsRDD[1] at textFile at <console>:29

scala> val counts = file.flatMap(_.split(" ")).map(word => (word,1)).reduceByKey(_ + _)
counts: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey at <console>:31

scala> counts.collect()
res0: Array[(String, Int)] = Array((executes,1), (is,1), (expressive,1), (real,1), ((clustered),1), ("",1), (apache,1), (computing,1), (fast,,1), (job,1), (environment.,1), (spark,1), (a,1), (in,1), (which,1), (extremely,1), (distributed,1), (time,1), (and,1), (system,1))