Mapreduce logs have weird behavior on HDP 2.3 Tez

When launching map reduce on Tez, we will not see our logs from HDP UI.  Click ‘History’ and got nothing although we can see Hadoop system logs.

But if you use following command, you will get map reduce logs from system stdout:

sudo -u hdfs yarn logs -applicationId application_1490133530166_0002

But the format is modified:

2017-03-21 15:04:24,708 [ERROR] [TezChild] |common.FindAndExitMapRunner|: caught throwable when run mapper: java.lang.UnsupportedOperationException: Input only available on map

The ‘source’ is changed to ‘TezChild’. 

Our package name is truncated to only have last part so our Java class name is not full name anymore. On this example, “com.xxx.hadoop.common.FindAndExitMapRunner” is changed to “common.FindAndExitMapRunner”

          To be compare with normal log (i.e. without Tez), here is what we should have from map reduce log (you see full package name and class name):

      2017-03-20 14:29:54,778 INFO [main] com.xxx.hadoop.common.ColumnMap: Columnar Mapper

Important! Important! Important! ---->
To review the log, you have to use exact user who launched such application: sudo -u hdfs"
Otherwise, you will see following error:
"Log aggregation has not completed or is not enabled."