When launching map reduce on Tez, we
will not see our logs from HDP UI. Click
‘History’ and got nothing although we can see Hadoop system logs.
But if you use following command, you will get map reduce logs
from system stdout:
sudo -u hdfs yarn logs -applicationId
application_1490133530166_0002
But the format is modified:
2017-03-21 15:04:24,708 [ERROR]
[TezChild] |common.FindAndExitMapRunner|: caught throwable when run mapper:
java.lang.UnsupportedOperationException: Input only available on map
The ‘source’ is changed to ‘TezChild’.
Our package name is truncated to only
have last part so our Java class name is not full name anymore. On this
example, “com.xxx.hadoop.common.FindAndExitMapRunner” is changed to “common.FindAndExitMapRunner”
To be
compare with normal log (i.e. without Tez), here is what we should have from map reduce log (you
see full package name and class name):
2017-03-20 14:29:54,778 INFO [main]
com.xxx.hadoop.common.ColumnMap: Columnar Mapper
Important! Important! Important! ---->
To review the log, you have to use exact user who launched such application: sudo -u hdfs"
Otherwise, you will see following error:
"Log aggregation has not completed or is not enabled."
Important! Important! Important! ---->
To review the log, you have to use exact user who launched such application: sudo -u hdfs"
Otherwise, you will see following error:
"Log aggregation has not completed or is not enabled."
