Access Hive collection type data in map reduce for Orc file

public void map(Object key, Writable value,
Context context) throws IOException, InterruptedException {

OrcStruct orcStruct = (OrcStruct)value;
int numberOfFields = orcStruct.getNumFields();

for (int i=0; i<numberOfFields; i++) {

// getFieldValue is private at this moment. Use reflection to access it.
Object field = null;
try {
field = getFieldValue.invoke(orcStruct, i);
} catch (Exception e) {
e.printStackTrace();
}
if (field==null) continue;

// process Hive collection type array, struct or map
if (field instanceof List) {
List list = (List)field;
for (int j=0; j<list.size(); j++)
System.out.println(list.get(j));
}
else if (field instanceof Map) {
Map map = (Map)field;
for (Iterator entries = map.entrySet().iterator(); entries.hasNext();) {
Map.Entry entry = (Entry) entries.next();
System.out.println("key="+entry.getKey()+",value="+entry.getValue());
}
}
else if (field instanceof OrcStruct) {
OrcStruct struct = (OrcStruct)field;
int numberOfField = struct.getNumFields();
for (int j=0; j<numberOfField; j++) {
try {
System.out.println("field"+j+"="+getFieldValue.invoke(struct, j));
} catch (Exception e) {
e.printStackTrace();
}
}
}
else {
System.out.println("Unknown type for field"+ field);
}
}
}

Troubleshooting version mismatch issue of map reduce job

When coding map reduce job and code to access Hadoop file system, you have to include hadoop jar files to make your java code to pass compiler. If the version of jar files are not compatible with the jar files of Hadoop system, the 500 error will be gotten when running your java code.

The 500 error code does not really tell anything about what is the problem.  This makes troubleshooting little bit hard.  To identify this kind of problem, we have to go to cluster to check logs.

For example, looking into hadoop-hdfs-namenode-*.log and try to find following error:

2014-08-14 14:21:42,386 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 192.168.72.1:15444 got version 7 expected version 4

Basically, this is telling us that cluster expects version 4, but client is using version 7.

Having this kind of information, lower the version of hadoop jar files.  Now my hadoop code works perfect fine with cluster.

Assigning a static IP to a VMware Workstation VM

http://bytealmanac.wordpress.com/2012/07/02/assigning-a-static-ip-to-a-vmware-workstation-vm/

Assumption: the VM is running a DHCP client and is assigned a dynamic IP by the DHCP service running on the host. My host machine is runs on Windows 7 Ultimate x64 with VMware Workstation 8.

Open C:\ProgramData\VMware\vmnetdhcp.conf as Administrator. This file follows the syntax of dhcpd.conf. Add the following lines (change host name, MAC, IP appropriately – these are shown in a different colour) under the correct section (for me it was for a NAT based network – VMnet8). The MAC can be found from the VM’s properties.

host ubuntu {
    hardware ethernet 00:0C:29:16:2A:D6;
    fixed-address 192.168.84.132;
}

Restart the VMware DHCP service. Use the following commands from an elevated prompt:
net stop vmnetdhcp
net start vmnetdhcp

On the VM, acquire a new lease using the below command (if VM runs Linux):

ifconfig eth0 down

ifconfig eth0 up

Cloudera CDH5 source code download

https://repository.cloudera.com/artifactory/public/org/apache/hadoop/hadoop-core/

zookeeper-env.sh issue when setting up HBase

Followed CDH4.2.2 installation guide to setup HBase.

Ran "service zookeeper-server start"

No error from command line.  But zookeeper.log says "nohup: failed to run command ‘java’: No such file or directory".

Interesting!  Java home is right by doing "echo $JAVA_HOME".

After about one hour troubleshooting and script checking, it turned out:

zookeeper-env.sh is needed under /etc/zookeeper/conf directory (as other Hadoop component).  But for somehow, zookeeper installer did not have such file by default.

I have to manually create such file and put following line into it:

export JAVA_HOME=/opt/jdk1.6.0_45/

After that, starting zookeeper works.  I can see it from 'jps':

[root@centos conf]# jps
2732 TaskTracker
4964 Jps
4776 QuorumPeerMain
3133 NameNode
2548 JobTracker
2922 DataNode

Eclipse debug step into, step over, etc. disappeared. How to bring them back?

http://stackoverflow.com/questions/12912896/eclipse-buttons-like-step-in-step-out-resume-etc-not-working

Write RC file

private static void testWrite() throws IOException {
Configuration conf = new Configuration();
conf.addResource(new Path("C:\\etc\\Hadoop\\conf\\core-site.xml"));
conf.addResource(new Path("C:\\etc\\Hadoop\\conf\\hdfs-site.xml"));
conf.addResource(new Path("C:\\etc\\Hadoop\\conf\\mapred-site.xml"));

FileSystem fs = null;
try {
fs = FileSystem.get(conf);
} catch (IOException e1) {
e1.printStackTrace();
}

// has to set column number manually
conf.setInt(RCFile.COLUMN_NUMBER_CONF_STR, 4);

RCFile.Writer rcWriter = new RCFile.Writer(fs, conf, new Path("/user/abc/output_rcwriter/output1"));

String[]  values =  
{"111222333,1200,999999.99,abc@yahoo.com",
"1112226666,1201,999999.99,abcdefg@yahoo.com"};

for (String value : values) {
String[] columns = value.split(",");

if (columns.length>0) {
BytesRefArrayWritable outputRow = new BytesRefArrayWritable(columns.length);
for (int i=0; i<columns.length; i++) {
BytesRefWritable column = new BytesRefWritable(columns[i].getBytes("UTF-8"));
outputRow.set(i, column);
}
rcWriter.append(outputRow);
}
}
rcWriter.close();
}