public void map(Object key, Writable value,
Context context) throws IOException, InterruptedException {
OrcStruct orcStruct = (OrcStruct)value;
int numberOfFields = orcStruct.getNumFields();
for (int i=0; i<numberOfFields; i++) {
// getFieldValue is private at this moment. Use reflection to access it.
Object field = null;
try {
field = getFieldValue.invoke(orcStruct, i);
} catch (Exception e) {
e.printStackTrace();
}
if (field==null) continue;
// process Hive collection type array, struct or map
if (field instanceof List) {
List list = (List)field;
for (int j=0; j<list.size(); j++)
System.out.println(list.get(j));
}
else if (field instanceof Map) {
Map map = (Map)field;
for (Iterator entries = map.entrySet().iterator(); entries.hasNext();) {
Map.Entry entry = (Entry) entries.next();
System.out.println("key="+entry.getKey()+",value="+entry.getValue());
}
}
else if (field instanceof OrcStruct) {
OrcStruct struct = (OrcStruct)field;
int numberOfField = struct.getNumFields();
for (int j=0; j<numberOfField; j++) {
try {
System.out.println("field"+j+"="+getFieldValue.invoke(struct, j));
} catch (Exception e) {
e.printStackTrace();
}
}
}
else {
System.out.println("Unknown type for field"+ field);
}
}
}
what is 'getFieldValue ' here?
ReplyDeletecould you provide the driver code for this MapReduce program?
ReplyDeleteYou left out the tricky part: assigning a value to getFieldValue. I found that the following simple statement didn't work: getFieldValueMethod = OrcStruct.class.getMethod("getFieldValue", new Class[] { int.class });
ReplyDeleteOK, cool, you need to use a different reflection method, getDeclaredMethod, and also to make the function accessible afterwards, how weird is this?
ReplyDeletegetFieldValue = OrcStruct.class.getDeclaredMethod("getFieldValue", int.class);
getFieldValue.setAccessible(true);