Monday, March 28, 2016

Two key properties to set if you write code for accessing Hadoop concurrently

›
conf.setBoolean("fs.hdfs.impl.disable.cache", true); conf.setBoolean("fs.maprfs.impl.disable.cache", true);   con...
Friday, February 19, 2016

Kerberos renewal trick when using keytab to login through proxy

›
When ugi returned by UserGroupInformation.createProxyUser is used to call "doAs", renewing ticket (through checkTGTAndReloginFrom...

Generate RC file from Java code

›
import java.util.ArrayList; import java.util.List; import java.util.Map; import java.util.Properties; import java.util.TreeMap; import...
Friday, December 18, 2015

A simple way for reducer to get how many records are processed in mapreduce job's mapper

›
JobClient client = new JobClient(jobConf); RunningJob mapreduceJob = client.getJob(JobID.forName( jobConf.get("mapred.job.id") ...
Wednesday, November 4, 2015

A simple way to calculate PageRank in R

›
library(igraph) source("http://michael.hahsler.net/SMU/ScientificCompR/code/map.R") # 1->2; 1->3, 2->1, 3->4, 4-...
Tuesday, October 27, 2015

Good articles for shingle, minhash and LSH

›
http://matthewcasperson.blogspot.com/2013/11/minhash-for-dummies.html http://infolab.stanford.edu/~ullman/mmds/bookL.pdf   (Chapter 3, rou...
Friday, September 25, 2015

How to override cluster jar files

›
If you need to use your own version of a jar file in your map reduce job, you can do so through following steps: 1. Bundle the jar file in...