Monday, February 01, 2010

Hadoop Combiners and Map-Reduce

According to the Hadoop documentation, a Hadoop Combiner can be used to speedup Hadoop Map-Reduce if the Reduce function is both commutative and associative. It would be interesting to write a program that introspects a Hadoop Reduce function to draw a conclusion regarding the validity of this constraint for the use of Hadoop Combiner but it will not be trivial.

No comments: