kMeans commandline introduction

This quick start page describes how to run the kMeans clustering algorithm on a Hadoop cluster.

Steps

Mahout’s k-Means clustering can be launched from the same command line invocation whether you are running on a single machine in stand-alone mode or on a larger Hadoop cluster. The difference is determined by the $HADOOP_HOME and $HADOOP_CONF_DIR environment variables. If both are set to an operating Hadoop cluster on the target machine then the invocation will run k-Means on that cluster. If either of the environment variables are missing then the stand-alone Hadoop configuration will be invoked instead.

./bin/mahout kmeans <OPTIONS>

In $MAHOUT_HOME/, build the jar containing the job (mvn install) The job will be generated in $MAHOUT_HOME/core/target/ and it’s name will contain the Mahout version number. For example, when using Mahout 0.3 release, the job will be mahout-core-0.3.job

Testing it on one single machine w/o cluster

• Put the data: cp testdata

• Run the Job:

  ./bin/mahout kmeans -i testdata -o output -c clusters -dm org.apache.mahout.common.distance.CosineDistanceMeasure -x 5 -ow -cd 1 -k 25

Running it on the cluster

• (As needed) Start up Hadoop: $HADOOP_HOME/bin/start-all.sh
• Put the data: $HADOOP_HOME/bin/hadoop fs -put testdata

• Run the Job:

  export HADOOP_HOME= export HADOOP_CONF_DIR=$HADOOP_HOME/conf ./bin/mahout kmeans -i testdata -o output -c clusters -dm org.apache.mahout.common.distance.CosineDistanceMeasure -x 5 -ow -cd 1 -k 25

• Get the data out of HDFS and have a look. Use bin/hadoop fs -lsr output to view all outputs.

Command line options

  --input (-i) input			       Path to job input directory. 
					       Must be a SequenceFile of    
					       VectorWritable		    
  --clusters (-c) clusters		       The input centroids, as Vectors. 
					       Must be a SequenceFile of    
					       Writable, Cluster/Canopy. If k  
					       is also specified, then a random 
					       set of vectors will be selected  
					       and written out to this path 
					       first			    
  --output (-o) output			       The directory pathname for   
					       output.			    
  --distanceMeasure (-dm) distanceMeasure      The classname of the	    
					       DistanceMeasure. Default is  
					       SquaredEuclidean 	    
  --convergenceDelta (-cd) convergenceDelta    The convergence delta value. 
					       Default is 0.5		    
  --maxIter (-x) maxIter		       The maximum number of	    
					       iterations.		    
  --maxRed (-r) maxRed			       The number of reduce tasks.  
					       Defaults to 2		    
  --k (-k) k				       The k in k-Means.  If specified, 
					       then a random selection of k 
					       Vectors will be chosen as the    
					       Centroid and written to the  
					       clusters input path.	    
  --overwrite (-ow)			       If present, overwrite the output 
					       directory before running job 
  --help (-h)				       Print out help		    
  --clustering (-cl)			       If present, run clustering after 
					       the iterations have taken place