ubuntu下的hadoop安装

1. 环境

ubuntu 14.04 64位
java 1.8.0_45（1.7即可）
ssh（sshd运行）

2. 安装版本

hadoop2.7.2（http://hadoop.apache.org/releases.html）

3. 安装过程

1.下载hadoop2.7.2的binary版本 http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz
2.解压，并将解压得到的hadoop-2.7.2文件夹移到/usr/local中
3.cd 到/usr/local/hadoop-2.7.2目录
4.设置JAVA_HOME参数：在当前目录的etc/hadoop/文件末尾添加

# set to the root of your Java installationexport JAVA_HOME=/usr/local/jdk1.8.0/

5.执行命令bin/hadoop，打印hadoop用法。（注意：不是/bin/hadoop）
6.此时，hadoop默认是Standalone单机模式，测试hadoop如下：

$ mkdir input$ cp etc/hadoop/*.xml input$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar grep input output 'dfs[a-z.]+'$ cat output/*

4. mapreduce简单示例

a、编写map，reduce和配置，保存成WordCount.java

import java.io.IOException;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;public class WordCount {  public static class TokenizerMapper       extends Mapper<Object, Text, Text, IntWritable>{    private final static IntWritable one = new IntWritable(1);    private Text word = new Text();    public void map(Object key, Text value, Context context                    ) throws IOException, InterruptedException {      StringTokenizer itr = new StringTokenizer(value.toString());      while (itr.hasMoreTokens()) {        word.set(itr.nextToken());        context.write(word, one);      }    }  }  public static class IntSumReducer       extends Reducer<Text,IntWritable,Text,IntWritable> {    private IntWritable result = new IntWritable();    public void reduce(Text key, Iterable<IntWritable> values,                       Context context                       ) throws IOException, InterruptedException {      int sum = 0;      for (IntWritable val : values) {        sum += val.get();      }      result.set(sum);      context.write(key, result);    }  }  public static void main(String[] args) throws Exception {    Configuration conf = new Configuration();    Job job = Job.getInstance(conf, "word count");    job.setJarByClass(WordCount.class);    job.setMapperClass(TokenizerMapper.class);    job.setCombinerClass(IntSumReducer.class);    job.setReducerClass(IntSumReducer.class);    job.setOutputKeyClass(Text.class);    job.setOutputValueClass(IntWritable.class);    FileInputFormat.addInputPath(job, new Path(args[0]));    FileOutputFormat.setOutputPath(job, new Path(args[1]));    System.exit(job.waitForCompletion(true) ? 0 : 1);  }}

b、编译WordCount.java

先设置环境变量：

export JAVA_HOME=/usr/local/jdk1.8.0export PATH=${JAVA_HOME}/bin:${PATH}export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar

其中，HADOOP_CLASSPATH是指定hadoop搜索哪些路径下的.class文件，下面用到的com.sun.tools.javac.Main就在${JAVA_HOME}/lib/tools.jar中。

然后，编译：

${HADOOP_HOME}/bin/hadoop com.sun.tools.javac.Main WordCount.javajar cf wc.jar WordCount*.class

当前目录下生成WordCount$IntSumReducer.class、WordCount$TokenizerMapper.class和WordCount.class三个文件；再用jar打包这三个文件，生成wc.jar。

c、运行

${HADOOP_HOME}/bin/hadoop jar wc.jar WordCount ./input ./output

输入文件放在input文件夹中，hadoop生成output目录保存结果。

d、歪楼

2中可以不打包class文件，只要把class所在文件夹的路径(当前路径)加到HADOOP_CLASSPATH中即可，也即

export HADOOP_CLASSPATH=.:${HADOOP_CLASSPATH}

然后直接运行即可：

${HADOOP_HOME}/bin/hadoop WordCount ./input ./output

参考：

[1] Hadoop: Setting up a Single Node Cluster.
[2] ubuntu下搭建JAVA开发环境
[3] MapReduce Tutorial