Hadoop node 部署步骤

 

1.        OS安装

a)        RHEL 6.2 X64

                         i.              刻录光盘安装（略）

b)        安装选项

                         i.              Basic server即可

                       ii.              设置hostname

                      iii.              安装完毕执行system-config-network，保存配置；如果网络命名:em1，修改/etc/sysconfig/network-scripts/ifcfg-em1:
ONBOOT=yes
使得网络自动启动

                      iv.              没有DNS的话编辑/etc/hosts，设置本机和其他机器的ip

2.        每个node配置ssh无密码登录，建议相互都可以做到无密码登录

a)        配置Root用户，方便后续安装部署

                         i.              在一台机器上对所有机器（包括自己）执行
# ssh crt-hadoop[01~06] ‘ssh-keygen –t rsa’  (提问直接enter到底)

                       ii.              把所有机器上的id_rsa.pub集中收集
# ssh crt-hadoop[01~06] ‘cat /root/.ssh/id_rsa.pub’ >> /root/.ssh/authorized_keys

                      iii.              复制authorized_keys和known_hosts到所有机器
# scp /root/.ssh/authorized_keys crt-hadoop[01~06]:/root/.ssh/
# scp /root/.ssh/known_hosts crt-hadoop[01~06]:/root/.ssh/
# ssh crt-hadoop[01~06] ‘chmod 600 /root/.ssh/authorized_keys’

b)        配置 hadoop用户，该用户将运行hadoop

                         i.              在一台机器上创建所有机器的hadoop用户
# ssh crt-hadoop[01~06] ‘adduser Hadoop;passwd Hadoop’

                       ii.              使用hadoop用户登录

                      iii.              使得hadoop用户也可以采用ssh无密码登录
$ ssh crt-hadoop[01~06] ‘cat /home/hadoop/.ssh/id_rsa.pub’ >> /home/hadoop/.ssh/authorized_keys

                      iv.              复制authorized_keys和known_hosts到所有机器
$ scp ~/.ssh/authorized_keys crt-hadoop[01~06]:~/.ssh/
$ scp ~/.ssh/known_hosts crt-hadoop[01~06]:~/.ssh/
$ ssh crt-hadoop[01~06] ‘chmod 600 ~/.ssh/authorized_keys’

3.        JDK安装

a)        下载Sun的jdk-6u30-linux-x64-rpm.bin

b)        在所有机器上安装jdk （root登录）
# scp jdk-6u30-linux-x64-rpm.bin crt-hadoop[01~06]:/home/Hadoop/
# ssh crt-hadoop[01~06] ‘chmod +x /home/Hadoop/jdk-6u30-linux-x64-rpm.bin; /home/Hadoop/jdk-6u30-linux-x64-rpm.bin

4.        Hadoop安装

a)        下载 hadoop-1.0.0-1.amd64.rpm

b)        在所有机器上安装hadoop（root登录）
# scp Hadoop-1.0.0-1.amd64.rpm crt-hadoop[01~06]:/home/Hadoop
# ssh crt-hadoop[01~06] ‘rpm –ivh /home/hadoop/Hadoop-1.0.0-1.amd64.rpm’

c)        设置hadoop脚本运行权限
# ssh crt-hadoop[01~06] ‘chmod +x /usr/sbin/*.sh’

d)        修改hadoop配置文件
/etc/hadoop/hdfs-size.xml
…
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
</configuration>

crt-hadoop01设置为master
/etc/hadoop/mapred-site.xml
…
<configuration>
    <property>
        <name>mapred.job.tracker</name>
        <value>crt-hadoop01:9001</value>
    </property>
</configuration>

/etc/hadoop/core-size.xml
…
<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://crt-hadoop01:9000</value>
    </property>
</configuration>

/etc/hadoop/slaves
crt-hadoop02
crt-hadoop03
crt-hadoop04
crt-hadoop05
crt-hadoop06

/etc/Hadoop/masters
crt-hadoop01

e)        复制配置文件到所有机器
# scp /etc/hadoop/* crt-hadoop[02~06]:/etc/hadoop/

f)         关闭防火墙
# ssh crt-hadoop[01~06] ‘ service iptables save; service iptables stop; chkconfig iptables off’

g)        运行hadoop example (hadoop登录)
$ hadoop namenode –format
$hadoop fs –put /etc/Hadoop test-data
$hadoop jar /usr/share/hadoop/hadoop-example-*.jar grep test-data out ‘dfs[a-z.]+’