win7+cygwin+hadoop亲测成功

Windows7 下安装Hadoop（一）

---在window下配置安装之cygwin的安装

如果想在window系统环境下运行hadoop就要安装Cygwin---一个模拟linux环境的软件。可以选择在线安装，本文假设读者已经将cygwin下载到了本地！！！

先决条件（重要！！！）：

1. 新建一个管理员用户，要有用户名+密码！！！

2. 安装了jdk1.6以上的版本，并且安装路径建议在分区根目录，原因是linux路径与windows风格不同！！！

3. 设置好了jdk的bin目录添加path，jre安装目录新建JAVA_HOME环境变量并添加！！！

 

===========================华丽的分割线==============================

 

1.双击setup.exe安装。如下图：

2.直接点击下一步后如下图：

3.直接点击下一步后如下图:

这里是要选择安装路径。建议设置到分区根目录如: C:\cygwin 。

4.设置好路径后下一步进入下图：

这是选择Cygwin安装源文件的路径。

5. 来到此图就要开始进行一些配置了。选择一下要安装的包。如下图：

首先：选择其中的Base Default，通常这里的包都已经选择上了。你要确保sed已选择上，这样你可以在eclipse中使用hadoop了。如下图这样即可：

其次：选择Devel Default，将其中的subversion选中第一个即可。如下图：

最后：选择Net default包，将其中的openssh及openssl选上。如下图：

经过上述包选择后点击下一步即可。

6. 来到这个图中如下：

都是些你不用也不能改的东西，包依赖的一些提示信息。直接点一下步。这是将进入下载及安装你刚刚所选择的包的过程。随后不用再说了。设置是否在桌面显示图标之类额。随后结束了。

 

===========================华丽的分割线==============================

 

7.千万别马上去运行配置cygwin！先在我的电脑->属性->高级->环境变量中配置加入变量CYGWIN=ntsec,在path环境变量中加入cygwin/bin路径, 例如:;c:/cygwin/bin

8. 给文件授权,以完成sshd 服务的安装

双击安装完成后桌面上出现的cygwin快捷图标运行cygwin，依次输入：

chmod +r /etc/group
chmod + rwx /etc/passwd
chmod +rwx /var

改变文件的访问权限，以完成ssh服务的安装配置！详细如下：

Copyingskeleton files.

These filesare for the user to personalise

their cygwinexperience.

These willnever be overwritten.

`./.bashrc'-> `/home/Administrator//.bashrc'

`./.bash_profile'-> `/home/Administrator//.bash_profile'

`./.inputrc'-> `/home/Administrator//.inputrc'

Administrator@backup~

$ chmod +r/etc/group # 对文件权限

Administrator@backup~

$ chmod +rwx /etc/passwd # 对文件权限

Administrator@backup~

$ chmod+rwx /var/ # 对文件权限

Administrator@backup~

$ssh-host-config # 引导SSH服务配置

*** Info:Generating /etc/ssh_host_key

*** Info:Generating /etc/ssh_host_rsa_key

*** Info:Generating /etc/ssh_host_dsa_key

*** Info:Creating default /etc/ssh_config file

*** Info:Creating default /etc/sshd_config file

*** Info:Privilege separation is set to yes by default since OpenSSH 3.3.

*** Info:However, this requires a non-privileged account called 'sshd'.

*** Info: Formore info on privilege separation read /usr/share/doc/openssh/README.privsep.

*** Query:Should privilege separation be used? (yes/no) yes #输入yes

*** Info:Updating /etc/sshd_config file

*** Warning:The following functions require administrator privileges!

*** Query: Doyou want to install sshd as a service?

*** Query:(Say "no" if it is already installed as a service) (yes/no) yes #输入yes

*** Info:Note that the CYGWIN variable must contain at least "ntsec"

*** Info: forsshd to be able to change user context without password.

*** Query:Enter the value of CYGWIN for the daemon: [ntsec] ntsec #输入ntsec

*** Info: OnWindows Server 2003, Windows Vista, and above, the

*** Info:SYSTEM account cannot setuid to other users -- a capability

*** Info:sshd requires. You need to have or to create a privileged

*** Info:account. This script will help you do so.

*** Info: Youappear to be running Windows 2003 Server or later. On 2003 and

*** Info:later systems, it's not possible to use the LocalSystem account

*** Info: forservices that can change the user id without an explicit password

*** Info:(such as passwordless logins [e.g. public key authentication] via sshd).

*** Info: If youwant to enable that functionality, it's required to create a new

*** Info:account with special privileges (unless a similar account already exists).

*** Info:This account is then used to run these special servers.

*** Info:Note that creating a new user requires that the current account have

*** Info:Administrator privileges itself.

*** Info: Noprivileged account could be found.

*** Info:This script plans to use 'cyg_server'.

*** Info:'cyg_server' will only be used by registered services.

*** Query: Doyou want to use a different name? (yes/no) no #输入no，不指定启动用户

*** Query:Create new privileged user account 'cyg_server'? (yes/no) no #输入no，不指定启动用户

*** ERROR:There was a serious problem creating a privileged user.

*** Query: Doyou want to proceed anyway? (yes/no) yes #输入yes

*** Warning:Expected privileged user 'cyg_server' does not exist.

*** Warning:Defaulting to 'SYSTEM'

*** Info: Thesshd service has been installed under the LocalSystem

*** Info:account (also known as SYSTEM). To start the service now, call

*** Info:`net start sshd' or `cygrunsrv -S sshd'. Otherwise, it

*** Info:will start automatically after the next reboot.

*** Info:Host configuration finished. Have fun!

Administrator@backup~

$ cygrunsrv.exe-S sshd # 启动 SSH服务

若以上详细有所不同，也不必在意，看着大意对就行了！

 

9. 输入“net start sshd”启动ssh服务！

10.输入“ssh localhost”登陆ssh服务端！并在提示下输入之前“ssh-host-config”配置的密码！登陆成功则后输入“ps”出现下图所示，则表示cygwin安装配置成功了！恭喜可以进入下一步安装配置hadoop了！

 

===========================华丽的分割线==============================

 

Windows7 下安装Hadoop（二）

---在window7 +cygwin下配置安装hadoop

1.下载hadoop安装包,我用的是hadoop-1.0.4。

2. 解压到一个目录下。随便选择按个目录。建议直接选择一个根目录。随后进行配置。这里配置的是伪分布式了。需要配置一个文件。都在conf目录下。分别为：hadoop-env.sh。下面说明：

hadoop-env.sh这个文件中主要配置JAVA_HOME路径。需要注意的是：

l 千万只能用记事本打开！！！否则如果用写字板之类的打开会自动在行尾添加换行转义符“/r”，导致后续运行报错！！！

l  jkd是1.6以上版本。

l  路径应该是linux风格的。打开后将export前面的#注释符去掉，路径为/cygdrive/你的jdk路径。

设置后如下图：

7.这就可以了。设置就结束了。下面运行试一试。先启动输入“net startsshd”启动服务，并输入“ssh localhost”登陆，进入到hadoop安装路径下的bin文件夹中。

首先格式化：./hadoop namenode –format如果让选择Y/N选择Y。如下图：

其次运行：./start-sh.all如下图则成功了。总共会启动五个节点，期间会要求出入之前配置的密码。详情如下所示：

kobbo@KYHTN3IKQDAHSH8~

$cd /cygdrive/c/hadoop-1.0.4/bin

 

kobbo@KYHTN3IKQDAHSH8/cygdrive/c/hadoop-1.0.4/bin

$ls

hadoop             start-all.sh               stop-balancer.sh

hadoop-config.sh   start-balancer.sh          stop-dfs.sh

hadoop-daemon.sh   start-dfs.sh               stop-jobhistoryserver.sh

hadoop-daemons.sh  start-jobhistoryserver.sh  stop-mapred.sh

rcc                start-mapred.sh            task-controller

slaves.sh          stop-all.sh

 

kobbo@KYHTN3IKQDAHSH8/cygdrive/c/hadoop-1.0.4/bin

$./hadoop namenode -format

13/05/0114:34:26 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG:Starting NameNode

STARTUP_MSG:   host = KYHTN3IKQDAHSH8/172.18.77.83

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 1.0.4

STARTUP_MSG:   build =https://svn.apache.org/repos/asf/hadoop/common/branches/b

ranch-1.0 -r1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012

************************************************************/

13/05/0114:34:26 INFO util.GSet: VM type       =32-bit

13/05/0114:34:26 INFO util.GSet: 2% max memory = 19.33375 MB

13/05/0114:34:26 INFO util.GSet: capacity      =2^22 = 4194304 entries

13/05/0114:34:26 INFO util.GSet: recommended=4194304, actual=4194304

13/05/0114:34:27 INFO namenode.FSNamesystem: fsOwner=kobbo

13/05/0114:34:27 INFO namenode.FSNamesystem: supergroup=supergroup

13/05/0114:34:27 INFO namenode.FSNamesystem: isPermissionEnabled=true

13/05/0114:34:27 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100

13/05/0114:34:27 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessK

eyUpdateInterval=0min(s), accessTokenLifetime=0 min(s)

13/05/0114:34:27 INFO namenode.NameNode: Caching file names occuring more than

10 times

13/05/0114:34:27 INFO common.Storage: Image file of size 111 saved in 0 seconds

.

13/05/0114:34:27 INFO common.Storage: Storage directory \tmp\hadoop-kobbo\dfs\n

ame has beensuccessfully formatted.

13/05/0114:34:27 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG:Shutting down NameNode at KYHTN3IKQDAHSH8/172.18.77.83

************************************************************/

 

kobbo@KYHTN3IKQDAHSH8/cygdrive/c/hadoop-1.0.4/bin

$./start-all.sh

startingnamenode, logging to /cygdrive/c/hadoop-1.0.4/libexec/../logs/hadoop-ko

bbo-namenode-KYHTN3IKQDAHSH8.out

kobbo@localhost'spassword:

localhost:Permission denied, please try again.

kobbo@localhost'spassword:

localhost:starting datanode, logging to /cygdrive/c/hadoop-1.0.4/libexec/../log

s/hadoop-kobbo-datanode-KYHTN3IKQDAHSH8.out

kobbo@localhost'spassword:

localhost:secondarynamenode running as process 3188. Stop it first.

startingjobtracker, logging to /cygdrive/c/hadoop-1.0.4/libexec/../logs/hadoop-

kobbo-jobtracker-KYHTN3IKQDAHSH8.out

kobbo@localhost'spassword:

localhost:starting tasktracker, logging to /cygdrive/c/hadoop-1.0.4/libexec/../

logs/hadoop-kobbo-tasktracker-KYHTN3IKQDAHSH8.out

最后：运行一下。

1.先在本地文件建立一个文件夹(最好建在hadoop的安装文件在同一个目录下)如testin随后在里面建几个文件txt或java的随意（也可以MP3什么都行）。在里面输入以下英文单字。如file1.java 内容Hello world ！file2.java 内容：I love you ！

2.在cygwin中将一个hadoophdfs上的目录如testin。命令如下：

./hadoop dfs –mkdir testin

3.将本地的两个文件上传到dfs上去。

./hadoop dfs –put /testin/*.java testin

此时运行./hadoop dfs –ls testin 就会就会发现已有文件了。也可以直接去目录下查看，如下图所示：