hadoop学习笔记资料.docx

一、环境（伪分布式） Centos、虚拟机、hadoop-1.0.1、jdk1.6（这两个的环境变量要设置）、myeclipse8.6 安装linux命令如下： 先授权 安装java1.6 ＃chmod +x jdk-6u23-linux-i586.bin 安装bin文件 ./jdk-6u23-linux-i586.bin linux 下设置JAVA 环境变量 1.cd /etc/profile.d 2.touch java.sh 3.在java.sh写入以下内容: vi java.sh #set java_environment export JAVA_HOME=/tools/jdk1.6.0_23 export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH:$HOME/bin 保存是: Esc到命令模式, 敲冒号:, 再wq!(3个字符),就保存退出了, [w是保存,q是退出,可单用,!是强制] 也可以是Shift+ZZ,保存退出的意思 4.chmod 777 java.sh --改为可执行文件 5.source java.sh --使生效 6.javac --看是否成功 Hadoop环境变量设置，但是此种方式重新启动后又将失效 [root@localhost ~]# cd tools/hadoop-1.0.1/bin [root@localhost bin]#export PATH=$PATH:/tools/hadoop-1.0.1/bin 修改配置： Configuration Use the following:?conf/core-site.xml: configuration property name/name valuehdfs://localhost:9000/value /property /configuration conf/hdfs-site.xml: configuration property namedfs.replication/name value1/value /property /configuration conf/mapred-site.xml: configuration property namemapred.job.tracker/name valuelocalhost:9001/value /property /configuration Setup passphraseless ssh Now check that you can ssh to the localhost without a passphrase:$ ssh localhost If you cannot ssh to localhost without a passphrase, execute the following commands:$ ssh-keygen -t dsa -P -f ~/.ssh/id_dsa?$ cat ~/.ssh/id_dsa.pub ~/.ssh/authorized_keys Format a new distributed-filesystem:$ bin/hadoop namenode -format Start the hadoop daemons:$ bin/start-all.sh The hadoop daemon log output is written to the?${HADOOP_LOG_DIR}?directory (defaults to?${HADOOP_HOME}/logs). Browse the web interface for the NameNode and the JobTracker; by default they are available at: NameNode?-? HYPERLINK http://localhost:50070/ http://localhost:50070/ JobTracker?-? HYPERLINK http://localhost:50030/ http://localhost:50030/ Copy the input files into the distributed filesystem:$ bin/hadoop fs -