如何二进制部署Apache Hadoop？
- By : Will
- Category : Apache-Hadoop
- Tags: Apache, Hadoop
Apache-Hadoop
1 基础知识
1.1 什么是Apache Hadoop
– Apache Hadoop项目是开发可靠、可扩展的分布式计算开源软件
– Apache Hadoop软件库是一个框架，框架允许以简单的编程模型实现处理跨计算机的分布式集群以处理大型数据集
– Apache Hadoop框架允许集群扩展到数千台计算机，每台计算机的计算与存储都能被集群使用
– Apache Hadoop软件库不依赖硬件实现高性能，而是通过检测和处理应用程的故障实现（实现高可用）
1.2 项目的模块
– Hadoop Common，支持其他Hadoop模块的常用使用程序
– Hadoop HDFS，分布式文件系统，为应用程序提供高吞吐量的数据访问
– Hadoop YARN，框架用于作业调度和管理集群资源
– Hadoop MapReduce，基于YARN系统，用于处理大型的并行数据集
– Hadoop Ozone，Hadoop的对象存储
– Hadoop Submarine，Hadoop的机器学习引擎
1.3 Hadoo的运行模式
– 非分布模式（默认模式，运行单独的Java进程，特别适合调试）
– 伪分布模式（单节点下运行，每个Hadoop守护进程运行于单独的Java进程中，适合测试）
– 全分布模式（几个或数千节点集群模式，适合生产环境）
1.4 Hadoo的单点安装
1.4.1 安装目标
– 配置单点Hadoop
– 配置Hadoop MapReduce
– 配置Hadoop HDFS（分布式文件系统）
1.4.2 支持的平台
– GNU/Linux（官方曾经做个2000个节点的Hadoop集群测试）
– Windows也支持Hadoop（有兴趣请参考Wiki）
1.4.3 依赖的软件
– 安装Java虚拟机（关于JDK与Hadoop的兼容关系，请参阅Hadoop Java Versions）
– 安装ssh却启动sshd服务
– 安装rsync
注：以上适用于Linux系统
1.5 相关项目
1.5.1 Ambari
– 基于Web的Hadoop管理工具
– 用于配置、管理和监控Apache Hadoop集群，支持Hadoop HDFS、Hadoop MapReduce、Hive、HCatalog、HBase、ZooKeeper、Oozie、Pig、Sqoop
– 提供查看集群运行状况的仪表板，支持热图方式查看MapReduce、Pig和Hive性能
1.5.2 Avro
– 数据系列化系统
1.5.3 Cassandra
– 可扩展的多主数据库
– 可防止单点故障的数据库
1.5.4 Chukwa
– 用于管理大型分布式系统的数据收集系统
1.5.5 HBase
– 可扩展的分布式数据库
– 支持大型表的结构化数据库
1.5.6 Hive
– 一种数据仓库基础架构
– 提供数据汇总和即时查询
1.5.7 Mahout
– 可扩展的机器学习和数据挖掘库
1.5.8 Pig
– 用于并行计算的高级数据流行语言和执行框架
1.5.9 Spark
– Hadoop数据的快速和通用计算引擎
– Spark提供一种简单而富有表现力的编程模型
– 支持广泛的应用程序，包括ETL、机器学习、流处理和图形计算
1.5.10 TEZ
– 基于Hadoop YARN的通用数据流程编程框架
– TEZ提供强大且灵活的引擎来执行任意的DAG任务（处理批量和交互式的用例数据）
– Tez将被Hive、Pig等其他框架采用，用于取代MapReduce作为底层执行引擎
1.5.11 Zookeeper
– 用于分布式应用程序的高性能协调服务
2 最佳实践
2.1 系统环境配置
2.1.1 系统信息
IP Address = 10.168.0.100
OS = CentOS 7.x x86_64
Host Name = hadoop.cmdschool.org
2.1.2 系统信息
yum install -y update
2.1.3 安装常用工具
yum install -y vim wget tree
2.2 软件环境配置
2.2.1 安装JDK
请参阅以下方法安装jdk-8u121-linux-x64，
https://www.cmdschool.org/archives/397
安装完成后，请使用如下命令检查JDK的安装，
java -version
命令显示如下，
java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
2.2.2 安装rsync
yum install -y rsync
2.3 部署Apache Hadoop软件包
2.3.1 下载二进制安装包
cd ~ wget https://archive.apache.org/dist/hadoop/common/hadoop-2.6.5/hadoop-2.6.5.tar.gz
其他版本请从以下链接下载，
https://archive.apache.org/dist/hadoop/common/
2.3.2 解压安装包
cd ~ tar -xf hadoop-2.6.5.tar.gz
2.3.2 部署软件包
cd ~ mv hadoop-2.6.5 /usr/
2.4 配置Apache Hadoop软件包
2.4.1 配置运行用户
groupadd hdfs useradd -g hdfs -d /var/lib/hadoop-hdfs/ hdfs
2.4.2 配置环境变量
vim /etc/profile.d/hadoop.sh
加入如下配置，
export HADOOP_HOME=/usr/hadoop-2.6.5
export HADOOP_PREFIX=${HADOOP_HOME}
export HADOOP_YARN_HOME=${HADOOP_HOME}
export PATH=${HADOOP_HOME}/bin:$PATH
export PATH=${HADOOP_HOME}/sbin:$PATH
export HADOOP_CONF_DIR=/etc/hadoop
export HADOOP_LOG_DIR=/var/log/hadoop
export HADOOP_PID_DIR=/var/run/hadoop
#export HADOOP_MASTER=localhost:${HADOOP_HOME}
export HADOOP_IDENT_STRING=$USER
export HADOOP_NICENESS=0
根据声明的目录配置权限，
mkdir -p /var/log/hadoop chown hdfs:hdfs /var/log/hadoop mkdir -p /var/run/hadoop chown hdfs:hdfs /var/run/hadoop
2.4.3 导入环境变量
source /etc/profile.d/hadoop.sh
2.4.4 测试安装部署
hadoop version
可见如下输出，
Hadoop 2.6.5 Subversion https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997 Compiled by sjlee on 2016-10-02T23:43Z Compiled with protoc 2.5.0 From source with checksum f05c9fa095a395faa9db9f7ba5d754 This command was run using /usr/hadoop-2.6.5/share/hadoop/common/hadoop-common-2.6.5.jar
3 非分布模式下运行
cd ${HADOOP_HOME}
mkdir input
cp etc/hadoop/*.xml input
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar grep input output 'dfs[a-z.]+'
cat output/*
4 伪分布模式下运行
4.1 定义配置
4.1.1 创建配置文件夹
mkdir -p /etc/hadoop
4.1.2 定义core-site.xml配置文件
cp ${HADOOP_HOME}/etc/hadoop/core-site.xml /etc/hadoop/
vim /etc/hadoop/core-site.xml
修改如下配置，
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
4.1.3 定义hdfs-site.xml配置文件
cp ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml /etc/hadoop/
vim /etc/hadoop/hdfs-site.xml
修改如下配置，
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
4.1.4 部署slaves配置文件
cp ${HADOOP_HOME}/etc/hadoop/slaves /etc/hadoop/
4.2 配置公钥认证
4.2.1 配置公钥认证
su - hdfs ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 0600 ~/.ssh/authorized_keys
4.2.2 测试公钥认证
su - hdfs ssh localhost ssh 0.0.0.0
注：以上，如果不用输入密码即可完成登录，则配置完成。
4.3 配置apache hadoop
4.3.1 格式化文件系统
rm -rf /tmp/* su - hdfs -c 'hadoop namenode -format'
为了便于分析，建议先清空临时目录（没有定义数据默认输出到临时目录），可使用如下命令检查命令行执行的结果，
ls -ld /tmp/*
可见如下输出，
drwxrwxr-x. 3 hdfs hdfs 17 Nov 14 23:22 /tmp/hadoop-hdfs drwxr-xr-x. 2 hdfs hdfs 6 Nov 14 23:22 /tmp/hsperfdata_hdfs
可使用如下命令查看目录的结构，
tree /tmp/
可见如下输出，
/tmp/ ├── hadoop-hdfs │ └── dfs │ └── name │ └── current │ ├── fsimage_0000000000000000000 │ ├── fsimage_0000000000000000000.md5 │ ├── seen_txid │ └── VERSION └── hsperfdata_hdfs 5 directories, 4 files
4.3.2 启动守护进程
su - hdfs -c '/usr/hadoop-2.6.5/sbin/start-dfs.sh'
可见如下输出，
Starting namenodes on [localhost] localhost: starting namenode, logging to /var/log/hadoop/hadoop-root-namenode-localhost.localdomain.out localhost: 2019-05-21 00:19:27,079 INFO [main] namenode.NameNode (StringUtils.java:startupShutdownMessage(633)) - STARTUP_MSG: localhost: /************************************************************ localhost: STARTUP_MSG: Starting NameNode localhost: STARTUP_MSG: host = localhost/127.0.0.1 localhost: STARTUP_MSG: args = [] localhost: STARTUP_MSG: version = 2.6.5 localhost: STARTUP_MSG: classpath = /etc/hadoop:/usr/hadoop-2.6.5/share/hadoop/common/lib/activation-1.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jackon-mapper-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/protobuf-java-2.5.0.jar:usr/hadoop-2.6.5/share/hadoop/common/lib/guava-11.0.2.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-net-3.1.jar:/usr/hadoop-2.6.5/share/hadoop/commonlib/jackson-jaxrs-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/httpclient-4.2.5.jar:/us/hadoop-2.6.5/share/hadoop/common/lib/xz-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/slf4-api-1.7.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/hadoop-2.6.5share/hadoop/common/lib/jersey-json-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/htrace-core-3.0.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/asm-3.2jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/hadoop-annotations-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/hadoo-2.6.5/share/hadoop/common/lib/curator-framework-2.6.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/hadoop-2.6.5/share/haoop/common/lib/commons-math3-3.1.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/apacheds-18n-2.0.0-M15.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/hadoo-2.6.5/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/hadoop-2.6.5/share/hadoop/commn/lib/jsr305-1.3.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/junit-4.11.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jettison-1.1.jar:/usr/hadoop-2.6.5/hare/hadoop/common/lib/jets3t-0.9.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/paranamer-2.3.jar/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-io-2.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/hadoop-2.6.5/share/hadoop/comon/lib/commons-el-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/adoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/comon/lib/commons-lang-2.6.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/hadoop-auth-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/avro-1.7.4.jar:/usr/haoop-2.6.5/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/hadoop-2.6.5/share/hadoop/common/ib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/netty-3.6..Final.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/gson-2.2.4.jar:/usr/hadoop-2.6.5/sare/hadoop/common/lib/commons-codec-1.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-httplient-3.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/curator-client-2.6.0.jar:/usr/hadop-2.6.5/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/hadoop-2.6.5/share/hadoop/comon/lib/jackson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/api-asn1-api-1..0-M20.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/hadoop-2.6.5/hare/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/snappy-jaa-1.0.4.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/hadoop-common-2.6.5-tests.jar:/usr/hadoop-2.6.5/share/hadoop/common/hadoop-nfs-2.6.5.jar:/usr/hadoop-2.6./share/hadoop/common/hadoop-common-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdoop-2.6.5/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/serlet-api-2.5.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/hadoop-2.6.5/sare/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jr:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lb/jsp-api-2.1.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/hadoop-2..5/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jetty-util-6..26.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/hadoop-2.6.5/share/haoop/hdfs/lib/commons-codec-1.4.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jckson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-2.6.5-tests.jar:/usr/hadoo-2.6.5/share/hadoop/hdfs/hadoop-hdfs-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/activatin-1.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/hadoop-26.5/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/servlet-ai-2.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/hadoop-2.6.5/share/hadoop/arn/lib/xz-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/hadoop-2.6./share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/asm-3.2.jar:/us/hadoop-2.6.5/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/arn/lib/jsr305-1.3.9.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/hadoop-2.6./share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-core-.9.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/hadoop-2.6.5/share/haoop/yarn/lib/jetty-util-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-guice-1.9.jar:usr/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/hadoop-2.6.5/share/hadoop/yar/lib/guice-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/hadoop-2.6.5/sare/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commonslogging-1.1.3.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/hadoop-2..5/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yar-common-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-appicationhistoryservice-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hdoop-yarn-api-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-registry-2.6.5jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.5.jar:/sr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-client-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-common-2.6.5.jar:/usr/hadoop-2.6.5/shar/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hadoop-.6.5/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/hadoop-2.6.5/share/hadoop/mapredce/lib/xz-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/hadoop-2.65/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapeduce/lib/junit-4.11.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/adoop-2.6.5/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/apreduce/lib/avro-1.7.4.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/netty-3.6.2.Final.ar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hadoop-2.6.5/share/hadop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/og4j-1.2.17.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.65.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6..jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclien-2.6.5-tests.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapredue-client-hs-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.5.jar localhost: STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997; compiled by 'sjlee' on 2016-10-02T23:43Z localhost: STARTUP_MSG: java = 1.8.0_121 localhost: ************************************************************/ localhost: starting datanode, logging to /var/log/hadoop/hadoop-root-datanode-localhost.localdomain.out localhost: 2019-05-21 00:19:31,485 INFO [main] datanode.DataNode (StringUtils.java:startupShutdownMessage(633)) - STARTUP_MSG: localhost: /************************************************************ localhost: STARTUP_MSG: Starting DataNode localhost: STARTUP_MSG: host = localhost/127.0.0.1 localhost: STARTUP_MSG: args = [] localhost: STARTUP_MSG: version = 2.6.5 localhost: STARTUP_MSG: classpath = /etc/hadoop:/usr/hadoop-2.6.5/share/hadoop/common/lib/activation-1.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jackon-mapper-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/protobuf-java-2.5.0.jar:usr/hadoop-2.6.5/share/hadoop/common/lib/guava-11.0.2.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-net-3.1.jar:/usr/hadoop-2.6.5/share/hadoop/commonlib/jackson-jaxrs-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/httpclient-4.2.5.jar:/us/hadoop-2.6.5/share/hadoop/common/lib/xz-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/slf4-api-1.7.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/hadoop-2.6.5share/hadoop/common/lib/jersey-json-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/htrace-core-3.0.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/asm-3.2jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/hadoop-annotations-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/hadoo-2.6.5/share/hadoop/common/lib/curator-framework-2.6.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/hadoop-2.6.5/share/haoop/common/lib/commons-math3-3.1.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/apacheds-18n-2.0.0-M15.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/hadoo-2.6.5/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/hadoop-2.6.5/share/hadoop/commn/lib/jsr305-1.3.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/junit-4.11.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jettison-1.1.jar:/usr/hadoop-2.6.5/hare/hadoop/common/lib/jets3t-0.9.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/paranamer-2.3.jar/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-io-2.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/hadoop-2.6.5/share/hadoop/comon/lib/commons-el-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/adoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/comon/lib/commons-lang-2.6.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/hadoop-auth-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/avro-1.7.4.jar:/usr/haoop-2.6.5/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/hadoop-2.6.5/share/hadoop/common/ib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/netty-3.6..Final.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/gson-2.2.4.jar:/usr/hadoop-2.6.5/sare/hadoop/common/lib/commons-codec-1.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-httplient-3.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/curator-client-2.6.0.jar:/usr/hadop-2.6.5/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/hadoop-2.6.5/share/hadoop/comon/lib/jackson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/api-asn1-api-1..0-M20.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/hadoop-2.6.5/hare/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/snappy-jaa-1.0.4.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/hadoop-common-2.6.5-tests.jar:/usr/hadoop-2.6.5/share/hadoop/common/hadoop-nfs-2.6.5.jar:/usr/hadoop-2.6./share/hadoop/common/hadoop-common-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdoop-2.6.5/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/serlet-api-2.5.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/hadoop-2.6.5/sare/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jr:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lb/jsp-api-2.1.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/hadoop-2..5/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jetty-util-6..26.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/hadoop-2.6.5/share/haoop/hdfs/lib/commons-codec-1.4.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jckson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-2.6.5-tests.jar:/usr/hadoo-2.6.5/share/hadoop/hdfs/hadoop-hdfs-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/activatin-1.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/hadoop-26.5/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/servlet-ai-2.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/hadoop-2.6.5/share/hadoop/arn/lib/xz-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/hadoop-2.6./share/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/asm-3.2.jar:/us/hadoop-2.6.5/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/arn/lib/jsr305-1.3.9.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/hadoop-2.6./share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-core-.9.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/hadoop-2.6.5/share/haoop/yarn/lib/jetty-util-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-guice-1.9.jar:usr/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/hadoop-2.6.5/share/hadoop/yar/lib/guice-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/hadoop-2.6.5/sare/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commonslogging-1.1.3.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/hadoop-2..5/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yar-common-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-appicationhistoryservice-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hdoop-yarn-api-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-registry-2.6.5jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.5.jar:/sr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-client-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-common-2.6.5.jar:/usr/hadoop-2.6.5/shar/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hadoop-.6.5/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/hadoop-2.6.5/share/hadoop/mapredce/lib/xz-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/hadoop-2.65/share/hadoop/mapreduce/lib/hadoop-annotations-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapeduce/lib/junit-4.11.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/adoop-2.6.5/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/apreduce/lib/avro-1.7.4.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/netty-3.6.2.Final.ar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hadoop-2.6.5/share/hadop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/og4j-1.2.17.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.65.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6..jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclien-2.6.5-tests.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapredue-client-hs-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.5.jar localhost: STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997; compiled by 'sjlee' on 2016-10-02T23:43Z localhost: STARTUP_MSG: java = 1.8.0_121 localhost: ************************************************************/ Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /var/log/hadoop/hadoop-root-secondarynamenode-localhost.localdomain.out 0.0.0.0: 2019-05-21 00:19:37,062 INFO [main] namenode.SecondaryNameNode (StringUtils.java:startupShutdownMessage(633)) - STARTUP_MSG: 0.0.0.0: /************************************************************ 0.0.0.0: STARTUP_MSG: Starting SecondaryNameNode 0.0.0.0: STARTUP_MSG: host = localhost/127.0.0.1 0.0.0.0: STARTUP_MSG: args = [] 0.0.0.0: STARTUP_MSG: version = 2.6.5 0.0.0.0: STARTUP_MSG: classpath = /etc/hadoop:/usr/hadoop-2.6.5/share/hadoop/common/lib/activation-1.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jackso-mapper-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/ur/hadoop-2.6.5/share/hadoop/common/lib/guava-11.0.2.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-net-3.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lb/jackson-jaxrs-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/httpclient-4.2.5.jar:/usr/adoop-2.6.5/share/hadoop/common/lib/xz-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/slf4j-pi-1.7.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/stax-api-1.0-2.jar:/usr/hadoop-2.6.5/sare/hadoop/common/lib/jersey-json-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/htrace-core-3.0.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/asm-3.2.jr:/usr/hadoop-2.6.5/share/hadoop/common/lib/hadoop-annotations-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/hadoop-.6.5/share/hadoop/common/lib/curator-framework-2.6.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/hadoop-2.6.5/share/hadop/common/lib/commons-math3-3.1.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/apacheds-i1n-2.0.0-M15.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/httpcore-4.2.5.jar:/usr/hadoop-.6.5/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/hadoop-2.6.5/share/hadoop/commonlib/jsr305-1.3.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/junit-4.11.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jettison-1.1.jar:/usr/hadoop-2.6.5/shre/hadoop/common/lib/jets3t-0.9.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/paranamer-2.3.jar:/sr/hadoop-2.6.5/share/hadoop/common/lib/commons-io-2.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/zookeeper-3.4.6.jar:/usr/hadoop-2.6.5/share/hadoop/commo/lib/commons-el-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/haoop-2.6.5/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/commn/lib/commons-lang-2.6.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/hadoop-auth-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/avro-1.7.4.jar:/usr/hadop-2.6.5/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/hadoop-2.6.5/share/hadoop/common/li/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/netty-3.6.2.inal.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/gson-2.2.4.jar:/usr/hadoop-2.6.5/shae/hadoop/common/lib/commons-codec-1.4.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-httpclent-3.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/curator-client-2.6.0.jar:/usr/hadoo-2.6.5/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/hadoop-2.6.5/share/hadoop/commn/lib/jackson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/hamcrest-core-1.3.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/api-asn1-api-1.0.-M20.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/hadoop-2.6.5/shre/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/hadoop-2.6.5/share/hadoop/common/lib/snappy-java1.0.4.1.jar:/usr/hadoop-2.6.5/share/hadoop/common/hadoop-common-2.6.5-tests.jar:/usr/hadoop-2.6.5/share/hadoop/common/hadoop-nfs-2.6.5.jar:/usr/hadoop-2.6.5/hare/hadoop/common/hadoop-common-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hadop-2.6.5/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/servlt-api-2.5.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/hadoop-2.6.5/shae/hadoop/hdfs/lib/htrace-core-3.0.4.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/libjsp-api-2.1.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-io-2.4.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/hadoop-2.6./share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jetty-util-6.1.6.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/hadoop-2.6.5/share/hadop/hdfs/lib/commons-codec-1.4.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/sr/hadoop-2.6.5/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/jacson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-2.6.5-tests.jar:/usr/hadoop-.6.5/share/hadoop/hdfs/hadoop-hdfs-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/hdfs/hadoop-hdfs-nfs-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/activation1.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/hadoop-2.65/share/hadoop/yarn/lib/guava-11.0.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/servlet-api2.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jline-0.9.94.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/hadoop-2.6.5/share/hadoop/yan/lib/xz-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-cli-1.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/hadoop-2.6.5/hare/hadoop/yarn/lib/stax-api-1.0-2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-json-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/asm-3.2.jar:/usr/adoop-2.6.5/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/yan/lib/jsr305-1.3.9.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jettison-1.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-io-2.4.jar:/usr/hadoop-2.6.5/hare/hadoop/yarn/lib/zookeeper-3.4.6.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-core-1..jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-client-1.9.jar:/usr/hadoop-2.6.5/share/hadop/yarn/lib/jetty-util-6.1.26.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-lang-2.6.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/ur/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/ib/guice-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-codec-1.4.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jetty-6.1.26.jar:/usr/hadoop-2.6.5/shae/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/commons-lgging-1.1.3.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/hadoop-2.6./share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-ommon-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-appliationhistoryservice-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadop-yarn-api-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-tests-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-registry-2.6.5.jr:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.6.5.jar:/us/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-client-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/yarn/hadoop-yarn-server-common-2.6.5.jar:/usr/hadoop-2.6.5/share/adoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hadoop-2..5/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduc/lib/xz-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/hadoop-2.6.5share/hadoop/mapreduce/lib/hadoop-annotations-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreuce/lib/junit-4.11.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/haoop-2.6.5/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/mareduce/lib/avro-1.7.4.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/netty-3.6.2.Final.ja:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hadoop-2.6.5/share/hadoo/mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/lo4j-1.2.17.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/ur/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.6.5jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.5.ar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-.6.5-tests.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduceclient-hs-2.6.5.jar:/usr/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.6.5.jar 0.0.0.0: STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e8c9fe0b4c252caf2ebf1464220599650f119997; compiled by 'sjlee' on 2016-10-02T23:43Z 0.0.0.0: STARTUP_MSG: java = 1.8.0_121 0.0.0.0: ************************************************************/
启动后，可使用如下命令再次查看并对比目录的变化，
tree /tmp/
可见如下输出，
/tmp/
├── hadoop-hdfs
│ └── dfs
│ ├── data
│ │ ├── current
│ │ │ ├── BP-2127593889-127.0.0.1-1573806732442
│ │ │ │ ├── current
│ │ │ │ │ ├── dfsUsed
│ │ │ │ │ ├── finalized
│ │ │ │ │ │ └── subdir0
│ │ │ │ │ │ └── subdir0
│ │ │ │ │ │ ├── blk_1073741825
│ │ │ │ │ │ ├── blk_1073741825_1001.meta
│ │ │ │ │ │ ├── blk_1073741826
│ │ │ │ │ │ ├── blk_1073741826_1002.meta
│ │ │ │ │ │ ├── blk_1073741827
│ │ │ │ │ │ └── blk_1073741827_1003.meta
│ │ │ │ │ ├── rbw
│ │ │ │ │ └── VERSION
│ │ │ │ ├── dncp_block_verification.log.curr
│ │ │ │ ├── dncp_block_verification.log.prev
│ │ │ │ └── tmp
│ │ │ └── VERSION
│ │ └── in_use.lock
│ ├── name
│ │ ├── current
│ │ │ ├── edits_0000000000000000001-0000000000000000002
│ │ │ ├── edits_inprogress_0000000000000000003
│ │ │ ├── fsimage_0000000000000000000
│ │ │ ├── fsimage_0000000000000000000.md5
│ │ │ ├── seen_txid
│ │ │ └── VERSION
│ │ └── in_use.lock
│ └── namesecondary
│ └── in_use.lock
├── hsperfdata_hdfs
│ ├── 9552
│ ├── 9672
│ └── 9823
├── Jetty_0_0_0_0_50070_hdfs____w2cu08
│ └── jsp
├── Jetty_0_0_0_0_50075_datanode____hwtdwq
│ └── jsp
└── Jetty_0_0_0_0_50090_secondary____y6aanv
└── jsp
21 directories, 23 files
可使用如下命令查看PID目录的文件创建，
ls /var/run/hadoop/
可见如下显示，
hadoop-hdfs-datanode.pid hadoop-hdfs-secondarynamenode.pid hadoop-hdfs-namenode.pid
从上面PID文件可以分析出，当前启用“datanode”，“namenode”，“secondarynamenode”三种角色，如果你想确认服务是否启用，请使用如下命令，
netstat -antp | grep java | grep LISTEN
可见如下输出，
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 18211/java tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 18488/java tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 18211/java tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 18335/java tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 18335/java tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 18335/java
测试到这里，请使用以下命令停止服务，
su - hdfs -c '/usr/hadoop-2.6.5/sbin/stop-dfs.sh'
然后，请使用以下命令测试服务的逐个启动，
su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh start namenode' su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh start datanode' su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh start secondarynamenode'
以上，每执行一条命令，我们建议你使用如下命令观察目录PID文件的创建，
ls /var/run/hadoop/
然后，请使用以下命令测试服务的逐个停止，
su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh stop namenode' su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh stop datanode' su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh stop secondarynamenode'
4.3.3 配置名称节点服务控制脚本
vim /usr/lib/systemd/system/hdfs-nn.service
可加入如下配置，
[Unit] Description=Apache HDFS namenode manager Wants=network.target Before=network.target After=network-pre.target Documentation=https://hadoop.apache.org/docs/ [Service] Type=forking ExecStartPre=/bin/sh -c 'mkdir -p /var/run/hadoop;chown hdfs:hdfs /var/run/hadoop;chmod 775 /var/run/hadoop' ExecStartPre=/bin/sh -c 'mkdir -p /var/log/hadoop;chown hdfs:hdfs /var/log/hadoop;chmod 775 /var/log/hadoop' ExecStart=/usr/bin/su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh start namenode' ExecStop=/usr/bin/su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh stop namenode' PIDFile=/var/run/hadoop/hadoop-hdfs-namenode.pid Restart=on-success [Install] WantedBy=multi-user.target
修改完脚本后，你需要使用如下命令重载服务，
systemctl daemon-reload
你可使用如下命令控制服务和查询状态，
systemctl start hdfs-nn.service systemctl status hdfs-nn.service systemctl stop hdfs-nn.service systemctl restart hdfs-nn.service
测试完毕，建议你使用如下命令设置服务自动启动，
systemctl enable hdfs-nn.service
4.3.4 配置数据节点服务控制脚本
vim /usr/lib/systemd/system/hdfs-dn.service
可加入如下配置，
[Unit] Description=Apache HDFS datanode manager Wants=network.target Before=network.target After=network-pre.target Documentation=https://hadoop.apache.org/docs/ [Service] Type=forking ExecStartPre=/bin/sh -c 'mkdir -p /var/run/hadoop;chown hdfs:hdfs /var/run/hadoop;chmod 775 /var/run/hadoop' ExecStartPre=/bin/sh -c 'mkdir -p /var/log/hadoop;chown hdfs:hdfs /var/log/hadoop;chmod 775 /var/log/hadoop' ExecStart=/usr/bin/su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh start datanode' ExecStop=/usr/bin/su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh stop datanode' PIDFile=/var/run/hadoop/hadoop-hdfs-datanode.pid Restart=on-success [Install] WantedBy=multi-user.target
修改完脚本后，你需要使用如下命令重载服务，
systemctl daemon-reload
你可使用如下命令控制服务和查询状态，
systemctl start hdfs-dn.service systemctl status hdfs-dn.service systemctl stop hdfs-dn.service systemctl restart hdfs-dn.service
测试完毕，建议你使用如下命令设置服务自动启动，
systemctl enable hdfs-dn.service
4.3.5 配置第二名称节点服务控制脚本
vim /usr/lib/systemd/system/hdfs-snn.service
可加入如下配置，
[Unit] Description=Apache HDFS secondarynamenode manager Wants=network.target Before=network.target After=network-pre.target Documentation=https://hadoop.apache.org/docs/ [Service] Type=forking ExecStartPre=/bin/sh -c 'mkdir -p /var/run/hadoop;chown hdfs:hdfs /var/run/hadoop;chmod 775 /var/run/hadoop' ExecStartPre=/bin/sh -c 'mkdir -p /var/log/hadoop;chown hdfs:hdfs /var/log/hadoop;chmod 775 /var/log/hadoop' ExecStart=/usr/bin/su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh start secondarynamenode' ExecStop=/usr/bin/su - hdfs -c '/usr/hadoop-2.6.5/sbin/hadoop-daemon.sh stop secondarynamenode' PIDFile=/var/run/hadoop/hadoop-hdfs-secondarynamenode.pid Restart=on-success [Install] WantedBy=multi-user.target
修改完脚本后，你需要使用如下命令重载服务，
systemctl daemon-reload
你可使用如下命令控制服务和查询状态，
systemctl start hdfs-snn.service systemctl status hdfs-snn.service systemctl stop hdfs-snn.service systemctl restart hdfs-snn.service
测试完毕，建议你使用如下命令设置服务自动启动，
systemctl enable hdfs-snn.service
4.3.6 开放名称节点的Web端口
firewall-cmd --permanent --add-port 50070/tcp firewall-cmd --reload firewall-cmd --list-all
4.3.7 访问名称节点的Web界面
http://10.168.0.100:50070/
可见如下页面，
4.4 测试apache hadoop
4.4.1 在hdfs中创建文件夹
su - hdfs hdfs dfs -ls / hdfs dfs -mkdir /user hdfs dfs -ls /
注：
– 第一条命令列出hdfs的根目录（空的，什么都没有）
– 第二条命令在hdfs的根目录下创建user文件夹
4.4.2 上传文件到hdfs
su - hdfs hdfs dfs -mkdir /input hdfs dfs -put /etc/hadoop/ /input hdfs dfs -ls /input
4.4.3 获取hdfs的文件
su - hdfs hdfs dfs -get /input/hadoop ls -l
4.4.4 使用cat读取hdfs的文件
su - hdfs hdfs dfs -cat /input/hadoop/*
参阅文档
=================
Apache Hadoop的部署文档
————————
https://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-common/SingleCluster.html
http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html
Apache Hadoop的软件下载
————————
https://archive.apache.org/dist/hadoop/common/
https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/
http://www.apache.org/dyn/closer.cgi/hadoop/common/
Hadoop的集群模式
————————
https://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-common/ClusterSetup.html
http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html
Apache Hadoop的官方主页
———————–
http://hadoop.apache.org/
没有评论