全球热文：hadoop3.2镜像挂载文件（附下载）

hadoop3.2.2集群部署

1.环境 redhat7.5+hadoop3.2.2+jdk1.81+

(资料图)

2.iso镜像挂载 0_master_local_yum.sh

#!/bin/shbase_path=$(cd `dirname $0`; pwd)iso_path=${base_path}/iso/rhel-server-7.5-x86_64-dvd.isomkdir -p /mnt/cdrom|mount -t iso9660 -o,loop ${iso_path} /mnt/cdrom/;touch /etc/yum.repos.d/local_os.repo;echo "[local_iso]" >> /etc/yum.repos.d/local_os.repo;echo "name=CentOS-$releasever - Media" >> /etc/yum.repos.d/local_os.repo;echo "baseurl=file:///mnt/cdrom" >> /etc/yum.repos.d/local_os.repo;echo "gpgcheck=0" >> /etc/yum.repos.d/local_os.repo;echo "enabled=1" >> /etc/yum.repos.d/local_os.repoecho "镜像挂载文件是5099则正确"yum repolistecho "Complete!"

3.解压hadoop3.2.2

echo "解压hadoop文件"tar -zxvf hadoop-3.2.2.tar.gz

4.防火墙相关命令（此处需要关闭防火墙）

添加端口firewall-cmd --zone=public --add-port=9091/tcp --permanent移除端口firewall-cmd --remove-port=9091/tcp --permanent查看开放端口firewall-cmd --zone=public --list-ports重载端口firewall-cmd --reload查看防火墙状态firewall-cmd --state systemctl status firewalld关闭防火墙systemctl stop firewalldservice firewalld stop开启防火墙systemctl start firewalldservice firewalld start配置文件vim /etc/selinux/config设置SELINUX=disabled

5.jdk安装 运行脚本 ./jdkinstall jdk-8u181-linux-x64.tar.gz 脚本内容如下

#!/bin/shecho "解压jdk-8u181-linux-x64.tar.gz包..."tar -xvf $1echo "将jdk1.8.0_181目录移到/usr/local/java/目录下..."mkdir /usr/local/javacp -r jdk1.8.0_181 /usr/local/java/echo "删除jdk1.8.0_181"rm -rf jdk1.8.0_181echo "设置环境变量"echo "#JAVA_HOME" >> /etc/profileecho "export JAVA_HOME=/usr/local/java/jdk1.8.0_181" >> /etc/profileecho "export JRE_HOME=${JAVA_HOME}/jre" >> /etc/profileecho "export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib" >> /etc/profileecho "export PATH=${JAVA_HOME}/bin:$PATH" >> /etc/profileecho "刷新环境变量"source /etc/profileecho "java安装成功..."java -version

6.hadoop环境变量

echo "#HADOOP_HOME" >> /etc/profileecho "export HADOOP_HOME=/apps/bigdata/hadoop-3.2.2" >> /etc/profileecho "export PATH=$PATH:${HADOOP_HOME}/bin" >> /etc/profileecho "export PATH=$PATH:${HADOOP_HOME}/sbin" >> /etc/profileecho "刷新环境变量"source /etc/profile

7.分发文件

同步jdk文件到Hadoop2,3上scp -r /usr/local/java root@hadoop2:/usr/local/javascp -r /etc/profile root@hadoop2:/etc/profilescp -r /usr/local/java root@hadoop3:/usr/local/javascp -r /etc/profile root@hadoop3:/etc/profile  同步hadoop文件到Hadoop2,3上mkdir -p /apps/bigdatascp -r /apps/bigdata/hadoop-3.2.2 root@hadoop2:/apps/bigdata/hadoop-3.2.2scp -r /apps/bigdata/hadoop-3.2.2 root@hadoop3:/apps/bigdata/hadoop-3.2.2或者同步语法rsync -rvl 其他的一致

8.xsync同步脚本

#!/bin/bash#1 获取输入参数个数，如果没有参数，直接退出pcount=$#if((pcount==0)); thenecho no args;exit;fi#2 获取文件名称p1=$1fname=`basename $p1`echo fname=$fname#3 获取上级目录到绝对路径pdir=`cd -P $(dirname $p1); pwd`echo pdir=$pdir#4 获取当前用户名称user=`whoami`#5 循环，这里host根据自己的节点数和主机名设置for((host=2; host<4; host++)); doecho --------------- hadoop$host ----------------rsync -rvl $pdir/$fname $user@hadoop$host:$pdirdone一般放置在/usr/local/bin目录下chmod 777 xsync例如：xsync 文件名

9.集群配置

核心配置文件：core-site.xml 参考配置文件地址:https://hadoop.apache.org/docs/r3.2.2/hadoop-project-dist/hadoop-common/core-default.xml

fs.defaultFShdfs://hadoop1:9000hadoop.tmp.dirfile:/apps/bigdata/hadoop-3.2.2/data/tmp

配置文件:hadoop-env.sh export JAVA_HOME=/usr/local/java/jdk1.8.0_181 配置文件:hdfs-site.xml 参考配置文件地址:https://hadoop.apache.org/docs/r3.2.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

dfs.namenode.http-addresshadoop1:9870dfs namenode web ui 将侦听的地址和基本端口。dfs.namenode.name.dirfile:/apps/bigdata/hadoop-3.2.2/data/dfs/name在本地文件系统上，namenode持久化存储命名空间和事务日志的路径。dfs.datanode.data.dirfile:/apps/bigdata/hadoop-3.2.2/data/dfs/dataDataNode的本地文件系统上应该存储块的逗号分隔的路径列表。dfs.namenode.secondary.http-addresshadoop3:9868辅助名称节点 http 服务器地址和端口。

配置文件:yarn-env.sh

export JAVA_HOME=/usr/local/java/jdk1.8.0_181

配置文件:yarn-site.xml 参考配置文件地址:https://hadoop.apache.org/docs/r3.2.2/hadoop-yarn/hadoop-yarn-common/yarn-default.xml

yarn.resourcemanager.hostnamehadoop2yarn.nodemanager.aux-servicesmapreduce_shuffleyarn.log-aggregation-enabletrueyarn.log-aggregation.retain-seconds604800

配置文件:mapred-env.sh

export JAVA_HOME=/usr/local/java/jdk1.8.0_181

配置文件:maperd-site.xml 参考配置文件地址:https://hadoop.apache.org/docs/r3.2.2/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml

mapreduce.framework.nameyarnmapreduce.jobhistory.addresshadoop2:10020MapReduce JobHistory 服务器主机：端口mapreduce.jobhistory.webapp.addresshadoop2:19888MapReduce JobHistory 服务器 Web UI主机：端口

hadoop 配置文件同步

xsync /apps/bigdata/hadoop-3.2.2/etc/hadoop/

sbin目录修改配置文件 start-dfs.sh和stop-dfs.sh

HDFS_DATANODE_USER=rootHDFS_DATANODE_SECURE_USER=hdfs  HDFS_NAMENODE_USER=rootHDFS_SECONDARYNAMENODE_USER=root

start-yarn.sh和stop-yarn.sh

YARN_RESOURCEMANAGER_USER=rootHADOOP_SECURE_DN_USER=yarnYARN_NODEMANAGER_USER=root

10.ssh配置

#hadoop1找到.ssh目录cd ~/.ssh文件known_hosts包含相应的主机访问记录不提示直接生成秘钥文件（官方文档的三步），公钥添加入authorized_keys文件，赋权ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsacat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keyschmod 0600 ~/.ssh/authorized_keys  或者提示生成秘钥文件 （enter回车三次）ssh-keygen -t rsa公钥id_rsa.pub私钥id_rsa将公钥放入主机访问记录,拷贝到hadoop2，hadoop3ssh-copy-id hadoop2ssh-copy-id hadoop3自己登陆自己免密ssh-copy-id hadoop1#hadoop2ssh-keygen -t rsassh-copy-id hadoop2ssh-copy-id hadoop3ssh-copy-id hadoop1

11.单机启动

#hadoop1 /apps/bigdata/hadoop-3.2.2/sbinecho "格式化namenode"hdfs namenode -format()此次一般不用) bin目录下的启动方式cd /apps/bigdata/hadoop-3.2.2/sbinecho "启动hdfs"hdfs --daemon start namenodehdfs --daemon start datanode#Hadoop2hdfs --daemon start datanode#hadoop3hdfs --daemon start datanode

12.时间同步

hadoop1查看ntp是否安装rpm -qa|grep ntp出现如下ntpdate-4.2.6p5-28.el7.x86_64fontpackages-filesystem-1.44-8.el7.noarch安装ntprpm -ivh autogen-libopts-5.18-5.el7.x86_64.rpmrpm -ivh ntp-4.2.6p5-29.el7.centos.2.x86_64.rpm  --nodeps修改配置文件vi /etc/ntp.conf授权网段192.168.1.0-255.255.255.0网段上的所有机器可以从这台机器上查询和时间同步#restrict 192.168.1.0 mask 255.255.255.0 nomodify notraprestrict 192.168.1.0 mask 255.255.255.0 nomodify notrap集群在局域网中，不使用其他互联网上的时间server 0.centos.pool.ntp.org iburstserver 1.centos.pool.ntp.org iburstserver 2.centos.pool.ntp.org iburstserver 3.centos.pool.ntp.org iburst注释#server 0.centos.pool.ntp.org iburst#server 1.centos.pool.ntp.org iburst#server 2.centos.pool.ntp.org iburst#server 3.centos.pool.ntp.org iburst当节点丢失网络连接，依然可以采用本地时间作为时间服务器为集群中的其他节点提供时间同步server127.127.1.0fudge 127.127.1.0 stratum 10修改/etc/sysconfig/ntpd硬件时间与系统时间一起同步SYNC_HWCLOCK=yesntpd启动systemctl start ntpd查询状态systemctl status ntpd设置开机自启systemctl enable ntpd其他机器配置5分钟与时间服务器同步一次 hadoop2,hadoop3定时器编辑crontab -e*/5 * * * * /usr/sbin/ntpdate hadoop1定时器查看crontab -l

13.集群启动 /apps/bigdata/hadoop-3.2.2/etc/hadoop

编辑集群配置文件：workers

hadoop1hadoop2hadoop3

分发脚本

xsync workers

hadoop1 启动集群namenode,datanode

cd /apps/bigdata/hadoop-3.2.2/sbin./start-dfs.shjps查看服务进程

hadoop2

启动resourcemanager./start-yarn.sh启动JobHistoryServer./mr-jobhistory-daemon.sh start historyserver启动日志job查看

查看页面

NameNode:http://192.168.189.10:9870/ 默认端口：9870

ResourceManager： http://192.168.189.11:8088/ 默认端口8088

MapReduce JobHistory Server: http://192.168.189.11:19888 默认端口：19888

!