Docker安装CDH6.3.1集群版

Docker安装CDH6.3.1集群版（内含安装包） #

一、前言 #

本次采用的在线安装方式，cdh为6.3.2版本，系统为centos7.4， docker节点可以为任意多个，下文将以3个docker容器为示例进行展示。此方法也可用在docker swarm上，docker容器能够互连，网络互通即可。安装前请确保内存够用，我的32g内存3个节点启动后内存使用率75%。

二、宿主机初始化 #

以下在docker的宿主机上进行操作。

2.1配置yum源 #

yum install -y wget \
&& mkdir -p /etc/yum.repos.d/repo_bak \
&& mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/repo_bak/ \
&& wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo \
&& wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo \
&& yum clean all \
&& yum makecache \
&& yum update –y

2.2安装docker-ce #

yum install -y yum-utils device-mapper-persistent-data lvm2 \
&& yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo\
&& yum makecache fast \
&& yum -y install docker-ce \
&& yum clean all \
&& docker version

2.3配置docker #

data-root：docker根目录路径
insecure-registries：docker私有仓库地址
registry-mirrors：公共镜像仓库加速地址

mkdir -p /etc/docker/ && \
(
cat <<EOF
{
  "data-root":"/home/docker",
  "registry-mirrors": ["https://1e7waog4.mirror.aliyuncs.com"]
}
EOF
) >>/etc/docker/daemon.json \
&& systemctl start docker \
&& systemctl enable docker \
&& cat /etc/docker/daemon.json \
&& docker version

安装docker命令补全工具

yum install -y bash-completion \
&& source /usr/share/bash-completion/completions/docker \
&& source /usr/share/bash-completion/bash_completion \
&& yum clean all

2.4安装基本工具 #

yum install -y vim wget ntp net-tools \
&& yum clean all

2.5关闭防火墙 #

systemctl stop firewalld \
&& systemctl disable firewalld \
&& systemctl status firewalld

2.6设置SELinux模式 #

setenforce 0 \
|| sed -i 's/enforcing/disabled/g' /etc/selinux/config \
&& sed -i 's/permissive/disabled/g' /etc/selinux/config \
&& getenforce

2.7配置时间同步 #

vim /etc/ntp.conf

将时钟服务器更改为如下4个

server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org   
server 3.cn.pool.ntp.org

注意：若为离线安装，则要将服务器地址改为内网时间服务器地址。

启动设置ntp服务

systemctl start ntpd \
&& systemctl enable ntpd \
&& ntpdate -u 0.cn.pool.ntp.org \
&& hwclock --systohc \
&& date

2.8创建自定义网络 #

docker network create --subnet=172.10.0.0/16 hadoop_net \
&& docker network ls

2.9下载CDH相关资源 #

由于CDH已经从2021年2月以后已经不提供免费下载了，所以我这里收集了一份已经下载好的安装必须的软件包集合（包括CDH，MYSQL，Dockerfile）。请大家自行下载
CDH资源下载提取码: u07l

2.10创建基础系统镜像 #

在下载后CDH文件夹中，Dockerfile同级目录执行:

docker build -t centos7-cdh .

2.11启动容器 #

如果想要开机自启动容器。可以在命令中加上--restart always

docker run -d \
--add-host cm.hadoop:172.10.0.2 \
--net hadoop_net \
--ip 172.10.0.2 \
-h cm.hadoop \
-p 10022:22 \
-p 7180:7180 \
--name cm.hadoop \
--privileged \
centos7-cdh \
/usr/sbin/init \
&& docker ps

2.12拷贝安装CDH需要的文件到容器 #

在下载后CDH文件夹中，hadoop_CDH文件夹同级目录执行:
{dockerId} 为2.11中创建的容器id

docker cp hadoop_CDH {dockerId}:/root/hadoop_CDH

三、容器安装ClouderaManager #

3.1初始化环境 #

进入第二步创建的容器中，并将root的登录密码改为root

docker exec –it {dockerId} /bin/bash
su root
passwd
root
root

安装基础环境

yum install -y kde-l10n-Chinese telnet reinstall glibc-common vim wget ntp net-tools \
&& yum clean all

3.2配置中文环境变量 #

(
cat <<EOF
export LC_ALL=zh_CN.utf8
export LANG=zh_CN.utf8
export LANGUAGE=zh_CN.utf8
EOF
) >> ~/.bashrc \
&& localedef -c -f UTF-8 -i zh_CN zh_CN.utf8 \
&& source ~/.bashrc \
&& echo $LANG

3.3设置NTP时间同步服务 #

vim /etc/ntp.conf

更改为以下四个时钟服务器

server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org

调整时区

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

启动ntp服务

systemctl start ntpd && \
systemctl enable ntpd && \
ntpdate -u 0.cn.pool.ntp.org && \
hwclock --systohc && \
date

3.4搭建本地yum源 #

yum -y install httpd createrepo \
&& systemctl start httpd \
&& systemctl enable httpd \
&& cd /root/hadoop_CDH/cloudera-repos/ && createrepo . \
&& mv /root/hadoop_CDH/cloudera-repos /var/www/html/ \
&& yum clean all \
&& ll /var/www/html/cloudera-repos

3.5安装jdk #

cd /var/www/html/cloudera-repos/;rpm -ivh oracle-j2sdk1.8-1.8.0update181-1.x86_64.rpm

3.6安装配置MySQL数据库 #

cd /root/hadoop_CDH/mysql/;tar -xvf mysql-5.7.27-1.el7.x86_64.rpm-bundle.tar \
&& yum install -y libaio numactl \
&& rpm -ivh mysql-community-common-5.7.27-1.el7.x86_64.rpm \
&& rpm -ivh mysql-community-libs-5.7.27-1.el7.x86_64.rpm \
&& rpm -ivh mysql-community-client-5.7.27-1.el7.x86_64.rpm \
&& rpm -ivh mysql-community-server-5.7.27-1.el7.x86_64.rpm \
&& rpm -ivh mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm \
&& echo character-set-server=utf8 >> /etc/my.cnf \
&& rm -rf /root/hadoop_CDH/mysql/ \
&& yum clean all \
&& rpm -qa |grep mysql

3.7数据库授权 #

(
cat <<EOF
set password for root@localhost = password('123456Aa.');
grant all privileges on *.* to 'root'@'%' identified by '123456Aa.';
flush privileges;
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456Aa.';
SHOW DATABASES;
EOF
) >> /root/c.sql

获取MySQL初始密码

systemctl start mysqld && grep password /var/log/mysqld.log | sed 's/.*\(............\)$/\1/'

执行SQL脚本 {password}为刚刚查询的初始密码

mysql -uroot –p‘{password}’

登陆后执行

source /root/c.sql

3.8配置mysql jdbc驱动 #

mkdir -p /usr/share/java/ \
&& cd /root/hadoop_CDH/mysql-jdbc/;tar -zxvf mysql-connector-java-5.1.48.tar.gz \
&& cp  /root/hadoop_CDH/mysql-jdbc/mysql-connector-java-5.1.48/mysql-connector-java-5.1.48-bin.jar /usr/share/java/mysql-connector-java.jar \
&& rm -rf /root/hadoop_CDH/mysql-jdbc/ \
&& ls /usr/share/java/

3.9安装Cloudera Manager #

(
cat <<EOF
[cloudera-manager]
name=Cloudera Manager 6.3.0
baseurl=http://172.10.0.2/cloudera-repos/
gpgcheck=0
enabled=1
EOF
) >> /etc/yum.repos.d/cloudera-manager.repo \
&& yum clean all \
&& yum makecache \
&& yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server \
&& yum clean all \
&& rpm -qa | grep cloudera-manager

3.10配置parcel库 #

cd /opt/cloudera/parcel-repo/;mv /root/hadoop_CDH/parcel/* ./ \
&& sha1sum CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel | awk '{ print $1 }' > CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha \
&& rm -rf /root/hadoop_CDH/parcel/ \
&& chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/* \
&& ll /opt/cloudera/parcel-repo/

3.11初始化scm库 #

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm 123456Aa.

3.12启动cloudera-server服务 #

systemctl start cloudera-scm-server \
&& sleep 2 \
&& tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log | grep "INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server"

四、配置CDH的worker节点 #

以下为worker容器的准备方式，若为多个时，重复执行以下步骤，创建多个worker节点。

4.1创建多个worker容器 #

创建2个work容器
Worker-1:

docker run -d \
--add-host cm.hadoop:172.10.0.2 \
--add-host cdh01.hadoop:172.10.0.3\
--net hadoop_net \
--ip 172.10.0.3 \
-h cdh01.hadoop \
-p 20022:22 \
--name cdh01.hadoop \
--privileged \
centos7-cdh \
/usr/sbin/init \
&& docker ps

Worker-2:

docker run -d \
--add-host cm.hadoop:172.10.0.2 \
--add-host cdh02.hadoop:172.10.0.4 \
--net hadoop_net \
--ip 172.10.0.4 \
-h cdh02.hadoop \
-p 30022:22 \
--name cdh02.hadoop \
--privileged \
centos7-cdh \
/usr/sbin/init \
&& docker ps

说明：
如果需要开机重启可以添加--restart always开启

4.2装基本工具 #

上一步创建的所有容器均执行，修改root的登录密码改为root

docker exec –it {dockerId} /bin/bash
su root
passwd
root
root

然后执行：

yum install -y kde-l10n-Chinese telnet reinstall glibc-common vim wget ntp net-tools \
&& yum clean all

4.3配置中文环境变量 #

(
cat <<EOF
export LC_ALL=zh_CN.utf8
export LANG=zh_CN.utf8
export LANGUAGE=zh_CN.utf8
EOF
) >> ~/.bashrc \
&& localedef -c -f UTF-8 -i zh_CN zh_CN.utf8 \
&& source ~/.bashrc \
&& echo $LANG

4.4配置NTP时间同步服务 #

vim /etc/ntp.conf

更改为以下四个时钟服务器

server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org

调整时区

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

启动ntp服务

systemctl start ntpd && \
systemctl enable ntpd && \
ntpdate -u 0.cn.pool.ntp.org && \
hwclock --systohc && \
date

4.5配置MySQL JDBC #

mkdir -p /usr/share/java/ \
&& wget -O /usr/share/java/mysql-connector-java-5.1.48.tar.gz \
https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.48.tar.gz \
&& cd /usr/share/java/;tar -zxvf mysql-connector-java-5.1.48.tar.gz \
&& cp /usr/share/java/mysql-connector-java-5.1.48/mysql-connector-java-5.1.48-bin.jar /usr/share/java/mysql-connector-java.jar \
&& rm -rf mysql-connector-java-5.1.48 mysql-connector-java-5.1.48.tar.gz \
&& ls /usr/share/java/

4.6修改CM主机的host文件 #

echo "172.10.0.3      cdh01.hadoop cdh01" >> /etc/hosts
echo "172.10.0.4      cdh02.hadoop cdh02" >> /etc/hosts

注意：不是在worker节点上操作，而是在2.11创建的cloud-manager节点上操作，有多少个work容器，执行多少次，内容按真实情况填写即可。

五、CM管理平台创建CDH集群 #

5.1登陆CM管理平台 #

http://IP:7180/cmf/login 账号密码：admin/admin

六、用户设置 #

用于spark等需要用hdfs用户，需要设置一下dhfs账户的登陆权限

以cm机器为例

docker exec -it {cdh01.containerId} bash
vim /etc/passwd
#搜有hdfs
:/hsfs
#修改 hdfs:x:996:993:Hadoop HDFS:/var/lib/hadoop-hdfs:/sbin/nologin
#为 hdfs:x:996:993:Hadoop HDFS:/var/lib/hadoop-hdfs:/bin/bash
:wq

之后测试一下

su hdfs
spark-shell