Cloudera Manager 6.2.1安装详解

一、环境准备

1、Linux环境

1.1关闭防火墙

#Centos7查看防火墙状态 systemctl status firewalld #临时关闭防火墙 systemctl stop firewalld #永久关闭防火墙 systemctl disable firewalld

1.2 关闭selinux

vim /etc/selinux/config

默认是关闭的

1.3 修改Linux swapping参数

#查看 cat /proc/sys/vm/swappiness #30 #修改当前 echo 0 > /proc/sys/vm/swappiness #添加到配置文件 echo "vm.swappiness=0" >> /etc/sysctl.conf

1.4 禁止透明页

echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled #添加到配置文件 echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local #避免重启后不生效 添加执行权限 chmod +x /etc/rc.d/rc.local

2、修改hostname

可以参考这个https://blog.csdn.net/qq_37056683/article/details/108508563

修改后的结果如下：

3、配置SSH免密

第一步:在本地机器上使用ssh-keygen产生公钥私钥对

ssh-keygen -t rsa #可以不用设置密码 一直enter

第二步:用ssh-copy-id将公钥复制到远程机器中

ssh-copy-id root@bigdata206 ssh-copy-id root@bigdata207 ssh-copy-id root@bigdata208

4、jdk安装

4.1 jdk版本要求

官网链接：https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_java_requirements.html#java_requirements

4.2 jdk安装

1.8.0_201版本 链接: https://pan.baidu.com/s/1XFAeaQp2WTCAYcfJznSKCw 提取码: n8gq

#把下载的tar包解压到/usr下 tar -zxf java.tar.gz -C /usr/ #添加环境变量 vim /etc/profile export JAVA_HOME=/usr/java/jdk1.8.0_201 export PATH=$PATH:$JAVA_HOME/bin #生效 source /etc/profile #校验 java -version

5、配置NTP服务器

#安装 yum -y install ntp #使用crontab 每半个小时同步一次内网的一台机器 #time 30 * * * * /usr/sbin/ntpdate 10.1.1.254; /sbin/hwclock -w #开启ntp服务 systemctl start ntpd.service systemctl enable ntpd.service

6、安装MySQL

需要先卸载主节点mariadb：

#查找 mariadb：rpm -qa|grep mariadb #卸载 rpm -e --nodeps mariadb-libs-5.5.64-1.el7.x86_64

可以参考MySQL5.7 yum方式安装： https://blog.csdn.net/qq_37056683/article/details/107809525

创建Cloudera Manager需要的数据库

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

修改配置文件

vim /etc/my.cnf [mysqld] basedir = /usr/local/mysql datadir = /var/lib/mysql port = 3306 socket=/var/lib/mysql/mysql.sock character-set-server=utf8 transaction-isolation = READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # to do so, uncomment this line: symbolic-links = 0 server_id=1 max-binlog-size = 500M log_bin=/var/lib/mysql/mysql_binary_log #binlog_format = mixed binlog_format = Row expire-logs-days = 14 max_connections = 550 read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M # InnoDB settings innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M [client] default-character-set=utf8 socket=/var/lib/mysql/mysql.sock [mysql] default-character-set=utf8 socket=/var/lib/mysql/mysql.sock [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid sql_mode=STRICT_ALL_TABLES skip-ssl

MySQL连接器配置

链接: https://pan.baidu.com/s/1KrAk0-fA6aGQ-Ngy6bAhaA 提取码: ge9k

#创建目录 mkdir -p /usr/share/java #移动到新建目录下 并改为mysql-connector-java.jar（重点） mv mysql-connector-java-5.1.34.jar /usr/share/java/mysql-connector-java.jar

安装第三方依赖

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb httpd mod_ssl portmap /lib/lsb/init-functions openssl-devel python-psycopg2 MySQL-python

二、Cloudera Manager安装

1、准备yum本地源

1.1 httpd服务

yum install -y httpd systemctl start httpd.service systemctl enable httpd.service

1.2 本地源

官网地址：https://archive.cloudera.com/cm6/6.2.1/redhat7/yum/RPMS/x86_64/

#创建目录 mkdir -p /var/www/html/cloudera-repos #下载文件 wget https://archive.cloudera.com/cm6/6.2.1/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm wget https://archive.cloudera.com/cm6/6.2.1/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm wget https://archive.cloudera.com/cm6/6.2.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.2.1-1426065.el7.x86_64.rpm

浏览器可以访问查看

2、CDH Parcel相关文件

#创建目录 mdkir -p /opt/cloudera/parcel-repo #下载文件 wget https://archive.cloudera.com/cdh6/6.2.1/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel wget https://archive.cloudera.com/cdh6/6.2.1/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel.sha1 wget https://archive.cloudera.com/cdh6/6.2.1/parcels/manifest.json #添加cloudera-scm用户 useradd --system --home=/opt/cloudera-manager/cm-6.2.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm #修改用户组 chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/ #校验parcel文件 sha1sum CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel #查看.sha1内容 cat CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel.sha1 #如果一样 直接修改.sha1 如果不一样将sha1sum后的值替换到.sha1文件中 并改为.sha后缀 mv CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel.sha1 CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel.sha

3、 Cloudera Manager安装

3.1 采用本地安装模式：

#因为依赖关系 注意安装顺序 yum -y localinstall http://10.0.128.210/cloudera-repos/cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm # 主节点安装server yum -y localinstall http://10.0.128.210/cloudera-repos/cloudera-manager-server-6.2.1-1426065.el7.x86_64.rpm #所有节点安装agent yum -y localinstall http://10.0.128.210/cloudera-repos/cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm

3.2 初始化Cloudera Manager Database

# /opt/cloudera/cm/schema/scm_prepare_database.sh 数据库类型 数据库名 用户名 密码 /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root root

3.3 启动Cloudera Manager

#启动server systemctl start cloudera-scm-server systemctl enable cloudera-scm-server #启动agent（所有节点） systemctl start cloudera-scm-agent systemctl enable cloudera-scm-agent

访问页面 http://10.0.128.206:7180/cmf/login

三 配置Cloudera Manager

1、Cloudera Manager 欢迎页

2、版本选择

3、集群欢迎页

4、配置集群主机节点

5、选择本地CDH版本

6、jdk环境

7、提供SSH登录凭据

8、主机相关服务的安装

9、CDH下载和分发

10、检查主机环境

11、CDH相关服务的安装

12、数据库设置

这个地方遇到了一个问题：Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200 一直测试不通过 因为用的是IE浏览器 后来改成Google浏览器就好了

13、完成

启动的时候遇到的问题

1.java.io.IOException: NameNode is not formatted.

#格式化 hdfs namenode -format

2、不良 : Canary 测试无法为 /tmp/.cloudera_health_monitoring_canary_files 创建父目录。

#检查/tmp目录是否存在,不存在,进行创建 hdfs dfs -ls /tmp #创建tmp目录 hdfs dfs -mkdir /tmp #修改权限 hdfs dfs -chmod -R 777 /tmp #如果执行权限不够往下看 #修改用户 sudo -u hdfs hadoop fs -chown hdfs:supergroup /tmp #或者执行下面这句 sudo -u hdfs hadoop fs -chmod -R 777 /tmp

3. Error creating done directory: [hdfs://bigdata206:8020/user/history/done]

sudo -u hdfs hadoop fs -mkdir -p /user/history sudo -u hdfs hadoop fs -chmod -R 1777 /user/history sudo -u hdfs hadoop fs -chown mapred:hadoop /user/history

---

CDH6.2.1 hdfs 高可用配置可以参考 ：https://blog.csdn.net/qq_37056683/article/details/109229949