Cloudera Manager(简称CM)是Cloudera公司开发的一款大数据集群安装部署利器，这款利器具有集群自动化安装、中心化管理、集群监控、报警等功能，使得安装集群从几天的时间缩短在几小时以内，运维人员从数十人降低到几人以内，极大的提高集群管理的效率。
CDH (Cloudera’s Distribution, including Apache Hadoop)，是Hadoop众多分支中的一种，由Cloudera维护，基于稳定版本的Apache Hadoop构建，并集成了很多补丁，可直接用于生产环境。
文档地址：官网安装教程
下载地址：CM 安装包下载
https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm另外需要下载 allkeys.asc
https://archive.cloudera.com/cm6/6.3.1/allkeys.asc下载地址：CDH 安装包下载
https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 https://archive.cloudera.com/cdh6/6.3.2/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256 https://archive.cloudera.com/cdh6/6.3.2/parcels/manifest.json注意：默认位置 /usr/share/java/ ，默认驱动名 mysql-connector-java.jar
# 创建文件夹 mkdir -p /usr/share/java/ # 进入 MySQL 驱动文件目录 cd /usr/share/java/ # 下载 MySQL 驱动 wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz # -bash: wget: 未找到命令 yum -y install wget # 解压压缩包 tar zxvf mysql-connector-java-5.1.46.tar.gz # 复制并且重命名 cp /usr/share/java/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar参考文章：CentOS7 部署 NTP 服务器进行时间同步
先省略，后补下载地址：https://www.oracle.com/technetwork/java/javase/archive-139210.html
# 创建文件夹 mkdir -p /usr/java/ # 上传 JDK8 压缩文件 # 解压 tar -zxvf jdk-8u131-linux-x64.tar.gz # 配置当前用户环境变量 vi /etc/profile # 在文件中添加如下命令，记得切换自己文件路径 export JAVA_HOME=/usr/java/jdk1.8.0_131 export PATH=$JAVA_HOME/bin:$PATH # 立即生效 source /etc/profile # 同步java文件夹到其它服务器 scp -r /usr/java root@cm02:/usr/ scp -r /usr/java root@cm03:/usr/ #同步/etc/profile文件 scp /etc/profile root@cm02:/etc/profile scp /etc/profile root@cm03:/etc/profile # 立即生效 source /etc/profile1、参考地址：官网 MySQL 安装教程
# 创建 mysql 文件夹 mkdir -p /var/www/html/cloudera-repos/mysql5/ # 进入目录 cd /var/www/html/cloudera-repos/mysql5/ # 下载 wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm rpm -ivh mysql-community-release-el7-5.noarch.rpm # 更新 yum update # 安装 mysql yum install mysql-server # 启动 mysql systemctl start mysqld # 开机自启动 systemctl enable mysqld2、初始化数据库
# 初始化数据库 /usr/bin/mysql_secure_installation [root@cm01 mysql5]# /usr/bin/mysql_secure_installation NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MySQL to secure it, we'll need the current password for the root user. If you've just installed MySQL, and you haven't set the root password yet, the password will be blank, so you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password ensures that nobody can log into the MySQL root user without the proper authorisation. Set root password? [Y/n] Y New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MySQL installation has an anonymous user, allowing anyone to log into MySQL without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] Y ... Success! Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] n ... skipping. By default, MySQL comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] Y - Dropping test database... ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist ... Failed! Not critical, keep moving... - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] Y ... Success! All done! If you've completed all of the above steps, your MySQL installation should now be secure. Thanks for using MySQL! Cleaning up... [root@cm01 mysql5]#3、备份数据库配置文件
cp /etc/my.cnf{,.bak}4、官网推荐配置
[mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock transaction-isolation = READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # to do so, uncomment this line: symbolic-links = 0 key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_cache_size = 64 query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1 max_connections = 550 #expire_logs_days = 10 #max_binlog_size = 100M #log_bin should be on a disk with enough free space. #Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your #system and chown the specified folder to the mysql user. log_bin=/var/lib/mysql/mysql_binary_log #In later versions of MySQL, if you enable the binary log and do not set #a server_id, MySQL will not start. The server_id must be unique within #the replicating group. server_id=1 binlog_format = mixed read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M # InnoDB settings innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid sql_mode=STRICT_ALL_TABLES6、创建数据库
ServiceDatabaseUserCloudera Manager ServerscmscmActivity MonitoramonamonReports ManagerrmanrmanHuehuehueHive Metastore ServermetastorehiveSentry ServersentrysentryCloudera Navigator Audit ServernavnavCloudera Navigator Metadata ServernavmsnavmsOozieoozieoozie # 登录 MySQL 数据库 mysql -uroot -p # 初始化 CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm@123'; CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon@123'; CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman@123'; CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue@123'; CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY 'metastore@123'; CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry@123'; CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav@123'; CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms@123'; CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie@123'; flush privileges; # 删除 drop database scm; drop database amon; drop database rman; drop database hue; drop database metastore; drop database sentry; drop database nav; drop database navms; drop database oozie; flush privileges; #开放mysql远程权限 GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; FLUSH PRIVILEGES;主节点
yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server所有从节点
yum install cloudera-manager-daemons cloudera-manager-agent参考文档：官网创建 CM 数据库教程
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm用户名：admin ，密码：admin
1、欢迎页面
2、接受最终用户许可条款和条件。
3、选择免费版本
1、欢迎页面
2、群集名称
3、扫描主机，输入主机名称点击搜索
4、选择存储库
自定义存储库：http://woniu/cloudera-repos/cm6/
远程 Parcel 存储库 URL ：http://woniu/cloudera-repos/cdh6/
保存更改
5、JDK 安装选项，跳过不需要勾选
6、提供 SSH 登录凭据
7、Install Agents ，自动安装
7、Install Parcels，自动安装
8、Inspect Cluster
先点击 Inspect Network Performance 再点击 Inspect Hosts
警告，点击 显示检查器结果
已启用透明大页面压缩，可能会导致重大性能问题。请运行“echo never > /sys/kernel/mm/transparent_hugepage/defrag”和“echo never > /sys/kernel/mm/transparent_hugepage/enabled”以禁用此设置，然后将同一命令添加到 /etc/rc.local 等初始化脚本中，以便在系统重启时予以设置
echo never>/sys/kernel/mm/transparent_hugepage/defrag echo never>/sys/kernel/mm/transparent_hugepage/enabled点击重新运行
1、Select Services
2、自定义角色分配
3、数据库设置
Databases for Cloudera Software 参考如下
点击测试连接
4、审核更改，默认设置
5、命令详细信息
6、汇总
