Hadoop分布式集群

it2024-12-27 11

Hadoop分布式集群

单机安装部署

购买云主机

主机IP地址配置hadoop1192.168.1.50最低配置2核2G

hadoop安装

拷贝云盘 public/hadoop/hadoop-2.7.7.tar.gz 到 hadoop1 上

[root@hadoop1 ~]# yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel [root@hadoop1 ~]# tar -zxf hadoop-2.7.7.tar.gz [root@hadoop1 ~]# mv hadoop-2.7.7 /usr/local/hadoop [root@hadoop1 ~]# chown -R 0.0 /usr/local/hadoop

配置JAVA运行环境

[root@hadoop1 ~]# vim /etc/hosts 192.168.1.50 hadoop1 [root@hadoop1 ~]# rpm -qa | grep java [root@hadoop1 ~]# rpm -ql java-1.8.0-openjdk [root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh 25: export JAVA_HOME="java-1.8.0-openjdk安装路径" 33: export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop" [root@hadoop1 ~]# /usr/local/hadoop/bin/hadoop version

热点词汇分析

[root@hadoop1 ~]# cd /usr/local/hadoop [root@hadoop1 hadoop]# mkdir input [root@hadoop1 hadoop]# cp *.txt input/ [root@hadoop1 hadoop]# ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar wordcount ./input ./output [root@hadoop1 hadoop]# cat ./output/*

集群安装部署

购买云主机

主机IP地址配置hadoop1192.168.1.50最低配置2核2Gnode-0001192.168.1.51最低配置2核2Gnode-0002192.168.1.52最低配置2核2Gnode-0003192.168.1.53最低配置2核2G

HDFS架构图

#mermaid-svg-xN536ItnLegxBrWj .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-xN536ItnLegxBrWj .label text{fill:#333}#mermaid-svg-xN536ItnLegxBrWj .node rect,#mermaid-svg-xN536ItnLegxBrWj .node circle,#mermaid-svg-xN536ItnLegxBrWj .node ellipse,#mermaid-svg-xN536ItnLegxBrWj .node polygon,#mermaid-svg-xN536ItnLegxBrWj .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-xN536ItnLegxBrWj .node .label{text-align:center;fill:#333}#mermaid-svg-xN536ItnLegxBrWj .node.clickable{cursor:pointer}#mermaid-svg-xN536ItnLegxBrWj .arrowheadPath{fill:#333}#mermaid-svg-xN536ItnLegxBrWj .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-xN536ItnLegxBrWj .flowchart-link{stroke:#333;fill:none}#mermaid-svg-xN536ItnLegxBrWj .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-xN536ItnLegxBrWj .edgeLabel rect{opacity:0.9}#mermaid-svg-xN536ItnLegxBrWj .edgeLabel span{color:#333}#mermaid-svg-xN536ItnLegxBrWj .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-xN536ItnLegxBrWj .cluster text{fill:#333}#mermaid-svg-xN536ItnLegxBrWj div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-xN536ItnLegxBrWj .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-xN536ItnLegxBrWj text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-xN536ItnLegxBrWj .actor-line{stroke:grey}#mermaid-svg-xN536ItnLegxBrWj .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-xN536ItnLegxBrWj .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-xN536ItnLegxBrWj #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-xN536ItnLegxBrWj .sequenceNumber{fill:#fff}#mermaid-svg-xN536ItnLegxBrWj #sequencenumber{fill:#333}#mermaid-svg-xN536ItnLegxBrWj #crosshead path{fill:#333;stroke:#333}#mermaid-svg-xN536ItnLegxBrWj .messageText{fill:#333;stroke:#333}#mermaid-svg-xN536ItnLegxBrWj .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-xN536ItnLegxBrWj .labelText,#mermaid-svg-xN536ItnLegxBrWj .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-xN536ItnLegxBrWj .loopText,#mermaid-svg-xN536ItnLegxBrWj .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-xN536ItnLegxBrWj .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-xN536ItnLegxBrWj .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-xN536ItnLegxBrWj .noteText,#mermaid-svg-xN536ItnLegxBrWj .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-xN536ItnLegxBrWj .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-xN536ItnLegxBrWj .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-xN536ItnLegxBrWj .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-xN536ItnLegxBrWj .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-xN536ItnLegxBrWj .section{stroke:none;opacity:0.2}#mermaid-svg-xN536ItnLegxBrWj .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-xN536ItnLegxBrWj .section2{fill:#fff400}#mermaid-svg-xN536ItnLegxBrWj .section1,#mermaid-svg-xN536ItnLegxBrWj .section3{fill:#fff;opacity:0.2}#mermaid-svg-xN536ItnLegxBrWj .sectionTitle0{fill:#333}#mermaid-svg-xN536ItnLegxBrWj .sectionTitle1{fill:#333}#mermaid-svg-xN536ItnLegxBrWj .sectionTitle2{fill:#333}#mermaid-svg-xN536ItnLegxBrWj .sectionTitle3{fill:#333}#mermaid-svg-xN536ItnLegxBrWj .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-xN536ItnLegxBrWj .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-xN536ItnLegxBrWj .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-xN536ItnLegxBrWj .grid path{stroke-width:0}#mermaid-svg-xN536ItnLegxBrWj .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-xN536ItnLegxBrWj .task{stroke-width:2}#mermaid-svg-xN536ItnLegxBrWj .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-xN536ItnLegxBrWj .taskText:not([font-size]){font-size:11px}#mermaid-svg-xN536ItnLegxBrWj .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-xN536ItnLegxBrWj .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-xN536ItnLegxBrWj .task.clickable{cursor:pointer}#mermaid-svg-xN536ItnLegxBrWj .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-xN536ItnLegxBrWj .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-xN536ItnLegxBrWj .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-xN536ItnLegxBrWj .taskText0,#mermaid-svg-xN536ItnLegxBrWj .taskText1,#mermaid-svg-xN536ItnLegxBrWj .taskText2,#mermaid-svg-xN536ItnLegxBrWj .taskText3{fill:#fff}#mermaid-svg-xN536ItnLegxBrWj .task0,#mermaid-svg-xN536ItnLegxBrWj .task1,#mermaid-svg-xN536ItnLegxBrWj .task2,#mermaid-svg-xN536ItnLegxBrWj .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-xN536ItnLegxBrWj .taskTextOutside0,#mermaid-svg-xN536ItnLegxBrWj .taskTextOutside2{fill:#000}#mermaid-svg-xN536ItnLegxBrWj .taskTextOutside1,#mermaid-svg-xN536ItnLegxBrWj .taskTextOutside3{fill:#000}#mermaid-svg-xN536ItnLegxBrWj .active0,#mermaid-svg-xN536ItnLegxBrWj .active1,#mermaid-svg-xN536ItnLegxBrWj .active2,#mermaid-svg-xN536ItnLegxBrWj .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-xN536ItnLegxBrWj .activeText0,#mermaid-svg-xN536ItnLegxBrWj .activeText1,#mermaid-svg-xN536ItnLegxBrWj .activeText2,#mermaid-svg-xN536ItnLegxBrWj .activeText3{fill:#000 !important}#mermaid-svg-xN536ItnLegxBrWj .done0,#mermaid-svg-xN536ItnLegxBrWj .done1,#mermaid-svg-xN536ItnLegxBrWj .done2,#mermaid-svg-xN536ItnLegxBrWj .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-xN536ItnLegxBrWj .doneText0,#mermaid-svg-xN536ItnLegxBrWj .doneText1,#mermaid-svg-xN536ItnLegxBrWj .doneText2,#mermaid-svg-xN536ItnLegxBrWj .doneText3{fill:#000 !important}#mermaid-svg-xN536ItnLegxBrWj .crit0,#mermaid-svg-xN536ItnLegxBrWj .crit1,#mermaid-svg-xN536ItnLegxBrWj .crit2,#mermaid-svg-xN536ItnLegxBrWj .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-xN536ItnLegxBrWj .activeCrit0,#mermaid-svg-xN536ItnLegxBrWj .activeCrit1,#mermaid-svg-xN536ItnLegxBrWj .activeCrit2,#mermaid-svg-xN536ItnLegxBrWj .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-xN536ItnLegxBrWj .doneCrit0,#mermaid-svg-xN536ItnLegxBrWj .doneCrit1,#mermaid-svg-xN536ItnLegxBrWj .doneCrit2,#mermaid-svg-xN536ItnLegxBrWj .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-xN536ItnLegxBrWj .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-xN536ItnLegxBrWj .milestoneText{font-style:italic}#mermaid-svg-xN536ItnLegxBrWj .doneCritText0,#mermaid-svg-xN536ItnLegxBrWj .doneCritText1,#mermaid-svg-xN536ItnLegxBrWj .doneCritText2,#mermaid-svg-xN536ItnLegxBrWj .doneCritText3{fill:#000 !important}#mermaid-svg-xN536ItnLegxBrWj .activeCritText0,#mermaid-svg-xN536ItnLegxBrWj .activeCritText1,#mermaid-svg-xN536ItnLegxBrWj .activeCritText2,#mermaid-svg-xN536ItnLegxBrWj .activeCritText3{fill:#000 !important}#mermaid-svg-xN536ItnLegxBrWj .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-xN536ItnLegxBrWj g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-xN536ItnLegxBrWj g.classGroup text .title{font-weight:bolder}#mermaid-svg-xN536ItnLegxBrWj g.clickable{cursor:pointer}#mermaid-svg-xN536ItnLegxBrWj g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-xN536ItnLegxBrWj g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-xN536ItnLegxBrWj .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-xN536ItnLegxBrWj .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-xN536ItnLegxBrWj .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-xN536ItnLegxBrWj .dashed-line{stroke-dasharray:3}#mermaid-svg-xN536ItnLegxBrWj #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-xN536ItnLegxBrWj #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-xN536ItnLegxBrWj #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-xN536ItnLegxBrWj #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-xN536ItnLegxBrWj #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-xN536ItnLegxBrWj #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-xN536ItnLegxBrWj #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-xN536ItnLegxBrWj #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-xN536ItnLegxBrWj .commit-id,#mermaid-svg-xN536ItnLegxBrWj .commit-msg,#mermaid-svg-xN536ItnLegxBrWj .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-xN536ItnLegxBrWj .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-xN536ItnLegxBrWj .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-xN536ItnLegxBrWj g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-xN536ItnLegxBrWj g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-xN536ItnLegxBrWj g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-xN536ItnLegxBrWj g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-xN536ItnLegxBrWj g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-xN536ItnLegxBrWj g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-xN536ItnLegxBrWj .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-xN536ItnLegxBrWj .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-xN536ItnLegxBrWj .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-xN536ItnLegxBrWj .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-xN536ItnLegxBrWj .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-xN536ItnLegxBrWj .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-xN536ItnLegxBrWj .edgeLabel text{fill:#333}#mermaid-svg-xN536ItnLegxBrWj .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-xN536ItnLegxBrWj .node circle.state-start{fill:black;stroke:black}#mermaid-svg-xN536ItnLegxBrWj .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-xN536ItnLegxBrWj #statediagram-barbEnd{fill:#9370db}#mermaid-svg-xN536ItnLegxBrWj .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-xN536ItnLegxBrWj .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-xN536ItnLegxBrWj .statediagram-state .divider{stroke:#9370db}#mermaid-svg-xN536ItnLegxBrWj .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-xN536ItnLegxBrWj .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-xN536ItnLegxBrWj .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-xN536ItnLegxBrWj .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-xN536ItnLegxBrWj .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-xN536ItnLegxBrWj .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-xN536ItnLegxBrWj .note-edge{stroke-dasharray:5}#mermaid-svg-xN536ItnLegxBrWj .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-xN536ItnLegxBrWj .error-icon{fill:#522}#mermaid-svg-xN536ItnLegxBrWj .error-text{fill:#522;stroke:#522}#mermaid-svg-xN536ItnLegxBrWj .edge-thickness-normal{stroke-width:2px}#mermaid-svg-xN536ItnLegxBrWj .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-xN536ItnLegxBrWj .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-xN536ItnLegxBrWj .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-xN536ItnLegxBrWj .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-xN536ItnLegxBrWj .marker{fill:#333}#mermaid-svg-xN536ItnLegxBrWj .marker.cross{stroke:#333} :root { --mermaid-font-family: "trebuchet ms", verdana, arial;} #mermaid-svg-xN536ItnLegxBrWj { color: rgba(0, 0, 0, 0.75); font: ; } hadoop1 NameNode SecondaryNamenode node-0001 Datanode node-0002 Datanode node-0003 Datanode

HDFS部署

以下操作所有机器都要执行

[root@hadoop1 ~]# yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel [root@hadoop1 ~]# vim /etc/hosts 192.168.1.50 hadoop1 192.168.1.51 node-0001 192.168.1.52 node-0002 192.168.1.53 node-0003

以下操作仅在 hadoop1 上执行

[root@hadoop1 ~]# vim /etc/ssh/ssh_config # 60行新添加 StrictHostKeyChecking no [root@hadoop1 ~]# ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa [root@hadoop1 ~]# for i in hadoop1 node-{0001..0003};do ssh-copy-id -i /root/.ssh/id_rsa.pub ${i} done

配置文件语法格式 – 官方手册

1、配置 hadoop-env.sh 参考配置JAVA运行环境案例

2、配置slaves

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/slaves node-0001 node-0002 node-0003

3、配置core-site.xml

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/core-site.xml <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hadoop1:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/var/hadoop</value> </property> </configuration>

4、配置hdfs-site.xml

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml <configuration> <property> <name>dfs.namenode.http-address</name> <value>hadoop1:50070</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>hadoop1:50090</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> </configuration>

5、启动集群 [以下操作仅在 hadoop1 上执行]

[root@hadoop1 ~]# for i in node-{0001..0003};do rsync -aXSH --delete /usr/local/hadoop ${i}:/usr/local/ done [root@hadoop1 ~]# mkdir /var/hadoop [root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs namenode -format [root@hadoop1 ~]# /usr/local/hadoop/sbin/start-dfs.sh

6、验证集群配置

[root@hadoop1 ~]# for i in hadoop1 node-{0001..0003};do echo ${i}; ssh ${i} jps; echo -e "\n"; done [root@hadoop1 ~]# /usr/local/hadoop/bin/hdfs dfsadmin -report ... ... ------------------------------------------------- Live datanodes (3):

mapreduce部署

[root@hadoop1 ~]# cd /usr/local/hadoop/etc/hadoop/ [root@hadoop1 hadoop]# cp mapred-site.xml.template mapred-site.xml [root@hadoop1 hadoop]# vim mapred-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>

YARN架构图

#mermaid-svg-h3SY7Xffdrpr00Vz .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .label text{fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .node rect,#mermaid-svg-h3SY7Xffdrpr00Vz .node circle,#mermaid-svg-h3SY7Xffdrpr00Vz .node ellipse,#mermaid-svg-h3SY7Xffdrpr00Vz .node polygon,#mermaid-svg-h3SY7Xffdrpr00Vz .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-h3SY7Xffdrpr00Vz .node .label{text-align:center;fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .node.clickable{cursor:pointer}#mermaid-svg-h3SY7Xffdrpr00Vz .arrowheadPath{fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-h3SY7Xffdrpr00Vz .flowchart-link{stroke:#333;fill:none}#mermaid-svg-h3SY7Xffdrpr00Vz .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-h3SY7Xffdrpr00Vz .edgeLabel rect{opacity:0.9}#mermaid-svg-h3SY7Xffdrpr00Vz .edgeLabel span{color:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-h3SY7Xffdrpr00Vz .cluster text{fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-h3SY7Xffdrpr00Vz .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-h3SY7Xffdrpr00Vz text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-h3SY7Xffdrpr00Vz .actor-line{stroke:grey}#mermaid-svg-h3SY7Xffdrpr00Vz .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-h3SY7Xffdrpr00Vz #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .sequenceNumber{fill:#fff}#mermaid-svg-h3SY7Xffdrpr00Vz #sequencenumber{fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz #crosshead path{fill:#333;stroke:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .messageText{fill:#333;stroke:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-h3SY7Xffdrpr00Vz .labelText,#mermaid-svg-h3SY7Xffdrpr00Vz .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-h3SY7Xffdrpr00Vz .loopText,#mermaid-svg-h3SY7Xffdrpr00Vz .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-h3SY7Xffdrpr00Vz .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-h3SY7Xffdrpr00Vz .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-h3SY7Xffdrpr00Vz .noteText,#mermaid-svg-h3SY7Xffdrpr00Vz .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-h3SY7Xffdrpr00Vz .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-h3SY7Xffdrpr00Vz .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-h3SY7Xffdrpr00Vz .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-h3SY7Xffdrpr00Vz .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-h3SY7Xffdrpr00Vz .section{stroke:none;opacity:0.2}#mermaid-svg-h3SY7Xffdrpr00Vz .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-h3SY7Xffdrpr00Vz .section2{fill:#fff400}#mermaid-svg-h3SY7Xffdrpr00Vz .section1,#mermaid-svg-h3SY7Xffdrpr00Vz .section3{fill:#fff;opacity:0.2}#mermaid-svg-h3SY7Xffdrpr00Vz .sectionTitle0{fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .sectionTitle1{fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .sectionTitle2{fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .sectionTitle3{fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-h3SY7Xffdrpr00Vz .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-h3SY7Xffdrpr00Vz .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-h3SY7Xffdrpr00Vz .grid path{stroke-width:0}#mermaid-svg-h3SY7Xffdrpr00Vz .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-h3SY7Xffdrpr00Vz .task{stroke-width:2}#mermaid-svg-h3SY7Xffdrpr00Vz .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-h3SY7Xffdrpr00Vz .taskText:not([font-size]){font-size:11px}#mermaid-svg-h3SY7Xffdrpr00Vz .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-h3SY7Xffdrpr00Vz .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-h3SY7Xffdrpr00Vz .task.clickable{cursor:pointer}#mermaid-svg-h3SY7Xffdrpr00Vz .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-h3SY7Xffdrpr00Vz .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-h3SY7Xffdrpr00Vz .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-h3SY7Xffdrpr00Vz .taskText0,#mermaid-svg-h3SY7Xffdrpr00Vz .taskText1,#mermaid-svg-h3SY7Xffdrpr00Vz .taskText2,#mermaid-svg-h3SY7Xffdrpr00Vz .taskText3{fill:#fff}#mermaid-svg-h3SY7Xffdrpr00Vz .task0,#mermaid-svg-h3SY7Xffdrpr00Vz .task1,#mermaid-svg-h3SY7Xffdrpr00Vz .task2,#mermaid-svg-h3SY7Xffdrpr00Vz .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-h3SY7Xffdrpr00Vz .taskTextOutside0,#mermaid-svg-h3SY7Xffdrpr00Vz .taskTextOutside2{fill:#000}#mermaid-svg-h3SY7Xffdrpr00Vz .taskTextOutside1,#mermaid-svg-h3SY7Xffdrpr00Vz .taskTextOutside3{fill:#000}#mermaid-svg-h3SY7Xffdrpr00Vz .active0,#mermaid-svg-h3SY7Xffdrpr00Vz .active1,#mermaid-svg-h3SY7Xffdrpr00Vz .active2,#mermaid-svg-h3SY7Xffdrpr00Vz .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-h3SY7Xffdrpr00Vz .activeText0,#mermaid-svg-h3SY7Xffdrpr00Vz .activeText1,#mermaid-svg-h3SY7Xffdrpr00Vz .activeText2,#mermaid-svg-h3SY7Xffdrpr00Vz .activeText3{fill:#000 !important}#mermaid-svg-h3SY7Xffdrpr00Vz .done0,#mermaid-svg-h3SY7Xffdrpr00Vz .done1,#mermaid-svg-h3SY7Xffdrpr00Vz .done2,#mermaid-svg-h3SY7Xffdrpr00Vz .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-h3SY7Xffdrpr00Vz .doneText0,#mermaid-svg-h3SY7Xffdrpr00Vz .doneText1,#mermaid-svg-h3SY7Xffdrpr00Vz .doneText2,#mermaid-svg-h3SY7Xffdrpr00Vz .doneText3{fill:#000 !important}#mermaid-svg-h3SY7Xffdrpr00Vz .crit0,#mermaid-svg-h3SY7Xffdrpr00Vz .crit1,#mermaid-svg-h3SY7Xffdrpr00Vz .crit2,#mermaid-svg-h3SY7Xffdrpr00Vz .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-h3SY7Xffdrpr00Vz .activeCrit0,#mermaid-svg-h3SY7Xffdrpr00Vz .activeCrit1,#mermaid-svg-h3SY7Xffdrpr00Vz .activeCrit2,#mermaid-svg-h3SY7Xffdrpr00Vz .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-h3SY7Xffdrpr00Vz .doneCrit0,#mermaid-svg-h3SY7Xffdrpr00Vz .doneCrit1,#mermaid-svg-h3SY7Xffdrpr00Vz .doneCrit2,#mermaid-svg-h3SY7Xffdrpr00Vz .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-h3SY7Xffdrpr00Vz .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-h3SY7Xffdrpr00Vz .milestoneText{font-style:italic}#mermaid-svg-h3SY7Xffdrpr00Vz .doneCritText0,#mermaid-svg-h3SY7Xffdrpr00Vz .doneCritText1,#mermaid-svg-h3SY7Xffdrpr00Vz .doneCritText2,#mermaid-svg-h3SY7Xffdrpr00Vz .doneCritText3{fill:#000 !important}#mermaid-svg-h3SY7Xffdrpr00Vz .activeCritText0,#mermaid-svg-h3SY7Xffdrpr00Vz .activeCritText1,#mermaid-svg-h3SY7Xffdrpr00Vz .activeCritText2,#mermaid-svg-h3SY7Xffdrpr00Vz .activeCritText3{fill:#000 !important}#mermaid-svg-h3SY7Xffdrpr00Vz .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-h3SY7Xffdrpr00Vz g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-h3SY7Xffdrpr00Vz g.classGroup text .title{font-weight:bolder}#mermaid-svg-h3SY7Xffdrpr00Vz g.clickable{cursor:pointer}#mermaid-svg-h3SY7Xffdrpr00Vz g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-h3SY7Xffdrpr00Vz g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-h3SY7Xffdrpr00Vz .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-h3SY7Xffdrpr00Vz .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-h3SY7Xffdrpr00Vz .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-h3SY7Xffdrpr00Vz .dashed-line{stroke-dasharray:3}#mermaid-svg-h3SY7Xffdrpr00Vz #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-h3SY7Xffdrpr00Vz #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-h3SY7Xffdrpr00Vz #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-h3SY7Xffdrpr00Vz #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-h3SY7Xffdrpr00Vz #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-h3SY7Xffdrpr00Vz #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-h3SY7Xffdrpr00Vz #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-h3SY7Xffdrpr00Vz #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-h3SY7Xffdrpr00Vz .commit-id,#mermaid-svg-h3SY7Xffdrpr00Vz .commit-msg,#mermaid-svg-h3SY7Xffdrpr00Vz .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-h3SY7Xffdrpr00Vz .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-h3SY7Xffdrpr00Vz .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-h3SY7Xffdrpr00Vz g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-h3SY7Xffdrpr00Vz g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-h3SY7Xffdrpr00Vz g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-h3SY7Xffdrpr00Vz g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-h3SY7Xffdrpr00Vz g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-h3SY7Xffdrpr00Vz .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-h3SY7Xffdrpr00Vz .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-h3SY7Xffdrpr00Vz .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-h3SY7Xffdrpr00Vz .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-h3SY7Xffdrpr00Vz .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-h3SY7Xffdrpr00Vz .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-h3SY7Xffdrpr00Vz .edgeLabel text{fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-h3SY7Xffdrpr00Vz .node circle.state-start{fill:black;stroke:black}#mermaid-svg-h3SY7Xffdrpr00Vz .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-h3SY7Xffdrpr00Vz #statediagram-barbEnd{fill:#9370db}#mermaid-svg-h3SY7Xffdrpr00Vz .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-h3SY7Xffdrpr00Vz .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-h3SY7Xffdrpr00Vz .statediagram-state .divider{stroke:#9370db}#mermaid-svg-h3SY7Xffdrpr00Vz .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-h3SY7Xffdrpr00Vz .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-h3SY7Xffdrpr00Vz .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-h3SY7Xffdrpr00Vz .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-h3SY7Xffdrpr00Vz .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-h3SY7Xffdrpr00Vz .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-h3SY7Xffdrpr00Vz .note-edge{stroke-dasharray:5}#mermaid-svg-h3SY7Xffdrpr00Vz .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-h3SY7Xffdrpr00Vz .error-icon{fill:#522}#mermaid-svg-h3SY7Xffdrpr00Vz .error-text{fill:#522;stroke:#522}#mermaid-svg-h3SY7Xffdrpr00Vz .edge-thickness-normal{stroke-width:2px}#mermaid-svg-h3SY7Xffdrpr00Vz .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-h3SY7Xffdrpr00Vz .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-h3SY7Xffdrpr00Vz .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-h3SY7Xffdrpr00Vz .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-h3SY7Xffdrpr00Vz .marker{fill:#333}#mermaid-svg-h3SY7Xffdrpr00Vz .marker.cross{stroke:#333} :root { --mermaid-font-family: "trebuchet ms", verdana, arial;} #mermaid-svg-h3SY7Xffdrpr00Vz { color: rgba(0, 0, 0, 0.75); font: ; } hadoop1 namenode secondarynamenode ResourceManager node-0001 DataNode NodeManager node-0002 DataNode NodeManager node-0003 DataNode NodeManager

Yarn部署

[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/yarn-site.xml <configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>hadoop1</value> </property>  <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>

启动集群 [以下操作仅在 hadoop1 上执行]

[root@hadoop1 ~]# for i in node-{0001..0003};do rsync -avXSH --delete /usr/local/hadoop/etc ${i}:/usr/local/hadoop/ done [root@hadoop1 ~]# /usr/local/hadoop/sbin/start-yarn.sh

验证集群

[root@hadoop1 ~]# for i in hadoop1 node-{0001..0003};do echo ${i}; ssh ${i} jps; echo -e "\n"; done [root@hadoop1 ~]# /usr/local/hadoop/bin/yarn node -list Total Nodes:3 Node-Id Node-State Node-Http-Address Number-of-Running-Containers node-0003:33212 RUNNING node-0003:8042 0 node-0001:40201 RUNNING node-0001:8042 0 node-0002:38830 RUNNING node-0002:8042 0

web页面访问

namenode: http://hadoop1:50070

secondarynamenode: http://hadoop1:50090

resourcemanager: http://hadoop1:8088/cluster

datanode: http://node-0001:50075

nodemanager: http://node-0001:8042/node

最新回复(0)