(shell脚本) 含特殊分隔符‘^‘的文件数据导入MongoDB(1)

it2024-11-12 7

文章目录

前言脚本简介分隔符替换（字符串）---1分隔符替换（文件）---2awk命令连接MongoDBJson数据导入MongoDB删除脚本生成的临时数据文件 shell完整代码

前言

本篇文章简单介绍使用shell脚本将文本数据转为JSON格式导入MongoDB。下篇文章介绍《将文件数据转为csv格式导入MongoDB（2）》链接： https://blog.csdn.net/qq_43380180/article/details/109364724 过程如下：

#mermaid-svg-XrVKGnMUVw533pjB .label{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);fill:#333;color:#333}#mermaid-svg-XrVKGnMUVw533pjB .label text{fill:#333}#mermaid-svg-XrVKGnMUVw533pjB .node rect,#mermaid-svg-XrVKGnMUVw533pjB .node circle,#mermaid-svg-XrVKGnMUVw533pjB .node ellipse,#mermaid-svg-XrVKGnMUVw533pjB .node polygon,#mermaid-svg-XrVKGnMUVw533pjB .node path{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-XrVKGnMUVw533pjB .node .label{text-align:center;fill:#333}#mermaid-svg-XrVKGnMUVw533pjB .node.clickable{cursor:pointer}#mermaid-svg-XrVKGnMUVw533pjB .arrowheadPath{fill:#333}#mermaid-svg-XrVKGnMUVw533pjB .edgePath .path{stroke:#333;stroke-width:1.5px}#mermaid-svg-XrVKGnMUVw533pjB .flowchart-link{stroke:#333;fill:none}#mermaid-svg-XrVKGnMUVw533pjB .edgeLabel{background-color:#e8e8e8;text-align:center}#mermaid-svg-XrVKGnMUVw533pjB .edgeLabel rect{opacity:0.9}#mermaid-svg-XrVKGnMUVw533pjB .edgeLabel span{color:#333}#mermaid-svg-XrVKGnMUVw533pjB .cluster rect{fill:#ffffde;stroke:#aa3;stroke-width:1px}#mermaid-svg-XrVKGnMUVw533pjB .cluster text{fill:#333}#mermaid-svg-XrVKGnMUVw533pjB div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:12px;background:#ffffde;border:1px solid #aa3;border-radius:2px;pointer-events:none;z-index:100}#mermaid-svg-XrVKGnMUVw533pjB .actor{stroke:#ccf;fill:#ECECFF}#mermaid-svg-XrVKGnMUVw533pjB text.actor>tspan{fill:#000;stroke:none}#mermaid-svg-XrVKGnMUVw533pjB .actor-line{stroke:grey}#mermaid-svg-XrVKGnMUVw533pjB .messageLine0{stroke-width:1.5;stroke-dasharray:none;stroke:#333}#mermaid-svg-XrVKGnMUVw533pjB .messageLine1{stroke-width:1.5;stroke-dasharray:2, 2;stroke:#333}#mermaid-svg-XrVKGnMUVw533pjB #arrowhead path{fill:#333;stroke:#333}#mermaid-svg-XrVKGnMUVw533pjB .sequenceNumber{fill:#fff}#mermaid-svg-XrVKGnMUVw533pjB #sequencenumber{fill:#333}#mermaid-svg-XrVKGnMUVw533pjB #crosshead path{fill:#333;stroke:#333}#mermaid-svg-XrVKGnMUVw533pjB .messageText{fill:#333;stroke:#333}#mermaid-svg-XrVKGnMUVw533pjB .labelBox{stroke:#ccf;fill:#ECECFF}#mermaid-svg-XrVKGnMUVw533pjB .labelText,#mermaid-svg-XrVKGnMUVw533pjB .labelText>tspan{fill:#000;stroke:none}#mermaid-svg-XrVKGnMUVw533pjB .loopText,#mermaid-svg-XrVKGnMUVw533pjB .loopText>tspan{fill:#000;stroke:none}#mermaid-svg-XrVKGnMUVw533pjB .loopLine{stroke-width:2px;stroke-dasharray:2, 2;stroke:#ccf;fill:#ccf}#mermaid-svg-XrVKGnMUVw533pjB .note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-XrVKGnMUVw533pjB .noteText,#mermaid-svg-XrVKGnMUVw533pjB .noteText>tspan{fill:#000;stroke:none}#mermaid-svg-XrVKGnMUVw533pjB .activation0{fill:#f4f4f4;stroke:#666}#mermaid-svg-XrVKGnMUVw533pjB .activation1{fill:#f4f4f4;stroke:#666}#mermaid-svg-XrVKGnMUVw533pjB .activation2{fill:#f4f4f4;stroke:#666}#mermaid-svg-XrVKGnMUVw533pjB .mermaid-main-font{font-family:"trebuchet ms", verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-XrVKGnMUVw533pjB .section{stroke:none;opacity:0.2}#mermaid-svg-XrVKGnMUVw533pjB .section0{fill:rgba(102,102,255,0.49)}#mermaid-svg-XrVKGnMUVw533pjB .section2{fill:#fff400}#mermaid-svg-XrVKGnMUVw533pjB .section1,#mermaid-svg-XrVKGnMUVw533pjB .section3{fill:#fff;opacity:0.2}#mermaid-svg-XrVKGnMUVw533pjB .sectionTitle0{fill:#333}#mermaid-svg-XrVKGnMUVw533pjB .sectionTitle1{fill:#333}#mermaid-svg-XrVKGnMUVw533pjB .sectionTitle2{fill:#333}#mermaid-svg-XrVKGnMUVw533pjB .sectionTitle3{fill:#333}#mermaid-svg-XrVKGnMUVw533pjB .sectionTitle{text-anchor:start;font-size:11px;text-height:14px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-XrVKGnMUVw533pjB .grid .tick{stroke:#d3d3d3;opacity:0.8;shape-rendering:crispEdges}#mermaid-svg-XrVKGnMUVw533pjB .grid .tick text{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-XrVKGnMUVw533pjB .grid path{stroke-width:0}#mermaid-svg-XrVKGnMUVw533pjB .today{fill:none;stroke:red;stroke-width:2px}#mermaid-svg-XrVKGnMUVw533pjB .task{stroke-width:2}#mermaid-svg-XrVKGnMUVw533pjB .taskText{text-anchor:middle;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-XrVKGnMUVw533pjB .taskText:not([font-size]){font-size:11px}#mermaid-svg-XrVKGnMUVw533pjB .taskTextOutsideRight{fill:#000;text-anchor:start;font-size:11px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-XrVKGnMUVw533pjB .taskTextOutsideLeft{fill:#000;text-anchor:end;font-size:11px}#mermaid-svg-XrVKGnMUVw533pjB .task.clickable{cursor:pointer}#mermaid-svg-XrVKGnMUVw533pjB .taskText.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-XrVKGnMUVw533pjB .taskTextOutsideLeft.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-XrVKGnMUVw533pjB .taskTextOutsideRight.clickable{cursor:pointer;fill:#003163 !important;font-weight:bold}#mermaid-svg-XrVKGnMUVw533pjB .taskText0,#mermaid-svg-XrVKGnMUVw533pjB .taskText1,#mermaid-svg-XrVKGnMUVw533pjB .taskText2,#mermaid-svg-XrVKGnMUVw533pjB .taskText3{fill:#fff}#mermaid-svg-XrVKGnMUVw533pjB .task0,#mermaid-svg-XrVKGnMUVw533pjB .task1,#mermaid-svg-XrVKGnMUVw533pjB .task2,#mermaid-svg-XrVKGnMUVw533pjB .task3{fill:#8a90dd;stroke:#534fbc}#mermaid-svg-XrVKGnMUVw533pjB .taskTextOutside0,#mermaid-svg-XrVKGnMUVw533pjB .taskTextOutside2{fill:#000}#mermaid-svg-XrVKGnMUVw533pjB .taskTextOutside1,#mermaid-svg-XrVKGnMUVw533pjB .taskTextOutside3{fill:#000}#mermaid-svg-XrVKGnMUVw533pjB .active0,#mermaid-svg-XrVKGnMUVw533pjB .active1,#mermaid-svg-XrVKGnMUVw533pjB .active2,#mermaid-svg-XrVKGnMUVw533pjB .active3{fill:#bfc7ff;stroke:#534fbc}#mermaid-svg-XrVKGnMUVw533pjB .activeText0,#mermaid-svg-XrVKGnMUVw533pjB .activeText1,#mermaid-svg-XrVKGnMUVw533pjB .activeText2,#mermaid-svg-XrVKGnMUVw533pjB .activeText3{fill:#000 !important}#mermaid-svg-XrVKGnMUVw533pjB .done0,#mermaid-svg-XrVKGnMUVw533pjB .done1,#mermaid-svg-XrVKGnMUVw533pjB .done2,#mermaid-svg-XrVKGnMUVw533pjB .done3{stroke:grey;fill:#d3d3d3;stroke-width:2}#mermaid-svg-XrVKGnMUVw533pjB .doneText0,#mermaid-svg-XrVKGnMUVw533pjB .doneText1,#mermaid-svg-XrVKGnMUVw533pjB .doneText2,#mermaid-svg-XrVKGnMUVw533pjB .doneText3{fill:#000 !important}#mermaid-svg-XrVKGnMUVw533pjB .crit0,#mermaid-svg-XrVKGnMUVw533pjB .crit1,#mermaid-svg-XrVKGnMUVw533pjB .crit2,#mermaid-svg-XrVKGnMUVw533pjB .crit3{stroke:#f88;fill:red;stroke-width:2}#mermaid-svg-XrVKGnMUVw533pjB .activeCrit0,#mermaid-svg-XrVKGnMUVw533pjB .activeCrit1,#mermaid-svg-XrVKGnMUVw533pjB .activeCrit2,#mermaid-svg-XrVKGnMUVw533pjB .activeCrit3{stroke:#f88;fill:#bfc7ff;stroke-width:2}#mermaid-svg-XrVKGnMUVw533pjB .doneCrit0,#mermaid-svg-XrVKGnMUVw533pjB .doneCrit1,#mermaid-svg-XrVKGnMUVw533pjB .doneCrit2,#mermaid-svg-XrVKGnMUVw533pjB .doneCrit3{stroke:#f88;fill:#d3d3d3;stroke-width:2;cursor:pointer;shape-rendering:crispEdges}#mermaid-svg-XrVKGnMUVw533pjB .milestone{transform:rotate(45deg) scale(0.8, 0.8)}#mermaid-svg-XrVKGnMUVw533pjB .milestoneText{font-style:italic}#mermaid-svg-XrVKGnMUVw533pjB .doneCritText0,#mermaid-svg-XrVKGnMUVw533pjB .doneCritText1,#mermaid-svg-XrVKGnMUVw533pjB .doneCritText2,#mermaid-svg-XrVKGnMUVw533pjB .doneCritText3{fill:#000 !important}#mermaid-svg-XrVKGnMUVw533pjB .activeCritText0,#mermaid-svg-XrVKGnMUVw533pjB .activeCritText1,#mermaid-svg-XrVKGnMUVw533pjB .activeCritText2,#mermaid-svg-XrVKGnMUVw533pjB .activeCritText3{fill:#000 !important}#mermaid-svg-XrVKGnMUVw533pjB .titleText{text-anchor:middle;font-size:18px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-XrVKGnMUVw533pjB g.classGroup text{fill:#9370db;stroke:none;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family);font-size:10px}#mermaid-svg-XrVKGnMUVw533pjB g.classGroup text .title{font-weight:bolder}#mermaid-svg-XrVKGnMUVw533pjB g.clickable{cursor:pointer}#mermaid-svg-XrVKGnMUVw533pjB g.classGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-XrVKGnMUVw533pjB g.classGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-XrVKGnMUVw533pjB .classLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.5}#mermaid-svg-XrVKGnMUVw533pjB .classLabel .label{fill:#9370db;font-size:10px}#mermaid-svg-XrVKGnMUVw533pjB .relation{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-XrVKGnMUVw533pjB .dashed-line{stroke-dasharray:3}#mermaid-svg-XrVKGnMUVw533pjB #compositionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-XrVKGnMUVw533pjB #compositionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-XrVKGnMUVw533pjB #aggregationStart{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-XrVKGnMUVw533pjB #aggregationEnd{fill:#ECECFF;stroke:#9370db;stroke-width:1}#mermaid-svg-XrVKGnMUVw533pjB #dependencyStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-XrVKGnMUVw533pjB #dependencyEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-XrVKGnMUVw533pjB #extensionStart{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-XrVKGnMUVw533pjB #extensionEnd{fill:#9370db;stroke:#9370db;stroke-width:1}#mermaid-svg-XrVKGnMUVw533pjB .commit-id,#mermaid-svg-XrVKGnMUVw533pjB .commit-msg,#mermaid-svg-XrVKGnMUVw533pjB .branch-label{fill:lightgrey;color:lightgrey;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-XrVKGnMUVw533pjB .pieTitleText{text-anchor:middle;font-size:25px;fill:#000;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-XrVKGnMUVw533pjB .slice{font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-XrVKGnMUVw533pjB g.stateGroup text{fill:#9370db;stroke:none;font-size:10px;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-XrVKGnMUVw533pjB g.stateGroup text{fill:#9370db;fill:#333;stroke:none;font-size:10px}#mermaid-svg-XrVKGnMUVw533pjB g.statediagram-cluster .cluster-label text{fill:#333}#mermaid-svg-XrVKGnMUVw533pjB g.stateGroup .state-title{font-weight:bolder;fill:#000}#mermaid-svg-XrVKGnMUVw533pjB g.stateGroup rect{fill:#ECECFF;stroke:#9370db}#mermaid-svg-XrVKGnMUVw533pjB g.stateGroup line{stroke:#9370db;stroke-width:1}#mermaid-svg-XrVKGnMUVw533pjB .transition{stroke:#9370db;stroke-width:1;fill:none}#mermaid-svg-XrVKGnMUVw533pjB .stateGroup .composit{fill:white;border-bottom:1px}#mermaid-svg-XrVKGnMUVw533pjB .stateGroup .alt-composit{fill:#e0e0e0;border-bottom:1px}#mermaid-svg-XrVKGnMUVw533pjB .state-note{stroke:#aa3;fill:#fff5ad}#mermaid-svg-XrVKGnMUVw533pjB .state-note text{fill:black;stroke:none;font-size:10px}#mermaid-svg-XrVKGnMUVw533pjB .stateLabel .box{stroke:none;stroke-width:0;fill:#ECECFF;opacity:0.7}#mermaid-svg-XrVKGnMUVw533pjB .edgeLabel text{fill:#333}#mermaid-svg-XrVKGnMUVw533pjB .stateLabel text{fill:#000;font-size:10px;font-weight:bold;font-family:'trebuchet ms', verdana, arial;font-family:var(--mermaid-font-family)}#mermaid-svg-XrVKGnMUVw533pjB .node circle.state-start{fill:black;stroke:black}#mermaid-svg-XrVKGnMUVw533pjB .node circle.state-end{fill:black;stroke:white;stroke-width:1.5}#mermaid-svg-XrVKGnMUVw533pjB #statediagram-barbEnd{fill:#9370db}#mermaid-svg-XrVKGnMUVw533pjB .statediagram-cluster rect{fill:#ECECFF;stroke:#9370db;stroke-width:1px}#mermaid-svg-XrVKGnMUVw533pjB .statediagram-cluster rect.outer{rx:5px;ry:5px}#mermaid-svg-XrVKGnMUVw533pjB .statediagram-state .divider{stroke:#9370db}#mermaid-svg-XrVKGnMUVw533pjB .statediagram-state .title-state{rx:5px;ry:5px}#mermaid-svg-XrVKGnMUVw533pjB .statediagram-cluster.statediagram-cluster .inner{fill:white}#mermaid-svg-XrVKGnMUVw533pjB .statediagram-cluster.statediagram-cluster-alt .inner{fill:#e0e0e0}#mermaid-svg-XrVKGnMUVw533pjB .statediagram-cluster .inner{rx:0;ry:0}#mermaid-svg-XrVKGnMUVw533pjB .statediagram-state rect.basic{rx:5px;ry:5px}#mermaid-svg-XrVKGnMUVw533pjB .statediagram-state rect.divider{stroke-dasharray:10,10;fill:#efefef}#mermaid-svg-XrVKGnMUVw533pjB .note-edge{stroke-dasharray:5}#mermaid-svg-XrVKGnMUVw533pjB .statediagram-note rect{fill:#fff5ad;stroke:#aa3;stroke-width:1px;rx:0;ry:0}:root{--mermaid-font-family: '"trebuchet ms", verdana, arial';--mermaid-font-family: "Comic Sans MS", "Comic Sans", cursive}#mermaid-svg-XrVKGnMUVw533pjB .error-icon{fill:#522}#mermaid-svg-XrVKGnMUVw533pjB .error-text{fill:#522;stroke:#522}#mermaid-svg-XrVKGnMUVw533pjB .edge-thickness-normal{stroke-width:2px}#mermaid-svg-XrVKGnMUVw533pjB .edge-thickness-thick{stroke-width:3.5px}#mermaid-svg-XrVKGnMUVw533pjB .edge-pattern-solid{stroke-dasharray:0}#mermaid-svg-XrVKGnMUVw533pjB .edge-pattern-dashed{stroke-dasharray:3}#mermaid-svg-XrVKGnMUVw533pjB .edge-pattern-dotted{stroke-dasharray:2}#mermaid-svg-XrVKGnMUVw533pjB .marker{fill:#333}#mermaid-svg-XrVKGnMUVw533pjB .marker.cross{stroke:#333} :root { --mermaid-font-family: "trebuchet ms", verdana, arial;} #mermaid-svg-XrVKGnMUVw533pjB { color: rgba(0, 0, 0, 0.75); font: ; } 源数据分隔符是否为特殊分隔符 ^ 特殊分隔符 ^ 数据格式预处理 awk处理数据转为Json Json文件入MongoDB 普通分隔符

脚本简介

分隔符替换（字符串）—1

while read line; do str = ${line//' '/''} str1 = ${str//'!^'/','} echo str1 done < ${pwd}/$1 > ${pwd_path}/process_file.txt

${str//'!^'/','} 字符替换，将 !^ 替换为 ,; < 表示从一个文件读内容； > 表示将内容写进一个文件，如果该文件不存在，会自动新建该文件，因此该操作会覆盖文件之前内容； >> 同 > ，追加模式，但是该操作不会覆盖之前文件内容。

分隔符替换（文件）—2

sed -i "/s/!^/,/g" cp_original.txt

替换文件中的字符串，将字符串!^替换为,

awk命令

#awk [-F|-f|-v] ‘数据处理’ file awk -F ',' -vq='"' ' function jsonStr(x,y){printf("%s:%s,"), x , q y q} function jsonDci(x,y){printf("%s:%.2f,"), x, y} function jsonEnd(x,y){printf("%s:%s)}, x, y} { printf "{" jsonStr("key", $1); jsonStr("key2", $2); jsonDci("key3", $3); jsonDci("key4", $4); jsonEnd("key5", $5); printf "}\n" } ' ${pwd_path}/process_file.txt > ${pwd_path}/process_json.json

该脚本使用awk将数据转为json格式，尝试使用 -F '!^' 做为分割符时，数据处理出现未分割的现象，原因是 ^ 在awk处理作为分隔符时，无法分辨，因此需要替换该分隔符。本次处理的文本数据分割符含有特殊分割符 ^ ，因此将分隔符替换为 , 。 -vq=' " ' 设置一个变量 q = "。 function··· 为数据处理过程，格式化为json。

连接MongoDB

mongo -u <user> -p <password> --host <host> --port port mongo << EOF use dbname db.{txt_name}.drop() EOF

EOF 之间可以执行mongo的任意操作命令。此处删除了集合名为 ${txt_name} 的集合（无论该集合是否存在，该操作不会影响脚本执行）。

Json数据导入MongoDB

mongoimpot --db dbname --collection ${txt_name} --file ${pwd_path}/process_json.json

mongoimport 将文件数据导入MongoDB，文件可以为json 或 csv 格式，此处使用的是json文件。

删除脚本生成的临时数据文件

rm -rf ${pwd_path}/process_file.txt ${pwd_path}/process_json.json

rm -rf file 强制删除文件file rm shell 删除文件（夹）的命令 -rf 该参数表示不询问强制删除，重要文件删除，不推荐使用该参数。

shell完整代码

#bin/bash txt_name = ${1%.*} echo ${txt_name} pwd_path = `pwd` #替换拆分符 '!^' 为 ',' while read line; do str = ${line//' '/''} str1 = ${str//'!^'/','} echo str1 done < ${pwd}/$1 > ${pwd_path}/process_file.txt #数据转化为json形式 awk -F ',' -vq='"' ' function jsonStr(x,y){printf("%s:%s,"), x , q y q} function jsonDci(x,y){printf("%s:%.2f,"), x, y} function jsonEnd(x,y){printf("%s:%s)}, x, y} { printf "{" jsonStr("key", $1); jsonStr("key2", $2); jsonDci("key3", $3); jsonDci("key4", $4); jsonEnd("key5", $5); printf "}\n" } ' ${pwd_path}/process_file.txt > ${pwd_path}/process_json.json #mongo -u <user> -p <password> --host <host> --port port mongo << EOF use dbname db.{txt_name}.drop() EOF #json文件导入mongo mongoimpot --db dbname --collection ${txt_name} --file ${pwd_path}/process_json.json #删除临时过程数据文件 rm -rf ${pwd_path}/process_file.txt ${pwd_path}/process_json.json #Finish

最新回复(0)