文章目录
实现步骤1.建立目录结构2.添加配置信息3.导入代码文件(dhlTest.scala)4. 编写文件 在路径:/usr/local/spark/StuGrade 下注意: 其他类似
5.打包并运行编译打包运行
独立运行文件: dhlTest2.scala
import org
.apache
.spark
.SparkContext
import org
.apache
.spark
.SparkContext
._
import org
.apache
.spark
.SparkConf
object dhlTest2
{
def main
(args
: Array
[String]): Unit = {
val conf
= new SparkConf
().setAppName
("BeyondNothing_dhl")
val sc
= new SparkContext
(conf
)
val dhltest
= sc
.textFile
("file:///usr/local/spark/StuGrade")
val res
= dhltest
.filter
(_
.trim
().length
>0).map
(line
=>(line
.split
(",")(0).trim
(),line
.split
(",")(1).trim
().toInt
)).groupByKey
().map
(x
=> {
var num
= 0.0
var sum
= 0
for(i
<- x
._2
){
sum
= sum
+ i
num
= num
+1
}
val avg
= sum
/num
val format
= f
"$avg%1.2f".toDouble
(x
._1
,format
)
}).collect
.foreach
(x
=> println
(x
._1
+"\t"+x
._2
))
}
}
实现步骤
1.建立目录结构
mkdir
./dhlTest2 # 创建应用程序根目录
mkdir
-p
./dhlTest2
/src
/main
/scala #创建所需的文件夹结构
2.添加配置信息
在dhlTest2目录下创建文件 simple.sbt 并且添加如下内容:
name
:= "dhl_BeyondNothing"
version
:= "1.0"
scalaVersion
:= "2.11.12"
libraryDependencies
+= "org.apache.spark" %% "spark-core" % "2.4.0"
3.导入代码文件(dhlTest.scala)
将代码文件放在scala目录下:
4. 编写文件 在路径:/usr/local/spark/StuGrade 下
注意: 其他类似
5.打包并运行
编译打包
进入到dhlTest2文档的路径下运行下列命令:
/usr
/local
/sbt
/sbt
package
运行
/usr
/local
/spark
/bin
/spark
-submit
--class "dhlTest2" ~/dhlTest2
/target
/scala
-2.11/dhl_beyondnothing_2
.11-1.0.jar
2>&1 | grep
"小"
