spark 同时读取多个路径的方法

1.传入多个参数

val result = spark.read.text("hdfs://hdfs-name/user/aa.txt","hdfs://hdfs-name/test/bb.txt")

2.正则

val result = spark.read.text("hdfs://hdfs-name/user/*")

3.文件列表

val path = "hdfs://hdfs-name/user/*.txt" val path2 = "hdfs://hdfs-name/test/*.txt" val arrPath = Array(path, path2) val ds = spark.read.textFile(arrPath:_*)