1.在idea中运行报错:scala.Product.$init$(Lscala/Product;)V 检查pom文件中scala的版本和idea中scala的版本是否一致 idea中scala版本为2.11 解决办法:<artifactId>spark-core_2.12</artifactId> -> <artifactId>spark-core_2.11</artifactId>
2.idea中 java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. 在程序main()方法下加入System.setProperty("hadoop.home.dir", "D:\\hadoop-2.9.2") D:\\hadoop-2.9.2 window环境中Hadoop文件的地址
3.Error:(17, 34) Unable to find encoder for type String. An implicit Encoder[String] is needed to store String instances in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases. val value = df1.createDataset(strings) 这里缺失了隐式转换 implicit val encoder=org.apache.spark.sql.Encoders.STRING
4.http调用livy时报错java.lang.NullPointerException http中的参数的前后不能有空格,检查是否有空格
5.livy session 返回值中文乱码 解决:
livy： 'conf': {'spark.driver.extraJavaOptions': '-Dfile.encoding=utf-8', 'spark.executor.extraJavaOptions': '-Dfile.encoding=utf-8'} spark: –conf spark.driver.extraJavaOptions=” -Dfile.encoding=utf-8 “ \ –conf spark.executor.extraJavaOptions=” -Dfile.encoding=utf-8 “ \6.livy session模式指定大jar包(使用的是batch模式的大jar包)导致jar包冲突 解决: 去掉batch模式大jar包中的和scala相关的依赖
<dependency> <groupId>org.json4s</groupId> <artifactId>json4s-jackson_2.11</artifactId> <version>3.2.11</version> <exclusions> <exclusion> <groupId>org.scala-lang</groupId> <artifactId>scala-reflect</artifactId> </exclusion> <exclusion> <groupId>org.scala-lang.modules</groupId> <artifactId>scala-xml_2.11</artifactId> </exclusion> <exclusion> <groupId>org.scala-lang</groupId> <artifactId>scala-compiler</artifactId> </exclusion> <exclusion> <groupId>org.scala-lang.modules</groupId> <artifactId>scala-parser-combinators_2.11</artifactId> </exclusion> <exclusion> <groupId>org.scala-lang</groupId> <artifactId>scalap</artifactId> </exclusion> </exclusions> </dependency>
