- JDK 1.8
- Spark 2.4.8
- Scala 2.11.8
运行Spark时
APP 2023-12-07 13:16:33,326 [main] [unknown] INFO org.apache.spark.SparkContext.logInfo:54 - Running Spark version 2.4.8
APP 2023-12-07 13:16:33,438 [main] [unknown] ERROR org.apache.hadoop.conf.Configuration.loadResource:2477 - Failed to set setXIncludeAware(true) for parser org.apache.xerces.jaxp.DocumentBuilderFactoryImpl@5bf0fe62:java.lang.UnsupportedOperationException: setXIncludeAware is not supported on this JAXP implementation or earlier: class org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
java.lang.UnsupportedOperationException: setXIncludeAware is not supported on this JAXP implementation or earlier: class org.apache.xerces.jaxp.DocumentBuilderFactoryImpl
at javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:584)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2475)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:968)
at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:987)
at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1388)
at org.apache.hadoop.security.SecurityUtil.<clinit>(SecurityUtil.java:70)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:272)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:789)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:774)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:647)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:293)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2526)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at Utils.SparkUtils.createSparkSession(SparkUtils.java:38)
at Utils.SparkUtils.createSparkSession(SparkUtils.java:26)
at Demo.SparkDemo.<clinit>(SparkDemo.java:10)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:65)
APP 2023-12-07 13:16:33,494 [main] [unknown] ERROR org.apache.hadoop.conf.Configuration.loadResource:2477 - Failed to set setXIncludeAware(true) for parser
网上说的是：jdk的版本、scala版本的子版本导致的。
我的解决：
本地xercesImpl是其他包里带过来的，版本是2.6.2版本
pom中添加如下版本就好了
<dependency>
<groupId>xerces</groupId>
<artifactId>xercesImpl</artifactId>
<version>2.11.0</version>
</dependency>