从spark2.4升级到spark3.0,在跑spark任务的时候,报了一下错误:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/internal/connector/SimpleTableProvider
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.filterImpl(TraversableLike.scala:247)
at scala.collection.TraversableLike$class.filter(TraversableLike.scala:259)
at scala.collection.AbstractTraversable.filter(Traversable.scala:104)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:630)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:245)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
at com.huitongjy.data.access.DeltaLake$.main(DeltaLake.scala:19)
at com.huitongjy.data.access.DeltaLake.main(DeltaLake.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.internal.connector.SimpleTableProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 41 more
解决办法: 使用命令spark-submit --version查看spark版本,看是否还是2.X的版本,然后查看环境变量。
相关推荐
该jar包解决报错java.lang.NoClassDefFoundError: org/apache/james/mime4j/MimeException,亲测可用。
java.lang.NoClassDefFoundError: de/javakaffee/kryoserializers/CurrencySerializer
jdk升级之后出现异常:java.lang.NoClassDefFoundError: javax/xml/ws/Service 1、解决方案jdk降级至1.8之前 2、由于JDK9以后,去除了javax.xml包需要引入对应的jar包 <groupId>javax.activation</groupId> ...
java.lang.NoClassDefFoundError: com/sun/activation/registries/LogSupport异常处理
JavaMail的java.lang.NoClassDefFoundError: com/sun/mail/util/LineInputStream错误 原因: MyEclipse6.5的javaee.jar中的mail包与JavaMail包有冲突。 解决: 在MyEclipse目录下(D:\Program Files\MyEclipse ...
NULL 博文链接:https://jaychang.iteye.com/blog/980159
NULL 博文链接:https://lym6520.iteye.com/blog/310725
出现Exception in threadmain java.lang.NoClassDefFoundError的各种可能情况.doc
NULL 博文链接:https://bijian1013.iteye.com/blog/2305804
NULL 博文链接:https://lyndon-lin.iteye.com/blog/856002
NULL 博文链接:https://chen106106.iteye.com/blog/1596135
jdk升级jdk10后,原本jdk自带的 webservice一些包确实,引起的一系列错误解决方案
jackson-annotations-2.10.2.jar jackson-core-2.10.2.jar jackson-databind-2.10.2.jar NoClassDefFoundError解决
在plugins org.eclipse.ui.workbench_3.3.1.M20070921-1200.jar下 打到org/eclipse/ui/internal/util/SWTResourceUtil.class 用winrar打开,同时也打开3.4的org.eclipse.ui.workbench_3.4.0.I20080606-1300.jar ...
java.lang.NoClassDefFoundError: org/apache/commons/lang/exception/NestableRuntimeException java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory 要在java里使用json用到的jar包怎么导入...
Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject,是因为缺少java-json.jar-附件资源
NULL 博文链接:https://chinrui.iteye.com/blog/1859820
NoClassDefFoundError: org/slf4j/LoggerFactory解决方案,需要添加的三个jar包都在,slf4j-api-1.5.2,slf4j-log4j12-1.5.2,log4j-1.2.12
Exception in thread "main" java.lang.NoClassDefFoundError: javax/mail/Message at com.bulktree.mail.SimpleMailTest.mail(SimpleMailTest.java:30) at com.bulktree.mail.SimpleMailTest.main(SimpleMailTest...