一. Spark shell配置
Spark shell默认就是可以访问的
spark-shell
spark.sql("select count(*) from test.t2").show()
二. Hive on Spark配置
2.1 问题描述
set hive.execution.engine=mr;
select count(*) from test.t2;
set hive.execution.engine=spark;
select count(*) from test.t2;
报错:
FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause.
2.2 解决方案
在hive中创建spark配置文件
cd /home/apache-hive-3.1.3-bin/conf
vim spark-defaults.conf
添加如下内容(在执行任务时,会根据如下参数执行)
spark.master yarn
spark.eventLog.enabled true
spark.eventLog.dir hdfs://hp5:8020/spark-history
spark.executor.memory 1g
spark.driver.memory 1g
vim /home/apache-hive-3.1.3-bin/conf/hive-site.xml
<!--Spark依赖位置(注意:端口号9000必须和namenode的端口号一致)-->
<property>
<name>spark.yarn.jars</name>
<value>hdfs://hp5:8020/spark-jars/*</value>
</property>
<!--Hive执行引擎-->
<property>
<name>hive.execution.engine</name>
<value>spark</value>
</property>
拷贝spark jar包到到hive的lib目录:
cd /home/spark-3.2.2-bin-hadoop3.2/jars
cp ./scala-library-2.12.15.jar /home/apache-hive-3.1.3-bin/lib/
cp ./spark-core_2.12-3.2.2.jar /home/apache-hive-3.1.3-bin/lib/
cp ./spark-network-common_2.12-3.2.2.jar /home/apache-hive-3.1.3-bin/lib/
Spark和Hive的新版本不兼容:
需要编译安装
https://blog.csdn.net/rfdjds/article/details/125389450
和我这边看到的报错一样,报的的
java.lang.NoSuchMethodError: org.apache.spark.api.java.JavaSparkContext.accumulator(Ljava/lang/Object;Ljava/lang/String;Lorg/apache/spark/AccumulatorParam;)Lorg/apache/spark/Accumulator;