Spark 3.1.3 HADOOP_CONF_DIR or YARN_CONF_DIR must be set

1. 现象

使用如下命令以 client 模式提交作业到 Yarn 上时：

spark-submit \
  --class com.spark.example.core.base.WordCount \
  --master yarn \
  --deploy-mode client \
  --executor-memory 2g \
  --num-executors 4 \
  --executor-cores 1 \
spark-example-3.1-1.0.jar \
/data/word-count/word-count-input /data/word-count/word-count-output

抛出如下异常：

Exception in thread "main" org.apache.spark.SparkException: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.
        at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala

这篇好文章是转载于：学新通技术网