I am getting Failed to create directory /tmp/spark-a12d2057-a5ea-4fb9-876f-382d41153eab error while setting up spark config for an ETL application , I am running this application using a docker container.This is the code for spark config:-
def spark_config(JAR_FILE, APP_NAME): conf = SparkConf() conf.set("spark.jars", JAR_FILE) spark = SparkSession.builder.appName(APP_NAME) \ .config("spark.sql.debug.maxToStringFields", 4000) \ .config("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false") \ .config("spark.executor.memory", "8G") \ .config("spark.driver.memory", "8G") \ .config("spark.driver.maxResultSize", "15G") \ .config("spark.sql.autoBroadcastJoinThreshold", -1) \ .config(conf=conf) \ .getOrCreate()Error:-
ERROR JavaUtils: Failed to create directory /tmp/spark-a12d2057-a5ea-4fb9-876f-382d41153eabjava.nio.file.AccessDeniedException: /tmpException in thread "main" java.io.IOException: Failed to create a temp directory (under /tmp) after 10 attempts!Any help or pointers will be appreciated!