I have used this question to set up pyspark timezone using configuration of the cluster with
spark.sql.session.timeZone Asia/HongkongThis, unfortunately does not set timezone for all nodes and therefore calling pythons datetime.datetime.now() will not include timezone set with spark session config.
Databricks is phasing out support of init scripts and therefore I cannot use it to set up timezone for all nodes.
How is it possible to do it during spark config?