in databricks, I created a spark dataframe, and need to convert to pandas dataframe,
sdf = spark.sql('select * from my_tbl') pdf = sdf.toPandas() but got error:ArrowInvalid: Casting from timestamp[us, tz=America/New_York] to timestamp[ns] would result in out of bounds timestamp: 253402214400000000File , line 1----> 1 pdf=sdf.toPandas()
just wonder how to fix, or is there anyway to generate a panda dataframe directly from invoking SQL in databricks?