Pyspark Error: Sparkexception: Exception Thrown In Future.get:

I have the following code: fact_item = ( spark.table('nn_team7_es.fact_table') .union(spark.table('nn_squad7_cs.fact_table')) .filter((f.col('date_key').between(start_py, end

Solution 1:

Check for datatypes for the joining columns. Also, you will see the error only while using display() or count(), the query would run as expected.

P.S: I was facing the same issue, once I resolved the datatypes mismatch, It was working fine.