How To Infer Types In Pandas Dataframe

I have a dataframe which I read in using pyspark with: df1 = spark.read.csv('/user/me/data/*').toPandas() Unfortunately, pyspark leaves all the types as Object, even numerical val

Solution 1:

If you have the same column names you could use pd.DataFrame.astype:

df1 = df1.astype(df2.dtypes)

Otherwise, you need to construct a dictionary where keys are the column names in df1 and the values are dtypes. You can start with d = df2.dtypes.to_dict() to see what it should look like. Then construct a new dictionary altering the keys where needed.

Once you've constructed the dictionary d, use:

df1 = df1.astype(d)