Converting List Of Dictionaries Into Single Dictionary In Python 3

I have a snippet of data from which I need to extract specific information. The Data looks like this: pid log Date 91 json D1 189 json D2

Solution 1:

I believe you need:

df = pd.DataFrame({'log':['{"Before":{"freq_term":"Daily","ideal_pmt":"637.5","datetime":"2015-01-08 06:26:11"},"After":{"freq_term":"Weekly","ideal_pmt":"3346.88","datetime":"2015-02-02 06:16:07"}}','{"Before":{"buy_rate":"1.180","irr":"31.63","uwfee":"","freq_term":"Weekly"}, "After":{"freq_term":"Bi-Monthly","ideal_pmt":"2583.33"}}']})
print (df)
                                                 log
0  {"Before":{"freq_term":"Daily","ideal_pmt":"63...
1  {"Before":{"buy_rate":"1.180","irr":"31.63","u...

---

First convert values to nested dictionaries and then filter by nested dict comprehension:

df['log'] = df['log'].apply(pd.io.json.loads)

L1 = ['Before','After']
L2 = ['freq_term','datetime']
f = lambda x: {k:{k1:v1 for k1,v1 in v.items() if k1 in L2} for k,v in x.items() if k in L1}
df['new'] = df['log'].apply(f)
print (df)

                                                 log  \
0  {'After': {'ideal_pmt': '3346.88', 'freq_term'...   
1  {'After': {'ideal_pmt': '2583.33', 'freq_term'...   

                                                 new  
0  {'After': {'freq_term': 'Weekly', 'datetime': ...  
1  {'After': {'freq_term': 'Bi-Monthly'}, 'Before...  

EDIT:

For find all rows with unparseable values is possible use:

def f(x):
    try:
        return ast.literal_eval(x)
    except:
        return 1

print (df[df['log'].apply(f) == 1])