How To Filter None Values Out Of Pcollection

My pubsub pull subscription is sending over the message and a None value for each message. I need to find a way to filter out the none values as part of my pipeline processing Of c

Solution 1:

Your approach to filter out None values looks good to me.

However, if I understand it correctly, when you are using testlogAndWrite and get the AttributeError you are keeping the "printHere" >> beam.Map(print_row) step in the pipeline.

print_row reads the messages and prints them but it does not output anything. Therefore, there will be no input for next step encode_here.

To solve this you can comment out that step or make sure that each element is returned:

defprint_row(row):
    print row
    printtype(row)
    return row

Output:

test1 message
<type'str'>
test2 message
<type'str'>
we found a none! get it out
test3 please work
<type'str'>