Macro Metrics (recall/f1...) For Multiclass Cnn

I use CNN for image classification on unbalance dataset. I'm totaly new with tensorflow backend. It's multiclass problem (not multilabel) and I have 16 classes. Class are one hot e

Solution 1:

let me answer both question but in the opposite order:

You can't use Recall as a base for a custom loss: It is not convex! If you do not fully understand why Recall or precision or f1 can't be used as a loss, please take the time to see the role of the loss (it is afterall a huge parameter in your model).

Indeed, the round is intended for a binary problem. As they say, if it's not you then it's the other. But in your case it's wrong. Let's go throw the code:

val_predict = (np.asarray(self.model.predict(self.validation_data[0]))).round()

from the inside out, he take the data (self.validation_data[0;]) and predict a number (1 neuron as output). As such he compute the probability of being a 1. If this probability is over 0.5, then the round transforms it into a 1, if it is under, it transforms it to a 0. As you can see, it is wrong for you. In some case you won't predict any class. Following this mistake, the rest is also wrong.

Now, the solution. You want to compute the mean Recall at every step. by the way, "but it only works on validation set". yes that is intended, you use the validation to validate the model, not the train, else it is cheating.

so Recall is equal to true positive over all positives. Let's do that!

def recall(y_true, y_pred):
     recall = 0
     pred = K.argmax(y_pred)
     true = K.argmax(y_true)
     for i in range(16):
         p = K.cast(K.equal(pred,i),'int32')
         t = K.cast(K.equal(true,i),'int32')
         # Compute the true positive
         common = K.sum(K.dot(K.reshape(t,(1,-1)),K.reshape(p,(-1,1))))
         # divide by all positives in t
         recall += common/ (K.sum(t) + K.epsilon)
     return recall/16

This gives you the mean recall for all classes. you could print the value for every class.

Tell me if you have any question!

for an implementation of the binary Recall, see this question from which the code is adapted.