Tensorflow & Keras Can't Load .ckpt Save

So I am using the ModelCheckpoint callback to save the best epoch of a model I am training. It saves with no errors, but when I try to load it, I get the error: 2019-07-27 22:58:04

Solution 1:

TLDR; you are saving whole model, while trying to load only weights, that's not how it works.

Explanation

Your model's fit:

model.fit(
    train_images,
    train_labels,
    epochs=100,
    callbacks=[
        keras.callbacks.ModelCheckpoint(
            "cp.ckpt", monitor="mean_absolute_error", save_best_only=True, verbose=1
        )
    ],
)

As save_weights=False by default in ModelCheckpoint, you are saving whole model to .ckpt.

BTW. File should be named .hdf5 or .hf5 as it's Hierarchical Data Format 5. As Windows is not extension-agnostic you may run into some problems if tensorflow / keras relies on extension on this OS.

On the other hand you are loading the model's weights only, while the file contains whole model:

model.load_weights("cp.ckpt")

Tensorflow's checkpointing (.cp) mechanism is different from Keras's (.hdf5), so watch out for that (there are plans to integrate them more closely, see here and here).

Solution

So, either use the callback as you currently do, BUT use model.load("model.hdf5") or add save_weights_only=True argument to ModelCheckpoint:

model.fit(
    train_images,
    train_labels,
    epochs=100,
    callbacks=[
        keras.callbacks.ModelCheckpoint(
            "weights.hdf5",
            monitor="mean_absolute_error",
            save_best_only=True,
            verbose=1,
            save_weights_only=True,  # Specify this
        )
    ],
)

and you can use your model.load_weights("weights.hdf5").

Solution 2:

model.load_weights will not work here. Reason is mentioned in the above answer. You can load weights by this code. Load your model first and than load weights. I hope this code will help you out

import tensorflow as tf

model=dense_net()
ckpt = tf.train.Checkpoint(
step=tf.Variable(1, dtype=tf.int64),  net=model)
ckpt.restore(tf.train.latest_checkpoint("/kaggle/working/training_1/cp.ckpt.data-00001-of-00002"))

Solution 3:

import tensorflow as tf

# Create some variables.
v1 = tf.Variable(tf.random_normal([784, 200], stddev=0.35), name="v1")
v2 = tf.Variable(tf.random_normal([784, 200], stddev=0.35), name="v2")

# Add an op to initialize the variables.
init_op = tf.global_variables_initializer()

# Add ops to save and restore all the variables.
saver = tf.train.Saver()

# Later, launch the model, initialize the variables, do some work, save the# variables to disk.with tf.Session() as sess:
  sess.run(init_op)
  # Do some work with the model.# Save the variables to disk.
  save_path = saver.save(sess, "/tmp/model.ckpt")
  print("Model saved in file: %s" % save_path)

# Later, launch the model, use the saver to restore variables from disk, and# do some work with the model.with tf.Session() as sess:
  # Restore variables from disk.
  saver.restore(sess, "/tmp/model.ckpt")
  print("Model restored.")
  # Do some work with the model

Source