How To Implement Network Using Bert As A Paragraph Encoder In Long Text Classification, In Keras?

I am doing a long text classification task, which has more than 10000 words in doc, I am planing to use Bert as a paragraph encoder, then feed the embeddings of paragraph to BiLSTM

Solution 1:

I believe you can check the following article. The author shows how to load a pre-trained BERT model, embed it into a Keras layer and use it into a customized Deep Neural Network. First install TensorFlow 2.0 Keras implementation of google-research/bert:

pip install bert-for-tf2

Then run:

import bert
import os

def createBertLayer():
    global bert_layer

    bertDir = os.path.join(modelBertDir, "multi_cased_L-12_H-768_A-12")

    bert_params = bert.params_from_pretrained_ckpt(bertDir)

    bert_layer = bert.BertModelLayer.from_params(bert_params, name="bert")

    bert_layer.apply_adapter_freeze()

def loadBertCheckpoint():

    modelsFolder = os.path.join(modelBertDir, "multi_cased_L-12_H-768_A-12")
    checkpointName = os.path.join(modelsFolder, "bert_model.ckpt")

    bert.load_stock_weights(bert_layer, checkpointName)