Python: Gensim Memory Error

import logging logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) from gensim import corpora, models, similarities from nltk.corpus import

Solution 1:

I have installed gensim in my windows computer successfully,but it also appears memoryError, when I set the topic numbers larger for big data. because the space complexity of gensim is O(K*V) where the K is topics numbers and V is the size of the dictionary, it depends on your computer RAM. so you can set the topic numbers to 50 or less than 100, which can solve it. maybe firstly you should test the example on the genism official website:http://radimrehurek.com/gensim/index.html

Solution 2:

The MemoryError appears because Gensim is trying to keep all of the data you need in memory while analyzing it. The solutions are scarse:

• Use a server with more memory (AWS machine, anything more powerful than your PC)
• Try a python interpreter in 64 bit
• Try reducing the size parameter in model.save(). This will lead to have less features representing your words
• Try increasing the min_count parameter in model.save(). This will make the model consider only words that appear at least min_count times

Be careful though, these last 2 advices will modify the characteristics of your model