Python: Gensim Memory Error
import logging logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO) from gensim import corpora, models, similarities from nltk.corpus import
Solution 1:
I have installed gensim in my windows computer successfully,but it also appears memoryError, when I set the topic numbers larger for big data. because the space complexity of gensim is O(K*V) where the K is topics numbers and V is the size of the dictionary, it depends on your computer RAM. so you can set the topic numbers to 50 or less than 100, which can solve it. maybe firstly you should test the example on the genism official website:http://radimrehurek.com/gensim/index.html
Solution 2:
The MemoryError
appears because Gensim is trying to keep all of the data you need in memory while analyzing it.
The solutions are scarse:
- Use a server with more memory (AWS machine, anything more powerful than your PC)
- Try a python interpreter in 64 bit
- Try reducing the
size
parameter inmodel.save()
. This will lead to have less features representing your words - Try increasing the
min_count
parameter inmodel.save()
. This will make the model consider only words that appear at leastmin_count
times
Be careful though, these last 2 advices will modify the characteristics of your model
Post a Comment for "Python: Gensim Memory Error"