Appengine Mapreduce Ndb, Deadlineexceedederror

we're trying to heavily use MapReduce in our project. Now we have this problem, there is a lots of 'DeadlineExceededError' errors in the log... One example of it ( traceback diffe

Solution 1:

Apparently you are doing too many puts than it is possible to insert in one datastore call. You have multiple options here:

1. If this is a relatively rare event - ignore it. Mapreduce will retry the slice and will lower put pool size. Make sure that your map is idempotent.
2. Take a look at http://code.google.com/p/appengine-mapreduce/source/browse/trunk/python/src/mapreduce/context.py - in your main.py you can lower DATASTORE_DEADLINE, MAX_ENTITY_COUNT or MAX_POOL_SIZE to lower the size of the pool for the whole mapreduce.

Solution 2:

If you're using an InputReader, you might be able to adjust the default batch_size to reduce the number of entities processed by each task.

I believe the task queue will retry tasks, but you probably don't want it to, since it'll likley hit the same DeadlineExceededError.

Data inconsistencies are possible.

See this question as well. App Engine - Task Queue Retry Count with Mapper API