Python - How Can I Open A File And Specify The Offset In Bytes?

I'm writing a program that will parse an Apache log file periodically to log it's visitors, bandwidth usage, etc.. The problem is, I don't want to open the log and parse data I've

Solution 1:

You can manage the position in the file thanks to the seek and tell methods of the file class see https://docs.python.org/2/tutorial/inputoutput.html

The tell method will tell you where to seek next time you open

Solution 2:

log = open('myfile.log')
pos = open('pos.dat','w')
printlog.readline()
pos.write(str(f.tell())
log.close()
pos.close()

log = open('myfile.log')
pos = open('pos.dat')
log.seek(int(pos.readline()))
printlog.readline()

Of course you shouldn't use it like that - you should wrap the operations up in functions like save_position(myfile) and load_position(myfile), but the functionality is all there.

Solution 3:

If your logfiles fit easily in memory (this is, you have a reasonable rotation policy) you can easily do something like:

log_lines =open('logfile','r').readlines()
last_line = get_last_lineprocessed() #Fromsome persistent storage
last_line = parse_log(log_lines[last_line:])
store_last_lineprocessed(last_line)

If you cannot do this, you can use something like (see accepted answer's use of seek and tell, in case you need to do it with them) Get last n lines of a file with Python, similar to tail

Solution 4:

If you're parsing your log line per line, you could juste save line number from the last parsing. You would juste have then to start read it from the good line the next time.

Seeking is more usefull when you have to be in a very specific place in the file.

Solution 5:

Note that you can seek() in python from the end of the file:

f.seek(-3, os.SEEK_END)

puts the read position 3 lines from the EOF.

However, why not use diff, either from the shell or with difflib?