Nonblocking Queue Of Threads

I want to create simple queue of threads. THREADS WILL START WITH POST REQUEST I created simple example without requests. I tried to join threads, but it doesn't work as i want. de

Solution 1:

You can pass a timeout argument to join(), this will block the calling thread maximal timeout seconds. 'Maximal' because the thread can terminate earlier. If you don't pass a timeout, it will block until the thread finishes. So what you would have to do in your example is join the thread twice, the first time with a timeout which allows printing test1.

from threading import Thread
fromtime import sleep

def a():
    print('start a')
    sleep(5)
    print('end a')


def b():
    print('start b')
    sleep(5)
    print('end b')


t = Thread(target=a)
t.start()
t.join(0.1)
print('test1')
t.join()

t = Thread(target=b)
t.start()
t.join(0.1)
print('test2')
t.join()

Output:

start a
test1
end a
start b
test2
end b

Apart from that, I'm in doubt this is a good approach if your concern is just limiting requests per second. You could reuse your threads instead of recreating them for every request this would spare you the creation overhead. You could use a ThreadPool and emit a request task into the pool with your specified heartbeat.

import logging
from time import sleep
from multiprocessing.pool import ThreadPool


defmake_request(*args):
    logger.debug(f'making request number {args[0]}')
    sleep(5)


definit_logging(log_level=logging.DEBUG):
    fmt = '[%(asctime)s %(levelname)-8s %(threadName)s' \
          ' %(funcName)s()] --- %(message)s'
    logging.basicConfig(format=fmt, level=log_level)


if __name__ == '__main__':

    N_THREADS = 5
    N_REQUESTS = 10

    arguments = [*zip(range(N_REQUESTS))]  # [(0,), (1,), (2,) ...]

    init_logging()
    logger = logging.getLogger()

    with ThreadPool(N_THREADS) as pool:
        for args in arguments:
            pool.apply_async(make_request, args=args)
            sleep(1)  # heartbeat

Example Output:

[2018-09-0903:17:06,303 DEBUG    Thread-1 make_request()] --- making request number 0
[2018-09-0903:17:07,304 DEBUG    Thread-2 make_request()] --- making request number 1
[2018-09-0903:17:08,306 DEBUG    Thread-3 make_request()] --- making request number 2
[2018-09-0903:17:09,307 DEBUG    Thread-4 make_request()] --- making request number 3
[2018-09-0903:17:10,308 DEBUG    Thread-5 make_request()] --- making request number 4
[2018-09-0903:17:11,309 DEBUG    Thread-1 make_request()] --- making request number 5
[2018-09-0903:17:12,310 DEBUG    Thread-2 make_request()] --- making request number 6
[2018-09-0903:17:13,311 DEBUG    Thread-3 make_request()] --- making request number 7
[2018-09-0903:17:14,312 DEBUG    Thread-4 make_request()] --- making request number 8
[2018-09-0903:17:15,313 DEBUG    Thread-5 make_request()] --- making request number 9

Process finished with exit code 0