FEV_KEGG.Util.Parallelism module

FEV_KEGG.Util.Parallelism.enableShallCancelThreads()

Set the global flag for cancelling all threads to “cancelling”.

FEV_KEGG.Util.Parallelism.getNumberOfThreadsDownload(isSSDB=False)

Number of threads allowed for downloading.

Depends on whether this function is executed inside a process pool, because all processes will have to share the total number of allowed download threads, in order not to exceed it.

Parameters:isSSDB (bool. optional) – IF True, uses a lower number of threads, because KEGG SSDB can not handle too many parallel connections. Returns:The number of allowed threads for downloading. Return type:int

FEV_KEGG.Util.Parallelism.getNumberOfThreadsFile()

Number of thread allowed for file access.

Depends on whether this function is executed inside a process pool, because all processes will have to share the total number of allowed file access threads, in order not to exceed it.

Returns:The number of allowed threads for file access. Return type:int

FEV_KEGG.Util.Parallelism.getShallCancelThreads()

Get the global flag for cancelling all threads.

Returns:Whether all tasks currently running in threads of this process shall abort and return early. Return type:bool

FEV_KEGG.Util.Parallelism.getTqdmPosition()

Get position where this process’ progress bar shall be printed.

Interprets current process’ name as the index for positioning its tqdm progress bar alongside the progress bar of all other processes.

Returns:Position index of this process’ progress bar, to be used by tqdm. Return type:int

Note

This function depends on the naming of processes of your OS! You might have to replace this function to get clean positioning of progress bars.

FEV_KEGG.Util.Parallelism.isMainProcess()

Check if this is running in the main process.

Returns:True if this is the main process (any thread), False if not. Return type:bool

FEV_KEGG.Util.Parallelism.isMainThread()

Check if this is running in the main thread.

Returns:True if this is the main thread (of any process), False if not. Return type:bool

FEV_KEGG.Util.Parallelism.isMainThreadInMainProcess()

Check if this is running in the main thread of the main process.

Returns:True if this is the main thread of the main process, False if not. Return type:bool

FEV_KEGG.Util.Parallelism.keyboardInterruptHandler(processPoolFutures=None, threadPool=None, threadPoolFutures=None, silent=False, terminateProcess=False)

Handle KeyboardInterrupt.

Tries to consistently handle KeyboardInterrupt in all current implementations of parallel computation. If this is inside a background process, cancel threadPool tasks, running and scheduled (if listed inside threadPoolFutures). If this is inside the main process, also cancel processPoolFutures and shut down the global process pool itself. May also exit this very process, if terminateProcess == True.

Parameters:

• processPoolFutures (Iterable, optional) – An Iterable of all Future currently scheduled in the process pool.
• threadPool (ThreadPoolExecutor, optional) – The thread pool currently running within the main process. This pool usually handles the process pool’s results in a parallel manner.
• threadPoolFutures (Iterable, optional) – The Iterable of all Future currently scheduled in the threadPool.
• silent (bool, optional) – If True, prints messages of the current state of handling the interrupt.
• terminateProcess (bool, optional) – If True, calls sys.exit() on this very process, at the end of this function.

Warning

Only ever call this from main thread (in any process)! Else, you might get unexpected and dangerous behaviour!

Note

terminateProcess is useful to cancel tasks already scheduled inside the call_queue of the pool, because there is no method to empty this queue and cancel already scheduled tasks. Keep in mind that this queue is always populated with p + 1 tasks, while p (different) tasks are being executed, with p being the number of processes in the pool. However, this only works as long as ProcessPoolExecutor does not re-create broken processes!

FEV_KEGG.Util.Parallelism.printBelowProgress(message)

Does a print() beneath any progress bars.

Simply prints as many newlines as there are processes in use by the process pool. Finally, prints the message.

Parameters:message (str) – Message to be printed below any progress bars.

Note

This is an indirect measure of currently possibly rendered progress bars, at best! There is no check if the possible progress bars are actually rendered. Therefore, a surplus of empty lines may appear, or there may not be enough.

FEV_KEGG.Util.Parallelism.processPool = <FEV_KEGG.lib.Python.concurrent.futures.InterruptibleProcessPoolExecutor object>

The global process pool.

If you want to run FEV@KEGG inside another thread or process, your will have to replace this with a concurrent.futures.Executor of your choice.

See also

startProcessPool

Default creator of a process pool.

FEV_KEGG.Util.Parallelism.resetShallCancelThreads()

Reset the global flag for cancelling all threads to “not cancelling”.

FEV_KEGG.Util.Parallelism.shallCancelThreads = False

Allows threads to be canceled in a timely manner, across the whole process.

Specifically, any task currently running in a thread of this process can check this variable to see whether it is supposed to abort and return early.

Note

This is necessary because concurrent.futures.ThreadPoolExecutor does not have support for neither pre-emptively aborting tasks (call_queue) nor emptying the pending task queue (pending_work_items).

FEV_KEGG.Util.Parallelism.startProcessPool()

Creates the process pool and stores it in processPool.

You might not want this when you already run your own concurrent.futures.Executor. If you run your own, let processPool point to it.

See also

FEV_KEGG.lib.Python.concurrent.futures.InterruptibleProcessPoolExecutor()

Our own executor used as a process pool, because it handles KeyboardInterrupt nicer than the default implementation.

Note

A process pool will only work from within the main thread in the main process of your application. If you want to run FEV@KEGG inside another thread or process, your will have to replace processPool with a concurrent.futures.Executor that allows running outside the main thread. This may be a dummy class.