FEV_KEGG.KEGG.File module

class FEV_KEGG.KEGG.File.CacheEntry(isCached: bool, absolutePath: str, result=None)

Bases: object

Abstract entry of the cache.

While cache() returns the content of a cache entry directly, cacheEntry() returns an instance of this class. This allows for delayed disk access, because results are always computed within cache() or cacheEntry(), but using this class, the actual reading/writing of the cache file is only started upon executing getResult().

Parameters:

• isCached (bool) – True if the underlying cache file already exists and, thus, will be read, not written.
• absolutePath (str) – The absolute path of the underlying cache file.
• result (Object, optional) – Contains the result of the computation to be written into the underlying cache file. Obviously only necessary if isCached == False.

Note

This is especially useful when calculating results within a process pool, to utilise parallelism, but writing results within a thread pool in the main process, keeping the process pool from idling, while waiting for the disk. All this is necessary, because neither ProcessPoolExecutor nor ThreadPoolExecutor support pre-emptive scheduling. The caveat of this approach is that the result of the computation, which might be several megabytes, has to be transferred between the background process and the main process via IPC, which is relatively slow. Still, on my laptop, this approach proved to be significantly faster overall.

getResult(noDiskIO=False)

Result of the cached computation.

Automatically performs read or write operation in this thread, depending on whether the computation was already cached or has only been computed just now.

Parameters:noDiskIO (bool, optional) – If True, the result currently available in memory is returned, without any disk access. If the computation was already cached, i.e. an underlying cache file exists, this is most likely None! Only use if you know exactly what you are doing! Returns:Result of the cached computation, no matter if it came from a cache file or from a fresh computation. Return type:Object Raises:OSError – File could not be opened. Only relevant if noDiskIO == False.

FEV_KEGG.KEGG.File.atomic_write(filename, text=True, keep=False, suffix='.bak', prefix='tmp')

Context manager for overwriting a file atomically.

Usage:

with atomic_write("myfile.txt") as f:
    f.write("data")

The context manager opens a temporary file for writing in the same directory as filename. On cleanly exiting the with-block, the temp file is renamed to the given filename. If the original file already exists, it will be overwritten and any existing contents replaced. (On POSIX systems, the rename is atomic. Other operating systems may not support atomic renames, in which case the function name is misleading.)

If an uncaught exception occurs inside the with-block, the original file is left untouched. By default the temporary file is not preserved. To keep the temp file, pass keep=True. Any errors in deleting the temp file are ignored. By default, the temp file is opened in text mode. To use binary mode, pass text=False as an argument. On some operating systems, this make no difference.

By default, the temp file will have a name starting with “tmp” and ending with “.bak”. You can vary that by passing strings as the suffix and prefix arguments.

Note

Copyright (c) 2017 ActiveState Software Inc.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

FEV_KEGG.KEGG.File.cache(folder_path, file_name)

Decorator for caching files on disk.

Checks if result of wrapped function has already been cached into the file specified by FEV_KEGG.settings.cachePath/folder_path/file_name. If yes, read file and return content. If no, execute wrapped function, write result to file, and then return result.

Parameters:

• folder_path (str) – Path of the file, in a format your OS understands. Something like ‘subfolder/another_folder/’ should most likely work. Remember, this is relative to your FEV_KEGG.settings.cachePath!
• file_name (str) – Name of the file itself.

Returns:

A decorator to automatically cache the decorated function’s return value into the file specified by folder_path/file_name.

Return type:

decorator

Raises:

• Error – If the decorated function raises any error. Only relevant if the decorated function is executed at all, which it only is if the result has not already been cached.
• OSError – File could not be opened.

FEV_KEGG.KEGG.File.cacheEntry(folder_path, file_name)

Decorator for caching files on disk, but returning only an intermediate CacheEntry.

Similar to cache decorator, but does not read content from file nor write result to file. Checks if result of wrapped function has already been cached into the file specified by FEV_KEGG.settings.cachePath/folder_path/file_name. If yes, return appropriate CacheEntry. This object provides all information and methods necessary to read/write the desired data from cache. If no, execute wrapped function, and then return appropriate CacheEntry.

Parameters:

• folder_path (str) – Path of the file, in a format your OS understands. Something like ‘subfolder/another_folder/’ should most likely work. Remember, this is relative to your FEV_KEGG.settings.cachePath!
• file_name (str) – Name of the file itself.

Returns:

A decorator to automatically return the delayed disk access in a CacheEntry instance.

Return type:

decorator

Raises:

• Error – If the decorated function raises any error. Only relevant if the decorated function is executed at all, which it only is if the result has not already been cached.
• OSError – File could not be opened.

Note

This is especially handy when splitting computation (in this function) from disk I/O (in CacheEntry) in a parallel computation environment.

FEV_KEGG.KEGG.File.createPath(fileName)

Extracts the path from fileName and ensures all folders exist.

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath.

Parameters:fileName (str) – Path and file name in a format your OS understands. Something like ‘subfolder/another_folder/myFile.txt’ should most likely work.

FEV_KEGG.KEGG.File.doesFileExist(fileName) → bool

Does fileName already exist AND is a file?

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath.

Parameters:fileName (str) – Path and name of the file, in a format your OS understands. Something like ‘subfolder/another_folder/myFile.txt’ should most likely work. Returns:Whether the file already exists. Return type:bool

FEV_KEGG.KEGG.File.doesFolderExist(folderName) → bool

Does fileName already exist AND is a folder?

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath.

Parameters:fileName (str) – Path and name of the folder, in a format your OS understands. Something like ‘subfolder/another_folder’ should most likely work. Returns:Whether the folder already exists. Return type:bool

FEV_KEGG.KEGG.File.getFileHandleRead(fileName) → io.IOBase

A low-level read-only handle to a text file.

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath.

Parameters:

fileName (str) – Path and name of the file, in a format your OS understands. Something like ‘subfolder/another_folder/myFile.txt’ should most likely work.

Returns:

A handle to the open file. Be sure to close it eventually!

Return type:

IOBase

Raises:

• ValueError – Decoding from UTF-8 failed.
• OSError – File could not be opened.

Warning

Make sure your code will close this handle eventually! Else, you will waste a lot of resources and eventually prevent your code from opening any other resources!

FEV_KEGG.KEGG.File.getFileHandleWrite(fileName) → io.IOBase

A low-level read-write handle to a text file.

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath. File will be overwritten completely, if already present.

Parameters:

fileName (str) – Path and name of the file, in a format your OS understands. Something like ‘subfolder/another_folder/myFile.txt’ should most likely work.

Returns:

A handle to the open file. Be sure to close it eventually!

Return type:

IOBase

Raises:

• ValueError – Decoding from UTF-8 failed.
• OSError – File could not be opened.

Warning

Make sure your code will close this handle eventually! Else, you will waste a lot of resources and eventually prevent your code from opening any other resources!

FEV_KEGG.KEGG.File.readBytesFromFileAtOnce(fileName) → bytes

Reads from a binary file.

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath. Reads the file fully at once (more memory intensive).

Parameters:

• data (bytes or bytearray) – Content of the file. Will not be encoded.
• fileName (str) – Path and name of the file, in a format your OS understands. Something like ‘subfolder/another_folder/myFile.txt’ should most likely work.
• atomic (bool, optional) – If True, write file atomically.

Returns:

Content of the file. Not decoded.

Return type:

bytes

Raises:

OSError – File could not be opened.

FEV_KEGG.KEGG.File.readGeneratorFromFileLinewise(fileName) → Generator[str, None, None]

Reads string-generator from a text file.

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath. Returns a generator (can only be read once!) for reading a file line by line (less memory intensive). Does NOT return the newline characters.

Parameters:

fileName (str) – Path and name of the file, in a format your OS understands. Something like ‘subfolder/another_folder/myFile.txt’ should most likely work.

Yields:

str – Line of content of the file. Will be decoded from UTF-8.

Raises:

• ValueError – Decoding from UTF-8 failed.
• OSError – File could not be opened.

FEV_KEGG.KEGG.File.readListFromFileAtOnce(fileName) → List[str]

Reads list from a text file.

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath. Reads the file fully at once (more memory intensive). Does NOT return the newline characters.

Parameters:

fileName (str) – Path and name of the file, in a format your OS understands. Something like ‘subfolder/another_folder/myFile.txt’ should most likely work.

Returns:

Content of the file as a list. Will be decoded from UTF-8.

Return type:

List[str]

Raises:

• ValueError – Decoding from UTF-8 failed.
• OSError – File could not be opened.

FEV_KEGG.KEGG.File.readSetFromFileAtOnce(fileName) → Set[str]

Reads set from a text file.

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath. Reads the file fully at once (more memory intensive). Does NOT return the newline characters.

Parameters:

fileName (str) – Path and name of the file, in a format your OS understands. Something like ‘subfolder/another_folder/myFile.txt’ should most likely work.

Returns:

Content of the file as a set. Will be decoded from UTF-8.

Return type:

Set[str]

Raises:

• ValueError – Decoding from UTF-8 failed.
• OSError – File could not be opened.

FEV_KEGG.KEGG.File.readStringFromFileAtOnce(fileName) → str

Reads from a text file.

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath. Reads the file fully at once (more memory intensive).

Parameters:

fileName (str) – Path and name of the file, in a format your OS understands. Something like ‘subfolder/another_folder/myFile.txt’ should most likely work.

Returns:

Content of the file. Will be decoded from UTF-8.

Return type:

str

Raises:

• ValueError – Decoding from UTF-8 failed.
• OSError – File could not be opened.

FEV_KEGG.KEGG.File.writeToFile(content: will be encoded in UTF-8, fileName: will be overwritten, if already present, atomic=True)

Writes content to a text file.

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath. File will be overwritten completely, if already present.

Parameters:

• content (str) – Content of the file. Will be encoded into UTF-8.
• fileName (str) – Path and name of the file, in a format your OS understands. Something like ‘subfolder/another_folder/myFile.txt’ should most likely work.
• atomic (bool, optional) – If True, write file atomically.

Raises:

• ValueError – Encoding into UTF-8 failed.
• OSError – File could not be opened.

FEV_KEGG.KEGG.File.writeToFileBytes(data, fileName: will be overwritten, if already present, atomic=True)

Writes content to a binary file.

fileName is relative to your cache folder! See FEV_KEGG.settings.cachePath. File will be overwritten completely, if already present.

Parameters:

• data (bytes or bytearray) – Content of the file. Will not be encoded.
• fileName (str) – Path and name of the file, in a format your OS understands. Something like ‘subfolder/another_folder/myFile.txt’ should most likely work.
• atomic (bool, optional) – If True, write file atomically.

Raises:

OSError – File could not be opened.