The caching_store Module

Created on Jun 10, 2013

@author: joe

class cloudfusion.store.caching_store.MultiprocessingCachingStore(store, cache_expiration_time=60, cache_size_in_mb=2000, cache_id=None, cache_dir='/tmp/cloudfusion/')

Bases: cloudfusion.store.store.Store

Like CachingStore, but does not make guarantees as to the consistency of the wrapped store. Use of best effort strategy to synchronize the store. Employs multiple threads for increased throughput. Therefore, it can only use stores with a thread-safe put_file method. Unlike CachingStore, guarantees that write operations do not block for transfer, until the cache size limit is reached. Unlike CachingStore, guarantees that no write operations on the wrapped store are invoked until a cached item expires.

Parameters:
  • store – the store whose access should be cached
  • cache_expiration_time – the time in seconds until any cache entry is expired
  • cache_size_in_mb – Approximate limit of the cache in MB.
  • cache_id – Serves as identifier for a persistent cache instance.
  • cache_dir – Cache directory on local hard drive disk, default value is /tmp/cloudfusion.
_get_actual_modified_date(path)
_is_valid_path(path)
_raise_error_if_invalid_path(path)
_refresh_cache(path_to_file)

Reloads the locally cached file path_to_file from the wrapped store, if the wrapped store version is newer. The cached file’s last updated time is set to the current point of time. Makes a new cache entry if it does not exist yet. If the file was changed in the wrapped store after the cached file’s modified time, the cached file...

  • is updated with the contents of the file in the wrapped store
  • is set to not dirty
  • gets the modified time stamp of the file of the wrapped stores
Raises:NoSuchFilesytemObjectError if file does not exist in wrapped store.

TODO: exception raising does not work and needs to be implemented

account_info()
create_directory(directory)
delete(path, is_dir)
duplicate(path_to_src, path_to_dest)
exists(path)
flush()
get_bytes(path)
get_cache_expiration_time()
Returns:the time in seconds until any cache entry is expired
get_directory_listing(directory)
get_file(path_to_file)

If the file was updated in the wrapped store, then its content in the cache will be updated if its entry is expired but not dirty. :returns: string – the data of the file with the path path_to_file

get_free_space()
get_logging_handler()
get_max_filesize()

Return maximum number of bytes per file

get_metadata(path)
get_modified(path)
get_name()
get_overall_space()
get_used_space()
is_cached_version_invalid(path)
Returns:True if the stores version is newer than the cached entry or does not exist and False otherwise.
is_dir(path)
move(path_to_src, path_to_dest)
reconnect()
store_file(path_to_file, dest_dir='/', remote_file_name=None)
store_fileobject(fileobject, path)

Stores a fileobject to the cloudfusion.util.cache.Cache and if the existing fileobject has expired it is also written to the wrapped store. The cached file’s updated and modified attributes will be reset to the current point of time. The cached file’s dirty flag is set to False if the entry has expired and was hence written to the store. Otherwise it is set to True.

Parameters:
  • fileobject – The file object with the method read() returning data as a string
  • path – The path where the file object’s data should be stored, including the filename

The metadata_caching_store Module

class cloudfusion.store.metadata_caching_store.Entry

Bases: object

add_to_listing(path)
remove_from_listing(path)
set_is_dir()
set_is_file()
set_modified(modified=None)
class cloudfusion.store.metadata_caching_store.MetadataCachingStore(store, cache_expiration_time=60)

Bases: cloudfusion.store.store.Store

_MetadataCachingStore__get_size(fileobject)
_add_parent_dir_listing(path)

Add listing for parent directory of path to cache if it does not yet exist

_add_to_parent_dir_listing(path)
_does_not_exist_in_parent_dir_listing(path)

‘:returns: True if path does not exist in the cached directory listing

_is_valid_path(path)
_prefetch_directory(path)
_prepare_entry(path, metadata)
_raise_error_if_invalid_path(path)
_remove_from_parent_dir_listing(path)
account_info()
create_directory(directory)
delete(path, is_dir)
duplicate(path_to_src, path_to_dest)
exists(path)
flush()
get_bytes(path)
get_directory_listing(directory)
get_file(path_to_file)
get_free_space()
get_logging_handler()
get_max_filesize()

Return maximum number of bytes per file

get_metadata(path)
get_modified(path)
get_name()
get_overall_space()
get_used_space()
is_dir(path)
move(path_to_src, path_to_dest)
store_file(path_to_file, dest_dir='/', remote_file_name=None, interrupt_event=None)
store_fileobject(fileobject, path, interrupt_event=None)

The store Module

Created on 08.04.2011

exception cloudfusion.store.store.AlreadyExistsError(msg, status=0)

Bases: cloudfusion.store.store.StoreAccessError

exception cloudfusion.store.store.DateParseError(msg)

Bases: exceptions.Exception

exception cloudfusion.store.store.InterruptedException(msg)

Bases: exceptions.Exception

exception cloudfusion.store.store.InvalidPathValueError(path)

Bases: exceptions.ValueError

exception cloudfusion.store.store.NoSuchFilesytemObjectError(path, status=0)

Bases: cloudfusion.store.store.StoreAccessError

class cloudfusion.store.store.Store

Bases: object

Central interface for any cloud storage provider. Any cloud storage provider that is used by CloudFusion needs to implement this interface. After implementing the interface for a new provider, you can add file system access to it by introducing a new branch to the if statement in cloudfusion.pyfusebox.configurable_pyfusebox.ConfigurablePyFuseBox.__get_new_store(). Advanced functionality such as caching and concurrency are supplied by wrappers, which are already implemented. Path parameters are always absolute paths of a file system, starting with a ‘/’

_is_valid_path(path)
_raise_error_if_invalid_path(path)
account_info()
Returns:a human readable string describing account info like provider, name, statistics
create_directory(directory)

Create the remote directory directory :param directory: the absolute path name of the directory to create :raises: AlreadyExistsError if the directory does already exist:

delete(path, is_dir)

Delete file or directory tree at path. :param path: path to the file or directory to delete :param is_dir: True iff path points to a directory :raises:[Errno 39] Directory not empty:

duplicate(path_to_src, path_to_dest)

Duplicate file or directory from path_to_src to path_to_dest. If path_to_dest exists before, it is deleted or overwritten. If path_to_src is a directory, the directory is duplicated with all its files and directories. Either this method or move() needs to be implemented in a subclass. :param path_to_src: must never be the same as path_to_dest :param path_to_dest: must end in the name of the child directory or the file specified by path_to_src

exists(path)
Returns:True if a remote file or directory exists at path, and False otherwise
flush()

Deprecated, do not overwrite this method.

get_bytes(path)
Returns:the number of bytes of the file at path, or 0 if path is a directory
get_directory_listing(directory)
Returns:list of absolute file paths of files in directory
get_file(path_to_file)
Returns:the data of the remote file at path_to_file as a string
Raises:NoSuchFilesytemObjectError if the object does not exist
get_free_space()
Returns:free space in bytes
get_logging_handler()

Get the name of the logging handler used by a subclass, so that the wrappers may use the same logger. Wrappers are responsible for extended functionality like caching data or concurrency i.e. cloudfusion.store.transparent_caching_store.TransparentMultiprocessingCachingStore. This method might simply return get_name(), even if the subclass does not use a logger. :return: the name of the logging handler used by a subclass and its wrappers.

get_max_filesize()

Return maximum number of bytes per file; Some cloud storages limit the size of files to be uploaded.

get_metadata(path)

This method is a hook that must be implemented by subclasses. If it is implemented, the methods exists(), get_bytes(), is_dir() work out of the box.

Returns:A dictionary with the keys ‘modified’, ‘bytes’ and ‘is_dir’ containing the corresponding metadata for path

The value for ‘modified’ is a date in seconds, stating when the object corresponding to path was last modified. The value for ‘bytes’ is the number of bytes of the object corresponding to path. It is 0 if the object is a directory. The value for ‘is_dir’ is True if the object is a directory and False otherwise.

Raises:NoSuchFilesytemObjectError if the object does not exist
get_modified(path)
Returns:the time path was modified in seconds from the epoche
get_name()
Returns:the name of the service; i.e. Amazon S3, or Dropbox
get_overall_space()
Returns:overall space in bytes
get_used_space()
Returns:space used by files in bytes
is_dir(path)
Returns:True if path is a remote file, and False if it is a remote directory
Raises:NoSuchFilesytemObjectError if the remote object does not exist
move(path_to_src, path_to_dest)

Rename a remote file or directory path_to_src to path_to_dest. If path_to_dest exists before, it is deleted or overwritten. If path_to_src is a directory, the directory is renamed to path_to_dest. Default implementation relies on an implementation of duplicate() in a subclass, but it should be overwritten. Either this method or duplicate() needs to be implemented in a subclass. :param path_to_src: path to a remote file or directory :param path_to_dest: path of the new remote file or directory

reconnect()

Try to reconnect to the service.

store_file(path_to_file, dest_dir='/', remote_file_name=None, interrupt_event=None)

Store the local file path_to_file to directory dest_dir on the store. :param path_to_file: local file path :param dest_dir: remote destination directory to store the contents of the local file to :param remote_file_name: the file name on the store; by default this is the original file name if this parameter is None. :param interrupt_event: (optional) If the value is not None, listen for an interrupt event with with interrupt_event.wait() until the file has been stored. Abort the upload if interrupt_event.wait() returns. :returns: (optional) the date in seconds, when the file was updated

store_fileobject(fileobject, path, interrupt_event=None)

Store the contents of fileobject to path on the store. :param fileobject: A file like object. The position of the fileobject needs to be at 0 (use fileobject.seek(0) before calling this method) :param path: The remote file path to store the contents of fileobject to :param interrupt_event: (optional) If the value is not None, listen for an interrupt event with with interrupt_event.wait() until the file has been stored. Abort the upload if interrupt_event.wait() returns. :returns: (optional) the date in seconds, when the file was updated

exception cloudfusion.store.store.StoreAccessError(msg, status=0)

Bases: exceptions.Exception

exception cloudfusion.store.store.StoreAutorizationError(msg, status=0)

Bases: cloudfusion.store.store.StoreAccessError

exception cloudfusion.store.store.StoreSpaceLimitError(msg='', status=0)

Bases: cloudfusion.store.store.StoreAccessError