configuration package

Example usage of config_parser_base

The following example is how to use dae.configuration.dae_config_parser.DAEConfigParser which inherit dae.configuration.config_parser_base.ConfigParserBase.

>>> from dae.configuration.dae_config_parser import DAEConfigParser

Content of the config which will be used in this example is:

  • DAE.conf:

    [DEFAULT]
    instance_id = data_hg19_startup
    
    [storage.genotype_impala]
    type = impala
    impala.host = localhost
    impala.port = 21050
    hdfs.host = localhost
    hdfs.port = 8020
    hdfs.base_dir = /user/%(instance_id)s/studies
    
    [studiesDB]
    confFile = %(wd)s/studiesDB.conf
    dir = %(wd)s/studies
    
    [datasetsDB]
    confFile = %(wd)s/datasetsDB.conf
    dir = %(wd)s/datasets
    
    [genomesDB]
    confFile = %(wd)s/genomesDB.conf
    
    [genomicScoresDB]
    confFile = %(wd)s/genomicScores.conf
    
    [annotation]
    confFile = %(wd)s/annotation.conf
    
    [phenoDB]
    dir = %(wd)s/pheno
    
    [geneInfoDB]
    confFile = %(wd)s/geneInfo.conf
    
    [defaultConfiguration]
    confFile = %(wd)s/defaultConfiguration.conf
    

Read and Parse file configuration

>>> os.environ['DAE_IMPALA_HOST'] = 'impala'
>>> config = DAEConfigParser.read_and_parse_file_configuration(
...     work_dir=fixtures_dir, environment_override=False
... )
>>> print(config.storage.genotype_impala.impala.host)
localhost

>>> print(config.storage.genotype_impala.impala.port)
21050

>>> print(config.storage.genotype_impala.impala.db)
gpf_variant_db
>>> print(config.gpfjs.permission_denied_prompt)
This is a default permission denied prompt. Please log in or register.

dae.configuration.config_parser_base - module for parsing configuration

class dae.configuration.config_parser_base.CaseSensitiveConfigParser(defaults=None, dict_type=<class 'collections.OrderedDict'>, allow_no_value=False, *, delimiters=('=', ':'), comment_prefixes=('#', ';'), inline_comment_prefixes=None, strict=True, empty_lines_in_values=True, default_section='DEFAULT', interpolation=<object object>, converters=<object object>)[source]

Bases: configparser.ConfigParser

Modified ConfigParser that allows case sensitive options.

optionxform(option)[source]

The default implementation returns a lower-case version of options.

class dae.configuration.config_parser_base.ConfigParserBase[source]

Bases: object

ConfigParserBase is a base class for all configuration parsers. It is responsible for parsing all of the system configuration. It can be inherited by a specific parser or be used alone. It has different kinds of methods for reading and parsing configurations. Any class which inherits from this parser may use its class properties which are used for parsing different kinds of properties from the configuration. The return value for all of the classmethods for reading a configuration is a Box object. All of its methods are class or static methods.

CAST_TO_BOOL = ()

Holds a tuple of configuration property names. Each property’s value will be cast from a string (you can find all possible values here) to a boolean value.

CAST_TO_INT = ()

Holds a tuple of configuration property names. Each property’s value will be cast from a string (which is number) to an integer.

ENABLED_DIR = '.'

Holds a path, relative to the configuration_dir parameter, passed in read_directory_configurations() and read_and_parse_directory_configurations() functions. This relative path is a directory of the enabled configurations.

FILTER_SELECTORS = {}

Holds a mapping of configuration property selectors to a list of their entries. All entries of the selectors which are not in this list or not a valid entry will be ignored. Invalid entries will always be filtered.

Note

Each entry which is not filtered will automatically receive an id property if one is not already set. The default value will be the name of the entry.

INCLUDE_PROPERTIES = ()

Holds a tuple of configuration property names. Any property name that is not inside this tuple will be omitted from the end result

SECTION = None

Holds the name of a configuration section which would be parsed by the parse() function. If it is None, then all of the sections in the configuration will be parsed one by one.

SPLIT_STR_LISTS = ()

Holds a tuple of configuration property names. Each property’s value will be converted from a comma-separated string to a list.

SPLIT_STR_SETS = ()

Holds a tuple of configuration property names. Each property’s value will be converted from a comma-separated string to a set.

VERIFY_VALUES = {}

Holds a mapping of configuration property names to functions. The values of the property names will be given as an input to the supplied functions, and the results of these functions will replace the original values. The messages of any exceptions that occur during the execution of the functions will be aggregated and displayed as the message of a raised VerificationError exception.

classmethod parse(config)[source]

Parse SECTION section from configuration if it is defined else parse all of the sections in the configuration.

Parameters

config (Box or dict) – configuration.

Returns

parsed configuration.

Return type

Box or dict or None

classmethod parse_section(config_section)[source]

Parse one section from configuration based on the SPLIT_STR_LISTS, SPLIT_STR_SETS, CAST_TO_BOOL, CAST_TO_INT, FILTER_SELECTORS and VERIFY_VALUES class properties. If enabled property is defined in the configuration section then it would be checked if the configuration section is enabled.

Parameters

config_section (Box or dict) – section from configuration.

Returns

parsed configuration section.

Return type

Box or dict or None

classmethod read_and_parse_directory_configurations(configurations_dir, defaults=None, fail_silently=False)[source]

Read and parse multiple configurations stored in a directory. This method will search recursively in the directory for configurations.

Parameters
  • configuration_dir (str) – directory which contains configurations.

  • defaults (dict or None) – default values which will be used when configuration file is readed.

  • fail_silently (bool) – flag which will indicate if it will be raised RuntimeError exception if combination of configuration_dir and ENABLED_DIR doesn’t exists.

Returns

list of read and parsed configurations.

Return type

list(Box)

Raises

RuntimeError – if fail_silently is False and combination of configuration_dir and ENABLED_DIR doesn’t exists.

classmethod read_and_parse_file_configuration(config_file, work_dir, defaults=None)[source]

Read and parse configuration stored in a file.

Parameters
  • config_file (str) – file which contains configuration.

  • work_dir (str) – working directory which will be added as work_dir and wd default values in the configuration.

  • defaults (dict or None) – default values which will be used when configuration file is readed.

Returns

read and parsed configuration.

Return type

Box or None

classmethod read_config(config_file, work_dir, defaults=None)[source]

Read configuration stored in a file. Delimiter used in this configuration file must be =. If SECTION is defined in the read configuration file then it would be checked if the configuration file is enabled.

Parameters
  • config_file (str) – file which contains configuration.

  • work_dir (str) – working directory which will be added as work_dir and wd default values in the configuration.

  • defaults (dict or None) –

    default values which will be used when configuration file is readed. If this parameter is dict then it can define the following properties:

    • values - this property define dict which would be added as default values for all of the configurational sections.

    • sections - with this property you can define dict containing default values for particular section in the configuration.

    • override - with this property you can define dict containing values which will be used for overriding values in the particular section in the configuration.

    • conf - with this property you can define configuration file. Values in the sections of this configuration are used as default values to the corresponding section in the read configuration.

Returns

read configuration.

Return type

dict or None

classmethod read_directory_configurations(configurations_dir, defaults=None, fail_silently=False)[source]

Read multiple configurations stored in a directory. This method will search recursively in the directory for configurations.

Parameters
  • configuration_dir (str) – directory which contains configurations.

  • defaults (dict or None) – default values which will be used when configuration file is readed.

  • fail_silently (bool) – flag which will indicate if it will be raised RuntimeError exception if combination of configuration_dir and ENABLED_DIR doesn’t exists.

Returns

list of read configurations.

Return type

list(Box)

Raises

RuntimeError – if fail_silently is False and combination of configuration_dir and ENABLED_DIR doesn’t exists.

classmethod read_file_configuration(config_file, work_dir, defaults=None)[source]

Read configuration stored in a file.

Parameters
  • config_file (str) – file which contains configuration.

  • work_dir (str) – working directory which will be added as work_dir and wd default values in the configuration.

  • defaults (dict or None) – default values which will be used when configuration file is readed.

Returns

read configuration.

Return type

Box or None

exception dae.configuration.config_parser_base.VerificationError(message)[source]

Bases: Exception

Exception to raise when there is some verification problem in the configuration.

Parameters

message (str) – message of the exeption

Variables

message (str) – message of the exeption

dae.configuration.dae_config_parser - module for parsing DAE Configuration

class dae.configuration.dae_config_parser.DAEConfigParser[source]

Bases: dae.configuration.config_parser_base.ConfigParserBase

DAEConfigParser is responsible for parsing DAE Configuration. It inherits ConfigParserBase and use some of its methods for reading and parsing configuration.

CAST_TO_INT = ('port',)
DEFAULT_SECTION_VALUES = {'gpfjs': {'permissionDeniedPrompt': 'This is a default permission denied prompt. Please log in or register.'}}

Holds a mapping of configuration section to a mapping of section property to a value of this property. This mapping will be used as default values for the configuration sections.

DEFAULT_VALUES = {'dir': '%(wd)s/studies', 'hdfs.base_dir': '/tmp', 'hdfs.port': '0', 'impala.db': 'gpf_variant_db', 'impala.port': '21050'}

Holds a a mapping of property to a value of this property. This mapping will be used as default values for the configuration sections.

FILTER_SELECTORS = {'storage': None}
classmethod parse(config, dae_data_dir=None)[source]

Parse all of the sections in the DAE configuration. This method overload parse from ConfigParserBase class by adding dae_data_dir parameter.

Parameters
  • config (Box or dict) – configuration.

  • dae_data_dir (str or None) – path of data directory

Returns

parsed configuration.

Return type

Box or dict or None

classmethod read_and_parse_file_configuration(config_file='DAE.conf', work_dir=None, defaults=None, environment_override=True)[source]

Read and parse DAE configuration stored in a file. This method overload read_and_parse_file_configuration from the ConfigParserBase class by adding environment_override parameter. It also combine default and override values from environment and from predifined in the class property DEFAULT_SECTION_VALUES above.

Parameters
  • config_file (str) – file which contains configuration.

  • work_dir (str) – working directory which will be added as work_dir and wd default values in the configuration.

  • defaults (dict or None) – default values which will be used when configuration file is readed.

  • environment_override (bool) – it shows if read_config will use override from defaults.

Returns

read and parsed configuration.

Return type

Box or None