scriptharness.config module

The goal of flexible configuration is to make each script useful in a variety of contexts and environments.

scriptharness.config.LOGGER_NAME

str

logging.getLogger name

scriptharness.config.OPTION_REGEX

re.compile

regular expression to validate a commandline option

scriptharness.config.VALID_ARGPARSE_ACTIONS

tuple

for validating the ConfigVariable action

scriptharness.config.STRINGS

dict

strings for ConfigVariable

scriptharness.config.DEFAULT_CONFIG_DEFINITION

dict

Config definition to create the default ConfigTemplate for all scriptharness scripts.

class scriptharness.config.ConfigTemplate(config_dict)

Bases: object

Short for Config Template Definition, or CTD. Because scriptharness scripts can take any arbitrary configuration variables or commandline options from various locations, it’s difficult to tell what requires what, what’s optional, and what’s extraneous.

By allowing the developer to create a config template definition, we can check for config well-formedness.

config_variables

dict

a name to ConfigVariable dictionary

parser

argparse.ArgumentParser

this is the commandline parser.

add_argument(*args, **kwargs)

Helper method to make ConfigTemplate usage more similar to ArgumentParser.

add_variable(definition, name=None)

Add a variable to the config template definition.

See scriptharness.config.ConfigVariable for the definition format.

Parameters:

• name (str) – the variable name. This maps to argparse’s dest
• definition (dict or ConfigVariable) – a ConfigVariable or the definition of the config variable.

all_options

Build and return set of all commandline options

Returns:options – all commandline options Return type:set

defaults()

Get the defaults for all the variables, even the non-commandline ones.

Returns:name to default value. Return type:dict

get_parser(**kwargs)

Create and populate the argparse.ArgumentParser for commandline parsing.

Parameters:**kwargs – keyword arguments to send to argparse.ArgumentParser. Returns:the commandline parser for this Config Template Return type:argparse.ArgumentParser

items()

Have ConfigTemplate act more like a dict.

Returns:self.config_variables.items()

remove_option(option)

Remove a commandline option from the ConfigTemplate.

Because we can add templates together, we may sometimes encounter conflicting commandline options. This method allows us to remove those options from one or both templates.

Parameters:option (str) – The commandline option to remove.

update(config_dict)

Update self with a new config_dict

Parameters:

• config_dict (dict) – A dict of ConfigVariables or dicts.
• strict (Optional[bool]) – When True, throw an exception when there’s a conflicting variable.

validate_config(config)

Validate a config dict against each ConfigVariable.validate_config check.

Parameters:config (dict) – the config dictionary to validate. Raises:scriptharness.exceptions.ScriptHarnessException – on error.

class scriptharness.config.ConfigVariable(name, definition)

Bases: object

This object defines what a single config variable looks like.

The variable is overridable from the commandline when when self.definition[‘options’] is defined. Otherwise the variable is only script-level and config-file-level settable.

The definition will look like this:

{
  # argparse-specific, for argparse.ArgumentParser.add_argument
  # if 'options' is not set, these will be ignored.
  'options': ['--foo', '-f'],
  'action': 'store',  # (None, 'store', 'store_const', 'store_true',
                      #  'store_false', 'append', 'append_const',
                      #  'count', 'help', 'version', 'parsers')
                      # defaults to 'store'

  # argparse-related
  # if 'options' is set, these will be used with
  # argparse.ArgumentParser.add_argument; otherwise they're here for
  # the non-commandline-config.
  'help': 'help string',  # not sure whether this should be required
                          # or highly recommended.
  'required': True,
  'default': 'bar',
  'parent_parser': 'parent',  # this is for argparse --help sorting
  'type': str,  # a python type
  'choices': [],  # enum / list of choices

  # Not related to argparse
  'validate_cb': None,  # optional, function to validate the
                        # config.  This function should take the args
                        # (name, parsed_args) and return a list of
                        # error message strings.
  'incompatible_vars': [],  # names of incompatible vars if this var
                            # is set
  'required_vars': [],  # names of other vars that are required to be
                        # set if this var is set
  'optional_vars': [],  # names of other vars that are optionally
                        # used in relation to this var.  This is purely
                        # informational.
}

name

str

the name of the variable. This corresponds to the argparse dest, or the config dict key.

definition

dict

the config definition for this variable. See above for the format.

add_argument(parser)

If self.definition[‘options’] is set, add the appropriate argument to the parser.

Parameters:parser (argparse.ArgumentParser) – the parser to add the argument to. Returns:on success. Return type:argparse.Action Raises:ScriptHarnessException – on argparse.ArgumentParser.add_argument error.

validate_config(config)

Once we build the config, we can validate it by sending the built config to each of these methods.

Parameters:config (dict) – the config built from build_config() Returns:messages – any error messages, if applicable. Return type:list of strings

scriptharness.config.action_config_template(all_actions)

Create an action option parser from the action list.

Actions to run are specified as the argparse.REMAINDER options.

Parameters:all_actions (iterable) – this is either all Action objects for the script, or a data structure of pairs of action_name:enabled to pass to iterate_pairs(). Returns:with action options Return type:ConfigTemplate

scriptharness.config.build_config(template, parsed_args, initial_config=None)

Build a configuration dict from the parser and initial config.

The configuration is built in this order:

• template defaults
• initial_config
• parsed_args.config_files, in order
• parsed_args.opt_config_files, in order, if they exist
• non-default parser args (cmdln_args)

So the commandline args can override everything else, as long as there are options to do so. (Commandline args will need to be a subset of the parser args). The final configuration file can override everything but the commandline args, and its config isn’t restricted as a subset of the parser options.

Parameters:

• parser (ArgumentParser) – the parser used to parse_args()
• parsed_args (argparse Namespace) – the results of parse_args()
• initial_config (Optional[dict]) – initial configuration to set before commandline args

scriptharness.config.download_url(url, path=None, timeout=None)

Download a url to a path.

Parameters:

• url (str) – the url to download
• path (Optional[str]) – the path to write the contents to.
• timeout (Optional[float]) – how long to wait before timing out.

Returns:

path – the path to the downloaded file.

Return type:

str

Raises:

scriptharness.exceptions.ScriptHarnessException – if there are download issues, or if we can’t write to path.

scriptharness.config.get_config_template(template=None, all_actions=None, definition=None)

Create a script ConfigTemplate.

If template is not defined, it will take the definition (defaults to DEFAULT_CONFIG_DEFINITION) and create a new ConfigTemplate. Otherwise it uses template.

If all_actions is defined, it will add an action ConfigTemplate to the template.

Parameters:

• template (Optional[ConfigTemplate]) – the ConfigTemplate to optionally append the action_template to. Defaults to None.
• all_actions (Optional[list]) – list of actions to generate an action ConfigTemplate. Defaults to None.
• definition (Optional[dict]) – config definition to prepopulate the ConfigTemplate with. Defaults to DEFAULT_CONFIG_DEFINITION.

Returns:

ConfigTemplate

scriptharness.config.get_filename_from_url(url)

Determine the filename of a file from its url.

Parameters:url (str) – the url to parse Returns:name – the name of the file Return type:str

scriptharness.config.get_list_actions_string(action_name, enabled, groups=None)

Build a string for –list-actions output.

Parameters:

• action_name (str) – name of the action
• enabled (bool) – whether the action is enabled by default
• groups (Optional[list]) – a list of action_group names that the action belongs to. Defaults to None.

Returns:

string – a line of –list-actions output.

Return type:

str

scriptharness.config.is_url(resource)

Is it a url?

Note

This function will return False for file:// strings

Parameters:resource (str) – possible url Returns:True if it’s a url, False otherwise. Return type:bool

scriptharness.config.parse_args(template, cmdln_args=None, **kwargs)

Parse the commandline args.

Parameters:

• template (ConfigTemplate) – specify the config template to use
• cmdln_args (Optional[list]) – override the commandline args with these
• **kwargs – sent to ConfigTemplate.get_parser() if parser is a ConfigTemplate

Returns:

tuple(ArgumentParser, parsed_args)

scriptharness.config.parse_config_file(path)

Read a config file and return a dictionary. For now, only support json.

Parameters:path (str) – path or url to config file. Returns:config – the parsed json dict. Return type:dict Raises:scriptharness.exceptions.ScriptHarnessException – if the path is unreadable or not valid json.

scriptharness.config.update_dirs(config, max_depth=2)

Directory paths for the script are defined in config. Absolute paths help avoid chdir issues.

scriptharness_base_dir (or any other directory path, or any config value) can be overridden during build_config(). Defining the directory paths as formattable strings is configurable but not overly complex.

Any key in config named scriptharness_SOMETHING_dir will be % formatted with the other dirs as the replacement dictionary.

Parameters:config (dict) – the config to parse for scriptharness_SOMETHING_dir keys.

scriptharness.config.validate_config_definition(name, definition)

Validate the ConfigVariable definition’s well-formedness.

Parameters:

• name (str) – the name of the variable
• definition (dict) – the definition to validate

Raises:

ScriptHarnessException – if there are any error messages