API reference

Decorators

@pystematic.experiment(name=None, inherit_params=None, defaults={})

Creates a new experiment with the decorated function as the main function.

Parameters

name (str, optional) – Name of the experiment. Defaults to the name of the main function.
inherit_params (Experiment, ExperimentGroup, List[Experiment], List[ExperimentGroup], optional) – An experiment, group or a list thereof to inherit parameters from. Defaults to None.
defaults (dict, optional) – A dict containing default values for parameters, will override any default set in the parameter declaration. Defaults to {}.

@pystematic.group(name=None, inherit_params=None)

Used to group experiments. This decorator is used on a function. Note that the decorated function will never be called. All parameters added to the group will be inherited by all experiments that are part of the group.

Parameters

name (str, optional) – The name of the group. Defaults to None.
inherit_params (Experiment, ExperimentGroup, List[Experiment], List[ExperimentGroup], optional) – An experiment, group or a list thereof to inherit parameters from. Defaults to None.

@pystematic.parameter(name: str, type: ~typing.Callable[[str], ~typing.Any] = <class 'str'>, default: ~typing.Optional[~typing.Union[~typing.Any, ~typing.Callable[[], ~typing.Any]]] = None, required: bool = False, allowed_values: ~typing.Optional[~typing.List[~typing.Any]] = None, is_flag: bool = False, multiple: bool = False, allow_from_file: bool = True, envvar: ~typing.Union[str, None, ~typing.Literal[False]] = None, help: ~typing.Optional[str] = None, default_help: ~typing.Optional[str] = None, hidden=False, behaviour=None)

Adds a parameter to an experiment.

Parameters

name (str) – The name of the parameter. The name must be a valid python identifier.
type (Callable[[str], Any], optional) – The type of the parameter. Must be a callable that takes a single string value argument and returns the converted value. Defaults to str.
default (Union[Any, Callable[[], Any], None], optional) – The default value of the parameter. Can be either a value or a zero arguments callable that returns the value. Defaults to None.
required (bool, optional) – Set to True if this parameter is required. If a default value is set, the parameter will effectively no longer be ‘required’, even if this options is set to True. Defaults to False.
allowed_values (list[Any], optional) – If given, the value must be in the list of allowed values. Defaults to None.
is_flag (bool, optional) – When set to True, this parameter is assumed to be a boolean flag. A flag parameter does not need to be given a value on the command line. Its mere presence on the command line will automatically assign it the value True. Defaults to False.
multiple (bool, optional) – When set to True, the parameter is assumed to be a list of zero or more values. Defaults to False.
allow_from_file (bool, optional) – Controls whether it should be allowed to load a value for this parameter from a params file. Defaults to True.
envvar (Union[str, None, Literal[False]], optional) – Name of the environment variable that the value for this parameter may be read from. Defaults to None.
help (Optional[str], optional) – A help text for the parameter that will be shown on the command line. Defaults to None.
default_help (Optional[str], optional) – A help text for the default value. If None, the default help text will be created by calling str(default_value). Useful when the default value is a callable. Defaults to None.

@pystematic.param_group(name, help=None, *parameters)

Defines a parameter group. Useful when you have many parameters and want to organize them. Parameter groups are not visible when passing parameters to, or reading parameters in the experiment. Their sole purpose is to make the CLI help output a bit more structured, which hopefully helps your colleagues when inspecting the experiment.

You define the parameters that you want to pass to this decorator with the normal parameter decorator, but without using the ‘@’ in front of the function. Like this:

import pystematic as ps

@ps.param_group(
    "param_group",
    ps.parameter(
        name="str_param",
        type=str
    ),
    ps.parameter(
        name="int_param",
        type=int
    )
)
@ps.experiment
def exp(params):
    print(f"str_param is {params["str_param"]} and int_param is {params["int_param"]}")

Parameters

name (str) – The name of the group
help (str, optional) – An optional description of this group. If provided, it has to be passed as the second argument to this decorator. Defaults to None.
*parameters (Parameter) – An arbirary number of parameters that should belong to this group. Use the parameter decorator - but without the ‘@’ - to create parameters that you pass as positional arguments to this function.

Experiment API

The experiment API is available for the currently running experiment. The use of the API when no experiment is running results in undefined behavior.

Attributes

These attributes holds information related to the current experiment. Note that they are uninitialized until an experiment has actually started.

pystematic.output_dir : pathlib.Path: Holds a pathlib.Path object that points to the current output directory. All output from an experiment should be written to this folder. All internal procedures that produce output will always write it to this folder. When you want to output something persistent from the experiment yourself, it is your responsibly to use this output directory.

pystematic.params : dict: Holds a dict of all parameters of the current experiment. It is the same dict that is passed to the main function. You should never modify this dict.

Functions

pystematic.new_seed(nbits=32) → int

Use this function to generate random numbers seeded by the experiment parameter random_seed. Expected use is to seed your own random number generators.

Parameters: nbits (int, optional) – The number of bits to use to represent the generated number. Defaults to 32.
Returns: A random number seeded by the experiment parameter random_seed.
Return type: int

pystematic.launch_subprocess(**additional_params) → Process

Launches a subprocess. The subprocess will be instructed to execute the main function of the currently running experiment, and have the same output directory and parameters as the current process.

Parameters: **additional_params – Any additional parameters that should be passed to the subprocess. Params given here takes precedence over the parameters copied from the current experiment.

Warning

The subprocess will be initialized with the same random seed as the current process. If this is not what you want, you should pass a new seed to this function in the random_seed parameter.

E.g.:

pystematic.launch_subprocess(random_seed=pystematic.new_seed())

pystematic.run_parameter_sweep(experiment, list_of_params, max_num_processes=1) → None

Runs an experiment multiple times with a set of different params. At most max_num_processes concurrent processes will be used. This call will block until all experiments have been run.

Parameters

experiment (Experiment) – The experiment to run.
list_of_params (list of dict) – A list of parameter dictionaries. Each corresponding to one run of the experiment. See pystematic.param_matrix() for a convenient way of generating such a list.
max_num_processes (int, optional) – The maximum number of concurrent processes to use for running the experiments. Defaults to 1.

pystematic.is_subprocess() → bool

Returns true if this process is a subprocess. I.e. it has been launched by a call to launch_subprocess() in a parent process.

Returns: Whether or not the current process is a subprocess.
Return type: bool

pystematic.local_rank()

Returns the local rank of the current process. The master process will always have rank 0, and every subprocess launched with pystematic.launch_subprocess() will be assigned a new local rank from an incrementing integer counter starting at 1.

Returns: The local rank of the current process.
Return type: int

pystematic.param_matrix(**param_values)

This function can be used to build parameter combinations to use when running parameter sweeps. It takes an arbitrary number of keywork arguments, where each argument is a parameter and a list of all values that you want to try for that parameter. It then builds a list of parameter dictionaries such that all combinations of parameter values appear once in the list. The output of this function can be passed directly to pystematic.run_parameter_sweep().

For example:

import pystematic as ps

param_list = ps.param_matrix(
    int_param=[1, 2],
    str_param=["hello", "world"]
)

assert param_list == [
    {
        "int_param": 1,
        "str_param": "hello"
    },
    {
        "int_param": 1,
        "str_param": "world"
    },
    {
        "int_param": 2,
        "str_param": "hello"
    },
    {
        "int_param": 2,
        "str_param": "world"
    }
]

Parameters: **param_values – A mapping from parameter name to a list of values to try for that parameter. If a value is not a list or tuple, it is assumed to be constant (its value will be the same in all combinations).
Returns: A list of parameter combinations created by taking the cartesian product of all values in the input.
Return type: list of dicts

Default parameters

The following parameters are added to all experiments by default. Note that these are also listed if you run an experiment from the command line with the --help option.

output_dir: Parent directory to store all run-logs in. Will be created if it does not exist. Default value is ./output.
random_seed: The value to seed the master random number generator with. Default is randomly generated.
params_file: Read experiment parameters from a yaml file, such as the one dumped in the output dir from an eariler run. When this option is set from the command line, any other options supplied after will override the ones loaded from the file.
debug: Sets debug flag ON/OFF. Configures the python logging mechanism to print all DEBUG messages. Default value is False.

Core types

These classes are not supposed to be instantiated manually, but only through their corresponding decorators.

class pystematic.core.Experiment(main_function, name=None, defaults_override={})

This is the class used to represent experiments. Note that you should not instantiate this class manually, but only through the pystematic.experiment() decorator.

cli(argv=None, exit_on_error=True)

Runs the experiment by parsing the parameters from the command line.

Parameters

argv (List[str], optional) – A list of command line arguments. If None, will use sys.argv. Defaults to None.
exit_on_error (bool, optional) – If an error occurs during cli parsing, the default behavior is to print the error and exit the program. You can disable this behavior by setting this parameter to False, in which case the error will be raised. Defaults to True.

get_param_groups(): Returns a list of all parameter groups for this experiment.

get_parameters(): Returns a list of all parameters registered with this experiment.

run(params={})

Runs the experiment in the current process with the provided parameters.

Parameters: params (dict, optional) – A dict containing values for the parameters. Defaults to {}.

run_in_new_process(params)

Runs the experiment in a new process with the parameters provided. Returns a handle to the process object used to run the experiment. If you want to wait for the experiment to finish you have to manually wait for the process to exit.

Parameters: params (dict) – A dict containing the values for the parameters.
Returns: The process object used to run the experiment
Return type: multiprocessing.Process

class pystematic.core.ExperimentGroup(main_function, name=None)

Use when you have many experiments and want to group them in some way.

cli(argv=None, exit_on_error=True)

Runs the group by parsing the parameters from the command line. The first argument should be the name of the experiment to run.

Parameters

argv (List[str], optional) – A list of command line arguments. If None, will use sys.argv. Defaults to None.
exit_on_error (bool, optional) – If an error occurs during cli parsing, the default behavior is to print the error and exit the program. You can disable this behavior by setting this parameter to False, in which case the error will be raised. Defaults to True.

experiment(name=None, inherit_params=None, defaults={}, inherit_params_from_group=True)

Creates a new experiment that is part of this group. See also pystematic.experiment().

Parameters

name (str, optional) – Name of the experiment. Defaults to the name of the main function.
inherit_params (Experiment, ExperimentGroup, List[Experiment], List[ExperimentGroup], optional) – An experiment, group or a list thereof to inherit parameters from. Defaults to None.
defaults (dict, optional) – A dict containing default values for parameters, will override any default set in the parameter declaration. Defaults to {}.
inherit_params_from_group (bool, optional) – By default, the experiment will inherit all params registered on the group. Set this option to False to disable this behavior. Defaults to True.

get_param_groups(): Returns a list of all parameter groups for this group.

get_parameters(): Returns a list of all parameters registered with this group.

group(name=None, inherit_params=None, inherit_params_from_group=True)

Creates a nested group. See also pystematic.group()

Parameters

name (str, optional) – Name of the group. Defaults to the name of the annotated function.
inherit_params (Experiment, ExperimentGroup, List[Experiment], List[ExperimentGroup], optional) – An experiment, group or a list thereof to inherit parameters from. Defaults to None.
inherit_params_from_group (bool, optional) – By default, the subgroup will inherit all params registered on the group. Set this option to False to disable this behavior. Defaults to True.

class pystematic.core.Parameter(name: str, type: ~typing.Callable[[str], ~typing.Any] = <class 'str'>, default: ~typing.Optional[~typing.Union[~typing.Any, ~typing.Callable[[], ~typing.Any]]] = None, required: bool = False, allowed_values: ~typing.Optional[~typing.List[~typing.Any]] = None, is_flag: bool = False, multiple: bool = False, allow_from_file: bool = True, envvar: ~typing.Union[str, None, ~typing.Literal[False]] = None, help: ~typing.Optional[str] = None, default_help: ~typing.Optional[str] = None, hidden=False, behaviour=None): Represents an experiment parameter. You will typically never interact with this class directly.

class pystematic.core.ParameterGroup(name, help, parameter_decorators)

class pystematic.core.PystematicApp

A single instance of this class is created when pystematic initializes. Its main purpose is to centralize extension management. When an extension is initialized it is handed a reference to that instance.

get_api_object()

Returns a handle to the pystematic global api. Extensions can retrieve the handle by calling this function, and modify the global api during initialization.

Returns: The api available under the pystematic global namespace.
Return type: module

on_after_experiment(callback, priority=50)

Adds a callback to the ‘after_experiment’ event. The event is triggered after the experiment main function has returned.

Parameters

callback (function) – A function that is called before an experiment is run. The callback must take a single argument which is the error that caused the experiment to end, or None if the experiment ended normally.
priority (int, optional) – The priority of the callback. Callbacks with lower priority are called before those with higher. Defaults to 50.

on_before_experiment(callback, priority=50)

Adds a callback to the ‘before_experiment’ event. The event is triggered before the experiment main function is called.

Parameters

callback (function) – A function that is called before an experiment is run. It should take two arguments; the experiment, and a dict of parameter values.
priority (int, optional) – The priority of the callback. Callbacks with lower priority are called before those with higher. Defaults to 50.

on_experiment_created(callback, priority=50)

Adds a callback to the ‘experiment_created’ event.

Parameters

callback (function) – A function that is called whenever a new experiment is defined. It should take the created experiment as a single argument.
priority (int, optional) – The priority of the callback. Callbacks with lower priority are called before those with higher. Defaults to 50.