Internal API

Warning

This API is for now considered completely internal and could change at any time, except for items listed in Public API

Engine

exception exekall.engine.NoOperatorError

Bases: Exception

Exception raised when no operator has been found to build objects of a needed type.

class exekall.engine.IndentationManager(style)

Bases: object

Manage the indentation level in a generated script.

indent()

dedent()

class exekall.engine.ValueDB(froz_val_seq_list, adaptor_cls=None)

Bases: object

Serializable object that contains a graph of FrozenExprVal.

This allows storing all the objects computed for each subexpression that was executed for later inspection.

Parameters:

• froz_val_seq_list (list(FrozenExprValSeq)) – List of FrozenExprValSeq to store at the root of the database.

• adaptor_cls (type) – A subclass of exekall.customization.AdaptorBase that was used when setting up the expressions. This class can provide hooks that are called when deserializing the database.

The values of each expression is recorded in a root list of FrozenExprValSeq.

PICKLE_PROTOCOL = 4

classmethod merge(db_list, roots_from=None)

Merge multiple databases together.

Parameters:

• db_list (list(ValueDB)) – Lists of DB to merge

• roots_from (ValueDB) – Only get the root values from the specified DB. The other DBs are only used to provide subexpressions values for expressions already present in that DB. That DB is also appended to db_list.

When two different FrozenExprVal are available for a given UUID, the one that contains the most information will be selected. This allows natural removal of values created using an PrebuiltOperator when the original object is also available.

classmethod from_path(path, relative_to=None)

Deserialize a ValueDB from a file.

At the moment, it is assumed to be an LZMA compressed Pickle file.

Parameters:

• path (str or pathlib.Path) – Path to the file containing the serialized ValueDB.

• relative_to (str or pathlib.Path) – If provided, path is interpreted as being relative to that folder, or to the containing folder if it is a file. This is mainly used to ease generation of scripts that can provide relative_to=__file__ so that the artifact folder can be moved around easily.

__reduce_ex__(protocol)

Provide custom serialization that will call the adaptor’s hooks.

to_path(path, optimize=True)

Write the DB to the given file.

Parameters:

• path (pathlib.Path or str) – path to file to write the DB into

• optimize (bool) – Optimize the representation of the DB. This may increase the dump time and memory consumption, but should speed-up loading/file size.

get_by_uuid(uuid)

Get a FrozenExprVal by its UUID.

get_by_predicate(predicate, flatten=True, deduplicate=False)

Get FrozenExprVal matching the predicate.

Parameters:

• predicate (collections.abc.Callable) – Predicate callable called with an instance of FrozenExprVal. If it returns True, the value is selected.

• flatten (bool) – If False, return a set of frozenset of objects. There is a frozenset set for each expression result that shared their parameters. If False, the top-level set is flattened into a set of objects matching the predicate.

• deduplicate (bool) – If True, there won’t be duplicates across nested sets.

get_roots(flatten=True)

Get all the root FrozenExprVal.

Parameters:

flatten (bool) – If True, a set of FrozenExprVal is returned, otherwise a set of frozensets of FrozenExprVal is returned. Each set will correspond to an expression, and values inside the frozenset will correspond to the values of that expression.

Root values are the result of full expression (as opposed to subexpressions).

prune_by_predicate(predicate)

Create a new ValueDB with all FrozenExprVal matching the predicate replaced by a terminal PrunedFrozVal.

Parameters:

predicate (collections.abc.Callable) – Predicate callable called with an instance of FrozenExprVal. If it returns True, the value is pruned out.

This allows trimming a ValueDB to a smaller size by removing non-necessary content.

get_all(**kwargs)

Get all FrozenExprVal contained in this database.

Variable keyword arguments:

Forwarded to ValueDB.get_by_predicate()

get_by_type(cls, include_subclasses=True, **kwargs)

Get all FrozenExprVal contained in this database which value has the specified type.

Parameters:

• cls (type) – Class to match.

• include_subclasses (bool) – If True, the check is done using isinstance, otherwise an exact type check is done using is.

Variable keyword arguments:

Forwarded to ValueDB.get_by_predicate()

Note

If a subexpressions had a exekall._utils.NoValue value, it will not be selected as type matching is done on the value itself, not the return type of the callable used for that sub expression.

get_by_id(id_pattern, qual=False, full_qual=False, **kwargs)

Get all FrozenExprVal contained in this database which ID matches the given pattern.

Parameters:

• id_pattern (str) – fnmatch.fnmatch() pattern used to match the ID.

• qual (bool) – If True, the match will be performed on the qualified ID.

• full_qual (bool) – If True, the match will be performed on the fully qualified ID.

Variable keyword arguments:

Forwarded to ValueDB.get_by_predicate()

class exekall.engine.ScriptValueDB(db, var_name='db')

Bases: object

Class tying together a generated script and a ValueDB.

Parameters:

• db (ValueDB) – ValueDB used.

• var_name (str) – Name of the variable used to represent the ValueDB in the generated script.

get_snippet(expr_val, attr)

exception exekall.engine.CycleError

Bases: Exception

Exception raised when a cyclic dependency is detected when building the Expression out of a set of callables.

exception exekall.engine.AlreadyVisitedException

Bases: Exception

Exception raised when a node has already been visited during a graph traversal.

class exekall.engine.ExprHelpers

Bases: Mapping

Helper class used by all expression-like classes.

It mainly implements the mapping protocol, with keys being parameters and values being subexpressions computing the value of these parameters.

__hash__()

Return hash(self).

fold(f, init=None, visit_once=False)

Fold the function f over the instance and all its parents listed in param_map attribute, deep first.

Parameters:

• f (collections.abc.Callable) –

  Function to execute for each instance. It must take two parameters:

  • as first parameter: the return value of the previous invocation of f. For the first call, init value is used.

  • as second parameter: the instance of ExprHelpers that is being visited.

• init (object) – Initial value passed to f.

• visit_once (bool) – If True, each ExprHelpers will only be visited once.

class exekall.engine.ExpressionBase(op, param_map)

Bases: ExprHelpers

Base class of all expressions proper.

Parameters:

• op (Operator) – Operator to call for that expression

• param_map (collections.OrderedDict) – Mapping of parameter names to other ExpressionBase. The mapping must maintain its order, so that it is possible to get the parameter list in the order it was defined in the sources.

classmethod cse(expr_list)

Apply a flavor of common subexpressions elimination to the list of ExpressionBase.

clone_by_predicate(predicate)

Create a new ExpressionBase, with the outer levels cloned and the inner sub expressions shared with the original one.

Parameters:

predicate (collections.abc.Callable) – Predicate called on ExpressionBase used to know at what level the sub expressions should not be cloned anymore, but instead shared with the original ExpressionBase. All parents of a shared expression will be shared no matter what to ensure consistent expressions.

format_structure(full_qual=True, graphviz=False)

Format the expression in a human readable way.

Parameters:

• full_qual (bool) – If True, use fully qualified IDs.

• graphviz – If True, return a graphviz description suitable for the dot graph rendering tool.

• graphviz – bool

get_id(*args, marked_expr_val_set={}, **kwargs)

Return the ID of the expression.

Parameters:

• marked_expr_val_set (set(ExpressionBase)) – If True, return a two-line strings with the second line containing marker characters under the ID of all ExpressionBase specified in that set.

• with_tags (bool) – Add the tags extracted from the values of each ExprVal.

• remove_tags (set(str)) – Do not add the specified tags values.

• qual (bool) – If True, return the qualified ID.

• full_qual (bool) – If True, return the fully qualified ID.

• style (str or None) – If "rst", return a Sphinx reStructuredText string with references to types.

• hidden_callable_set (set(collections.abc.Callable)) – Hide the ID of all callables given in that set, including their parent’s ID.

get_script(*args, **kwargs)

Return a script equivalent to that ExpressionBase.

Variable keyword arguments:

Forwarded to get_all_script().

classmethod get_all_script(expr_list, prefix='value', db_path='VALUE_DB.pickle.xz', db_relative_to=None, db=None, adaptor_cls=None)

Return a script equivalent to executing the specified ExpressionBase.

Parameters:

• expr_list (list(ExpressionBase)) – List of ExpressionBase to turn into a script

• prefix (str) – Prefix used to name variables containing the values of expressions of expr_list.

• db_path (str) – Path to the serialized ValueDB that contains the FrozenExprVal of the expressions in expr_list. This is used to generate commented-out code allowing to deserialize values instead of calling the operator again.

• relative_to (str) – Passed to ValueDB.from_path() when the ValueDB is opened at the beginning of the script. This can typically be set to __file__, so that the script will be able to refer to the ValueDB using a relative path.

• db (ValueDB or None) – ValueDB containing the FrozenExprVal that were computed when computing expressions of expr_list. If None is provided, a new ValueDB object will be built assuming expr_list is a list of ComputableExpression.

• adaptor_cls (type) – If db=None, used to build a new ValueDB.

EXPR_DATA_VAR_NAME = 'EXPR_DATA'

class exekall.engine.ComputableExpression(op, param_map, data=None)

Bases: ExpressionBase

Expression that also contains its computed values.

Parameters:

data (ExprData or None) – ExprData to use when computing the values of the expression. The data of the root ComputableExpression will be used for all the subexpressions as well during the execution.

See also

ExpressionBase

Instances of this class contains values, whereas Expression do not.

classmethod from_expr(expr, **kwargs)

Build an instance from an ExpressionBase

Variable keyword arguments:

Forwarded to __init__

classmethod from_expr_list(expr_list)

Build an a list of instances from a list of ExpressionBase.

Note

Common Subexpression Elimination using ExpressionBase.cse() will be applied on the resulting list.

get_id(mark_excep=False, marked_expr_val_set={}, **kwargs)

Return the ID of the expression.

Parameters:

• mark_excep (bool) – Mark expressions listed by get_excep().

• marked_expr_val_set (bool) – If mark_excep=False mark these exceptions, otherwise it is ignored.

See also

ExpressionBase.get_id()

find_expr_val_seq_list(param_map)

Return a list of ExprValSeq that were computed using the given parameters.

Parameters:

param_map (collections.OrderedDict) – Mapping of parameter names to values

Note

param_map will be checked as a subset of the parameters.

classmethod execute_all(expr_list, *args, **kwargs)

Execute all expressions of expr_list after applying Common Expression Elimination and yield tuples of (ExpressionBase, ExprVal).

Parameters:

expr_list (list(ExpressionBase)) – List of expressions to execute

Variable keyword arguments:

Forwarded to execute().

prepare_execute()

Prepare the expression for execution.

This includes appropriate cloning of expressions using ConsumerOperator and ExprData.

Note

Calling this method manually is only useful to get more accurate graphs when showing the structure of the expression, since it is done in any case by execute().`

execute(post_compute_cb=None)

Execute the expression and yield its ExprVal.

Parameters:

post_compute_cb (collections.abc.Callable) – Callback called after every computed value. It takes two parameters: 1) the ExprVal that was just computed and 2) a boolean that is True if the ExprVal was merely reused and False if it was actually computed.

Note

The prepare_execute() is called prior to executing.

get_all_vals()

Get all ExprVal that were computed for that expression.

get_excep()

Get all ExprVal containing an exception that are reachable from ExprVal computed for that expression.

class exekall.engine.ClassContext(op_map, cls_map)

Bases: object

Collect callables and types that put together will be used to create Expression.

Parameters:

• op_map (dict(type, list(Operator))) – Mapping of types to list of Operator that can produce that type.

• cls_map (dict(type, list(type))) – Mapping of types to a list of compatible types. A common “compatibility” relation is issubclass. In that case, keys are classes and values are the list of all (direct and indirect) subclasses.

classmethod from_op_set(op_set, forbidden_pattern_set={}, restricted_pattern_set={}, compat_cls=<built-in function issubclass>)

Build an ClassContext out of a set of Operator.

Parameters:

• op_set (set(Operator)) – Set of Operator to consider.

• forbidden_pattern_set (set(str)) – Set fnmatch.fnmatch() type name patterns that are not allowed to be produced.

• compat_cls (collections.abc.Callable) – Callable defining the compatibility relation between two classes. It will be called on two classes and shall return True if the classes are compatible, False otherwise.

Parm restricted_pattern_set:

Set of fnmatch.fnmatch() Operator ID pattern. Operators matching that pattern will be the only one allowed to produce the type they are producing, or any other compatible type.

build_expr_list(result_op_set, non_produced_handler='raise', cycle_handler='raise')

Build a list of consistent Expression.

Parameters:

• result_op_set (set(Operator)) – Set of Operator that will constitute the roots of expressions.

• non_produced_handler (str or collections.abc.Callable) –

  Handler to be used when a needed type is produced by no Operator:

  • raise: will raise a NoOperatorError exception

  • ignore: the expression will be ignored

  • a callback: called with the following parameters:

    • __qualname__ of the type that cannot be produced.

    • name of the Operator for which a value of the type was needed.

    • name of the parameter for which a value of the type was needed.

    • a stack (tuple) of callables which is the path leading from the root expression to the operator for which the type was needed.

• cycle_handler (str or collections.abc.Callable) –

  Handler to be used when a cycle is detected in the built Expression:

  • raise: will raise a CycleError exception

  • ignore: the expression will be ignored

  • a callback: called with a tuple of callables constituting the cycle.

All combinations of compatible classes and operators will be generated.

class exekall.engine.Expression(op, param_map)

Bases: ExpressionBase

Static subclass ExpressionBase tying Operator with its parameters.

Instances of this class do not contain any computed values, they can be considered as read-only structure.

See also

ComputableExpression.

validate(cls_map)

Check that the Expression does not involve two classes that are compatible.

This ensures that only one class of each “category” will be used in each expression, so that all references to that class will point to the same expression after ExpressionBase.cse() is applied.

exception exekall.engine.AnnotationError

Bases: Exception

Exception raised when there is a missing or invalid PEP 484 annotation.

exception exekall.engine.PartialAnnotationError

Bases: AnnotationError

Exception raised when there is a missing PEP 484 annotation, but other parameters are annotated.

This usually indicates a missing annotation in a function otherwise supposed to be annotated.

class exekall.engine.ForcedParamType

Bases: object

Base class for types placeholders used when forcing the value of a parameter using Operator.force_param().

class exekall.engine.UnboundMethod(callable_, cls)

Bases: object

Wrap a function in a similar way to Python 2 unbound methods.

Parameters:

• callable (collections.abc.Callable) – method to wrap.

• cls (type) – Class on which the method is available.

Note

It is generally assumed that if a given method is wrapped in an UnboundMethod, all subclasses will also have that method wrapped the same way.

Note

UnboundMethod can wrap things such as staticmethod or classmethod as well, it is simply a wrapper used to attach the class of origin.

__call__(*args, **kwargs)

Call self as a function.

__hash__()

Return hash(self).

class exekall.engine.Operator(callable_, non_reusable_type_set=None, tags_getter=None)

Bases: object

Wrap a callable.

Parameters:

• callable (collections.abc.Callable) – callable to represent.

• non_reusable_type_set (set(type)) – Set of non reusable types. If the callable produces a subclass of these types, it will be considered as non-reusable.

• tags_getter (collections.abc.Callable) – Callback used to get the tags for the objects returned by the callable. It takes the object as argument, and is expected to return a mapping of tags names to values.

property callable_globals

Returns a dictionnary of global variables as seen by the callable.

force_param(param_value_map, tags_getter=None)

Force the value of a given parameter of the callable.

Parameters:

• param_value_map (dict(str, list(object))) – Mapping of parameter names to list of values that this parameter should take.

• tags_getter (collections.abc.Callable) – Callable used to return the tags for the values of the parameter. Same as Operator’s __init__ parameter.

property resolved_callable

Fully unwrapped callable. If the callable is a class, it’s __init__ will be returned.

property unwrapped_callable

Fully unwrapped callable.

See also

functools.wraps()

get_name(*args, **kwargs)

Get the name of the callable, or None if no name can be retrieved.

get_id(full_qual=True, qual=True, style=None)

Get the ID of the operator.

Parameters:

• full_qual (bool) – Fully qualified name, including the module name.

• qual (bool) – Qualified name.

• style (str or None) – If rst, a Sphinx reStructuredText string is returned.

property name

Same as get_name()

property mod_name

Name of the module the callable is defined in.

If the callable is an UnboundMethod, the module of its class is returned.

Note

The callable is first unwrapped.

property src_loc

Get the source location of the unwrapped callable.

See also

exekall._utils.get_src_loc()

property value_type

Annotated return type of the callable.

property is_genfunc

True if the callable is a generator function.

property is_class

True if the callable is a class.

property is_static_method

True if the callable is a staticmethod.

property is_method

True if the callable is a plain method.

property is_cls_method

True if the callable is a classmethod.

property is_factory_cls_method

True if the callable is a factory classmethod, i.e. a classmethod that returns objects of the class it is defined in (or of a subclass of it), or has a return annotation of typing.Self.

make_expr_val_iter(expr, param_map)

Make an iterator that will yield the computed ExprVal.

class exekall.engine.PrebuiltOperator(obj_type, obj_list, id_=None, **kwargs)

Bases: Operator

Operator that injects prebuilt objects.

Parameters:

• obj_type (type) – Type of the objects that are injected.

• obj_list (list(object)) – List of objects to inject

• id (str or None) – ID of the operator.

Variable keyword arguments:

Forwarded to Operator constructor.

get_name(*args, **kwargs)

Get the name of the callable, or None if no name can be retrieved.

get_id(*args, style=None, **kwargs)

Get the ID of the operator.

Parameters:

• full_qual (bool) – Fully qualified name, including the module name.

• qual (bool) – Qualified name.

• style (str or None) – If rst, a Sphinx reStructuredText string is returned.

property src_loc

Get the source location of the unwrapped callable.

See also

exekall._utils.get_src_loc()

property is_genfunc

True if the callable is a generator function.

property is_method

True if the callable is a plain method.

make_expr_val_iter(expr, param_map)

Make an iterator that will yield the computed ExprVal.

class exekall.engine.ConsumerOperator(consumer=None)

Bases: PrebuiltOperator

Placeholder operator used to represent the consumer of the an expression asking for it.

Parameters:

consumer (collections.abc.Callable) – Callable that will consume the value of the expression refering to its consumer.

get_name(*args, **kwargs)

Get the name of the callable, or None if no name can be retrieved.

property consumer

class exekall.engine.ExprDataOperator(data=None)

Bases: PrebuiltOperator

Placeholder operator for ExprData.

The ExprData that will be used is the same throughout an expression, and is the one of the root expression.

property data

property uuid_list

class exekall.engine.ExprValSeq(expr, iterator, param_map, post_compute_cb=None)

Bases: object

Sequence of ExprVal produced by an ComputableExpression.

Parameters:

• expr (ComputableExpression) – ComputableExpression that was used to compute the values.

• iterator (collections.abc.Iterator) – Iterator that yields the ExprVal. This is used when the expressions are being executed.

• param_map (collections.OrderedDict) – Ordered mapping of parameters name to ExprVal used to compute the recored ExprVal.

• post_compute_cb (collections.abc.Callable) – See ComputableExpression.execute()

Since ComputableExpression can represent generator functions, they are allowed to create multiple ExprVal.

classmethod from_one_expr_val(expr_val)

Build an ExprValSeq out of a single ExprVal.

See also

ExprValSeq for parameters description.

classmethod from_expr(expr, param_map, **kwargs)

Build an ExprValSeq out of a single ComputableExpression.

See also

ExprValSeq for parameters description.

iter_expr_val()

Iterate over the iterator and yield ExprVal.

post_compute_cb will be called when a value is computed or reused.

class exekall.engine.ExprValParamMap

Bases: OrderedDict

Mapping of parameters to ExprVal used when computing the value of a ComputableExpression.

is_partial(ignore_error=False)

Return True if the map is partial, i.e. some parameters don’t have a value.

That could be because one of them could not be computed due to an exception, or because it was skipped since it could not lead to a result anyway.

classmethod from_gen_map(param_gen_map)

Build a ExprValParamMap out of a mapping of parameters names and expressions to generators.

Parameters:

param_gen_map (collections.OrderedDict) – Mapping of tuple(param_name, param_expr) to an iterator that is ready to generate the possible values for the generator.

Generators are assumed to only yield once.

See also

from_gen_map_product() for cases where the generator is expected to yield more than once.

classmethod from_gen_map_product(param_gen_map)

Yield collections.OrderedDict for each combination of parameter values.

Parameters:

param_gen_map (collections.OrderedDict) – Mapping of tuple(param_name, param_expr) to an iterator that is ready to generate the possible values for the generator.

class exekall.engine.ExprValBase(param_map, value, excep, uuid, duration, log)

Bases: ExprHelpers

Base class for classes representing the value of an expression.

Parameters:

• param_map (dict(str, ExprValBase)) – Map of parameter names of the Operator that gave this value to their ExprValBase value.

• value (Exception) – Value that was computed. If no value was computed, exekall._utils.NoValue will be used.

• excep – Exception that was raised while computing the value. If no excpetion was raised, exekall._utils.NoValue will be used.

• uuid (str) – UUID of the ExprValBase

• duration (float or None) – Time it took to compute the value or the exception in seconds.

• log (ExprValLog) – Log collected during the computation of the value.

property cumulative_duration

Sum of the duration of all ExprValBase that were involved in the computation of that one.

get_full_log(level)

Reconstruct a consistent log output at the given level by stitching the logs of all parent ExprValBase that were involved in the computation of that value.

Parameters:

level (str) – Logging level to reconstruct.

get_by_predicate(predicate)

Get a list of parents ExprValBase for which the predicate returns True.

get_excep()

Get all the parents ExprValBase for which an exception was raised.

get_by_type(cls, include_subclasses=True, **kwargs)

Get a list of parents ExprValBase having a value of the given type.

Parameters:

• cls (type) – Type to look for.

• include_subclasses (bool) – If True, the check is done using isinstance, otherwise an exact type check is done using is.

format_structure(full_qual=True)

Format the value and its parents in a human readable way.

Parameters:

full_qual (bool) – If True, use fully qualified IDs.

class exekall.engine.FrozenExprVal(param_map, value, excep, uuid, duration, log, callable_qualname, callable_name, recorded_id_map, tags)

Bases: ExprValBase

Serializable version of ExprVal.

Parameters:

• uuid (str) – UUID of the ExprVal

• duration (float or None) – Time it took to compute the value or the exception in seconds.

• log (ExprValLog) – Log collected during the computation of the value.

• callable_qualname (str) – Qualified name of the callable that was used to compute the value, including module name.

• callable_name (str) – Name of the callable that was used to compute the value.

• recorded_id_map (dict) – Mapping of ExprVal.get_id() parameters to the corresponding ID. The parameters

The most important instance attributes are:

value	Value that was computed, or NoValue if it was not computed. This could be because of an exception when computing it, or because computing the value was skipped.
excep	Exception that was raised when trying to compute the value, or NoValue.
uuid	String UUID of that value. This is unique and can be used to correlate with logs, or deduplicate across multiple ValueDB.
duration	Time it took to compute the value in seconds.

Since it is a subclass of ExprValBase, the FrozenExprVal value of the parameters of the callable that was used to compute it can be accessed using the subscript operator [].

Instances of this class will not refer to the callable that was used to create the values, and will record the ID of the values instead of recomputing it from the graph of ExpressionBase. This allows manipulating FrozenExprVal as standalone objects, with minimal references to the code, which improves robustness against API change that would make deserializing them impossible.

See also

ExprValBase

property callable_

Callable that produced the value.

If it cannot be found, an AttributeError is raised.

property type_

Type of the value, as reported by the return annotation of the callable that produced it.

If the callable cannot be found, the actual value type is used. If the value type is compatible with the return annotation of the callable (i.e. is a subclass), the value type is returned as well.

property type_names

classmethod from_expr_val(expr_val, hidden_callable_set=None)

Build a FrozenExprVal from one ExprVal.

Parameters:

hidden_callable_set (set(collections.abc.Callable)) – Set of callables that should not appear in the ID.

get_id(full_qual=True, qual=True, with_tags=True, remove_tags={})

Return recorded IDs generated using ExprVal.get_id().

get_tags()

class exekall.engine.PrunedFrozVal(froz_val)

Bases: FrozenExprVal

Placeholder introduced by ValueDB.prune_by_predicate() when a FrozenExprVal is pruned.

property cumulative_duration

Sum of the duration of all ExprValBase that were involved in the computation of that one.

class exekall.engine.FrozenExprValSeq(froz_val_list, param_map)

Bases: Sequence

Sequence of FrozenExprVal analogous to ExprValSeq.

Parameters:

• froz_val_list (list(FrozenExprVal)) – List of FrozenExprVal.

• param_map (dict) – Parameter map that was used to compute the ExprVal. See ExprValSeq.

Since it inherits from collections.abc.Sequence, it can be iterated over directly.

classmethod from_expr_val_seq(expr_val_seq, **kwargs)

Build a FrozenExprValSeq from an ExprValSeq.

Variable keyword arguments:

Forwarded to FrozenExprVal.from_expr_val().

classmethod from_expr_list(expr_list, **kwargs)

Build a list of FrozenExprValSeq from an list of ComputableExpression.

Parameters:

expr_list (list(ComputableExpression)) – List of ComputableExpression to extract the ExprVal from.

Variable keyword arguments:

Forwarded to from_expr_val_seq().

class exekall.engine.ExprValLog(log_map, utc_datetime)

Bases: object

Logging output created when computing an ExprValBase.

Parameters:

• log_map (dict(str, str)) – Mapping of log level name to log content.

• utc_datetime (datetime.datetime) – UTC timestamp as a datetime object corresponding to the beginning of the log.

class exekall.engine.ExprVal(expr, param_map, value=NoValue, excep=NoValue, uuid=None, duration=None, log=None)

Bases: ExprValBase

Value computed when executing ComputableExpression.

Parameters:

• expr (ExpressionBase) – Expression for which this value was computed

• uuid (str) – UUID of the value.

See also

ExprValBase for the other parameters.

get_tags()

Return a dictionary of the tags.

format_tags(remove_tags={})

Return a formatted string for the tags of that ExprVal.

Parameters:

remove_tags (set(str)) – Do not include those tags

classmethod validate(expr_val_list)

Check that the list contains only one ExprVal for each ComputableExpression, unless it is non reusable.

get_id(*args, with_tags=True, **kwargs)

See ExpressionBase.get_id.

class exekall.engine.UnEvaluatedExprVal(expr)

Bases: ExprVal

Placeholder ExprVal created when computing the value was known to not lead to anything useful.

class exekall.engine.Consumer

Bases: object

Placeholder type used in PEP 484 annotations by callables to refer to the callable that will use their value.

Note

This leads to cloning the expression refering to its consumer for each different consumer.

class exekall.engine.ExprData

Bases: dict

Placeholder type used in PEP 484 annotations by callables to refer to the expression-wide data dictionnary.

Customization

class exekall.customization.AdaptorBase(args)

Bases: object

Base class of all adaptors.

Parameters:

args (argparse.Namespace) – Command line argument namespace as returned by argparse.ArgumentParser.parse_args(). Depending on the subcommand used, it could be the arguments of exekall run or exekall compare (or any other subcommand).

An adaptor is a class providing a number of hooks to customize the behaviour of exekall on a given codebase. It should be implemented in a module called exekall_customize in order to be found by exekall run.

name = 'default'

get_non_reusable_type_set()

Return a set of non-reusable types.

Defaults to an empty set.

static get_tags(value)

Returns a dictionnary of tag names to their value.

Default to empty set of tags, unless value is a numbers.Number in which case the value of the number is used.

filter_op_set(op_set)

Returns a potentially filtered set of exekall.engine.Operator.

This allows removing some operators from the ones that will be used to build expressions. Defaults to filtering out operators without any parameter, since the “spark” values are usually introduced by the adaptor directly, instead of calling functions from the code base.

format_expr_list(expr_list, verbose=0)

Return a string that is printed right after the list of executed expressions.

Parameters:

expr_list (list(exekall.engine.ExpressionBase)) – List of exekall.engine.ExpressionBase that will be executed. Note that this list has not yet undergone CSE, cloning for multiple iterations or other transformations.

This can be used to add some information about the expressions that are about to be executed.

get_prebuilt_op_set()

Returns a set of exekall.engine.PrebuiltOperator.

This allows injecting any “spark” value that is needed to build the expressions, like configuration objects. These values are usually built out of the custom CLI parameters added by the adaptor.

get_hidden_op_set(op_set)

Returns the set of hidden exekall.engine.Operator.

This allows hiding parts of the IDs that would not add much information but clutter them.

static register_run_param(parser)

Register CLI parameters for the run subcommand.

Parameters:

parser (argparse.ArgumentParser) – Parser of the run subcommand to add arguments onto.

static register_compare_param(parser)

Register CLI parameters for the compare subcommand.

Parameters:

parser (argparse.ArgumentParser) – Parser of the compare subcommand to add arguments onto.

compare_db_list(db_list)

Compare databases listed in db_list.

Parameters:

db_list (list(exekall.engine.ValueDB)) – List of exekall.engine.ValueDB to compare.

This is called by exekall compare to actually do something useful.

static register_show_param(parser)

Register CLI parameters for the show subcommand.

Parameters:

parser (argparse.ArgumentParser) – Parser of the show subcommand to add arguments onto.

show_db(db)

Show the content of the database.

Parameters:

db (exekall.engine.ValueDB) – exekall.engine.ValueDB to show.

This is called by exekall show to actually do something useful.

static get_default_type_goal_pattern_set()

Returns a set of patterns that will be used as the default value for exekall run --goal.

classmethod reload_db(db, path=None)

Hook called when reloading a serialized exekall.engine.ValueDB. The returned database will be used.

Parameters:

• db (exekall.engine.ValueDB) – exekall.engine.ValueDB that has just been deserialized.

• path (str or None) – Path of the file of the serialized database if available.

finalize_expr(expr)

Finalize an exekall.engine.ComputableExpression right after all its values have been computed.

format_result(expr_val)

Format an exekall.engine.ExprVal that is the result of an expression. It should return a (short) string that will be displayed at the end of the computation.

get_summary(result_map)

Return the summary of an exekall run session as a string.

Parameters:

result_map (dict(exekall.engine.ComputableExpression, list(exekall.engine.ExprVal))) – Dictionary of expressions to the list of their values.

get_run_exit_code(result_map)

Return an integer used as exekall run exit status.

Parameters:

result_map (dict(exekall.engine.ComputableExpression, exekall.engine.ExprVal)) – Dictionary of expressions to the list of their values.

classmethod get_adaptor_cls(name=None)

Return the adaptor class that has the name name.

Note

This is not intended to be overriden by subclasses.

Utils

exekall.utils.get_callable_set(module_set, verbose=False)

Get the set of callables defined in all modules of module_set.

We ignore any callable that is defined outside of the modules’ package.

Parameters:

module_set (set(types.ModuleType)) – Set of modules to scan.

exekall.utils.sweep_param(callable_, param, start, stop, step=1)

Used to generate a stream of numbers or strings to feed to a callable.

Parameters:

• callable (collections.abc.Callable) – Callable the numbers will be used by.

• param (str) – Name of the parameter of the callable the numbers will be providing values for.

• start (str) – Starting value.

• stop (str) – End value (inclusive)

• step (str) – Increment step.

If start == stop, only that value will be yielded, and it can be of any type.

The type used will either be one that is annotated on the callable, or the one from the default value if no annotation is available, or float if no default value is found. The value will then be built by passing the string to the type as only parameter.

exekall.utils.get_method_class(function)

Get the class in which a function is defined.

exception exekall._utils.NotSerializableError

Bases: Exception

exekall._utils.get_class_from_name(cls_name, module_map=None)

Get a class object from its full name (including the module name).

exekall._utils.create_uuid()

Creates a UUID.

exekall._utils.get_mro(cls)

Wrapper on top of inspect.getmro() that recognizes None as a type (treated like type(None)).

exekall._utils.get_method_class(function)

Get the class of a method by analyzing its name.

exekall._utils.get_name(obj, full_qual=True, qual=True, pretty=False)

Get a name for obj (function or class) that can be used in a generated script.

Parameters:

• full_qual (bool) – Full name of the object, including its module name.

• qual (bool) – Qualified name of the object

• pretty (bool) – If True, will show a prettier name for some types, although it is not guarnateed it will actually be a type name. For example, type(None) will be shown as None instead of NoneType.

exekall._utils.get_toplevel_module(obj)

Return the outermost module object in which obj is defined as a tuple of source file path and line number.

This is usually a package.

exekall._utils.get_src_loc(obj, shorten=True)

Get the source code location of obj

Parameters:

shorten (bool) – Shorten the paths of the source files by only keeping the part relative to the top-level package.

class exekall._utils.ExceptionPickler(*args, **kwargs)

Bases: Pickler

Fix pickling of exceptions so they can be reloaded without troubles, even when called with keyword arguments:

https://bugs.python.org/issue40917

reducer_override(obj)

classmethod dump_bytestring(obj, **kwargs)

classmethod dump_file(f, obj, **kwargs)

classmethod is_serializable(obj, raise_excep=False)

Try to Pickle the object to see if that raises any exception.

exekall._utils.is_serializable(obj, raise_excep=False)

Try to Pickle the object to see if that raises any exception.

exekall._utils.once(callable_)

Call the given function at most once per set of parameters

exekall._utils.remove_indices(iterable, ignored_indices)

Filter the given iterable by removing listed in ignored_indices.

exekall._utils.get_module_basename(path)

Get the module name of the module defined in source path.

exekall._utils.iterate_cb(iterator, pre_hook=None, post_hook=None)

Iterate over iterator, and call some callbacks.

Parameters:

• pre_hook (collections.abc.Callable) – Callback called right before getting a new value.

• post_hook (collections.abc.Callable) – Callback called right after getting a new value.

exekall._utils.format_exception(e)

Format the traceback of the exception e in a string.

exekall._utils.LOGGING_OUT_LEVEL = 60

Log level used for the OUT level.

This allows sending all the output through the logging module instead of using print(), so it can easily be recorded to a file

class exekall._utils.ExekallFormatter(fmt, *args, **kwargs)

Bases: Formatter

Custom logging.Formatter that takes care of OUT level.

This OUT level allows using logging instead of print() so it can be redirected to a file easily.

Initialize the formatter with specified format strings.

Initialize the formatter either with the specified format string, or a default as described above. Allow for specialized date formatting with the optional datefmt argument. If datefmt is omitted, you get an ISO8601-like (or RFC 3339-like) format.

Use a style parameter of ‘%’, ‘{’ or ‘$’ to specify that you want to use one of %-formatting, str.format() ({}) formatting or string.Template formatting in your format string.

Changed in version 3.2: Added the style parameter.

format(record)

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

exekall._utils.setup_logging(log_level, debug_log_file=None, info_log_file=None, verbose=0)

Setup the logging module.

Parameters:

• log_level (str) – Lowest log level name to display.

• debug_log_file (str) – Path to a file where logs are collected at the DEBUG level.

• info_log_file (str) – Path to a file where logs are collected at the INFO level.

• verbose (int) – Verbosity level. The format string for log entries will contain more information when the level increases.`

exekall._utils.out(msg)

To be used as a replacement of print().

This allows easy redirection of the output to a file.

exekall._utils.info(msg)

Write a log message at the INFO level.

exekall._utils.debug(msg)

Write a log message at the DEBUG level.

exekall._utils.warn(msg)

Write a log message at the WARNING level.

exekall._utils.error(msg)

Write a log message at the ERROR level.

exekall._utils.infer_mod_name(python_src, package_roots=None, excep_handler=None)

Compute the module name of a Python source file by inferring its top-level package

exekall._utils.find_customization_module_set(module_set)

Find all customization modules, where subclasses of exekall.customization.AdaptorBase are expected to be found.

It looks for modules named exekall_customize present in any enclosing package of modules in module_set.

exekall._utils.import_modules(paths_or_names, excep_handler=None)

Import the modules in the given list of paths.

If a folder is passed, all Python sources are recursively imported.

exekall._utils.import_name_recursively(name, excep_handler=None)

Import a module by its name.

Parameters:

name (str) – Full name of the module.

If it’s a package, import all submodules recursively.

exekall._utils.import_folder(path, package_roots=None, excep_handler=None)

Import all modules contained in the given folder, recurisvely.

exekall._utils.import_file(python_src, *args, excep_handler=None, **kwargs)

Import a module.

Parameters:

• python_src (str or pathlib.Path) – Path to a Python source file.

• module_name (str) – Name under which to import the module. If None, the name is inferred using infer_mod_name()

• package_roots (list(str)) – Paths to the root of the package, used by infer_mod_name(). A namespace package can have multiple roots.

• is_package (bool) – True if the module is a package. If a folder or __init__.py is passed, this is forcefully set to True.

exekall._utils.flatten_seq(seq, levels=1)

Flatten a nested sequence, up to levels levels.

exekall._utils.take_first(iterable)

Pick the first item of iterable.

exekall._utils.NoValue = NoValue

Singleton with similar purposes as None.

class exekall._utils.RestartableIter(it)

Bases: object

Wrap an iterator to give a new iterator that is restartable.

exekall._utils.get_froz_val_set_set(db, uuid_seq=None, type_pattern_seq=None)

Get a set of sets of exekall.engine.FrozenExprVal.

Parameters:

• db (exekall.engine.ValueDB) – exekall.engine.ValueDB to look into

• uuid_seq (list(str)) – Sequence of UUIDs to select.

• type_pattern_seq (list(str)) – Sequence of fnmatch.fnmatch() patterns matching type names (including module name).

exekall._utils.match_base_cls(cls, pattern_list)

Match the name of the class of the object and all its base classes.

exekall._utils.match_name(name, pattern_list)

Return True if name is matched by any pattern in pattern_list.

If a pattern starts with !, it is taken as a negative pattern.

exekall._utils.get_common_base(cls_list)

Get the most derived common base class of classes in cls_list.

exekall._utils.get_subclasses(cls)

Get all the (direct and indirect) subclasses of cls.

exekall._utils.get_package(module)

Find the package name of a module. If the module has no package, its own name is taken.

exekall._utils.get_recursive_module_set(module_set, package_set, visited_module_set=None)

Retrieve the set of all modules recursively imported from the modules in module_set if they are (indirectly) part of one of the packages named in package_set.

exekall._utils.disable_gc()

Context manager to disable garbage collection.

This can result in significant speed-up in code creating a lot of objects, like pickle.load().

exekall._utils.render_graphviz(expr)

Render the structure of an expression as a graphviz description or SVG.

Returns:

A tuple(bool, content) where the boolean is True if SVG could be rendered or False if it still a graphviz description.

Parameters:

expr (exekall.engine.ExpressionBase) – Expression to render

exekall._utils.add_argument(parser, *args, help, **kwargs)

Equivalent to argparse.ArgumentParser.add_argument(), with help formatting.

This allows using parsers setup using raw formatters.

exekall._utils.create_adaptor_parser_group(parser, adaptor_cls)

exekall._utils.powerset(iterable)

Powerset of the given iterable ::

powerset([1,2,3]) –> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)

exekall._utils.measure_time(iterator)

Measure how long it took to yield each value of the given iterator.

exekall._utils.capture_log(iterator)

class exekall._utils.OrderedSetBase(items=[])

Bases: object

Base class for custom ordered sets.

class exekall._utils.FrozenOrderedSet(*args, **kwargs)

Bases: OrderedSetBase, Set

Like a regular frozenset, but iterating over it will yield items in insertion order.

__hash__()

Return hash(self).

class exekall._utils.OrderedSet(items=[])

Bases: OrderedSetBase, MutableSet

Like a regular set, but iterating over it will yield items in insertion order.

add(item)

Add an element.

update(*sets)

discard(item)

Remove an element. Do not raise an exception if absent.

exekall._utils.utc_datetime()

Return a UTC datetime.datetime.