Kernel testing

Introduction

The LISA kernel tests are mostly meant for regression testing, or for supporting new submissions to LKML. The existing infrastructure can also be used to hack up workloads that would allow you to poke specific areas of the kernel.

Tests do not have to target Arm platforms nor the task scheduler. The only real requirement is to be able to abstract your target through devlib, and from there you are free to implement tests as you see fit.

They are commonly split into two steps:

1. Collect some data by doing work on the target

2. Post-process the collected data

In our case, the data usually consists of Ftrace traces that we then parse into pandas.DataFrame.

See also

Kernel trace analysis

Available tests

The following tests are available. They can be used as:

• direct execution using lisa-test command (LISA shell) and exekall (see Automated testing)

• the individual classes/methods they are composed of can be used in custom scripts/jupyter notebooks (see ipynb/tests/synthetics_example.ipynb)

Note

LISA used to ship a lisa_tests package containing a number of scheduler-related kernel tests. As these tests are not maintained anymore, they have been removed. However, the infrastructure to write such tests still exists and is part of the public lisa API.

Running tests

From the CLI

The shortest path to executing a test from a shell is:

1. Update the target_conf.yml file located at the root of the repository with the credentials to connect to the development board (see TargetConf keys for more information)

2. Run the following:

# To run all tests
lisa-test
# To list available tests
lisa-test --list
# To run a test matching a pattern
lisa-test '*test_task_placement'

# For reStructuredText to have its bullet point closing blank line
printf "\n"

More advanced workflows are described at Automated testing.

From a python environment

See the usage example of TestBundle

Writing tests

Concepts

Writing scheduler tests can be tricky, especially when you’re trying to make them work without relying on custom tracepoints (which is what you should aim for). Sometimes, a good chunk of the test code will be about trying to get the target in an expected initial state, or preventing some undesired mechanic from barging in. That’s why we rely on the freezer cgroup to reduce the amount of noise introduced by the userspace, but it’s not solving all of the issues. As such, your tests should be designed to:

1. minimize the amount of non test-related noise (e.g. freezer)

2. withstand events we can’t control (use error margins, averages…)

The main class of the kernel tests is TestBundle. Have a look at its documentation for implementation and usage examples.

The relationship between the test classes has been condensed into this diagram, although you’ll find more details in the API documentation of these classes.

Implementations of _from_target can execute any sort of arbitry Python code. This means that you are free to manipulate sysfs entries, or to execute arbitray binaries on the target. The Workload class has been created to facilitate the execution of commands/binaries on the target.

An important daughter class of Workload is RTA, as it facilitates the creation and execution of rt-app workloads. It is very useful for scheduler-related tests, as it makes it easy to create tasks with a pre-determined utilization.

Example

Here is a commented example of an rt-app-based test, showcasing the APIs that are commonly used to write such tests.

It can be executed using:

exekall run lisa lisa_tests.test_example --conf $LISA_CONF

API

Base classes

class lisa.tests.base.TestMetric(data, units=None)

Bases: object

A storage class for metrics used by tests

Parameters:

• data – The data to store. Can be any base type or dict(TestMetric)

• units (str) – The data units

pretty_format(multiline=True)

Pretty print the metrics.

Parameters:

multiline (bool) – If True, use a multiline format.

class lisa.tests.base.Result(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: Enum

A classification of a test result

PASSED = 1

The test has passed

FAILED = 2

The test has failed

UNDECIDED = 3

The test data could not be used to decide between PASSED or FAILED

SKIPPED = 4

The test does not make sense on this platform and should therefore be skipped.

Note

UNDECIDED should be used when the data are inconclusive but the test still makes sense on the target.

property lower_name

Return the name in lower case

exception lisa.tests.base.ResultBundleBase

Bases: Exception

Base class for all result bundles.

Note

__init__ is not provided as some classes uses properties to provide some of the attributes.

__bool__()

True if the result is Result.PASSED, False otherwise.

pretty_format(multiline=True)

add_metric(name, data, units=None)

Lets you append several test TestMetric to the bundle.

Parameters:

TestMetric parameters

display_and_exit() → None

exception lisa.tests.base.ResultBundle(result, utc_datetime=None, context=None)

Bases: ResultBundleBase

Bundle for storing test results

Parameters:

• result (Result) – Indicates whether the associated test passed. It will also be used as the truth-value of a ResultBundle.

• utc_datetime (datetime.datetime) – UTC time at which the result was collected, or None to record the current datetime.

• context (dict(str, object)) – Contextual information to attach to the bundle. Keep the content small, as size of ResultBundle instances matters a lot for storing long test sessions results.

TestMetric can be added to an instance of this class. This can make it easier for users of your tests to understand why a certain test passed or failed. For instance:

def test_is_noon():
    now = time.localtime().tm_hour
    res = ResultBundle(Result.PASSED if now == 12 else Result.FAILED)
    res.add_metric("current time", now)

    return res

>>> res_bundle = test_is_noon()
>>> print(res_bundle.result.name)
FAILED

# At this point, the user can wonder why the test failed.
# Metrics are here to help, and are printed along with the result:
>>> print(res_bundle)
FAILED: current time=11

classmethod from_bool(cond, *args, **kwargs)

Alternate constructor where ResultBundle.result is determined from a bool

classmethod raise_skip(msg, from_=None, **kwargs)

Raise an ResultBundle with the Result.SKIPPED result, thereby short-circuiting the rest of the test.

Parameters:

• msg (str) – Reason why the test is skipped

• from (Exception or None) – Other exception that lead to the test being skipped. It will be used as the Y in raise X from Y.

This is typically used as a way to bail out while indicating to the user that the test has essentially been skipped because the target does not support what the test is testing.

exception lisa.tests.base.AggregatedResultBundle(result_bundles, name_metric=None, result=None, context=None)

Bases: ResultBundleBase

Aggregates many ResultBundle into one.

Parameters:

• result_bundles (list(ResultBundle)) – List of ResultBundle to aggregate.

• name_metric (str) – Metric to use as the “name” of each result bundle. The value of that metric will be used as top-level key in the aggregated metrics. If not provided, the index in the result_bundles list will be used.

• result (Result) – Optionally, force the self.result attribute to that value. This is useful when the way of combining the result bundles is not the default one, without having to make a whole new subclass.

• context (dict(str, object)) – Contextual information to attach to the bundle. Keep the content small, as size of ResultBundle instances matters a lot for storing long test sessions results.

This is useful for some tests that are naturally decomposed in subtests.

Note

Metrics of aggregated bundles will always be shown, but can be augmented with new metrics using the usual API.

property utc_datetime

Use the earliest utc_datetime among the aggregated bundles.

property context

Merge the context of all the aggregated bundles, with priority given to last in the list.

property result

property metrics

class lisa.tests.base.TestBundleMeta(cls_name, bases, dct, **kwargs)

Bases: ABCMeta

Metaclass of TestBundleBase.

Method with a return annotation of ResultBundleBase are wrapped to:

• Update the context attribute of a returned ResultBundleBase

• Add an undecided_filter attribute, with add_undecided_filter() decorator, so that any test method can be used as a pre-filter for another one right away.

• Wrap _from_target to provide a single collector parameter, built from the composition of the collectors provided by _make_collector methods in the base class tree.

If _from_target is defined in the class but from_target is not, a stub is created and the annotation of _from_target is copied to the stub. The annotation is then removed from _from_target so that it is not picked up by exekall.

The signature of from_target is the result of merging the original cls.from_target parameters with the ones defined in _from_target.

Test methods:

• test_method()

classmethod test_method(func)

Decorator to intercept returned ResultBundle and attach some contextual information.

classmethod collector_factory(f)

static add_undecided_filter(func)

Turn any method returning a ResultBundleBase into a decorator that can be used as a test method filter.

The filter decorator is accessible as the undecided_filter attribute of the decorated method.

Once a test is decorated, the filter method will be run in addition to the wrapped test, and if the filter does not succeed, the ResultBundleBase result will be set to Result.UNDECIDED.

Example:

class Foo(TestBundle):
    @TestBundle.add_undecided_filter
    def test_foo(self, xxx=42, ...):
        ...

    # Alternatively, ResultBundle return annotation will
    # automatically decorate the method with TestBundleMeta
    # metaclass.
    def test_foo(self, xxx=42, ...) -> ResultBundle:
        ...

class Bar(Foo):
    # Set xxx=55 as default, but this can be overriden when
    # test_bar() is called.
    @Foo.test_foo.undecided_filter(xxx=77)
    def test_bar(self, yyy=43, ...) -> ResultBundle:
        ...

The resulting decorated method can take the union of keyword parameters:

bar = Bar()
bar.test_bar(xxx=33, yyy=55)
# Same as
bar.test_bar(33, yyy=55)
# But this fails, since only keyword arguments can be passed to the
# wrapping pre-test
bar.test_bar(33, 55)

If there is a parameter conflict, it is detected at import time and will result in a TypeError.

Note

Even if the pre-test does not succeed, the wrapped test is still executed, so that the ResultBundle metrics are updated and the artifacts still produced. This can be important in order to manually analyse results in case the pre-filter was overly conservative and marked a usable result as UNDECIDED.

classmethod __prepare__() → dict

used to create the namespace for the class statement

static __new__(metacls, cls_name, bases, dct, **kwargs)

class lisa.tests.base.TestBundleBase(res_dir, plat_info)

Bases: Serializable, ExekallTaggable, ABC, _DocstringAppend

A LISA test bundle.

Parameters:

• res_dir (str) – Directory in which the target execution artifacts reside. This will also be used to dump any artifact generated in the test code.

• plat_info (lisa.platforms.platinfo.PlatformInfo) – Various informations about the platform, that is available to all tests.

The point of a TestBundleBase is to bundle in a single object all of the required data to run some test assertion (hence the name). When inheriting from this class, you can define test methods that use this data, and return a ResultBundle.

Thanks to Serializable, instances of this class can be serialized with minimal effort. As long as some information is stored within an object’s member, it will be automagically handled.

Please refrain from monkey-patching the object in from_target(). Data required by the object to run test assertions should be exposed as __init__ parameters.

Note

All subclasses are considered as “application” code, as opposed to most of the rest of lisa which is treated as a library. This means that the classes and their API is subject to change when needs evolve, which is not always backward compatible. It’s rarely an issue since these classes are used “manually” mostly for debugging, which is a version-specific activity. Likewise, the set of tests will evolve as existing tests are replaced by more general implementations, that could be organized and named differently.

Design notes:

• from_target() will collect whatever artifacts are required from a given target, and will then return a TestBundleBase. Note that a default implementation is provided out of _from_target.

• from_dir() will use whatever artifacts are available in a given directory (which should have been created by an earlier call to from_target() and then to_dir()), and will then return a TestBundleBase.

• VERIFY_SERIALIZATION is there to ensure the instances can serialized and deserialized without error.

• res_dir parameter of __init__ must be stored as an attribute without further processing, in order to support result directory relocation.

• Test methods should have a return annotation for the ResultBundle to be picked up by the test runners.

Implementation example:

from lisa.target import Target
from lisa.platforms.platinfo import PlatformInfo
from lisa.utils import ArtifactPath

class DummyTestBundle(TestBundle):

    def __init__(self, res_dir, plat_info, shell_output):
        super().__init__(res_dir, plat_info)

        self.shell_output = shell_output

    @classmethod
    def _from_target(cls, target:Target, *, res_dir:ArtifactPath) -> 'DummyTestBundle':
        output = target.execute('echo $((21+21))').split()
        return cls(res_dir, target.plat_info, output)

    def test_output(self) -> ResultBundle:
        return ResultBundle.from_bool(
            any(
                '42' in line
                for line in self.shell_output
            )
        )

Usage example:

# Creating a Bundle from a live target
bundle = TestBundle.from_target(target, plat_info=plat_info, res_dir="/my/res/dir")
# Running some test on the bundle
res_bundle = bundle.test_foo()

# Saving the bundle on the disk
bundle.to_dir("/my/res/dir")

# Reloading the bundle from the disk
bundle = TestBundle.from_dir("/my/res/dir")
# The reloaded object can be used just like the original one.
# Keep in mind that serializing/deserializing this way will have a
# similar effect than a deepcopy.
res_bundle = bundle.test_foo()

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

Note

As a subclass of lisa.tests.base.TestBundleBase, this class is considered as “application” and its API is therefore more subject to change than other parts of lisa.

VERIFY_SERIALIZATION = True

When True, this enforces a serialization/deserialization step in from_target().

Note

The deserialized instance is thrown away in order to avoid using what is in effect a deepcopy of the original bundle. Using that deepcopy greatly increases the memory consumption of long running processes.

get_tags()

Returns:

Dictionary of tags and tag values

Return type:

dict(str, object)

abstract classmethod _from_target(target, *, res_dir)

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

classmethod check_from_target(target)

Check whether the given target can be used to create an instance of this class

Raises:

lisa.tests.base.ResultBundleBase with result as lisa.tests.base.Result.SKIPPED if the check fails

This method should be overriden to check your implementation requirements

classmethod can_create_from_target(target)

Returns:

Whether the given target can be used to create an instance of this class

Return type:

bool

check_from_target() is used internally, so there shouldn’t be any

need to override this.

classmethod from_target(target: Target, *, res_dir: ArtifactPath = None, **kwargs)

Factory method to create a bundle using a live target

Parameters:

• target (lisa.target.Target) – Target to connect to.

• res_dir (str or lisa.utils.ArtifactPath) – Host result directory holding artifacts.

• custom_collector (lisa.trace.CollectorBase) – Custom collector that will be used as a context manager when calling the workload.

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundleBase._from_target() instead.

classmethod _get_filepath(res_dir)

Returns the path of the file containing the serialized object in res_dir folder.

property _children_test_bundles

List of references to TestBundleBase instances self relies on (directly and indirectly).

This is used for some post-deserialization fixup that need to walk the whole graph of TestBundleBase.

classmethod from_dir(res_dir, update_res_dir=True)

Wrapper around lisa.utils.Serializable.from_path().

It uses _get_filepath() to get the name of the serialized file to reload.

to_dir(res_dir)

See lisa.utils.Serializable.to_path()

class lisa.tests.base.FtraceTestBundleBase(res_dir, plat_info)

Bases: TestBundleBase

Base class for test bundles needing ftrace traces.

Optionally, an FTRACE_CONF class attribute can be defined to hold additional FTrace configuration used to record a trace while the synthetic workload is being run. By default, the required events are extracted from decorated test methods.

This base class ensures that each subclass will get its own copy of FTRACE_CONF attribute, and that the events specified in that configuration are a superset of what is needed by methods using the family of decorators lisa.trace.requires_events(). This makes sure that the default set of events is always enough to run all defined methods, without duplicating that information. That means that trace events are “inherited” at the same time as the methods that need them.

The FTRACE_CONF attribute is typically built by merging these sources:

• Existing FTRACE_CONF class attribute on the RTATestBundle subclass

• Events required by methods using lisa.trace.requires_events() decorator (and equivalents).

• lisa.trace.FtraceConf specified by the user and passed to lisa.tests.base.TestBundleBase.from_target() as ftrace_conf parameter.

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

Note

As a subclass of lisa.tests.base.TestBundleBase, this class is considered as “application” and its API is therefore more subject to change than other parts of lisa.

TRACE_PATH = 'trace.dat'

Path to the trace-cmd trace.dat file in the result directory.

classmethod __init_subclass__(**kwargs)

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

property trace_path

Path to the trace-cmd report trace.dat file.

property trace

Returns:

a lisa.trace._TraceView

All events specified in FTRACE_CONF are parsed from the trace, so it is suitable for direct use in methods.

Having the trace as a property lets us defer the loading of the actual trace to when it is first used. Also, this prevents it from being serialized when calling lisa.utils.Serializable.to_path() and allows updating the underlying path before it is actually loaded to match a different folder structure.

get_trace(events=None, **kwargs)

Returns:

a lisa.trace.Trace collected in the standard location.

Variable keyword arguments:

Forwarded to lisa.trace.Trace.

class lisa.tests.base.FtraceTestBundle(res_dir, plat_info)

Bases: FtraceTestBundleBase

Dummy subclass of FtraceTestBundleBase to be inherited from to override OptionalFtraceTestBundle in the inheritance tree.

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

Note

As a subclass of lisa.tests.base.TestBundleBase, this class is considered as “application” and its API is therefore more subject to change than other parts of lisa.

FTRACE_CONF = <lisa.trace.FtraceConf object>

ftrace_conf = <lisa.trace.FtraceConf object>

class lisa.tests.base.OptionalFtraceTestBundle(res_dir, plat_info)

Bases: FtraceTestBundleBase, Loggable

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

Note

As a subclass of lisa.tests.base.TestBundleBase, this class is considered as “application” and its API is therefore more subject to change than other parts of lisa.

FTRACE_CONF = <lisa.trace.FtraceConf object>

ftrace_conf = <lisa.trace.FtraceConf object>

class lisa.tests.base.TestConfBase(conf=None, src='user', add_default_src=True)

Bases: SimpleMultiSrcConf

Base class for test configurations.

This class will add a test-conf top-level key above the level specified by the class, so that if the class specifies a TopLevelKeyDesc('foo'), the actual top-level key will be test-conf/foo.

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

classmethod __init_subclass__(**kwargs)

This method is called when a class is subclassed.

The default implementation does nothing. It may be overridden to extend subclasses.

class lisa.tests.base.DmesgTestConf(conf=None, src='user', add_default_src=True)

Bases: TestConfBase

Configuration class for lisa.tests.base.DmesgTestBundle.test_dmesg().

• dmesg: Dmesg test configuration

• ignored-patterns (Sequence): List of Python regex matching dmesg entries content to be ignored (see devlib.collector.dmesg.KernelLogEntry for how the message is split).

Example YAML:

# Dmesg test configuration
test-conf:
    dmesg:

        # List of Python regex matching dmesg entries *content* to be
        # ignored (see :class:`devlib.collector.dmesg.KernelLogEntry`
        # for how the message is split)
        # type: Sequence
        ignored-patterns: _

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

STRUCTURE = <lisa.conf.TopLevelKeyDesc object>

class IgnoredPatterns

Bases: HideExekallID

List of Python regex matching dmesg entries content to be ignored (see devlib.collector.dmesg.KernelLogEntry for how the message is split)

class lisa.tests.base.DmesgTestBundleBase(res_dir, plat_info)

Bases: TestBundleBase

Abstract Base Class for TestBundles based on dmesg output.

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

Note

As a subclass of lisa.tests.base.TestBundleBase, this class is considered as “application” and its API is therefore more subject to change than other parts of lisa.

Test methods:

• test_dmesg()

DMESG_PATH = 'dmesg.log'

Path to the dmesg log in the result directory.

CANNED_DMESG_IGNORED_PATTERNS = {'EAS-schedutil': 'Disabling EAS, schedutil is mandatory', 'executable-stack': 'started with executable stack'}

Mapping of canned patterns to avoid repetition while defining lisa.tests.base.DmesgTestBundleBase.DMESG_IGNORED_PATTERNS in subclasses.

DMESG_IGNORED_PATTERNS = ['started with executable stack']

List of patterns to ignore in addition to the ones passed to test_dmesg().

property dmesg_path

Path to the dmesg output log file

property dmesg_entries

List of parsed dmesg output entries devlib.collector.dmesg.KernelLogEntry.

test_dmesg(level='warn', facility=None, ignored_patterns: IgnoredPatterns = None) → ResultBundle

Basic test on kernel dmesg output.

Parameters:

• level (str) – Any dmesg entr with a level more critical than (and including) that will make the test fail.

• facility (str or None) – Only select entries emitted by the given dmesg facility like kern. Note that not all versions of dmesg are able to print it, so specifying it may lead to no entry being inspected at all. If None, the facility is ignored.

• ignored_patterns (list or None) – List of regexes to ignore some messages. The pattern list is combined with DMESG_IGNORED_PATTERNS class attribute.

class lisa.tests.base.DmesgTestBundle(res_dir, plat_info)

Bases: DmesgTestBundleBase

Dummy subclass of DmesgTestBundleBase to be inherited from to override OptionalDmesgTestBundle in the inheritance tree.

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

Note

As a subclass of lisa.tests.base.TestBundleBase, this class is considered as “application” and its API is therefore more subject to change than other parts of lisa.

Test methods:

• test_dmesg()

test_dmesg(level='warn', facility=None, ignored_patterns: IgnoredPatterns = None) → ResultBundle

Basic test on kernel dmesg output.

Parameters:

• level (str) – Any dmesg entr with a level more critical than (and including) that will make the test fail.

• facility (str or None) – Only select entries emitted by the given dmesg facility like kern. Note that not all versions of dmesg are able to print it, so specifying it may lead to no entry being inspected at all. If None, the facility is ignored.

• ignored_patterns (list or None) – List of regexes to ignore some messages. The pattern list is combined with DMESG_IGNORED_PATTERNS class attribute.

class lisa.tests.base.OptionalDmesgTestBundle(res_dir, plat_info)

Bases: DmesgTestBundleBase, Loggable

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

Note

As a subclass of lisa.tests.base.TestBundleBase, this class is considered as “application” and its API is therefore more subject to change than other parts of lisa.

Test methods:

• test_dmesg()

test_dmesg(level='warn', facility=None, ignored_patterns: IgnoredPatterns = None) → ResultBundle

Basic test on kernel dmesg output.

Parameters:

• level (str) – Any dmesg entr with a level more critical than (and including) that will make the test fail.

• facility (str or None) – Only select entries emitted by the given dmesg facility like kern. Note that not all versions of dmesg are able to print it, so specifying it may lead to no entry being inspected at all. If None, the facility is ignored.

• ignored_patterns (list or None) – List of regexes to ignore some messages. The pattern list is combined with DMESG_IGNORED_PATTERNS class attribute.

class lisa.tests.base.RTATestBundle(res_dir, plat_info, rtapp_profile_kwargs=None)

Bases: FtraceTestBundle, DmesgTestBundle

Abstract Base Class for lisa.wlgen.rta.RTA-powered TestBundles

Parameters:

rtapp_profile_kwargs (collections.abc.Mapping or None) – Keyword arguments to pass to lisa.tests.base.RTATestBundle._get_rtapp_profile() when called from the lisa.tests.base.RTATestBundle._get_rtapp_profile() property.

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

Note

As a subclass of lisa.tests.base.TestBundleBase, this class is considered as “application” and its API is therefore more subject to change than other parts of lisa.

Test methods:

• test_dmesg()

• test_noisy_tasks()

TASK_PERIOD = 0.016

A task period in seconds you can re-use for your lisa.wlgen.rta.RTATask definitions.

NOISE_ACCOUNTING_THRESHOLDS = {'^irq/\\d+-.*$': 1.5, '^sugov:\\d+$': 5, (0, None): 100}

PID/comm specific tuning for test_noisy_tasks()

• keys can be PIDs, comms, or regexps for comms.

• values are noisiness thresholds (%), IOW below that runtime threshold the associated task will be ignored in the noise accounting.

trace_window(trace)

The time window to consider for this RTATestBundle

Returns:

a (start, stop) tuple

Since we’re using rt-app profiles, we know the name of tasks we are interested in, so we can trim our trace scope to filter out the setup/teardown events we don’t care about.

Override this method if you need a different trace trimming.

Warning

Calling self.trace here will raise an AttributeError exception, to avoid entering infinite recursion.

Required trace events:

• userspace@rtapp_loop

• sched_switch

property rtapp_profile

Compute the RTapp profile based on plat_info.

property rtapp_task_ids_map

Mapping of task names as specified in the rtapp profile to list of lisa.analysis.tasks.TaskID names found in the trace.

If the task forked, the list will contain more than one item.

Required trace events:

• sched_switch

property rtapp_task_ids

The rtapp task lisa.analysis.tasks.TaskID as found from the trace in this bundle.

Returns:

the list of actual trace task lisa.analysis.tasks.TaskID

Required trace events:

• sched_switch

property rtapp_tasks_map

Same as rtapp_task_ids_map() but with list of strings for values.

Required trace events:

• sched_switch

property rtapp_tasks

Same as rtapp_task_ids() but as a list of string.

Returns:

the list of actual trace task names

Required trace events:

• sched_switch

property cgroup_configuration

Compute the cgroup configuration based on plat_info

FTRACE_CONF = <lisa.trace.FtraceConf object>

classmethod from_target(target: Target, *, res_dir: ArtifactPath = None, custom_collector=None, ftrace_conf: FtraceConf = None) → RTATestBundle

Factory method to create a bundle using a live target

Parameters:

• target (lisa.target.Target) – Target to connect to.

• res_dir (str or lisa.utils.ArtifactPath) – Host result directory holding artifacts.

• custom_collector (lisa.trace.CollectorBase) – Custom collector that will be used as a context manager when calling the workload.

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundleBase._from_target() instead.

ftrace_conf = <lisa.trace.FtraceConf object>

property trace

Returns:

a lisa.trace._TraceView cropped to the window given by trace_window().

See also

FtraceTestBundleBase.trace

df_noisy_tasks(with_threshold_exclusion=True)

Returns:

a DataFrame containing all tasks that participate to the test noise. i.e. all non rt-app tasks.

Parameters:

with_threshold_exclusion – When set to True, known noisy services will be ignored.

test_noisy_tasks(*, noise_threshold_pct=None, noise_threshold_ms=None)

Test that no non-rtapp (“noisy”) task ran for longer than the specified thresholds

Parameters:

• noise_threshold_pct (float) – The maximum allowed runtime for noisy tasks in percentage of the total rt-app execution time

• noise_threshold_ms (float) – The maximum allowed runtime for noisy tasks in ms

If both are specified, the smallest threshold (in seconds) will be used.

Required trace events:

• sched_switch

• sched_wakeup

• one group of: task_rename

• optional: sched_wakeup_new

classmethod unscaled_utilization(plat_info, cpu, utilization_pct)

Convert utilization scaled to a CPU to a ‘raw’, unscaled one.

Parameters:

• capacity (int) – The CPU against which utilization_pct` is scaled

• utilization_pct (int) – The scaled utilization in %

classmethod get_rtapp_profile(plat_info, **kwargs)

Returns a dict with task names as keys and lisa.wlgen.rta.RTATask as values.

The following modifications are done on the profile returned by _get_rtapp_profile():

• A buffer phase may be inserted at the beginning of each task in order to stabilize some kernel signals.

• A from_test meta key is added to each lisa.wlgen.rta.RTAPhase with a boolean value that is True if the phase comes from the test itself and False if it was added here (e.g. the buffer phase). This allows future-proof filtering of phases in the test code when inspecting the profile by looking at phase['meta']['from_test'].

Note

If you want to override the method in a subclass, override _get_rtapp_profile() instead.

abstract classmethod _get_rtapp_profile(plat_info)

Returns:

a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

classmethod get_cgroup_configuration(plat_info)

Returns:

a dict representing the configuration of a particular cgroup.

This is a method you may optionally override to configure a cgroup for the synthetic workload.

Example of return value:

{
    'name': 'lisa_test',
    'controller': 'schedtune',
    'attributes' : {
        'prefer_idle' : 1,
        'boost': 50
    }
}

classmethod run_rtapp(target, res_dir, profile=None, collector=None, cg_cfg=None, wipe_run_dir=True, update_cpu_capacities=None)

Run the given RTA profile on the target, and collect an ftrace trace.

Parameters:

• target (lisa.target.Target) – target to execute the workload on.

• res_dir (str or lisa.utils.ArtifactPath) – Artifact folder where the artifacts will be stored.

• profile (dict(str, lisa.wlgen.rta.RTATask)) – rt-app profile, as a dictionary of dict(task_name, RTATask). If None, get_rtapp_profile() is called with target.plat_info.

• collector (lisa.trace.ComposedCollector) – Context manager collector to use while running rt-app.

• cg_cfg (dict) – CGroup configuration dictionary. If None, lisa.tests.base.RTATestBundle.get_cgroup_configuration() is called with target.plat_info.

• wipe_run_dir (bool) – Remove the run directory on the target after execution of the workload.

• update_cpu_capacities (bool) – Attempt to update the CPU capacities based on the calibration values of rtapp to get the most accurate reproduction of duty cycles.

classmethod _run_rtapp(*args, **kwargs)

Has been renamed to run_rtapp(), as it really is part of the public API.

classmethod _from_target(target, *, custom_collector=None, ftrace_conf: FtraceConf = None, res_dir: ArtifactPath = None) → RTATestBundle

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

class lisa.tests.base.TestBundle(res_dir, plat_info)

Bases: OptionalFtraceTestBundle, OptionalDmesgTestBundle, TestBundleBase

Dummy class used as a base class for all tests.

Warning

Arbitrary code can be executed while loading an instance from a YAML or Pickle file. To include untrusted data in YAML, use the !untrusted tag along with a string

Note

As a subclass of lisa.tests.base.TestBundleBase, this class is considered as “application” and its API is therefore more subject to change than other parts of lisa.

Test methods:

• test_dmesg()

FTRACE_CONF = <lisa.trace.FtraceConf object>

ftrace_conf = <lisa.trace.FtraceConf object>

classmethod check_from_target(target)

Check whether the given target can be used to create an instance of this class

Raises:

lisa.tests.base.ResultBundleBase with result as lisa.tests.base.Result.SKIPPED if the check fails

This method should be overriden to check your implementation requirements