Kernel testing

Introduction

The LISA kernel tests are mostly meant for regression testing, or for supporting new submissions to LKML. The existing infrastructure can also be used to hack up workloads that would allow you to poke specific areas of the kernel.

Tests do not have to target Arm platforms nor the task scheduler. The only real requirement is to be able to abstract your target through devlib, and from there you are free to implement tests as you see fit.

They are commonly split into two steps:
  1. Collect some data by doing work on the target
  2. Post-process the collected data

In our case, the data usually consists of Ftrace traces that we then parse into pandas.DataFrame.

Available tests

The following tests are available. They can be used as:

  • direct execution using lisa-test command (LISA shell) and exekall (see Automated testing)
  • the individual classes/methods they are composed of can be used in custom scripts/jupyter notebooks (see ipynb/tests/synthetics_example.ipynb)

Running tests

From the CLI

The shortest path to executing a test from a shell is:

  1. Update the target_conf.yml file located at the root of the repository with the credentials to connect to the development board (see TargetConf keys for more information)
  2. Run the following:
# To run all tests
lisa-test
# To list available tests
lisa-test --list
# To run a test matching a pattern
lisa-test '*test_task_placement'

More advanced workflows are described at Automated testing.

From a python environment

See the usage example of TestBundle

Writing tests

Concepts

Writing scheduler tests can be tricky, especially when you’re trying to make them work without relying on custom tracepoints (which is what you should aim for). Sometimes, a good chunk of the test code will be about trying to get the target in an expected initial state, or preventing some undesired mechanic from barging in. That’s why we rely on the freezer cgroup to reduce the amount of noise introduced by the userspace, but it’s not solving all of the issues. As such, your tests should be designed to:

  1. minimize the amount of non test-related noise (e.g. freezer)
  2. withstand events we can’t control (use error margins, averages…)

The main class of the kernel tests is TestBundle. Have a look at its documentation for implementation and usage examples.

The relationship between the test classes has been condensed into this diagram, although you’ll find more details in the API documentation of these classes.

class TestMetric {
      + data
      + units
}


note bottom of TestMetric {
     TestMetrics serve to answer
     <b>"Why did my test fail/pass ?"</b>.
     They are free-form, so they can be
     error counts, durations, stats...
}

class Result {
      PASSED
      FAILED
      UNDECIDED
}

class ResultBundle {
      + result : Result
      + add_metric()
}

ResultBundle "1" *- "1" Result
' This forces a longer arrow ------------v
ResultBundle "1" *- "1..*" TestMetric : "          "

class TestBundle {
    # _from_target() : TestBundle
    + from_target() : TestBundle
    + from_dir() : TestBundle
}

note right of TestBundle {
    Methods returning <b>TestBundle</b>
    are alternative constructors

    <b>from_target()</b> does some generic
    work, then calls <b>_from_target()</b>. You'll
    have to override it depending on what
    you want to execute on the target.
}

class MyTestBundle {
      # _from_target() : TestBundle
      + test_foo_is_bar() : ResultBundle
}

note right of MyTestBundle {
    Non-abstract <b>TestBundle</b> classes
    must define test methods that return
    a <b>ResultBundle</b>
}

TestBundle <|-- MyTestBundle
MyTestBundle .. ResultBundle

Implementations of _from_target can execute any sort of arbitry Python code. This means that you are free to manipulate sysfs entries, or to execute arbitray binaries on the target. The Workload class has been created to facilitate the execution of commands/binaries on the target.

An important daughter class of Workload is RTA, as it facilitates the creation and execution of rt-app workloads. It is very useful for scheduler-related tests, as it makes it easy to create tasks with a pre-determined utilization.

Example

Here is a commented example of an rt-app-based test, showcasing the APIs that are commonly used to write such tests.

It can be executed using:

exekall run lisa.test_example --conf $LISA_CONF
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
class ExampleTestBundle(RTATestBundle):
    """
    The test bundle contains the data the test will work on. See
    :class:`lisa.tests.base.TestBundle` for design notes.

    This example derives from :class:`lisa.tests.base.RTATestBundle`, so it
    gains some ``rt-app``-specific and ftrace capabilities.
    """

    task_prefix = 'exmpl'
    "Prefix used for rt-app task names"

    # res_dir and plat_info are "mandatory" parameters of all TestBundle, but
    # the other ones are specific to a given use case.
    def __init__(self, res_dir, plat_info, shell_output):
        # This must be called, don't set res_dir or plat_info yourself
        super().__init__(res_dir, plat_info)

        self.shell_output = shell_output

    @classmethod
    def _from_target(cls, target: Target, *, res_dir: ArtifactPath, ftrace_coll: FtraceCollector = None) -> 'ExampleTestBundle':
        """
        This class method is the main way of creating a :class:`ExampleTestBundle`.

        It takes a first (positional) ``target`` parameter, which is a live
        :class:`lisa.target.Target` object. It can be used to manipulate a
        remote device such as a development board, to run workloads on it,
        manipulate sysfs entries and so on.

        **All other parameters are keyword-only**
        This means they must appear after the lone ``*`` in the parameter list.

        ``res_dir`` stands for "result directory" and is a location where the
        bundle can store some artifacts collected from the target. The bundle
        can rely on that folder being populated by this method.

        The "'ExampleTestBundle'" return annotation tells the test runner that
        this class method acts as a factory of :class:`ExampleTestBundle`, so it
        will be used to assemble the test case.

        .. seealso:: The class :class:`lisa.platforms.platinfo.PlatformInfo`
            provides information about a device that are usually needed in
            tests.

        .. seealso: This methods provides an easy way of running an rt-app
            workload on the target device
            :meth:`lisa.tests.base.RTATestBundle.run_rtapp`
        """
        # PlatformInfo
        # https://lisa-linux-integrated-system-analysis.readthedocs.io/en/master/target.html#lisa.platforms.platinfo.PlatformInfo
        #
        # It's a central piece of LISA: it holds all the information about a
        # given device. Use it to access any data it contains rather than
        # fetching them yourselves, as the final user will have ways of
        # providing values in case auto-detection fails, and logging of all the
        # data it contains is provided out of the box.
        plat_info = target.plat_info

        # The rt-app profile defines the rt-app workload that will be run
        # note: If None is given to run_rtapp(), it will default to calling
        # get_rtapp_profile()
        rtapp_profile = cls.get_rtapp_profile(plat_info)

        # Here, we wanted to make sure the cpufreq governor is schedutil, since
        # that's what we want to test. This is achieved through the used of
        # devlib modules:
        # https://devlib.readthedocs.io/en/latest/modules.html
        with target.cpufreq.use_governor("schedutil"):
            # RTATestBundle.run_rtapp()
            # https://lisa-linux-integrated-system-analysis.readthedocs.io/en/master/kernel_tests.html#lisa.tests.base.RTATestBundle.run_rtapp
            #
            # It allows running the rt-app profile on the target. ftrace_coll
            # is the object used to control the recording of the trace, and is
            # setup by the test runner. This allows the final user to extend
            # the list of ftrace events collected. If no collector is provided,
            # a default one will be created by run_rtapp() based on the
            # @requires_events() decorators used on method of that
            # ExampleTestBundle. Note that it will also freeze all the tasks on
            # the target device, so that the scheduler signals are not
            # disturbed. Some critical tasks are not frozen though.
            cls.run_rtapp(target, res_dir, rtapp_profile, ftrace_coll=ftrace_coll)

        # Execute a silly shell command on the target device as well
        output = target.execute('echo $((21+21))').split()

        # Logging must be done through the provided logger, so it integrates well in LISA.
        cls.get_logger().info('Finished doing stuff')

        # Actually create a ExampleTestBundle by calling the class.
        return cls(res_dir, plat_info, output)

    @classmethod
    def get_rtapp_profile(cls, plat_info):
        """
        This class method is in charge of generating an rt-app profile, to
        configure the workload that will be run using
        :meth:`lisa.tests.base.RTATestBundle.run_rtapp`.

        It can access any information in the given
        :class:`lisa.platforms.PlatformInfo` in order to obtain a workload
        tailored to the capacity of the CPUs of the target, the available
        frequencies and so on.
        """

        # Build a list of the CPU IDs that are available
        cpus = list(range(plat_info['cpus-count']))

        # The profile is a dictionary of task names (keys) to
        # lisa.wlgen.RTATask instances
        # https://lisa-linux-integrated-system-analysis.readthedocs.io/en/master/workloads.html
        profile = {}

        for cpu in cpus:
            # Compute a utilization needed to fill 50% of ``cpu`` capacity.
            util = cls.unscaled_utilization(plat_info, cpu, 50)

            # A Periodic task has a period, and a duty_cycle (which really is a
            # target utilization). LISA will run rt-app calibration if needed
            # (it can be provided by the user in the platform information)
            profile["{}_{}".format(cls.task_prefix, cpu)] = Periodic(
                duty_cycle_pct=util,
                duration_s=1,
                period_ms=cls.TASK_PERIOD_MS,
                cpus=[cpu]
            )

        return profile

    # ftrace events necessary for that test method to run must be specified here.
    # This information will be used in a number of places:
    # * To build the ExampleTestBundle.ftrace_conf attribute, which is then used by RTATestBundle.run_rtapp()
    # * To parse the ftrace trace
    # * In the Sphinx documentation.
    # * To check that the events are available in the trace. A clear exception
    #   is raised if an even is missing.
    # Note: Other decorators can be used to express optional events or
    # alternatives, see lisa.trace module.
    @requires_events('sched_switch', 'sched_wakeup')
    # This allows referencing the @requires_events() of
    # LoadTrackingAnalysis.df_tasks_signal(), so we don't duplicate that
    # information here in case it changes in the future. Use that when you
    # don't use the events directly in your code.
    @LoadTrackingAnalysis.df_tasks_signal.used_events
    # This decorator allows checking that there was no background noise (other
    # tasks executing) while running the workload. If that was the case, the
    # returned result_bundle.result will be set to Result.UNDECIDED, expressing
    # that the data don't allow drawing a pass/fail conclusion.
    @RTATestBundle.check_noisy_tasks(noise_threshold_pct=1)
    def test_output(self, util_margin=50) -> ResultBundle:
        """
        Actual test method that looks at the collected data and draws a
        conclusion based on it.

        The return annotation "'ResultBundle'" is used by the test runner to
        assemble the test cases, since it's driven by types and what function
        can produce them.

        .. seealso:: :class:`lisa.tests.base.ResultBundle`
        """

        # Get the pandas DataFrame of tasks utilisation.
        #
        # self.trace: This is a lisa.trace.Trace object, with all the events
        # specified using @requires_events() on methods of this class. For
        # subclasses of RTATestBundle, self.trace is actually a TraceView
        # object, restricting the time range to when the rt-app tasks were
        # executing. The test methods can therefore work on minimal and
        # hopefully clean/relevant data.
        #
        # self.trace.analyis: A number of analysis objects are available,
        # giving df_* methods that return various dataframes, and plot_*
        # functions that can do various plots.
        # https://lisa-linux-integrated-system-analysis.readthedocs.io/en/master/trace_analysis.html
        df = self.trace.analysis.load_tracking.df_tasks_signal('util')

        # "resolve" the task names into (pid, comm) tuples. If there is any
        # ambiguity because of the same name is reused in different PIDs, an
        # exception will be raised.
        # self.rtapp_tasks gives the list of task names as defined in
        # get_rtapp_profile().
        task_ids = [self.trace.get_task_id(task) for task in self.rtapp_tasks]

        util_means = {}

        # Example test that checks the tasks' average utilisation is as expected
        def check_task_util(task_id):
            # Only keep the data about the tasks we care about.
            _df = df_filter_task_ids(df, [task_id])
            avg = _df['util'].mean()
            util_means[task_id.comm] = avg
            # Util is not supposed to be higher than 512 given what we asked for in get_rtapp_profile()
            return avg < (512 + util_margin)

        # Will be True if all check_task_util() calls are True
        ok = all(check_task_util(task_id) for task_id in task_ids)

        # Create a pass/fail ResultBundle.
        res_bundle = ResultBundle.from_bool(ok)

        # Named metrics (with a optional unit) can be attached to the
        # ResultBundle, and will be reported to whoever runs the test. Good
        # practice for threshold-based tests is to add one metric for the
        # computed value, and one for the threshold.
        # Extra metrics can be very helpful when doing initial investigations
        # on a test failure, so it's better to be more verbose than not.
        res_bundle.add_metric('expected util', 512)
        for task, util_mean in util_means.items():
            res_bundle.add_metric('{} util'.format(task), util_mean)

        return res_bundle

API

Base classes

lisa.tests.base._nested_formatter(multiline)
class lisa.tests.base.TestMetric(data, units=None)

Bases: object

A storage class for metrics used by tests

Parameters:
  • data – The data to store. Can be any base type or dict(TestMetric)
  • units (str) – The data units
pretty_format(multiline=True)

Pretty print the metrics.

Parameters:multiline (bool) – If True, use a multiline format.
class lisa.tests.base.Result

Bases: enum.Enum

A classification of a test result

PASSED = 1

The test has passed

FAILED = 2

The test has failed

UNDECIDED = 3

The test data could not be used to decide between PASSED or FAILED

lower_name

Return the name in lower case

class lisa.tests.base.ResultBundleBase

Bases: object

Base class for all result bundles.

Note

__init__ is not provided as some classes uses properties to provide some of the attributes.

pretty_format(multiline=True)
_repr_pretty_(p, cycle)

Pretty print instances in Jupyter notebooks

add_metric(name, data, units=None)

Lets you append several test TestMetric to the bundle.

Parameters:TestMetric parameters
display_and_exit() → None
class lisa.tests.base.ResultBundle(result, utc_datetime=None, context=None)

Bases: lisa.tests.base.ResultBundleBase

Bundle for storing test results

Parameters:
  • result (Result) – Indicates whether the associated test passed. It will also be used as the truth-value of a ResultBundle.
  • utc_datetime (datetime.datetime) – UTC time at which the result was collected, or None to record the current datetime.
  • context (dict(str, object)) – Contextual information to attach to the bundle. Keep the content small, as size of ResultBundle instances matters a lot for storing long test sessions results.

TestMetric can be added to an instance of this class. This can make it easier for users of your tests to understand why a certain test passed or failed. For instance:

def test_is_noon():
    now = time.localtime().tm_hour
    res = ResultBundle(Result.PASSED if now == 12 else Result.FAILED)
    res.add_metric("current time", now)

    return res

>>> res_bundle = test_is_noon()
>>> print(res_bundle.result.name)
FAILED

# At this point, the user can wonder why the test failed.
# Metrics are here to help, and are printed along with the result:
>>> print(res_bundle)
FAILED: current time=11
classmethod from_bool(cond, *args, **kwargs)

Alternate constructor where ResultBundle.result is determined from a bool

class lisa.tests.base.AggregatedResultBundle(result_bundles, name_metric=None, result=None, context=None)

Bases: lisa.tests.base.ResultBundleBase

Aggregates many ResultBundle into one.

Parameters:
  • result_bundles (list(ResultBundle)) – List of ResultBundle to aggregate.
  • name_metric (str) – Metric to use as the “name” of each result bundle. The value of that metric will be used as top-level key in the aggregated metrics. If not provided, the index in the result_bundles list will be used.
  • result (Result) – Optionally, force the self.result attribute to that value. This is useful when the way of combining the result bundles is not the default one, without having to make a whole new subclass.
  • context (dict(str, object)) – Contextual information to attach to the bundle. Keep the content small, as size of ResultBundle instances matters a lot for storing long test sessions results.

This is useful for some tests that are naturally decomposed in subtests.

Note

Metrics of aggregated bundles will always be shown, but can be augmented with new metrics using the usual API.

utc_datetime

Use the earliest utc_datetime among the aggregated bundles.

context

Merge the context of all the aggregated bundles, with priority given to last in the list.

result
_
metrics
exception lisa.tests.base.CannotCreateError

Bases: RuntimeError

Something prevented the creation of a TestBundle or ResultBundleBase instance.

class lisa.tests.base.TestBundleMeta

Bases: abc.ABCMeta

Metaclass of TestBundle.

Method with a return annotation of ResultBundleBase are wrapped to update the context attribute of a returned ResultBundleBase.

If _from_target is defined in the class but from_target is not, a stub is created and the annotation of _from_target is copied to the stub. The annotation is then removed from _from_target so that it is not picked up by exekall.

The signature of from_target is the result of merging the original cls.from_target parameters with the ones defined in _from_target.

Test methods:
static test_method(func)

Decorator to intercept returned ResultBundle and attach some contextual information.

class lisa.tests.base.TestBundle(res_dir, plat_info)

Bases: lisa.utils.Serializable, lisa.utils.ExekallTaggable, abc.ABC

A LISA test bundle.

Parameters:
  • res_dir (str) – Directory in which the target execution artifacts reside. This will also be used to dump any artifact generated in the test code.
  • plat_info (lisa.platforms.platinfo.PlatformInfo) – Various informations about the platform, that is available to all tests.

The point of a TestBundle is to bundle in a single object all of the required data to run some test assertion (hence the name). When inheriting from this class, you can define test methods that use this data, and return a ResultBundle.

Thanks to Serializable, instances of this class can be serialized with minimal effort. As long as some information is stored within an object’s member, it will be automagically handled.

Please refrain from monkey-patching the object in from_target(). Data required by the object to run test assertions should be exposed as __init__ parameters.

Design notes:

  • from_target() will collect whatever artifacts are required from a given target, and will then return a TestBundle. Note that a default implementation is provided out of _from_target.
  • from_dir() will use whatever artifacts are available in a given directory (which should have been created by an earlier call to from_target() and then to_dir()), and will then return a TestBundle.
  • VERIFY_SERIALIZATION is there to ensure the instances can serialized and deserialized without error.
  • res_dir parameter of __init__ must be stored as an attribute without further processing, in order to support result directory relocation.
  • Test methods should have a return annotation for the ResultBundle to be picked up by the test runners.

Implementation example:

from lisa.target import Target
from lisa.platforms.platinfo import PlatformInfo
from lisa.utils import ArtifactPath

class DummyTestBundle(TestBundle):

    def __init__(self, res_dir, plat_info, shell_output):
        super().__init__(res_dir, plat_info)

        self.shell_output = shell_output

    @classmethod
    def _from_target(cls, target:Target, *, res_dir:ArtifactPath) -> 'DummyTestBundle':
        output = target.execute('echo $((21+21))').split()
        return cls(res_dir, target.plat_info, output)

    def test_output(self) -> ResultBundle:
        return ResultBundle.from_bool(
            any(
                '42' in line
                for line in self.shell_output
            )
        )

Usage example:

# Creating a Bundle from a live target
bundle = TestBundle.from_target(target, plat_info=plat_info, res_dir="/my/res/dir")
# Running some test on the bundle
res_bundle = bundle.test_foo()

# Saving the bundle on the disk
bundle.to_dir("/my/res/dir")

# Reloading the bundle from the disk
bundle = TestBundle.from_dir("/my/res/dir")
# The reloaded object can be used just like the original one.
# Keep in mind that serializing/deserializing this way will have a
# similar effect than a deepcopy.
res_bundle = bundle.test_foo()
VERIFY_SERIALIZATION = True

When True, this enforces a serialization/deserialization step in from_target().

Note

The deserialized instance is thrown away in order to avoid using what is in effect a deepcopy of the original bundle. Using that deepcopy greatly increases the memory consumption of long running processes.

get_tags()
Returns:Dictionary of tags and tag values
Return type:dict(str, object)
classmethod _from_target(target, *, res_dir)

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...
classmethod check_from_target(target)

Check whether the given target can be used to create an instance of this class

Raises:CannotCreateError if the check fails

This method should be overriden to check your implementation requirements

classmethod can_create_from_target(target)
Returns:Whether the given target can be used to create an instance of this class
Return type:bool
check_from_target() is used internally, so there shouldn’t be any
need to override this.
classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None)

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...
classmethod _get_filepath(res_dir)

Returns the path of the file containing the serialized object in res_dir folder.

classmethod from_dir(res_dir, update_res_dir=True)

Wrapper around lisa.utils.Serializable.from_path().

It uses _get_filepath() to get the name of the serialized file to reload.

classmethod _get_filepath(res_dir)

Returns the path of the file containing the serialized object in res_dir folder.

to_dir(res_dir)

See lisa.utils.Serializable.to_path()

_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
class lisa.tests.base.FtraceTestBundleMeta

Bases: lisa.tests.base.TestBundleMeta

Metaclass of FtraceTestBundle.

This metaclass ensures that each class will get its own copy of ftrace_conf attribute, and that the events specified in that configuration are a superset of what is needed by methods using the family of decorators lisa.trace.requires_events(). This makes sure that the default set of events is always enough to run all defined methods, without duplicating that information. That means that trace events are “inherited” at the same time as the methods that need them.

The ftrace_conf attribute is typically built by merging these sources:

Test methods:
class lisa.tests.base.FtraceTestBundle(res_dir, plat_info)

Bases: lisa.tests.base.TestBundle

Abstract Base Class for lisa.wlgen.rta.RTA-powered TestBundles

Optionally, an ftrace_conf class attribute can be defined to hold additional FTrace configuration used to record a trace while the synthetic workload is being run. By default, the required events are extracted from decorated test methods.

TRACE_PATH = 'trace.dat'

Path to the trace-cmd trace.dat file in the result directory.

trace_path

Path to the trace-cmd report trace.dat file.

trace
Returns:a lisa.trace.TraceView

All events specified in ftrace_conf are parsed from the trace, so it is suitable for direct use in methods.

Having the trace as a property lets us defer the loading of the actual trace to when it is first used. Also, this prevents it from being serialized when calling lisa.utils.Serializable.to_path() and allows updating the underlying path before it is actually loaded to match a different folder structure.

get_trace(**kwargs)
Returns:a lisa.trace.Trace collected in the standard location.
Variable keyword arguments:
 Forwarded to lisa.trace.Trace.
_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.base.DmesgTestConf(conf=None, src='user', add_default_src=True)

Bases: lisa.conf.SimpleMultiSrcConf

Configuration class for lisa.tests.base.DmesgTestBundle.test_dmesg().

  • dmesg-test-conf: Dmesg test configuration
  • ignored-patterns (TypedList[str]): List of Python regex matching dmesg entries content to be whitelisted.
STRUCTURE = <lisa.conf.TopLevelKeyDesc object>
class IgnoredPatterns

Bases: lisa.utils.HideExekallID

List of Python regex matching dmesg entries content to be whitelisted

_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 231
_abc_registry = <_weakrefset.WeakSet object>
_get_typed_key_IgnoredPatterns() → lisa.tests.base.DmesgTestConf.IgnoredPatterns
class lisa.tests.base.DmesgTestBundle(res_dir, plat_info)

Bases: lisa.tests.base.TestBundle

Abstract Base Class for TestBundles based on dmesg output.

Test methods:
DMESG_PATH = 'dmesg.log'

Path to the dmesg log in the result directory.

CANNED_DMESG_IGNORED_PATTERNS = {'EAS-schedutil': 'Disabling EAS, schedutil is mandatory', 'executable-stack': 'started with executable stack'}

Mapping of canned patterns to avoid repetition while defining DMESG_IGNORED_PATTERNS in subclasses.

DMESG_IGNORED_PATTERNS = ['started with executable stack']

List of patterns to ignore in addition to the ones passed to test_dmesg().

dmesg_path

Path to the dmesg output log file

dmesg_entries

List of parsed dmesg output entries devlib.collector.dmesg.KernelLogEntry.

test_dmesg(level='warn', facility=None, ignored_patterns: lisa.tests.base.DmesgTestConf.IgnoredPatterns = None) → lisa.tests.base.ResultBundle

Basic test on kernel dmesg output.

Parameters:
  • level (str) – Any dmesg entr with a level more critical than (and including) that will make the test fail.
  • facility (str or None) – Only select entries emitted by the given dmesg facility like kern. Note that not all versions of dmesg are able to print it, so specifying it may lead to no entry being inspected at all. If None, the facility is ignored.
  • ignored_patterns (list or None) – List of regexes to ignore some messages. The pattern list is combined with DMESG_IGNORED_PATTERNS class attribute.
_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
class lisa.tests.base.RTATestBundle(res_dir, plat_info)

Bases: lisa.tests.base.FtraceTestBundle, lisa.tests.base.DmesgTestBundle

Abstract Base Class for lisa.wlgen.rta.RTA-powered TestBundles

Test methods:
TASK_PERIOD_MS = 16

A task period you can re-use for your lisa.wlgen.rta.RTATask definitions.

NOISE_ACCOUNTING_THRESHOLDS = {TaskID(pid=0, comm=None): 100, '^sugov:\\d+$': 5, '^irq/\\d+-mhu_link$': 1.5}

PID/comm specific tuning for test_noisy_tasks()

  • keys can be PIDs, comms, or regexps for comms.
  • values are noisiness thresholds (%), IOW below that runtime threshold the associated task will be ignored in the noise accounting.
_BUFFER_PHASE_DURATION_S = 0.5

Duration of the initial buffer phase; this is a phase that copies the first phase of each task, and that is prepended to the relevant task - this means all task in the profile get a buffer phase.

trace_window(trace)

The time window to consider for this RTATestBundle

Returns:a (start, stop) tuple

Since we’re using rt-app profiles, we know the name of tasks we are interested in, so we can trim our trace scope to filter out the setup/teardown events we don’t care about.

Override this method if you need a different trace trimming.

Warning

Calling self.trace here will raise an AttributeError exception, to avoid entering infinite recursion.

Required trace events:
 
  • sched_switch
  • userspace@rtapp_loop
rtapp_profile

Compute the RTapp profile based on plat_info.

_rtapp_tasks_events = <lisa.trace.AndTraceEventChecker object>
rtapp_task_ids_map

Mapping of task names as specified in the rtapp profile to list of lisa.trace.TaskID names found in the trace.

If the task forked, the list will contain more than one item.

rtapp_task_ids

The rtapp task lisa.trace.TaskID as found from the trace in this bundle.

Returns:the list of actual trace task lisa.trace.TaskID
rtapp_tasks_map

Same as rtapp_task_ids_map() but with list of strings for values.

rtapp_tasks

Same as rtapp_task_ids() but as a list of string.

Returns:the list of actual trace task names
_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
cgroup_configuration

Compute the cgroup configuration based on plat_info

classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, ftrace_coll: lisa.trace.FtraceCollector = None) → lisa.tests.base.RTATestBundle

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Factory method to create a bundle using a live target

This will execute the rt-app workload described in get_rtapp_profile()

ftrace_conf = <lisa.trace.FtraceConf object>
trace
Returns:a lisa.trace.TraceView cropped to the window given by trace_window().

All events specified in ftrace_conf are parsed from the trace, so it is suitable for direct use in methods.

Having the trace as a property lets us defer the loading of the actual trace to when it is first used. Also, this prevents it from being serialized when calling lisa.utils.Serializable.to_path() and allows updating the underlying path before it is actually loaded to match a different folder structure.

test_noisy_tasks(noise_threshold_pct=None, noise_threshold_ms=None)

Test that no non-rtapp (“noisy”) task ran for longer than the specified thresholds

Parameters:
  • noise_threshold_pct (float) – The maximum allowed runtime for noisy tasks in percentage of the total rt-app execution time
  • noise_threshold_ms (float) – The maximum allowed runtime for noisy tasks in ms

If both are specified, the smallest threshold (in seconds) will be used.

Required trace events:
 
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
classmethod check_noisy_tasks(noise_threshold_pct=None, noise_threshold_ms=None)

Decorator that applies test_noisy_tasks() to the trace of the TestBundle returned by the underlying method. The Result will be changed to Result.UNDECIDED if that test fails.

We also expose test_noisy_tasks() parameters to the decorated function.

classmethod unscaled_utilization(plat_info, cpu, utilization_pct)

Convert utilization scaled to a CPU to a ‘raw’, unscaled one.

Parameters:
  • capacity (int) – The CPU against which utilization_pct` is scaled
  • utilization_pct (int) – The scaled utilization in %
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

classmethod get_cgroup_configuration(plat_info)
Returns:a dict representing the configuration of a particular cgroup.

This is a method you may optionally override to configure a cgroup for the synthetic workload.

Example of return value:

{
    'name': 'lisa_test',
    'controller': 'schedtune',
    'attributes' : {
        'prefer_idle' : 1,
        'boost': 50
    }
}
classmethod _target_configure_cgroup(target, cfg)
classmethod run_rtapp(target, res_dir, profile=None, ftrace_coll=None, cg_cfg=None, wipe_run_dir=True, update_cpu_capacities=None)

Run the given RTA profile on the target, and collect an ftrace trace.

Parameters:
  • target (lisa.target.Target) – target to execute the workload on.
  • res_dir (str or lisa.utils.ArtifactPath) – Artifact folder where the artifacts will be stored.
  • profile (dict(str, lisa.wlgen.rta.RTATask)) – rt-app profile, as a dictionary of dict(task_name, RTATask). If None, get_rtapp_profile() is called with target.plat_info.
  • ftrace_coll (lisa.trace.FtraceCollector) – Ftrace collector to use to record the trace. This allows recording extra events compared to the default one, which is based on the ftrace_conf class attribute.
  • cg_cfg (dict) – CGroup configuration dictionary. If None, lisa.tests.base.RTATestBundle.get_cgroup_configuration() is called with target.plat_info.
  • wipe_run_dir (bool) – Remove the run directory on the target after execution of the workload.
  • update_cpu_capacities (bool) – Attempt to update the CPU capacities based on the calibration values of rtapp to get the most accurate reproduction of duty cycles.
classmethod _run_rtapp(*args, **kwargs)

Has been renamed to run_rtapp(), as it really is part of the public API.

classmethod _from_target(target, *, res_dir, ftrace_coll=None)

Factory method to create a bundle using a live target

This will execute the rt-app workload described in get_rtapp_profile()

Scheduler tests

EAS tests

Inheritance diagram of lisa.tests.scheduler.eas_behaviour

class lisa.tests.scheduler.eas_behaviour.EASBehaviour(res_dir, plat_info)

Bases: lisa.tests.base.RTATestBundle

Abstract class for EAS behavioural testing.

Parameters:nrg_model (EnergyModel) – The energy model of the platform the synthetic workload was run on

This class provides test_task_placement() to validate the basic behaviour of EAS. The implementations of this class have been developed to verify patches supporting Arm’s big.LITTLE in the Linux scheduler. You can see these test results being published here.

Test methods:
nrg_model
classmethod get_big_duty_cycle(plat_info, utilization_pct=None)

Returns a duty cycle for lisa.wlgen.rta.Periodic that will guarantee placement on a big CPU.

Parameters:utilization_pct (int or None) – If None, the duty cycle will be chosen so that the task will not fit on the second to biggest CPUs in the system, thereby forcing up-migration while minimizing the thermal impact. If a number is passed, the value will be used as a percentage of utilization of a big CPU, and will be converted to a duty cycle.
classmethod get_little_duty_cycle(plat_info, utilization_pct)

Returns the duty-cycle percentage equivalent to utilization_pct on little CPUs.

classmethod check_from_target(target)

Check whether the given target can be used to create an instance of this class

Raises:CannotCreateError if the check fails

This method should be overriden to check your implementation requirements

test_task_placement(energy_est_threshold_pct=5, nrg_model: lisa.energy_model.EnergyModel = None, capacity_margin_pct=20, *, noise_threshold_pct=1, noise_threshold_ms=None) → lisa.tests.base.ResultBundle

Test that task placement was energy-efficient

Parameters:
  • nrg_model (EnergyModel) – Allow using an alternate EnergyModel instead of nrg_model`
  • energy_est_threshold_pct (int) – Allowed margin for estimated vs optimal task placement energy cost

Compute optimal energy consumption (energy-optimal task placement) and compare to energy consumption estimated from the trace. Check that the estimated energy does not exceed the optimal energy by more than energy_est_threshold_pct` percents.

Added by lisa.tests.base.RTATestBundle.test_noisy_tasks():

The returned ResultBundle.result will be changed to UNDECIDED if the environment was too noisy: Test that no non-rtapp (“noisy”) task ran for longer than the specified thresholds

Parameters:
  • noise_threshold_pct (float) – The maximum allowed runtime for noisy tasks in percentage of the total rt-app execution time
  • noise_threshold_ms (float) – The maximum allowed runtime for noisy tasks in ms

If both are specified, the smallest threshold (in seconds) will be used.

Required trace events:
 
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
  • userspace@rtapp_loop
test_slack(negative_slack_allowed_pct=15) → lisa.tests.base.ResultBundle

Assert that the RTApp workload was given enough performance

Parameters:negative_slack_allowed_pct (int) – Allowed percentage of RT-app task activations with negative slack.

Use lisa.analysis.rta.RTAEventsAnalysis to find instances where the RT-App workload wasn’t able to complete its activations (i.e. its reported “slack” was negative). Assert that this happened less than negative_slack_allowed_pct percent of the time.

Required trace events:
 
  • userspace@rtapp_stats
classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, ftrace_coll: lisa.trace.FtraceCollector = None) → lisa.tests.scheduler.eas_behaviour.EASBehaviour

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Factory method to create a bundle using a live target

This will execute the rt-app workload described in get_rtapp_profile()

(above inherited from lisa.tests.base.RTATestBundle.from_target())

Factory method to create a bundle using a live target

This will execute the rt-app workload described in lisa.tests.base.RTATestBundle.get_rtapp_profile()

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.eas_behaviour.OneSmallTask(res_dir, plat_info)

Bases: lisa.tests.scheduler.eas_behaviour.EASBehaviour

A single ‘small’ task

Test methods:
task_name = 'small'
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.eas_behaviour.ThreeSmallTasks(res_dir, plat_info)

Bases: lisa.tests.scheduler.eas_behaviour.EASBehaviour

Three ‘small’ tasks

Test methods:
task_prefix = 'small'
test_task_placement(energy_est_threshold_pct=20, nrg_model: lisa.energy_model.EnergyModel = None, noise_threshold_pct=1, noise_threshold_ms=None, capacity_margin_pct=20) → lisa.tests.base.ResultBundle

Same as EASBehaviour.test_task_placement() but with a higher default threshold

The energy estimation for this test is probably not very accurate and this isn’t a very realistic workload. It doesn’t really matter if we pick an “ideal” task placement for this workload, we just want to avoid using big CPUs in a big.LITTLE system. So use a larger energy threshold that hopefully prevents too much use of big CPUs but otherwise is flexible in allocation of LITTLEs.

Required trace events:
 
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
  • userspace@rtapp_loop
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.eas_behaviour.TwoBigTasks(res_dir, plat_info)

Bases: lisa.tests.scheduler.eas_behaviour.EASBehaviour

Two ‘big’ tasks

Test methods:
task_prefix = 'big'
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.eas_behaviour.TwoBigThreeSmall(res_dir, plat_info)

Bases: lisa.tests.scheduler.eas_behaviour.EASBehaviour

A mix of ‘big’ and ‘small’ tasks

Test methods:
small_prefix = 'small'
big_prefix = 'big'
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.eas_behaviour.EnergyModelWakeMigration(res_dir, plat_info)

Bases: lisa.tests.scheduler.eas_behaviour.EASBehaviour

One task per big CPU, alternating between two phases:

  • Low utilization phase (should run on a LITTLE CPU)
  • High utilization phase (should run on a big CPU)
Test methods:
task_prefix = 'emwm'
classmethod check_from_target(target)

Check whether the given target can be used to create an instance of this class

Raises:CannotCreateError if the check fails

This method should be overriden to check your implementation requirements

classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.eas_behaviour.RampUp(res_dir, plat_info)

Bases: lisa.tests.scheduler.eas_behaviour.EASBehaviour

A single task whose utilization slowly ramps up

Test methods:
task_name = 'up'
test_task_placement(energy_est_threshold_pct=15, nrg_model: lisa.energy_model.EnergyModel = None, noise_threshold_pct=1, noise_threshold_ms=None, capacity_margin_pct=20) → lisa.tests.base.ResultBundle

Same as EASBehaviour.test_task_placement() but with a higher default threshold.

The main purpose of this test is to ensure that as it grows in load, a task is migrated from LITTLE to big CPUs on a big.LITTLE system. This migration naturally happens some time _after_ it could possibly be done, since there must be some hysteresis to avoid a performance cost. Therefore allow a larger energy usage threshold

Required trace events:
 
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
  • userspace@rtapp_loop
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.eas_behaviour.RampDown(res_dir, plat_info)

Bases: lisa.tests.scheduler.eas_behaviour.EASBehaviour

A single task whose utilization slowly ramps down

Test methods:
task_name = 'down'
test_task_placement(energy_est_threshold_pct=18, nrg_model: lisa.energy_model.EnergyModel = None, noise_threshold_pct=1, noise_threshold_ms=None, capacity_margin_pct=20) → lisa.tests.base.ResultBundle

Same as EASBehaviour.test_task_placement() but with a higher default threshold

The main purpose of this test is to ensure that as it reduces in load, a task is migrated from big to LITTLE CPUs on a big.LITTLE system. This migration naturally happens some time _after_ it could possibly be done, since there must be some hysteresis to avoid a performance cost. Therefore allow a larger energy usage threshold

The number below has been found by trial and error on the platform generally used for testing EAS (at the time of writing: Juno r0, Juno r2, Hikey960 and TC2). It would be better to estimate the amount of energy ‘wasted’ in the hysteresis (the overutilized band) and compute a threshold based on that. But implementing this isn’t easy because it’s very platform dependent, so until we have a way to do that easily in test classes, let’s stick with the arbitrary threshold.

Required trace events:
 
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
  • userspace@rtapp_loop
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

ftrace_conf = <lisa.trace.FtraceConf object>

Load tracking tests

Inheritance diagram of lisa.tests.scheduler.load_tracking

lisa.tests.scheduler.load_tracking.UTIL_CONVERGENCE_TIME_S = 0.331923351977693

Time in seconds for util_avg to converge (i.e. ignored time)

class lisa.tests.scheduler.load_tracking.LoadTrackingHelpers

Bases: object

Common bunch of helpers for load tracking tests.

MAX_RTAPP_CALIB_DEVIATION = 0.03

Blacklist CPUs that have a RTapp calibration value that deviates too much from the average calib value in their capacity class.

classmethod _get_blacklisted_cpus(plat_info)

Consider some CPUs as blacklisted when the load would not be proportionnal to utilization on them.

That happens for CPUs that are busy executing other code than the test workload, like handling interrupts. It is detect that by looking at the RTapp calibration value and we blacklist outliers.

classmethod filter_capacity_classes(plat_info)

Filter out capacity-classes key of plat_info to remove blacklisted CPUs provided by:

classmethod _get_blacklisted_cpus(plat_info)

Consider some CPUs as blacklisted when the load would not be proportionnal to utilization on them.

That happens for CPUs that are busy executing other code than the test workload, like handling interrupts. It is detect that by looking at the RTapp calibration value and we blacklist outliers.

classmethod correct_expected_pelt(plat_info, cpu, signal_value)

Correct an expected PELT signal from rt-app based on the calibration values.

Since the instruction mix of rt-app might not be the same as the benchmark that was used to establish CPU capacities, the duty cycle of rt-app will only be accurate on big CPUs. When we know on which CPU the task actually executed, we can correct the expected value based on the ratio of calibration values and CPU capacities.

class lisa.tests.scheduler.load_tracking.LoadTrackingBase(res_dir, plat_info)

Bases: lisa.tests.base.RTATestBundle, lisa.tests.scheduler.load_tracking.LoadTrackingHelpers

Base class for shared functionality of load tracking tests

Test methods:
cpufreq_conf = {'governor': 'performance'}

The cpufreq configuration used while the synthetic workload is being run. Items are arguments to devlib.module.cpufreq.CpufreqModule.use_governor().

classmethod _from_target(target, *, res_dir=None, ftrace_coll=None)

Factory method to create a bundle using a live target

This will execute the rt-app workload described in get_rtapp_profile()

static is_almost_equal(target, value, allowed_delta_pct)

Verify that value` is reasonably close to target`

Returns:A tuple (bool, delta_pct)
_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, ftrace_coll: lisa.trace.FtraceCollector = None) → lisa.tests.scheduler.load_tracking.LoadTrackingBase

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Factory method to create a bundle using a live target

This will execute the rt-app workload described in get_rtapp_profile()

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.load_tracking.InvarianceItem(res_dir, plat_info, cpu, freq, freq_list)

Bases: lisa.tests.scheduler.load_tracking.LoadTrackingBase, lisa.utils.ExekallTaggable

Basic check for CPU and frequency invariant load and utilization tracking

Expected Behaviour:

Load tracking signals are scaled so that the workload results in roughly the same util & load values regardless of compute power of the CPU used and its frequency.

Test methods:
task_prefix = 'invar'
cpufreq_conf = {'governor': 'userspace'}
rtapp_profile

Compute the RTapp profile based on plat_info.

task_name

The name of the only task this test uses

wlgen_task

The lisa.wlgen.rta.RTATask description of the only rt-app task, as specified in the profile.

get_tags()
Returns:Dictionary of tags and tag values
Return type:dict(str, object)
classmethod get_rtapp_profile(plat_info, cpu, freq)

Get a specification for a rt-app workload with the specificied duty cycle, pinned to the given CPU.

classmethod _from_target(target, *, cpu, freq, freq_list=None, res_dir=None, ftrace_coll=None)
Parameters:
  • cpu (int or None) – CPU to use, or None to automatically choose an appropriate set of CPUs.
  • freq (int or None) – Frequency to run at in kHz. It is only relevant in combination with cpu.
static _get_freq_capa(cpu, freq, plat_info)
get_simulated_pelt(task, signal_name)

Simulate a PELT signal for a given task.

Parameters:
  • task (int or str or tuple(int, str)) – task to look for in the trace.
  • signal_name (str) – Name of the PELT signal to simulate.
Returns:

A pandas.DataFrame with a simulated column containing the simulated signal, along with the column of the signal as found in the trace.

Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
_plot_pelt(task, signal_name, simulated, test_name)
_add_cpu_metric(res_bundle)
_test_behaviour(signal_name, error_margin_pct)
Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
_test_correctness(signal_name, mean_error_margin_pct, max_error_margin_pct)
Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
test_util_correctness(mean_error_margin_pct=2, max_error_margin_pct=5) → lisa.tests.base.ResultBundle

Check that the utilization signal is as expected.

Parameters:
  • mean_error_margin_pct (float) – Maximum allowed difference in the mean of the actual signal and the simulated one, as a percentage of utilization scale.
  • max_error_margin_pct (float) – Maximum allowed difference between samples of the actual signal and the simulated one, as a percentage of utilization scale.
Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
test_load_correctness(mean_error_margin_pct=2, max_error_margin_pct=5) → lisa.tests.base.ResultBundle

Same as test_util_correctness() but checking the load.

Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
test_util_behaviour(error_margin_pct=5, *, noise_threshold_pct=1, noise_threshold_ms=None) → lisa.tests.base.ResultBundle

Check the utilization mean is linked to the task duty cycle.

Note

That is not really the case, as the util of a task is not updated when the task is sleeping, but is fairly close to reality as long as the task period is small enough.

Parameters:error_margin_pct (float) – Allowed difference in percentage of utilization scale.

Added by lisa.tests.base.RTATestBundle.test_noisy_tasks():

The returned ResultBundle.result will be changed to UNDECIDED if the environment was too noisy: Test that no non-rtapp (“noisy”) task ran for longer than the specified thresholds

Parameters:
  • noise_threshold_pct (float) – The maximum allowed runtime for noisy tasks in percentage of the total rt-app execution time
  • noise_threshold_ms (float) – The maximum allowed runtime for noisy tasks in ms

If both are specified, the smallest threshold (in seconds) will be used.

Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
test_load_behaviour(error_margin_pct=5, *, noise_threshold_pct=1, noise_threshold_ms=None) → lisa.tests.base.ResultBundle

Same as test_util_behaviour() but checking the load.

Added by lisa.tests.base.RTATestBundle.test_noisy_tasks():

The returned ResultBundle.result will be changed to UNDECIDED if the environment was too noisy: Test that no non-rtapp (“noisy”) task ran for longer than the specified thresholds

Parameters:
  • noise_threshold_pct (float) – The maximum allowed runtime for noisy tasks in percentage of the total rt-app execution time
  • noise_threshold_ms (float) – The maximum allowed runtime for noisy tasks in ms

If both are specified, the smallest threshold (in seconds) will be used.

Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, ftrace_coll: lisa.trace.FtraceCollector = None, cpu: int, freq: int, freq_list=None) → lisa.tests.scheduler.load_tracking.InvarianceItem

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Factory method to create a bundle using a live target

This will execute the rt-app workload described in get_rtapp_profile()

(above inherited from lisa.tests.scheduler.load_tracking.LoadTrackingBase.from_target())

Parameters:
  • cpu (int or None) – CPU to use, or None to automatically choose an appropriate set of CPUs.
  • freq (int or None) – Frequency to run at in kHz. It is only relevant in combination with cpu.
ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.load_tracking.Invariance(res_dir, plat_info, invariance_items)

Bases: lisa.tests.base.TestBundle, lisa.tests.scheduler.load_tracking.LoadTrackingHelpers

Basic check for frequency invariant load and utilization tracking

This test runs the same workload on one CPU of each capacity available in the system at a cross section of available frequencies.

This class is mostly a wrapper around InvarianceItem, providing a way to build a list of those for a few frequencies, and providing aggregated versions of the tests. Calling the tests methods on the items directly is recommended to avoid the unavoidable loss of information when aggregating the Result of each item.

invariance_items instance attribute is a list of instances of InvarianceItem.

Test methods:
ftrace_conf = <lisa.trace.FtraceConf object>
NR_FREQUENCIES = 8

Maximum number of tested frequencies.

classmethod _build_invariance_items(target, res_dir, ftrace_coll)

Yield a InvarianceItem for a subset of target’s frequencies, for one CPU of each capacity class.

This is a generator function.

Return type:Iterator[InvarianceItem]
iter_invariance_items() → lisa.tests.scheduler.load_tracking.InvarianceItem
classmethod _from_target(target, *, res_dir=None, ftrace_coll=None)

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...
get_trace(cpu, freq)
Returns:The trace generated when running at a given frequency
test_util_correctness(mean_error_margin_pct=2, max_error_margin_pct=5) → lisa.tests.base.AggregatedResultBundle

Aggregated version of InvarianceItem.test_util_correctness()

Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
test_load_correctness(mean_error_margin_pct=2, max_error_margin_pct=5) → lisa.tests.base.AggregatedResultBundle

Aggregated version of InvarianceItem.test_load_correctness()

Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
test_util_behaviour(error_margin_pct=5) → lisa.tests.base.AggregatedResultBundle

Aggregated version of InvarianceItem.test_util_behaviour()

Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
test_load_behaviour(error_margin_pct=5) → lisa.tests.base.AggregatedResultBundle

Aggregated version of InvarianceItem.test_load_behaviour()

Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
_test_all_freq(item_test)

Apply the item_test function on all instances of InvarianceItem and aggregate the returned ResultBundle into one.

UNDECIDED is ignored.

test_cpu_invariance() → lisa.tests.base.AggregatedResultBundle

Check that items using the max freq on each CPU is passing util avg test.

There could be false positives, but they are expected to be relatively rare.

Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
test_freq_invariance() → lisa.tests.base.ResultBundle

Check that at least one CPU has items passing for all tested frequencies.

Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, ftrace_coll: lisa.trace.FtraceCollector = None) → lisa.tests.scheduler.load_tracking.Invariance

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...
class lisa.tests.scheduler.load_tracking.CPUMigrationBase(res_dir, plat_info)

Bases: lisa.tests.scheduler.load_tracking.LoadTrackingBase

Base class for migration-related load tracking tests

The idea here is to run several rt-app tasks and to have them pinned to a single CPU for a single phase. They can change CPUs in a new phase, and we can then inspect the CPU utilization - it should match the sum of the utilization of all the tasks running on it.

Design notes:

Since we sum up the utilization of each task, make sure not to overload the CPU - IOW, there should always be some idle cycles.

The code assumes all tasks have the same number of phases, and that those phases are all aligned.

Test methods:
PHASE_DURATION_S = 0.9957700559330791

The duration of a single phase

TASK_PERIOD_MS = 16

The average value of the runqueue PELT signals is very dependent on the task period, so it’s important to set it to a known validate value in that class.

get_nr_required_cpu(plat_info)

The number of CPUs of same capacity involved in the test

classmethod run_rtapp(target, res_dir, profile, ftrace_coll, cgroup=None)

Run the given RTA profile on the target, and collect an ftrace trace.

Parameters:
  • target (lisa.target.Target) – target to execute the workload on.
  • res_dir (str or lisa.utils.ArtifactPath) – Artifact folder where the artifacts will be stored.
  • profile (dict(str, lisa.wlgen.rta.RTATask)) – rt-app profile, as a dictionary of dict(task_name, RTATask). If None, get_rtapp_profile() is called with target.plat_info.
  • ftrace_coll (lisa.trace.FtraceCollector) – Ftrace collector to use to record the trace. This allows recording extra events compared to the default one, which is based on the ftrace_conf class attribute.
  • cg_cfg (dict) – CGroup configuration dictionary. If None, lisa.tests.base.RTATestBundle.get_cgroup_configuration() is called with target.plat_info.
  • wipe_run_dir (bool) – Remove the run directory on the target after execution of the workload.
  • update_cpu_capacities (bool) – Attempt to update the CPU capacities based on the calibration values of rtapp to get the most accurate reproduction of duty cycles.
cpus

All CPUs used by RTapp workload.

classmethod check_from_target(target)

Check whether the given target can be used to create an instance of this class

Raises:CannotCreateError if the check fails

This method should be overriden to check your implementation requirements

classmethod get_migration_cpus(plat_info)
Returns:N CPUs of same capacity, with N set by get_nr_required_cpu().
get_expected_cpu_util()

Get the per-phase average CPU utilization expected from the duty cycle of the tasks found in the trace.

Returns:A dict of the shape {cpu : {phase_id : expected_util}}

Note

This is more robust than just looking at the duty cycle in the task profile, since rtapp might not reproduce accurately the duty cycle it was asked.

Required trace events:
 
  • optional: (cpu_frequency or userspace@cpu_frequency_devlib)
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
  • userspace@rtapp_loop
get_trace_cpu_util()

Get the per-phase average CPU utilization read from the trace

Returns:

A dict of the shape {cpu : {phase_id : trace_util}}

Required trace events:
 
  • optional: (sched_load_avg_cpu or sched_load_cfs_rq or sched_pelt_cfs) , cpu_capacity , sched_util_est_cfs
_plot_util()
Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
test_util_task_migration(allowed_error_pct=3, *, noise_threshold_pct=1, noise_threshold_ms=None) → lisa.tests.base.ResultBundle

Test that a migrated task properly propagates its utilization at the CPU level

Parameters:allowed_error_pct (float) – How much the trace averages can stray from the expected values

Added by lisa.tests.base.RTATestBundle.test_noisy_tasks():

The returned ResultBundle.result will be changed to UNDECIDED if the environment was too noisy: Test that no non-rtapp (“noisy”) task ran for longer than the specified thresholds

Parameters:
  • noise_threshold_pct (float) – The maximum allowed runtime for noisy tasks in percentage of the total rt-app execution time
  • noise_threshold_ms (float) – The maximum allowed runtime for noisy tasks in ms

If both are specified, the smallest threshold (in seconds) will be used.

Required trace events:
 
  • optional: (sched_load_avg_cpu or sched_load_cfs_rq or sched_pelt_cfs) , cpu_capacity , sched_util_est_cfs
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: (cpu_frequency or userspace@cpu_frequency_devlib)
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
  • userspace@rtapp_loop
_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.load_tracking.OneTaskCPUMigration(res_dir, plat_info)

Bases: lisa.tests.scheduler.load_tracking.CPUMigrationBase

Some tasks on two big CPUs, one of them migrates in its second phase.

Test methods:
classmethod get_nr_required_cpu(plat_info)

The number of CPUs of same capacity involved in the test

classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.load_tracking.NTasksCPUMigrationBase(res_dir, plat_info)

Bases: lisa.tests.scheduler.load_tracking.CPUMigrationBase

N tasks on N CPUs, with all the migration permutations.

Test methods:
_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
ftrace_conf = <lisa.trace.FtraceConf object>
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

class lisa.tests.scheduler.load_tracking.TwoTasksCPUMigration(res_dir, plat_info)

Bases: lisa.tests.scheduler.load_tracking.NTasksCPUMigrationBase

Two tasks on two big CPUs, swap their CPU in the second phase

Test methods:
_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
ftrace_conf = <lisa.trace.FtraceConf object>
classmethod get_nr_required_cpu(plat_info)

The number of CPUs of same capacity involved in the test

class lisa.tests.scheduler.load_tracking.NTasksCPUMigration(res_dir, plat_info)

Bases: lisa.tests.scheduler.load_tracking.NTasksCPUMigrationBase

N tasks on N CPUs, and try all permutations of tasks and CPUs.

Test methods:
_abc_cache = <_weakrefset.WeakSet object>
_abc_negative_cache = <_weakrefset.WeakSet object>
_abc_negative_cache_version = 223
_abc_registry = <_weakrefset.WeakSet object>
ftrace_conf = <lisa.trace.FtraceConf object>
classmethod get_nr_required_cpu(plat_info)

Select the maximum number of CPUs the tests can handle.

test_util_task_migration(allowed_error_pct=8) → lisa.tests.base.ResultBundle

Relax the margins compared to the super-class version.


Inheritance diagram of lisa.tests.scheduler.util_tracking

class lisa.tests.scheduler.util_tracking.UtilTrackingBase(res_dir, plat_info)

Bases: lisa.tests.base.RTATestBundle, lisa.tests.scheduler.load_tracking.LoadTrackingHelpers

Base class for shared functionality of utilization tracking tests

Test methods:
classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, ftrace_coll: lisa.trace.FtraceCollector = None) → lisa.tests.scheduler.util_tracking.UtilTrackingBase

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Factory method to create a bundle using a live target

This will execute the rt-app workload described in get_rtapp_profile()

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.util_tracking.PhaseStats(start, end, mean_util, mean_enqueued, mean_ewma, issue)

Bases: collections.abc.Mapping

class lisa.tests.scheduler.util_tracking.ActivationSignals(time, util, enqueued, ewma, issue)

Bases: collections.abc.Mapping

class lisa.tests.scheduler.util_tracking.UtilConvergence(res_dir, plat_info)

Bases: lisa.tests.scheduler.util_tracking.UtilTrackingBase

Basic checks for estimated utilization signals

Expected Behaviour:

The estimated utilization of a task is properly computed starting form its util value at the end of each activation.

Two signals composes the estimated utlization of a task:

  • enqueued : is expected to match the max between util and ewma at the end of the previous activation
  • ewma : is expected to track an Exponential Weighted Moving Average of the util signal sampled at the end of each activation.

Based on these two invariant, this class provides a set of tests to verify these conditions using different methods and sampling points.

Test methods:
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

fast_ramp
test_means(*, noise_threshold_pct=1, noise_threshold_ms=None) → lisa.tests.base.ResultBundle

Test signals are properly “dominated”.

The mean of enqueued is expected to be always not smaller than that of util, since this last is subject to decays while the first not.

The mean of enqueued is expected to be always greater or equal than the mean of util, since this util is subject to decays while enqueued not.

On fast-ramp systems, the ewma signal is never smaller then the enqueued, thus his mean is expected to be bigger.

On non fast-ramp systems instead, the ewma is expected to be smaller then enqueued in ramp-up phases, or bigger in ramp-down phases.

Those conditions are checked on a single execution of a task which has three main behaviours:

  • STABLE: periodic big task running for a relatively long period to ensure util saturation.
  • DOWN: periodic ramp-down task, to slowly decay util
  • UP: periodic ramp-up task, to slowly increase util

Added by lisa.tests.base.RTATestBundle.test_noisy_tasks():

The returned ResultBundle.result will be changed to UNDECIDED if the environment was too noisy: Test that no non-rtapp (“noisy”) task ran for longer than the specified thresholds

Parameters:
  • noise_threshold_pct (float) – The maximum allowed runtime for noisy tasks in percentage of the total rt-app execution time
  • noise_threshold_ms (float) – The maximum allowed runtime for noisy tasks in ms

If both are specified, the smallest threshold (in seconds) will be used.

Required trace events:
 
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_util_est_se
  • sched_wakeup
  • userspace@rtapp_loop
test_activations(*, noise_threshold_pct=1, noise_threshold_ms=None) → lisa.tests.base.ResultBundle

Test signals are properly “aggregated” at enqueue/dequeue time.

On fast-ramp systems, enqueud is expected to be always smaller than ewma.

On non fast-ramp systems, the enqueued is expected to be smaller then ewma in ramp-down phases, or bigger in ramp-up phases.

Those conditions are checked on a single execution of a task which has three main behaviours:

  • STABLE: periodic big task running for a relatively long period to ensure util saturation.
  • DOWN: periodic ramp-down task, to slowly decay util
  • UP: periodic ramp-up task, to slowly increase util

Added by lisa.tests.base.RTATestBundle.test_noisy_tasks():

The returned ResultBundle.result will be changed to UNDECIDED if the environment was too noisy: Test that no non-rtapp (“noisy”) task ran for longer than the specified thresholds

Parameters:
  • noise_threshold_pct (float) – The maximum allowed runtime for noisy tasks in percentage of the total rt-app execution time
  • noise_threshold_ms (float) – The maximum allowed runtime for noisy tasks in ms

If both are specified, the smallest threshold (in seconds) will be used.

Required trace events:
 
  • optional: sched_wakeup_new
  • sched_switch
  • sched_util_est_se
  • sched_wakeup
ftrace_conf = <lisa.trace.FtraceConf object>

Misfit tests

Inheritance diagram of lisa.tests.scheduler.misfit

class lisa.tests.scheduler.misfit.MisfitMigrationBase(res_dir, plat_info)

Bases: lisa.tests.base.RTATestBundle

Abstract class for Misfit behavioural testing

This class provides some helpers for features related to Misfit.

Test methods:
ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.scheduler.misfit.StaggeredFinishes(res_dir, plat_info)

Bases: lisa.tests.scheduler.misfit.MisfitMigrationBase

One 100% task per CPU, with staggered completion times.

By spawning one task per CPU on an asymmetric system, we expect the tasks running on the higher-performance CPUs to complete first. At this point, the misfit logic should kick in and they should pull tasks from lower-performance CPUs.

The tasks have staggered completion times to prevent having several of them completing at the same time, which can cause some unwanted noise (e.g. some sshd or systemd activity at the end of the task).

The end result should look something like this on big.LITTLE:

a,b,c,d are CPU-hogging tasks
_ signifies idling

LITTLE_0 | a a a a _ _ _
LITTLE_1 | b b b b b _ _
---------|--------------
  big_0  | c c c c a a a
  big_1  | d d d d d b b
Test methods:
task_prefix = 'msft'
PIN_DELAY_S = 0.001

How long the tasks will be pinned to their “starting” CPU. Doesn’t have to be long (we just have to ensure they spawn there), so arbitrary value

IDLING_DELAY_S = 1

A somewhat arbitray delay - long enough to ensure rq->avg_idle > sysctl_sched_migration_cost

src_cpus
dst_cpus
end_time
duration
start_time

The tasks don’t wake up at the same exact time, find the task that is the last to wake up. We don’t want to redefine trace_window() here because we still need the first wakeups to be visible.

classmethod check_from_target(target)

Check whether the given target can be used to create an instance of this class

Raises:CannotCreateError if the check fails

This method should be overriden to check your implementation requirements

classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

test_preempt_time(allowed_preempt_pct=1) → lisa.tests.base.ResultBundle

Test that tasks are not being preempted too much

Required trace events:
 
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
ftrace_conf = <lisa.trace.FtraceConf object>
test_throughput(allowed_idle_time_s=None, *, noise_threshold_pct=1, noise_threshold_ms=None) → lisa.tests.base.ResultBundle

Test that big CPUs are not idle when there are misfit tasks to upmigrate

Parameters:allowed_idle_time_s (int) –

How much time should be allowed between a big CPU going idle and a misfit task ending on that CPU. In theory a newidle balance should lead to a null delay, but in practice there’s a tiny one, so don’t set that to 0 and expect the test to pass.

Furthermore, we’re not always guaranteed to get a newidle pull, so allow time for a regular load balance to happen.

When None, this defaults to (1ms x number_of_cpus) to mimic the default balance_interval (balance_interval = sd_weight), see kernel/sched/topology.c:sd_init().

Added by lisa.tests.base.RTATestBundle.test_noisy_tasks():

The returned ResultBundle.result will be changed to UNDECIDED if the environment was too noisy: Test that no non-rtapp (“noisy”) task ran for longer than the specified thresholds

Parameters:
  • noise_threshold_pct (float) – The maximum allowed runtime for noisy tasks in percentage of the total rt-app execution time
  • noise_threshold_ms (float) – The maximum allowed runtime for noisy tasks in ms

If both are specified, the smallest threshold (in seconds) will be used.

Required trace events:
 
  • cpu_idle
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup

Sanity tests

Inheritance diagram of lisa.tests.scheduler.sanity

class lisa.tests.scheduler.sanity.CapacitySanity(res_dir, plat_info, capacity_work)

Bases: lisa.tests.base.TestBundle

A class for making sure capacity values make sense on a given target

Parameters:

capacity_work (dict) – A description of the amount of work done on the target, per capacity value ({capacity : work})

Test methods:
test_capacity_sanity() → lisa.tests.base.ResultBundle

Assert that higher CPU capacity means more work done

classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None) → lisa.tests.scheduler.sanity.CapacitySanity

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Factory method to create a bundle using a live target

Hotplug tests

Inheritance diagram of lisa.tests.hotplug.torture

exception lisa.tests.hotplug.torture.CPUHPSequenceError

Bases: Exception

class lisa.tests.hotplug.torture.HotplugBase(plat_info, target_alive, hotpluggable_cpus, live_cpus)

Bases: lisa.tests.base.TestBundle

Test methods:
classmethod cpuhp_seq(nr_operations, hotpluggable_cpus, max_cpus_off, random_gen)

Yield a consistent random sequence of CPU hotplug operations

Parameters:
  • nr_operations – Number of operations in the sequence
  • max_cpus_off – Max number of CPUs plugged-off
  • random_gen (random.Random) – A random generator instance

“Consistent” means that a CPU will be plugged-in only if it was plugged-off before (and vice versa). Moreover the state of the CPUs once the sequence has completed should the same as it was before.

test_target_alive() → lisa.tests.base.ResultBundle

Test that the hotplugs didn’t leave the target in an unusable state

test_cpus_alive() → lisa.tests.base.ResultBundle

Test that all CPUs came back online after the hotplug operations

classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, seed=None, nr_operations=100, sleep_min_ms=10, sleep_max_ms=100, max_cpus_off=9223372036854775807) → lisa.tests.hotplug.torture.HotplugBase

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Parameters:
  • seed (int) – Seed of the RNG used to create the hotplug sequences
  • nr_operations (int) – Number of operations in the sequence
  • sleep_min_ms (int) – Minimum sleep duration between hotplug operations
  • sleep_max_ms (int) – Maximum sleep duration between hotplug operations (0 would lead to no sleep)
  • max_cpus_off (int) – Maximum number of CPUs hotplugged out at any given moment
class lisa.tests.hotplug.torture.HotplugTorture(plat_info, target_alive, hotpluggable_cpus, live_cpus)

Bases: lisa.tests.hotplug.torture.HotplugBase

Test methods:
classmethod cpuhp_seq(nr_operations, hotpluggable_cpus, max_cpus_off, random_gen)

FIXME: is that actually still true ? The actual length of the sequence might differ from the requested one by 1 because it’s easier to implement and it shouldn’t be an issue for most test cases.

Cpufreq tests

Inheritance diagram of lisa.tests.cpufreq.sanity

class lisa.tests.cpufreq.sanity.UserspaceSanityItem(res_dir, plat_info, cpu, freq, work)

Bases: lisa.tests.base.TestBundle

Record the number of sysbench events on a given CPU at a given frequency.

classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, cpu, freq, switch_governor=True) → lisa.tests.cpufreq.sanity.UserspaceSanityItem

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Create a UserspaceSanityItem from a live lisa.target.Target.

Parameters:
  • cpu (int) – CPU to run on.
  • freq (int) – Frequency to run at.
  • switch_governor (bool) – Switch the governor to userspace, and undo it at the end. If that has been done in advance, not doing it for every item saves substantial time.
class lisa.tests.cpufreq.sanity.UserspaceSanity(res_dir, plat_info, sanity_items)

Bases: lisa.tests.base.DmesgTestBundle

A class for making sure the userspace governor behaves sanely

Parameters:

sanity_items (list(UserspaceSanityItem)) – A list of UserspaceSanityItem.

Test methods:
DMESG_IGNORED_PATTERNS = ['started with executable stack', 'Disabling EAS, schedutil is mandatory']
test_performance_sanity() → lisa.tests.base.ResultBundle

Assert that higher CPU frequency leads to more work done

classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, freq_count_limit=5) → lisa.tests.cpufreq.sanity.UserspaceSanity

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Factory method to create a bundle using a live target

Parameters:freq_count_limit (int) – The maximum amount of frequencies to test

This will run Sysbench at different frequencies using the userspace governor

Staged tests

Those are tests that have been merged into LISA but whose behaviour are being actively evaluated.

class lisa.tests.staging.sched_android.SchedTuneItemBase(res_dir, plat_info, boost, prefer_idle)

Bases: lisa.tests.base.RTATestBundle

Abstract class enabling rtapp execution in a schedtune group

Parameters:
  • boost (int) – The boost level to set for the cgroup
  • prefer_idle (bool) – The prefer_idle flag to set for the cgroup
Test methods:
cgroup_configuration

Compute the cgroup configuration based on plat_info

classmethod get_cgroup_configuration(plat_info, boost, prefer_idle)
Returns:a dict representing the configuration of a particular cgroup.

This is a method you may optionally override to configure a cgroup for the synthetic workload.

Example of return value:

{
    'name': 'lisa_test',
    'controller': 'schedtune',
    'attributes' : {
        'prefer_idle' : 1,
        'boost': 50
    }
}
classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, ftrace_coll: lisa.trace.FtraceCollector = None, boost, prefer_idle)

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Factory method to create a bundle using a live target

This will execute the rt-app workload described in get_rtapp_profile()

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.staging.sched_android.SchedTuneBase(res_dir, plat_info, test_bundles)

Bases: lisa.tests.base.TestBundle

Abstract class enabling the aggregation of SchedTuneItemBase

Parameters:test_bundles (list) – a list of test bundles generated by multiple SchedTuneItemBase instances
classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, ftrace_coll: lisa.trace.FtraceCollector = None) → lisa.tests.staging.sched_android.SchedTuneBase

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Creates a SchedTuneBase bundle from the target.

class lisa.tests.staging.sched_android.SchedTuneFreqItem(res_dir, plat_info, boost, prefer_idle)

Bases: lisa.tests.staging.sched_android.SchedTuneItemBase

Runs a tiny RT rtapp task pinned to a big CPU at a given boost level and checks the frequency selection was performed accordingly.

Test methods:
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

trace_window(trace)

Set the boundaries of the trace window to cpu_frequency events before/after the task’s start/end time

Required trace events:
 
  • cpu_frequency or userspace@cpu_frequency_devlib
  • cpu_frequency
  • sched_switch
  • userspace@rtapp_loop
test_stune_frequency(freq_margin_pct=10) → lisa.tests.base.ResultBundle

Test that frequency selection followed the boost

Param:freq_margin_pct: Allowed margin between estimated and measured average frequencies

Compute the expected frequency given the boost level and compare to the real average frequency from the trace. Check that the difference between expected and measured frequencies is no larger than freq_margin_pct.

Required trace events:
 
  • cpu_frequency or userspace@cpu_frequency_devlib
ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.staging.sched_android.SchedTuneFrequencyTest(res_dir, plat_info, test_bundles)

Bases: lisa.tests.staging.sched_android.SchedTuneBase

Runs multiple SchedTuneFreqItem tests at various boost levels ranging from 20% to 100%, then checks all succedeed.

Test methods:
ftrace_conf = <lisa.trace.FtraceConf object>
test_stune_frequency(freq_margin_pct=10) → lisa.tests.base.AggregatedResultBundle
class lisa.tests.staging.sched_android.SchedTunePlacementItem(res_dir, plat_info, boost, prefer_idle)

Bases: lisa.tests.staging.sched_android.SchedTuneItemBase

Runs a tiny RT-App task marked ‘prefer_idle’ at a given boost level and tests if it was placed on big-enough CPUs.

Test methods:
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

test_stune_task_placement(bad_cpu_margin_pct=10) → lisa.tests.base.ResultBundle

Test that the task placement satisfied the boost requirement

Check that top-app tasks spend no more than bad_cpu_margin_pct of their time on CPUs that don’t have enough capacity to serve their boost.

Required trace events:
 
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.staging.sched_android.SchedTunePlacementTest(res_dir, plat_info, test_bundles)

Bases: lisa.tests.staging.sched_android.SchedTuneBase

Runs multiple SchedTunePlacementItem tests with prefer_idle set and typical top-app boost levels, then checks all succedeed.

Test methods:
ftrace_conf = <lisa.trace.FtraceConf object>
test_stune_task_placement(margin_pct=10) → lisa.tests.base.AggregatedResultBundle
class lisa.tests.staging.schedutil.RampBoostTestBase(res_dir, plat_info, cpu)

Bases: lisa.tests.base.RTATestBundle

Test schedutil’s ramp boost feature.

Test methods:
estimate_nrg()
Required trace events:
 
  • cpu_frequency
  • cpu_idle
  • sched_wakeup
get_avg_slack(only_negative=False)
df_ramp_boost()

Return a dataframe with schedutil-related signals, sampled at the frequency decisions timestamps for the CPU this bundle was executed on.

Note

The computed columns only take into account the CPU the test was executing on. It currently does not handle multi-task workloads.

Required trace events:
 
  • optional: (sched_load_avg_cpu or sched_load_cfs_rq or sched_pelt_cfs) , cpu_capacity , sched_util_est_cfs
  • schedutil_em
test_ramp_boost(cost_threshold_pct=0.1, bad_samples_threshold_pct=0.1) → lisa.tests.base.ResultBundle

Test that the energy boost feature is triggering as expected.

Required trace events:
 
  • cpu_frequency or userspace@cpu_frequency_devlib
  • optional: (sched_load_avg_cpu or sched_load_cfs_rq or sched_pelt_cfs) , cpu_capacity , sched_util_est_cfs
  • optional: (sched_load_avg_task or sched_load_se or sched_pelt_se) , sched_util_est_se
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
  • schedutil_em
  • userspace@rtapp_stats
ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.staging.schedutil.LargeStepUp(res_dir, plat_info, cpu, nr_steps)

Bases: lisa.tests.staging.schedutil.RampBoostTestBase

A single task whose utilization rises extremely quickly

Test methods:
task_name = 'step_up'
rtapp_profile

Compute the RTapp profile based on plat_info.

classmethod get_rtapp_profile(plat_info, cpu, nr_steps, min_util=5, max_util=75)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

classmethod from_target(target: lisa.target.Target, *, res_dir: lisa.utils.ArtifactPath = None, ftrace_coll: lisa.trace.FtraceCollector = None, cpu=None, nr_steps=1) → lisa.tests.staging.schedutil.LargeStepUp

Factory method to create a bundle using a live target

Parameters:

This is mostly boiler-plate code around _from_target(), which lets us introduce common functionalities for daughter classes. Unless you know what you are doing, you should not override this method, but the internal lisa.tests.base.TestBundle._from_target() instead.

(above inherited from lisa.tests.base.TestBundle.from_target())

Internals of the target factory method.

Note

This must be a classmethod, and all parameters except target must be keyword-only, i.e. appearing after args* or a lonely *:

@classmethod
def _from_target(cls, target, *, foo=33, bar):
    ...

(above inherited from lisa.tests.base.TestBundle.from_target())

Factory method to create a bundle using a live target

This will execute the rt-app workload described in get_rtapp_profile()

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.staging.numa_behaviour.NUMABehaviour(res_dir, plat_info)

Bases: lisa.tests.base.RTATestBundle

Abstract class for NUMA related scheduler testing.

Test methods:
classmethod check_from_target(target)

Check whether the given target can be used to create an instance of this class

Raises:CannotCreateError if the check fails

This method should be overriden to check your implementation requirements

test_task_remains() → lisa.tests.base.ResultBundle

Test that task remains on the same core

Required trace events:
 
  • optional: sched_wakeup_new
  • sched_switch
  • sched_wakeup
ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.staging.numa_behaviour.NUMASmallTaskPlacement(res_dir, plat_info)

Bases: lisa.tests.staging.numa_behaviour.NUMABehaviour

A single task with 50% utilization

Test methods:
task_prefix = 'tsk'
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

ftrace_conf = <lisa.trace.FtraceConf object>
class lisa.tests.staging.numa_behaviour.NUMAMultipleTasksPlacement(res_dir, plat_info)

Bases: lisa.tests.staging.numa_behaviour.NUMABehaviour

Multiple tasks with 50% utilization

Test methods:
task_prefix = 'tsk'
classmethod get_rtapp_profile(plat_info)
Returns:a dict with task names as keys and lisa.wlgen.rta.RTATask as values

This is the method you want to override to specify what is your synthetic workload.

ftrace_conf = <lisa.trace.FtraceConf object>