Synthetic test example

import logging
from lisa.utils import setup_logging
setup_logging()

import holoviews as hv
hv.extension('bokeh')

2021-07-01 17:29:37,151 INFO    : root         : Using LISA logging configuration: /home/dourai01/Work/lisa/lisa/logging.conf

Target configuration

from lisa.target import Target, TargetConf

# target = Target(
#     kind='linux',
#     name='myhikey960',
#     host='192.168.0.1',
#     username='root',
#     password='root',
# )
# Uses settings from target_conf.yml
target = Target.from_default_conf()

2021-07-01 17:25:16,435 INFO    : lisa.target.Target : Target configuration:
+- devlib:
    |- excluded-modules from default (list): []
    +- platform:
        |- class from default (str): devlib.platform.Platform
|- host from user (str): 10.7.6.218
|- kind from user (str): linux
|- lazy-platinfo from default (bool): False
|- name from user (str): juno-r0
|- password from user (str): <password>
|- strict-host-check from user (bool): False
|- tools from default (list): []
|- username from user (str): root
+- wait-boot:
    |- enable from default (bool): True
    |- timeout from default (int): 10
2021-07-01 17:25:16,541 INFO    : lisa.target.Target : Creating result directory: /home/dourai01/Work/lisa/lisa/results/Target-juno-r0-20210701_172516.541661
2021-07-01 17:25:16,542 INFO    : lisa.target.Target : User-defined platform information:

2021-07-01 17:25:22,004 INFO    : lisa.target.Target : Connected to target juno-r0
2021-07-01 17:25:22,421 INFO    : lisa.energy_model.EnergyModel.from_target : Attempting to load EM using LinuxEnergyModel
2021-07-01 17:25:24,041 INFO    : lisa.target.Target : Loading target devlib module cpuidle
2021-07-01 17:25:26,428 INFO    : lisa.target.Target : Loading target devlib module cpufreq
2021-07-01 17:25:28,027 INFO    : lisa.target.Target : Loading target devlib module sched
2021-07-01 17:25:29,836 INFO    : sched        : Scheduler sched_domain procfs entries not found
2021-07-01 17:25:29,838 INFO    : sched        : Detected kernel compiled with SCHED_DEBUG=n
2021-07-01 17:25:29,843 INFO    : sched        : CPU capacity sysfs entries found
2021-07-01 17:25:33,699 INFO    : lisa.target.Target : Effective platform information:
|- abi from target (str): arm64
+- cpu-capacities:
    |- orig from target (dict): {0: 446, 1: 1024, 2: 1024, 3: 446, 4: 446, 5: 446}
    |- writeable from target (bool): True
    |- rtapp from target(platform-info/rtapp/calib),target(platform-info/cpu-capacities/orig),target(platform-info/cpu-capacities/writeable) (str): <depends on lazy keys: platform-info/rtapp/calib>
|- cpus-count from target (int): 6
|- freq-domains from target (list): [[0, 3, 4, 5], [1, 2]]
|- freqs from target (dict): {0: [450000, 575000, 700000, 775000, 850000], 1: [450000, 625000, 800000, 950000, 1100000], 2: [450000, 625000, 800000, 950000, 1100000], 3: [450000, 575000, 700000, 775000, 850000], 4: [450000, 575000, 700000, 775000, 850000], 5: [450000, 575000, 700000, 775000, 850000]}
+- kernel:
    |- config from target (TypedKernelConfig): <kernel config>
    |- symbols-address from target (DeferredValue): <symbols address>
    |- version from target (KernelVersion): 5.13.0-rc3-g5c95b23cc7dd 1 SMP PREEMPT Thu Jul 1 16:07:56 BST 2021
|- name from target-conf (str): juno-r0
|- nrg-model from target (LinuxEnergyModel): <lisa.energy_model.LinuxEnergyModel object at 0x7f8cd41111f0>
|- numa-nodes-count from target (int): 1
|- os from target (str): linux
+- rtapp:
    |- calib from target (DeferredValue): <lazy value of RTA.get_cpu_calibrations>
|- capacity-classes from target(platform-info/cpu-capacities/orig) (list): [[0, 3, 4, 5], [1, 2]]

Live target experimentation

from lisa.tests.scheduler.eas_behaviour import EnergyModelWakeMigration

We want to create a TestBundle from a live target, so we can use the from_target alternative constructor:

bundle = EnergyModelWakeMigration.from_target(target)

2021-07-01 17:25:35,657 INFO    : lisa.target.Target : Creating result directory: /home/dourai01/Work/lisa/lisa/results/Target-juno-r0-20210701_172516.541661/EnergyModelWakeMigration-20210701_172535.656944
2021-07-01 17:25:36,547 INFO    : lisa.trace.FtraceCollector : Ftrace configuration:
|- buffer-size from user+EnergyModelWakeMigration (int): 10240
|- events from EnergyModelWakeMigration (list): ['sched_switch', 'sched_wakeup', 'sched_wakeup_new', 'task_rename', 'userspace@rtapp_loop', 'userspace@rtapp_stats']
|- saved-cmdlines-nr from user+EnergyModelWakeMigration (int): 8192
2021-07-01 17:25:44,154 INFO    : lisa.target.Target : Disabling idle states for all domains
2021-07-01 17:25:50,544 INFO    : lisa.wlgen.rta.RTA : Created workload's run target directory: /root/devlib-target/lisa/wlgen/20210701_172550_b9c2870159b745d79bc4aa2e022bde65
2021-07-01 17:25:51,202 INFO    : lisa.target.Target : Loading target devlib module cgroups
2021-07-01 17:25:51,404 INFO    : CGroups      : Available controllers:
2021-07-01 17:25:51,843 INFO    : CGroups      :   cpuset       : /root/devlib-target/cgroups/devlib_cgh1
2021-07-01 17:25:52,307 INFO    : CGroups      :   cpu          : /root/devlib-target/cgroups/devlib_cgh1
2021-07-01 17:25:52,788 INFO    : CGroups      :   cpuacct      : /root/devlib-target/cgroups/devlib_cgh1
2021-07-01 17:25:53,255 INFO    : CGroups      :   blkio        : /root/devlib-target/cgroups/devlib_cgh1
2021-07-01 17:25:53,769 INFO    : CGroups      :   memory       : /root/devlib-target/cgroups/devlib_cgh1
2021-07-01 17:25:54,249 INFO    : CGroups      :   devices      : /root/devlib-target/cgroups/devlib_cgh1
2021-07-01 17:25:54,741 INFO    : CGroups      :   freezer      : /root/devlib-target/cgroups/devlib_cgh1
2021-07-01 17:25:55,292 INFO    : CGroups      :   perf_event   : /root/devlib-target/cgroups/devlib_cgh1
2021-07-01 17:25:55,822 INFO    : CGroups      :   hugetlb      : /root/devlib-target/cgroups/devlib_cgh1
2021-07-01 17:25:56,617 INFO    : CGroups      :   pids         : /root/devlib-target/cgroups/devlib_cgh1
2021-07-01 17:25:56,742 INFO    : lisa.target.Target : Freezing all tasks except: init,systemd[^-],dbus,sh,ssh,rsyslogd,jbd2
2021-07-01 17:26:00,369 INFO    : lisa.wlgen.rta.RTA : Execution start: rt-app /root/devlib-target/lisa/wlgen/20210701_172550_b9c2870159b745d79bc4aa2e022bde65/rta_calib_cpu0.json 2>&1
2021-07-01 17:26:06,952 INFO    : lisa.target.Target : Un-freezing userspace tasks
2021-07-01 17:26:07,580 INFO    : lisa.wlgen.rta.RTA : Wiping target run directory: /root/devlib-target/lisa/wlgen/20210701_172550_b9c2870159b745d79bc4aa2e022bde65
2021-07-01 17:26:07,880 INFO    : lisa.wlgen.rta.RTA : CPU0 calibration=335
2021-07-01 17:26:08,224 INFO    : lisa.wlgen.rta.RTA : Created workload's run target directory: /root/devlib-target/lisa/wlgen/20210701_172607_1ceb744c7c7e4f878053d4fede5bfe2a
2021-07-01 17:26:08,646 INFO    : lisa.target.Target : Freezing all tasks except: init,systemd[^-],dbus,sh,ssh,rsyslogd,jbd2
2021-07-01 17:26:11,984 INFO    : lisa.wlgen.rta.RTA : Execution start: rt-app /root/devlib-target/lisa/wlgen/20210701_172607_1ceb744c7c7e4f878053d4fede5bfe2a/rta_calib_cpu1.json 2>&1
2021-07-01 17:26:18,459 INFO    : lisa.target.Target : Un-freezing userspace tasks
2021-07-01 17:26:19,095 INFO    : lisa.wlgen.rta.RTA : Wiping target run directory: /root/devlib-target/lisa/wlgen/20210701_172607_1ceb744c7c7e4f878053d4fede5bfe2a
2021-07-01 17:26:19,393 INFO    : lisa.wlgen.rta.RTA : CPU1 calibration=204
2021-07-01 17:26:19,878 INFO    : lisa.wlgen.rta.RTA : Created workload's run target directory: /root/devlib-target/lisa/wlgen/20210701_172619_a348cc3bac984f659de36482b81fcb09
2021-07-01 17:26:20,227 INFO    : lisa.target.Target : Freezing all tasks except: init,systemd[^-],dbus,sh,ssh,rsyslogd,jbd2
2021-07-01 17:26:23,584 INFO    : lisa.wlgen.rta.RTA : Execution start: rt-app /root/devlib-target/lisa/wlgen/20210701_172619_a348cc3bac984f659de36482b81fcb09/rta_calib_cpu2.json 2>&1
2021-07-01 17:26:31,135 INFO    : lisa.target.Target : Un-freezing userspace tasks
2021-07-01 17:26:31,824 INFO    : lisa.wlgen.rta.RTA : Wiping target run directory: /root/devlib-target/lisa/wlgen/20210701_172619_a348cc3bac984f659de36482b81fcb09
2021-07-01 17:26:32,074 INFO    : lisa.wlgen.rta.RTA : CPU2 calibration=194
2021-07-01 17:26:32,566 INFO    : lisa.wlgen.rta.RTA : Created workload's run target directory: /root/devlib-target/lisa/wlgen/20210701_172632_860c8ce187844f47920f7f2701d25551
2021-07-01 17:26:32,959 INFO    : lisa.target.Target : Freezing all tasks except: init,systemd[^-],dbus,sh,ssh,rsyslogd,jbd2
2021-07-01 17:26:36,298 INFO    : lisa.wlgen.rta.RTA : Execution start: rt-app /root/devlib-target/lisa/wlgen/20210701_172632_860c8ce187844f47920f7f2701d25551/rta_calib_cpu3.json 2>&1
2021-07-01 17:26:42,900 INFO    : lisa.target.Target : Un-freezing userspace tasks
2021-07-01 17:26:43,750 INFO    : lisa.wlgen.rta.RTA : Wiping target run directory: /root/devlib-target/lisa/wlgen/20210701_172632_860c8ce187844f47920f7f2701d25551
2021-07-01 17:26:44,028 INFO    : lisa.wlgen.rta.RTA : CPU3 calibration=335
2021-07-01 17:26:44,589 INFO    : lisa.wlgen.rta.RTA : Created workload's run target directory: /root/devlib-target/lisa/wlgen/20210701_172644_6dc247bcbc7e473ca7e0696b242f7946
2021-07-01 17:26:44,941 INFO    : lisa.target.Target : Freezing all tasks except: init,systemd[^-],dbus,sh,ssh,rsyslogd,jbd2
2021-07-01 17:26:48,374 INFO    : lisa.wlgen.rta.RTA : Execution start: rt-app /root/devlib-target/lisa/wlgen/20210701_172644_6dc247bcbc7e473ca7e0696b242f7946/rta_calib_cpu4.json 2>&1
2021-07-01 17:26:54,984 INFO    : lisa.target.Target : Un-freezing userspace tasks
2021-07-01 17:26:55,828 INFO    : lisa.wlgen.rta.RTA : Wiping target run directory: /root/devlib-target/lisa/wlgen/20210701_172644_6dc247bcbc7e473ca7e0696b242f7946
2021-07-01 17:26:56,167 INFO    : lisa.wlgen.rta.RTA : CPU4 calibration=335
2021-07-01 17:26:56,495 INFO    : lisa.wlgen.rta.RTA : Created workload's run target directory: /root/devlib-target/lisa/wlgen/20210701_172656_a0da6bd975794f639744c68b81709aa3
2021-07-01 17:26:56,861 INFO    : lisa.target.Target : Freezing all tasks except: init,systemd[^-],dbus,sh,ssh,rsyslogd,jbd2
2021-07-01 17:27:00,118 INFO    : lisa.wlgen.rta.RTA : Execution start: rt-app /root/devlib-target/lisa/wlgen/20210701_172656_a0da6bd975794f639744c68b81709aa3/rta_calib_cpu5.json 2>&1
2021-07-01 17:27:06,706 INFO    : lisa.target.Target : Un-freezing userspace tasks
2021-07-01 17:27:07,421 INFO    : lisa.wlgen.rta.RTA : Wiping target run directory: /root/devlib-target/lisa/wlgen/20210701_172656_a0da6bd975794f639744c68b81709aa3
2021-07-01 17:27:07,709 INFO    : lisa.wlgen.rta.RTA : CPU5 calibration=335
2021-07-01 17:27:07,760 INFO    : lisa.wlgen.rta.RTA : Platform info rt-app calibration configuration:
platform-info:
    conf:
        cpu-capacities: {}
        kernel: {}
        rtapp:
            calib:
                0: 335
                1: 204
                2: 194
                3: 335
                4: 335
                5: 335

2021-07-01 17:27:07,793 INFO    : lisa.wlgen.rta.RTA : CPU capacities according to rt-app workload: {0: 608, 3: 608, 4: 608, 5: 608, 1: 1024, 2: 1024}
2021-07-01 17:27:07,794 WARNING : lisa.wlgen.rta.RTA : The calibration values are not inversely proportional to the CPU capacities, the duty cycles will be up to 36.32% off on some CPUs: {0: 136.32286995515693, 1: 100.0, 2: 100.0, 3: 136.32286995515693, 4: 136.32286995515693, 5: 136.32286995515693}
2021-07-01 17:27:07,795 WARNING : lisa.wlgen.rta.RTA : The calibration values are not inversely proportional to the CPU capacities. Either rt-app calibration failed, or the rt-app busy loops has a very different instruction mix compared to the workload used to establish the CPU capacities: {0: 136.32286995515693, 1: 100.0, 2: 100.0, 3: 136.32286995515693, 4: 136.32286995515693, 5: 136.32286995515693}
2021-07-01 17:27:07,795 INFO    : lisa.target.Target : Re-enabling idle states for all domains
2021-07-01 17:27:15,898 INFO    : lisa.target.Target : Disabling idle states for all domains
2021-07-01 17:27:28,787 INFO    : lisa.tests.scheduler.eas_behaviour.EnergyModelWakeMigration : rt-app workload:
Task emwm_0:
    Phase buffer:
        cpus=None
        meta={'from_test': False}
        numa_nodes_membind=None
        wload=PeriodicWload(duty_cycle_pct=20, period=0.016, duration=0.5, scale_for_cpu=0)

    Phase test:
        Phase test/0:
            meta={'from_test': True}
            wload=PeriodicWload(duty_cycle_pct=20, period=0.016, duration=2, scale_for_cpu=0)

        Phase test/1:
            meta={'from_test': True}
            wload=PeriodicWload(duty_cycle_pct=59.375, period=0.016, duration=2)

        Phase test/2:
            meta={'from_test': True}
            wload=PeriodicWload(duty_cycle_pct=20, period=0.016, duration=2, scale_for_cpu=0)

        Phase test/3:
            meta={'from_test': True}
            wload=PeriodicWload(duty_cycle_pct=59.375, period=0.016, duration=2)
Task emwm_1:
    Phase buffer:
        cpus=None
        meta={'from_test': False}
        numa_nodes_membind=None
        wload=PeriodicWload(duty_cycle_pct=20, period=0.016, duration=0.5, scale_for_cpu=0)

    Phase test:
        Phase test/0:
            meta={'from_test': True}
            wload=PeriodicWload(duty_cycle_pct=20, period=0.016, duration=2, scale_for_cpu=0)

        Phase test/1:
            meta={'from_test': True}
            wload=PeriodicWload(duty_cycle_pct=59.375, period=0.016, duration=2)

        Phase test/2:
            meta={'from_test': True}
            wload=PeriodicWload(duty_cycle_pct=20, period=0.016, duration=2, scale_for_cpu=0)

        Phase test/3:
            meta={'from_test': True}
            wload=PeriodicWload(duty_cycle_pct=59.375, period=0.016, duration=2)
2021-07-01 17:27:28,907 INFO    : lisa.target.Target : Freezing all tasks except: init,systemd[^-],dbus,sh,ssh,rsyslogd,jbd2
2021-07-01 17:27:33,218 INFO    : lisa.wlgen.rta.RTA : Created workload's run target directory: /root/devlib-target/lisa/wlgen/20210701_172727_0c93c83c4ffd403ea6a84f5fae986b65
2021-07-01 17:27:33,580 INFO    : lisa.wlgen.rta.RTA : Will update CPU capacities in sysfs: {0: 608, 3: 608, 4: 608, 5: 608, 1: 1024, 2: 1024}
2021-07-01 17:27:44,257 INFO    : lisa.wlgen.rta.RTA : Execution start: rt-app /root/devlib-target/lisa/wlgen/20210701_172727_0c93c83c4ffd403ea6a84f5fae986b65/rta_energymodelwakemigration.json 2>&1
2021-07-01 17:28:04,391 INFO    : lisa.wlgen.rta.RTA : Wiping target run directory: /root/devlib-target/lisa/wlgen/20210701_172727_0c93c83c4ffd403ea6a84f5fae986b65
2021-07-01 17:28:04,715 INFO    : lisa.target.Target : Un-freezing userspace tasks
2021-07-01 17:28:07,241 INFO    : lisa.target.Target : Re-enabling idle states for all domains

We now have a Target instance. We’re free to play around with its data, more specifically to execute some tests related to this data:

print(bundle.test_slack())

PASSED: {emwm_0-0 delayed activations=1.1976047904191616 %, emwm_1-1 delayed activations=1.1976047904191616 %}

Whenever possible, we want to expose test margins to the end user. Here, we can change the failure threshold, and for the sake of demonstration we’ll set it way too low (and expect a failure)

print(bundle.test_slack(negative_slack_allowed_pct=1))

FAILED: {emwm_0-0 delayed activations=1.1976047904191616 %, emwm_1-1 delayed activations=1.1976047904191616 %}

This test also produces a plot, let’s run it and have a look

print(bundle.test_task_placement())

2021-07-01 17:28:39,856 INFO    : lisa.tests.scheduler.eas_behaviour.EnergyModelWakeMigration : Ignored PIDs for noise contribution: [10103:emwm_0-0], [10104:emwm_1-1], [0]

PASSED: {energy threshold=2767.3222838127895 bogo-joules, estimated energy=2588.307295510689 bogo-joules, noisiest task={comm=kworker/u12:1, duration (abs)=0.0030110400007288263 s, duration (rel)=0.037382648575806546 %, pid=9506}}

bundle.trace.df_all_events()

2021-07-01 17:28:40,600 WARNING : py.warnings  : /tmp/ipykernel_108817/2525158218.py:1: DeprecationWarning: lisa.trace.TraceBase.df_all_events is deprecated and will be removed in version 3.0, use lisa.analysis.notebook.NotebookAnalysis.df_all_event instead: This method has been deprecated and is an alias for "trace.ana.notebook.df_all_events()"
  bundle.trace.df_all_events()

	info
Time
0.003987	sched_switch : __comm=sshd __pid=8159 ...
0.004115	sched_switch : __comm=trace-cmd __pid=...
0.020523	sched_wakeup : __comm=<idle> __pid=0 _...
0.020687	sched_switch : __comm=kworker/0:1 __pi...
0.021014	sched_switch : __comm=ksoftirqd/0 __pi...
...	...
9.552784	sched_wakeup : __comm=rt-app __pid=101...
9.552805	sched_switch : __comm=rt-app __pid=101...
9.552809	sched_switch : __comm=emwm_0-0 __pid=1...
9.552830	print : __comm=emwm_1-1 __pid=1...
9.552830	userspace@rtapp_loop : event=end phase_loop=12...

9244 rows × 1 columns

import os
from IPython.display import IFrame

image_path = os.path.join(bundle.res_dir, "expected_placement.html")
IFrame(src=image_path)

Offline replaying

Say someone executed some tests, and ran into some issues. They can create an archive of the results directory and send it your way, and you’ll be able to reload it automagically. This also works if you are working on the test code itself, and don’t want to re-execute the workload every single time you change the code.

# Let's just reuse what was generated in the previous run
archive_dir = bundle.res_dir

reloaded_bundle = EnergyModelWakeMigration.from_dir(archive_dir)

# Just to prove we're not cheating!
reloaded_bundle == bundle

False

print(reloaded_bundle.test_task_placement())

PASSED: {energy threshold=2767.3222838127895 bogo-joules, estimated energy=2588.307295510689 bogo-joules, noisiest task={comm=kworker/u12:1, duration (abs)=0.0030110400007288263 s, duration (rel)=0.037382648575806546 %, pid=9506}}