APIs stability

APIs inside LISA are split between private and public ones:

  • Public APIs can be expected to stay stable, or undergo a deprecation cycle where they will trigger an DeprecationWarning and be documented as such before being removed. Exceptions to that rule are documented explicitly as such.

  • Private APIs can be changed at all points.

Public APIs consist of classes and functions with names not starting with an underscore, defined in modules with a name not starting with an underscore (or any of its parent modules or containing class).

Everything else is private.

Note

User subclassing is usually more at risk of breakage than other uses of the APIs. Behaviors are usually not restricted to a single method, which means the subclass would have to override multiple of them to preserve important API laws. This is unfortunately not future-proof, as new versions can add new methods that would also require being overridden and kept in sync. If for some reason subclassing is required, please get in touch in the GitLab issue tracker before relying on that for production.

Note

Instance attributes are considered public following the same convention as functions and classes. Only reading from them is expected in user code though, any attempt to modify or delete them is outside of the bounds of what the public API exposes (unless stated explicitly otherwise). This means that a minor version change could swap an instance attribute for a read-only property. It also means that any problem following the modification of an attribute by a user will not be considered as a bug.

Versioning

LISA releases on PyPI are done following semantic versioning as defined in https://semver.org/. As pointed by api-stability, classes are split on the following axes for the purpose of semver tracking:

  • A set of methods and attributes in general: Adding a method entails a minor version bump, even though it can technically cause a breaking change in a user subclass that happened to use the same name.

  • Inheritance tree: the MRO of a class is not considered as part of the stable public API and can therefore change at any point. Classes named *Base can usually be relied on for issubclass() and isinstance() but that is not a hard rule. The reason behind that is that even adding a class to the hierarchy can break existing uses of isinstance() so there is essentially no way of making any change to the inheritance tree that is not a breaking change.

Changelog

Next release

Feature

  • doc/setup.rst: Put the config fragment in a separate file

  • doc/setup.rst: Remove manual and in-tree module build doc

  • install_base.sh: Add --install-kernel-build-dependencies

  • lisa-build-asset: Add x86 and ppc64le arch

  • lisa-buildroot-update-kernel-config: Do not run make command

  • lisa._assets.binaries: Remove pahole

  • lisa._assets.kmodules.lisa: Add DEFINE_TP_DEPRECATED_EVENT_FEATURE

  • lisa._assets.kmodules.lisa: Allow C type introspection

  • lisa._assets.kmodules.lisa: Allow attaching probe to multiple tracepoints

  • lisa._assets.kmodules.lisa: Check for exported symbols and types

  • lisa._assets.kmodules.lisa: Extend type introspection to the entire kernel

  • lisa._assets.kmodules.lisa: Log content of module sources

  • lisa._assets.kmodules.lisa: Make a synthetic vmlinux for in-tree builds

  • lisa._assets.kmodules.lisa: Reduce event sizes

  • lisa._assets.kmodules.lisa: Show compilation warning when introspection info is not available

  • lisa._assets.kmodules.sched_tp: Add "lisa__" prefix to all event names

  • lisa._assets.kmodules.sched_tp: Add device field to pixel6_emeter

  • lisa._assets.kmodules.sched_tp: Change subsystem of ftrace events

  • lisa._assets.kmodules.sched_tp: Extend pixel6_emeter to support all channels

  • lisa._assets.kmodules: Add kallsyms introspection support

  • lisa._assets: Add ABI_BINARIES_FOLDER

  • lisa._btf: Add BTF parser

  • lisa._cli_tools.lisa_load_kmod: Display events supported by the module

  • lisa._generic: Add OneOf

  • lisa._kmod: Add build-env settings

  • lisa._kmod: Add util_est_unified trace events

  • lisa._kmod: Capture dmesg output when loading kmod

  • lisa._kmod: Ensure kernel tree is clean

  • lisa._kmod: Give priority to toolchain that was used for the kernel

  • lisa._kmod: Improve toolchain detection logging

  • lisa._kmod: KmodSrc: Separate checksum and checksum_sources

  • lisa._kmod: LISAFtraceDynamicKmod: Detect modules built in-tree

  • lisa._kmod: Log KmodSrc file list

  • lisa._kmod: Refactor toolchain selection

  • lisa._kmod: Remove CONFIG_CFI_CLANG check in FtraceDynamicKmod

  • lisa._kmod: Rename LISAFtraceDynamicKmod to LISADynamicKmod

  • lisa._kmod: Use Alpine 3.18 for the chroot

  • lisa._kmod: Use rmmod -f if rmmod failed

  • lisa.analysis.base: Make TraceAnalysisBase.cache() more powerful

  • lisa.analysis.pixel6: Add GPU to emeter channel names

  • lisa.conf: Add DelegatedLevelKeyDesc

  • lisa.conf: Add FilteredDeferredValue

  • lisa.conf: Add MultiSrcConf.add_src(inplace=True) parameter

  • lisa.conf: Add VariadicLevelKeyDesc

  • lisa.conf: Allow LevelKeyDesc to accept leaf values

  • lisa.conf: Improve FilteredDeferredValue display

  • lisa.conf: Support typing.Union

  • lisa.conf: Use typeguard to validate conf key types

  • lisa.conf: Use unicode for MultiSrcConf display

  • lisa.datautils: Add series_update_duplicates()

  • lisa.datautils: Enable some functions for polars objects

  • lisa.target: Add Target._from_devlib_target()

  • lisa.target: Add Target.closing()

  • lisa.target: Allow repeated --conf in TargetConf.from_cli()

  • lisa.target: Forward kwargs to ctor in Target.from_conf()

  • lisa.target: Split module build conf

  • lisa.target: Use lisa._generic.OneOf in TargetConf

  • lisa.tests.staging.utilclamp: Add expected signal plot

  • lisa.trace: Add Trace(..., namespaces=("lisa", None)) parameter

  • lisa.trace: Add TraceBase.with_time_offset()

  • lisa.trace: Add TraceView(process_df=...) parameter

  • lisa.trace: Add _Trace._parse_all()

  • lisa.trace: Allow preloading all available events

  • lisa.trace: Allow str sources for meta events

  • lisa.trace: Do not apply event namespace if the event is already namespaces

  • lisa.trace: Gather the list of available events from the parser

  • lisa.trace: Generalize TraceCache to other data type

  • lisa.trace: Improve MissingTraceEventError display

  • lisa.trace: Make all parser metadata cacheable

  • lisa.trace: Make metadata writing non-blocking

  • lisa.trace: Only show closest matches in MissingTraceEventError

  • lisa.trace: Preload trace metadata

  • lisa.trace: Switch to TraceDumpTraceParser by default

  • lisa.utils.Serializable: Allow !call to take positional args

  • lisa.utils: Add !untrusted and !include-untrusted YAML tags to Serializable

  • lisa.utils: Add is_link_dead()

  • lisa.utils: Add mp_spawn_pool()

  • lisa.utils: Add subprocess_detailed_excep() context manager

  • lisa.utils: Add typing hint support to get_cls_name()

  • lisa.utils: Add update_params_from()

  • lisa.utils: Generalize group_by_value()

  • lisa.utils: Log the DirCache key and token upon lookup

  • lisa.utils: Make set_nested_key() return mapping

  • lisa.utils: Normalize DirCache keys

  • lisa.utils: Read LISA_CACHE_HOME env var

  • lisa.wa.plugins: Add LISA kmod workload-automation plugin

  • lisa.wlgen.rta: Add WloadPropertyBase.__rmul__()

  • lisa.wlgen.rta: Use ComposableMultiConcretePropertyBase._ATTRIBUTES types

  • lisa/_assets/kmodules/lisa: Add version check

  • lisa/_assets/kmodules/lisa: Support in-tree builds

  • lisa: Rename the module to "lisa" across all workflows

  • lisa_tests.test_example: Comment-out a return annotation to hide it to exekall

  • setup.py: Bump required Python to >= 3.8

  • tests: Add BTF tests

  • tools/batch-rebase: Add topics/*/nr-commits key

  • tools/buildroot_config: Sync rootfs to devlib busybox & add '-o nice' to ps

  • tools/exekall: Add run --dependency option

  • tools/exekall: Handle namespace packages

  • tools/kmodules: Add helpers for in-tree module builds

  • tools/lisa-buildroot-update-kernel-config: Apply the LISA module kernel fragment

  • tools/lisa-make-preview: Rename to lisa-combine-pr and add CLI args

  • tools/recipes/pahole.recipe: Update pahole recipe

  • tools/recipes/rt-app.recipe: Update rt-app

  • tools/trace-parser: Add Rust trace.dat parser

  • tools/trace-parser: Allow adjustable row-group size

  • tools/trace-parser: Allow changing compression on CLI

  • tools/trace-parser: Allow changing the chunk size on CLI

  • tools/trace-parser: Only compile-in compression codec that are used

  • tools/trace-parser: Scan the trace for metadata

Fix

  • Fix a couple of GroupBy.apply() deprecations

  • doc/conf.py: Ensure we fetch tags when building doc

  • doc/conf.py: prepare() only once per build

  • doc/kernel_tests.rst: Fix reStructuredText warning

  • doc/setup.rst: Update the manual module build documentation

  • doc/traces: Amend trace_pixel6.dat to avoid dropped events

  • doc: Use best effort import of lisa submodules

  • doc: re-enable Sphinx nitpicky mode

  • exekall: Fix handling of parametrized classes

  • exekall: Only consider callables in the packages under test

  • install_base.sh: follow symlink when trying determining JAVA_HOME

  • lisa._assets.binaries: Update pahole

  • lisa._assets.binaries: Update trace-cmd

  • lisa._assets.kmodules.kconfig_fragment.config: Add CONFIG_FTRACE=y

  • lisa._assets.kmodules.lisa: Adapt to new locations of some macros

  • lisa._assets.kmodules.lisa: Add include guard to private_types.h

  • lisa._assets.kmodules.lisa: Allow module-specific make variables

  • lisa._assets.kmodules.lisa: Create temp file in $(MODULE_OBJ)

  • lisa._assets.kmodules.lisa: Ensure warnings dont break build

  • lisa._assets.kmodules.lisa: Fail module init if unknown feature is requested

  • lisa._assets.kmodules.lisa: Fix conditional presence of lisa__sched_cpu_capacity.capacity_orig field

  • lisa._assets.kmodules.lisa: Fix exported symbols detection

  • lisa._assets.kmodules.lisa: Fix introspect_header.py

  • lisa._assets.kmodules.lisa: Fix lisa__sched_overutilized.overutilized field type

  • lisa._assets.kmodules.lisa: Fix module checksum

  • lisa._assets.kmodules.lisa: Fix rq->uclamp guards

  • lisa._assets.kmodules.lisa: Fix sha1sum invocation

  • lisa._assets.kmodules.lisa: Harden current CPU capacity computation

  • lisa._assets.kmodules.lisa: Load all features by default

  • lisa._assets.kmodules.lisa: Make introspect_header.py a no-op when imported

  • lisa._assets.kmodules.lisa: Refcount explicit feature enabling

  • lisa._assets.kmodules.lisa: Rename CFS_UCLAMP into RQ_UCLAMP

  • lisa._assets.kmodules.lisa: Rename feature event__pixel6_emeter to event__lisa__pixel6_emeter

  • lisa._assets.kmodules.lisa: Rename some helpers to remove lisa_ prefix

  • lisa._assets.kmodules.lisa: Replace strlcpy() with strscpy()

  • lisa._assets.kmodules.lisa: Shield against KBUILD_ABS_SRCTREE

  • lisa._assets.kmodules.lisa: Shield against missing arch_freq_scale

  • lisa._assets.kmodules.lisa: Update tracepoint probes prototype

  • lisa._assets.kmodules.lisa: Use introspect_header.py to avoid type name clash

  • lisa._assets.kmodules.lisa: Use kallsyms symbol address

  • lisa._btf: Ensure the BTF types are dumped as deeply as possible when forward decls are used

  • lisa._btf: Fix alignment inference

  • lisa._btf: Fix comment

  • lisa._btf: Fix nested anonymous struct/union dumping

  • lisa._btf: Fix renaming of BTFForwardDecl

  • lisa._doc.helpers: Fix "exec" directive

  • lisa._doc.helpers: Fix regex pattern

  • lisa._git: Default to including binary files in patches

  • lisa._git: Fix get_uncommited_patch() output decoding

  • lisa._git: Use core.checkStat=minimal

  • lisa._kmod: Add missing "perl" chroot kernel build dependency

  • lisa._kmod: Add x86 32bit support

  • lisa._kmod: Allow aarch64-none-elf- toolchain autodetection

  • lisa._kmod: Autodetect arm-none-eabi- toolchain for armv7

  • lisa._kmod: Avoid raising exception when target ABI is unknown

  • lisa._kmod: Cache compilation on cc --version

  • lisa._kmod: Check for file availibity rather than kernel config

  • lisa._kmod: Do not download kernel.org tarball for rc kernels

  • lisa._kmod: Do not include __pycache__ in kmodule sources

  • lisa._kmod: Do not install gcc explicitly on clang Alpine chroot

  • lisa._kmod: Do not use module symbols from kallsyms

  • lisa._kmod: Ensure all module sources are checksumed

  • lisa._kmod: Filter kallsyms from lisa module symbols

  • lisa._kmod: Fix Alpine clang package name

  • lisa._kmod: Fix atomic headers sha1 fixup

  • lisa._kmod: Fix pre-existing .ko loading

  • lisa._kmod: Fix preinstalled kmod path when base path exists

  • lisa._kmod: Fix use of arch vs abi

  • lisa._kmod: Force tools when LLVM=-N is used

  • lisa._kmod: Harden against KCONFIG_CONFIG != .config

  • lisa._kmod: Honor LLVM=-XXX in alpine build env

  • lisa._kmod: Make it clear that some module setup are unsupported

  • lisa._kmod: Make toolchain selection reproducible

  • lisa._kmod: Rationalize the Alpine chroot arch selection

  • lisa._kmod: Remove duplicates in cross_compiles list

  • lisa._kmod: Repair atomic headers fixup

  • lisa._kmod: Sanitize CC and make vars handling

  • lisa._kmod: Support paths for CC make var

  • lisa._kmod: Use busybox insmod

  • lisa._kmod: Use kernel version info when choosing alpine toolchain

  • lisa._kmod: Use target.pull(as_root=True) when necessary

  • lisa._kmod: Warn about non ideal toolchain in more situations

  • lisa._typeclass: Fix BuiltinFromStringInstance

  • lisa._typeclass: Make all instances private

  • lisa._unshare: Ensure logging is setup before unpickling function

  • lisa.analysi._proxy: Fix _AnalysisPreset unpickling

  • lisa.analysis.base: Disable undo/redo bokeh tools

  • lisa.analysis.tasks: Fix plot_task_residency()

  • lisa.analysis.tasks: Fix plot_tasks_activation()

  • lisa.analysis.tasks: Fix stringify_task_state_series()

  • lisa.conf: Deal with import errors more cleanly.

  • lisa.conf: Fix an exception message

  • lisa.conf: Forward memo in __deepcopy__

  • lisa.conf: Remove debug logging when getting/setting a key

  • lisa.datautils: Fix _pandas_find_unique_bool_vector()

  • lisa.datautils: Update for pandas 2.1.0

  • lisa.pelt: Fix plet_settling_time()

  • lisa.platforms.platinfo: Handle better unknown cpu-capacities/writeable

  • lisa.target: Add missing f-string

  • lisa.target: Allow mount to work on Android

  • lisa.target: Fix adb-over-ip connection

  • lisa.target: Improve Target.from_cli() --conf handling

  • lisa.tests.base: Fix FtraceTestBundle.trace events namespaces

  • lisa.tests.staging.utilclamp: Avoid overlapping CPU capacities

  • lisa.tests.staging.utilclamp: Choose uclamp so that freq is in the middle of a band

  • lisa.tests.staging.utilclamp: Fix UtilClamp expected frequency

  • lisa.tests.staging.utilclamp: Fix debug plot

  • lisa.tests.staging.utilclamp: Fix uclamp value type

  • lisa.tests.starting.utilclamp: Use square brackets for df column access

  • lisa.trace: Allow parsers to return LazyFrames based on scans of files not in the cache

  • lisa.trace: Bind "path" in TraceParserBase.__init__()

  • lisa.trace: Compress cached parquet files

  • lisa.trace: Do not query parser if available events set is known

  • lisa.trace: Ensure MissingTraceEventError has accurate event info

  • lisa.trace: Ensure TxtTraceParser.from_data(events=None) works

  • lisa.trace: Fix FtraceCollector error when non-existant event is required

  • lisa.trace: Fix FtraceCollector(events=['print'])

  • lisa.trace: Fix FtraceConf merging

  • lisa.trace: Fix FtraceConf.add_merged_src()

  • lisa.trace: Fix FtraceConf.merge()

  • lisa.trace: Fix Trace data swap size estimation

  • lisa.trace: Fix TraceEventCheckerBase.check_events() with namespaces

  • lisa.trace: Fix cpus-count metadata on trace-cmd parser

  • lisa.trace: Fix dataframe prefiltering for userspace ftrace events

  • lisa.trace: Fix meta event availability detection

  • lisa.trace: Fix parquet loading from cache

  • lisa.trace: Fix parser identification

  • lisa.trace: Fix parser metadata handling

  • lisa.trace: Fix parser parquet stealing

  • lisa.trace: Fix polars LazyFrame caching

  • lisa.trace: Fix polars scan path handling

  • lisa.trace: Fix task comm extraction

  • lisa.trace: Fix text trace parser regex

  • lisa.trace: Fix trace-id metadata

  • lisa.trace: Improve missing optional events FtraceCollector reporting

  • lisa.trace: Only update metadata JSON if necessary

  • lisa.trace: Reduce FtraceConf.add_merged_src() debug output

  • lisa.trace: Remove dataframe columns that should not be exposed

  • lisa.trace: Rename FtraceConf event-namespaces key

  • lisa.trace: Update Trace.df_event() docstring

  • lisa.utils.Serializable: Only allow custom YAML tags in unsafe mode.

  • lisa.utils: Fix optional_kwargs __doc__ forwarding

  • lisa.utils: Hide warnings in import_all_submodules()

  • lisa.utils: Make memoized decorator thread-safe

  • lisa.version: Take binary files diff into account

  • lisa.wa: Fix WATraceCollector.traces

  • lisa.wlgen.rta: Fix ComposableMultiConcretePropertyBase._ATTRIBUTES type hint

  • lisa.wlgen.rta: Fix PeriodicWload(run_wload=...) expected type

  • lisa.wlgen.rta: Fix PeriodicWload.unscaled_duty_cycle_pct()

  • lisa.wlgen.rta: Fix docstring of PeriodicWload

  • lisa.wlgen.rta: Fix double self parameter passing

  • lisa.wlgen.rta: Fix rt-app calibration debug logging

  • lisa/_assets/kmodules/lisa: Add symbol namespace for kernel_read()

  • lisa: Address some pandas FutureWarning

  • lisa: Remove use of pandas.Series.fillna(method=...)

  • lisa: Use typing.Sequence instead of typing.List

  • lsia.trace: Fix events-namespaces FtraceConf key type

  • shell/lisa_shell: Avoid error if ANDROID_HOME does not exist

  • shell/lisa_shell: Create git hooks dir

  • shell/lisa_shell: Fix hook install in git worktrees

  • shell: Fix git hook install location

  • tests: Fix task PID/comm mapping unit test

  • tools/batch-rebase: Allow any branch name

  • tools/batch-rebase: Cherry pick commits in correct order

  • tools/batch-rebase: Fix cherry picking in some scenarios

  • tools/batch-rebase: Support Path in json dump

  • tools/batch-rebase: Use typing instead of lisa._generic

  • tools/batch-rebase: require subcommand in CLI

  • tools/bisector: Fix ruamel usage of DEFAULT_MAPPING_TAG

  • tools/buildroot_config: Remove deprecated config

  • tools/exekall: Allow imported modules to manipulate sys.modules

  • tools/exekall: Do not consider inherited methods defined outside of the package set

  • tools/exekall: Fix use of Path as str

  • tools/exekall: Restrict selected callables

  • tools/lisa-build-asset: Assume BROKEN_CROSS_COMPILATION=1

  • tools/lisa-build-asset: Avoid emptying --build-dir

  • tools/lisa-build-asset: Try harder to remove bind mounts

  • tools/lisa-build-asset: Use lazy umount

  • tools/lisa-combine-gitlab-mr: Use merge request reference

  • tools/lisa-combine-pr: Use logging module

  • tools/lisa-make-preview: Fix case where there is no preview PR

  • tools/lisa-make-preview: Use branch sha1 for remote name

  • tools/recipes/busybox.recipe: Fix insmod applet

  • tools/recipes/pahole.manifest.yaml: Add typedef fix

  • tools/recipes/trace-cmd.recipe: Avoid building shared object

  • tools/recipes/trace-cmd.recipe: Enable zlib

  • tools/trace-parser: Do not emit event info key if event did not occur

  • tools/trace-parser: Fix event selection

  • tools/trace-parser: Propagate writer thread error

  • tools: Add missing __init__.py

  • tools: Remove main() call

Breaking change

  • lisa.datautils: Disallow df_window(clip_window=False)

  • lisa.notebook: Do not initialize holoviews

  • lisa.trace: Convert TxtTraceParserBase internals to polars

  • lisa.trace: Make TraceCache private

  • lisa.trace: Move event sanitization to parser classes

  • lisa.trace: Remove Trace.df_event(raw=...) parameter

  • lisa.trace: Remove TrappyTraceParser

  • lisa.trace: Remove cpumask sanitization

  • lisa.trace: Remove optional field support in meta events

  • lisa.trace: Remove trace_printk@ meta event bputs support

  • lisa.trace: Use Categorical for polars LazyFrame string columns

  • lisa: Move test application code into its own package

  • lisa_tests: Split off lisa_tests

v3.1.0

Feature

  • lisa._assets.kmodules.sched_tp: Add Pixel 6 integrated energy meter support

  • lisa._assets: Make HOST_PATH depend on LISA_USE_SYSTEM_BIN

  • lisa._kmod: Add DynamicKmod.mod_name property

  • lisa._kmod: Add a from_proc_config() kernel tree loader

  • lisa._kmod: Bump Alpine to v3.16

  • lisa._kmod: Bump to Alpine v3.17.3

  • lisa._kmod: Explicitly pass module/kernel path when building module

  • lisa._kmod: Improve kernel tree preparation exception reporting

  • lisa._kmod: Update Alpine to v3.17.1

  • lisa.analysis.latency: Include CPUs in latency DFs

  • lisa.analysis: Add Pixel 6 analysis

  • lisa.notebook: Add dimension parameter to plot_signal

  • lisa.target: Add Target.from_custom_cli(description=...)

  • lisa.target: Expose devlib's Target(max_async=...)

  • lisa.tests.base.RTATestBundle: Add rtapp_profile_kwargs ctor param

  • lisa.trace: Allow CollectorBase.get_data(path=None)

  • lisa.trace: Collect metadata when using concurrent parsing

  • lisa.trace: Improve FtraceCollector docstring

  • lisa.trace: Improve FtraceCollector kmod build error error reporting

  • lisa.trace: Use lisa._assets.get_bin() for trace-cmd

  • lisa.utils: Add LazyMapping

  • lisa.utils: Add destroyablecontextmanager

  • lisa.utils: Add ignore_exceps() context manager combinator

  • lisa.wa: Expose jobs, outputs in WAOutput & traces in WATraceCollector

  • lisa.wlgen.rta: Add RTA(log_level=None) parameter

  • shell/lisa_shell: Allow lisa-install to upgrade venv

  • tools/check-setuppy: Allow watching multiple patterns

  • tools/exekall: Support PEP 673 Self return annotation

  • tools/lisa-whatsnew: Add lisa-whatsnew

  • tools/recipes/rt-app.recipe: Update versions

  • tools/recipes/trace-cmd.recipe: Update to trace-cmd 3.1.5

  • tools/recipes/trace-cmd.recipe: Update to trace-cmd to use libtraceevent 1.7.0

  • tools/recipes: Add rt-app.manifest.yaml

Fix

  • .gitignore: Add cache/ and kernel module artifacts

  • doc/conf.py: Fix holoviews intersphinx inventory location

  • install_base.sh: Do not install selenium geckodriver on recent Ubuntu

  • ipynb: Update hv.extension() calls

  • lisa.__init__: Ignore pending deprecation warnings in matplotlib

  • lisa._assets.kmodules.sched_tp: Add MODULE_IMPORT_NS()

  • lisa._assets.kmodules.sched_tp: Fix symbol namespace extraction

  • lisa._git: Avoid warning when git notes are not available

  • lisa._kmod: Enable KBUILD_MODPOST_WARN=1

  • lisa._kmod: Fix try_loaders() exception handling

  • lisa._kmod: Ignore non-existing include/linux/atomic

  • lisa._kmod: Workaround broken atomic headers in IKHEADERS

  • lisa.analysis.pixel6: Fix df_power_meter() with pandas 2.0.0

  • lisa.analysis.rta: Silence pandas warning

  • lisa.analysis.tasks: Fix plot_tasks_activation() with trace window

  • lisa.analysis: Fix doc comments relating to task_states

  • lisa.analysis: Fix lisa.trace.Trace doc links

  • lisa.conf: Avoid kwargs in exceptions

  • lisa.datautils: Fix :type ...: in docstrings

  • lisa.datautils: Speedup and fix df_update_duplicates()

  • lisa.monad: Fix module docstring

  • lisa.notebook: Disable pagination in Tabulator

  • lisa.notebook: Fix plot UI table scroll

  • lisa.target: Fix TargetConf max-async key type

  • lisa.target: Fix max_async=None

  • lisa.target: Fix nargs=argparse.REMAINDER in Target.from_custom_cli()

  • lisa.target: Pass max_async to devlib Target

  • lisa.target: Restore compat with non-async devlib

  • lisa.tests.scheduler.load_tracking: Add missing noise check

  • lisa.tests.scheduler.load_tracking: Fix noise check usage

  • lisa.tests.scheduler.sanity: Fix use of collector

  • lisa.trace: Add compatiblity with recent trace-cmd

  • lisa.trace: Avoid broken trace in FtraceCollector.get_data()

  • lisa.trace: Avoid nested multiprocessing.Pool

  • lisa.trace: Ensure FtraceCollector context manager is reusable

  • lisa.trace: Fix HRTxtTraceParser regex parsing

  • lisa.trace: Fix Trace.from_target() with kernel module

  • lisa.trace: Fix TxtEventParser regex for named ring buffer

  • lisa.trace: Fix automatic Trace parser detection

  • lisa.trace: Fix cpuhp_enter fun field dtype

  • lisa.trace: Preserve time-range metadata in TxtTraceParserBase

  • lisa.trace: Update HRTxtTraceParser skeleton regex

  • lisa.trace: Use SysTraceParser.from_html() for .html files

  • lisa.utils: Fix PartialInit.__getattribute__

  • lisa.utils: Fix destroyablecontextmanager exception handling

  • lisa.wa: Fix caching of _get_job_df()

  • lisa.wa: Fix getting jobs in WACollectorBase

  • lisa.wlgen.sysbench: Fix docstring

  • lisa: Avoid using DataFrame.iteritems()

  • lisa: Fix iteration over DataFrame.groupby([x])

  • lisa: Fix pandas warnings

  • lisa: Specify DataFrame/Series.groupby(group_keys=False)

  • lisa: Use lisa.utils.destroyablecontextmanager

  • pytest.ini: Ignore deprecation warnings comming from devlib

  • pytest.ini: Ignore deprecation warnings in dependencies

  • pytest.ini: Ignore some Jupyterlab internal deprecation warnings

  • tools/check-setuppy: Fix CLI path detection

  • tools/check-setuppy: Watch git hooks

  • tools/exekall/exekall/engine.py: Fix support of typing.Self

  • tools/exekall: Allow class without __init__

Breaking change

  • lisa.analysis.pixel6: Change df_power_meter() "value" column to "power"

v3.0.1

Fix

  • doc: Set Sphinx language

  • exekall.engine: Fix FrozenExprVal.callable_ property

  • lisa._assets.kmodules.sched_tp: Fix Makefile vmlinux.h dependency

  • lisa._doc.helpers: Escape changelog entries

v3.0.0

Feature

  • .github: Add Python 3.10 workflow

  • doc/changes.rst: Add Versioning section

  • doc/changes.rst: Document instance attributes stability

  • doc/changes.rst: Document release process

  • doc/setup.rst: Add paragraph on Python version requirement.

  • lisa-version: Display LISA python package version

  • lisa._assets.kmodules.sched_tp: Do not fail module load when no feature was manually selected

  • lisa._assets.kmodules.sched_tp: Refactor kernel module

  • lisa._assets: Add HOST_PATH

  • lisa._doc.helpers: Make use of refs/notes/changelog

  • lisa._git: Add find_root()

  • lisa._git: Allow get_commit_message to display notes

  • lisa._kmod: Add FtraceDynamicKmod

  • lisa._kmod: Add LISAFtraceDynamicKmod

  • lisa._kmod: Allow passing parameters to insmod

  • lisa._kmod: Introduce dynamic kernel module building

  • lisa._kmod: Rename lisa kernel module to "lisa"

  • lisa._kmod: Speedup copy overlay backend

  • lisa._kmod: Use LLVM make var to infer CC

  • lisa._unshare: Add lisa._unshare module

  • lisa.fuzz: Add lisa.fuzz module

  • lisa.monad: Replace the State monad implementation

  • lisa.notebook: Use Tabulator panel widget

  • lisa.target: Add Target.cached_pull()

  • lisa.target: Add Target.get_kmod()

  • lisa.target: Allow pickling Target

  • lisa.tests.scheduler.eas_behaviour: Frequency based duty cycle

  • lisa.trace: Add DynamicTraceEventChecker

  • lisa.trace: Add TraceEventCheckerBase.check_events(..., optional=False)

  • lisa.trace: Add modules/auto-load key to FtraceConf

  • lisa.trace: Allow TraceEventCheckerBase for FtraceCollector events

  • lisa.trace: Allow TraceEventCheckerBase in Trace(events=...)

  • lisa.trace: Hook LISAFtraceDynamicKmod into FtraceCollector

  • lisa.trace: Log missing optional events in FtraceCollector

  • lisa.trace: Make TraceEventCheckerBase a Sequence

  • lisa.trace: Make kmod build failure an error when possible

  • lisa.trace: Try building the kmod for optional events

  • lisa.trace: Use kmod features in FtraceCollector._get_kmods()

  • lisa.utils: Add DirCache class

  • lisa.utils: Add LISA_CACHE_HOME

  • lisa.utils: Add LISA_HOST_ABI

  • lisa.utils: Add SerializeViaConstructor class

  • lisa.utils: Add chain_cm() helper

  • lisa.utils: Add instancemethod decorator

  • lisa.utils: Add subprocess_log

  • lisa.version: Allow overriding VERSION_TOKEN

  • lisa.wlgen.rta: Add RTAPhaseBase.get_rtapp_repr(no_force_default_keys=None)

  • lisa.wlgen.rta: Add async API

  • lisa.wlgen.rta: Add leaf_precedence()

  • tools/batch-rebase: Make "name" topic key optional

  • tools/batch-rebase: Make "rr-cache" key optional

  • tools/recipes/pahole.recipe: Add pahole binary to LISA

  • tools/recipes/pahole: Add the batch-rebase conf and update it

Fix

  • .github: Add read permissions to debugfs

  • bisector: Do not return None in Report.load()

  • bisector: Use the new location for exekall's ExceptionPickler

  • doc/conf.py: Fix matplotlib intersphinx inventory URL

  • doc/setup.rst: Fix manual kmod setup instruction

  • doc: Fix broken matplotlib docstring xref

  • doc: Ignore some broken references

  • exekall._utils: Fix ExceptionPickler memory leak

  • exekall: Add missing import

  • exekall: Fix standalone module handling passed on CLI

  • exekall: Set package attribute when importing submodule

  • external/subtrees.conf: Revert to devlib master

  • lisa-doc-build: Handle change in argparse --help output

  • lisa-version: Fix python version display

  • lisa.__init__: Ignore spurious CryptographyDeprecationWarning

  • lisa._assets.binaries: Update rt-app

  • lisa._assets.kmodules.sched_tp: Fix failed feature dependency

  • lisa._assets.kmodules.sched_tp: tp.h: Fix tp feature init

  • lisa._assets.kmodules: Add missing static in sched_helpers.h

  • lisa._kmod: Avoid broken split .BTF section in .ko

  • lisa._kmod: Control env vars

  • lisa._kmod: Fix DynamicKmod._do_compile()

  • lisa._kmod: Fix KernelTree docstring

  • lisa._kmod: Fix PATH in Alpine chroot

  • lisa._kmod: Fix copy overlay backend

  • lisa._kmod: Fix misleading log

  • lisa._kmod: Fix module with subfolders build

  • lisa._kmod: Fix undefined variable in DynamicKmod

  • lisa._kmod: Forward kwargs in KmodSrc.from_path()

  • lisa._kmod: Handle non-str make variables

  • lisa._kmod: Propagate overlay_backend to _make_chroot()

  • lisa.analysis.frequency: Add missing type annotation for plot_peripheral_frequency

  • lisa.analysis.frequency: Fix df_peripheral_clock_effective_rate

  • lisa.analysis.tasks: Best effort in plot_tasks_activations()

  • lisa.analysis.tasks: Fix plot_tasks_activation()

  • lisa.conf: Allow empty levels in YAML

  • lisa.datautils: Add SignalDesc for clk_* events

  • lisa.datautils: Avoid deprecated use of Index.__and__

  • lisa.datautils: Replace pd.Float64Index by pd.Index

  • lisa.energy_model: Fix estimate_from_trace

  • lisa.fuzz: Update docstring

  • lisa.stats: Fix plot_stats() X scale

  • lisa.target: Allow None for target-conf/kernel/src config

  • lisa.target: Fix tools install

  • lisa.target: Remove adb rooting attempt

  • lisa.tests.base: Add OptionalFtraceTestBundle

  • lisa.tests.base: Fix FtraceTestBundle docstring

  • lisa.tests.base: Fix as_root value passed to wload

  • lisa.tests.base: Use Series.str.replace(regex=False)

  • lisa.tests.hotplug: Fix HotplugRollback MRO

  • lisa.tests.scheduler.eas_behaviour: Use a custom RTAPhase Buffer

  • lisa.trace: Add TraceEventCheckerBase.__bool__

  • lisa.trace: De-hardcode tracefs location

  • lisa.trace: Ensure Trace.trace_path is a string or None

  • lisa.trace: Fix parsing of sched_pelt_se/sched_util_est_se

  • lisa.trace: Fix parsing of thermal events

  • lisa.utils: Fix PartialInit param binding

  • lisa.utils: Fix SimpleHash.__eq__

  • lisa.utils: Fix sig_bind()

  • lisa.utils: Speedup Loggable.get_logger()

  • lisa.wlgen.rta: Cope with broken ulimit -Hr

  • lisa.wlgen.rta: Fix PlaceHolderValue.__str__()

  • lisa.wlgen.rta: Fix recursive calibration

  • lisa.wlgen.rta: Fix with_props() examples

  • lisa.wlgen.rta: Use a separate result folder for each CPU calibration workload

  • lisa/_assets/kmodules/sched_tp/Makefile: Do not use pahole --forward_decl

  • lisa: Ban DataFrame.append

  • lisa: Fix uses of Index.get_indexer()

  • lisa: Remove deprecated use of pandas.Index.get_loc()

  • tests.__init__: Move test-specific setup from lisa to tests

  • tests/test_wlgen_rtapp.py: Fix workload cleanup

  • tools/batch-rebase: Fix git cherry-pick --continue error handling

  • tools/batch-rebase: Fix tag action

  • tools/kmodules/: Make a symlink per file

  • tools/recipes/pahole.manifest.yaml: Add pointer size fix

Breaking change

  • lisa.fuzz: Port lisa.fuzz.Gen over the new lisa.monad API

  • lisa.monad: Replace the State monad implementation

  • lisa.target_script: Remove deprecated module

  • lisa.tests.base: Require root for ftrace

  • lisa.trace: Drop df.name attribute

  • lisa.utils: Replace serialized_whitelist/blacklist/placeholders by ATTRIBUTES_SERIALIZATION

v2.0.0

Breaking change

  • lisa-plot: Update to use refurbished analysis plot backends

  • lisa.analysis.base: Move COLOR_CYCLES to notebook and rename to COLOR_CYCLE

  • lisa.analysis.functions: Move ksym resolution from event sanitization to analysis

  • lisa.analysis.functions: Rename FunctionsAnalysis

  • lisa.analysis.rta: Rename RTAEventsAnalysis.tasks_window() into task_window()

  • lisa.analysis.status: Move sched_overutilized synthetic "len" column to analysis

  • lisa.analysis.tasks: Make df_task_activation() duty_cycle aware of preemption

  • lisa.assets: Rename to lisa._assets

  • lisa.colors: Remove lisa.colors as it is unused

  • lisa.datautils: Update df_window_signals to take a list of signals

  • lisa.doc: Rename to lisa._doc

  • lisa.generic: Move generic classes from lisa.conf to lisa.generic

  • lisa.generic: Rename to lisa._generic

  • lisa.git: Rename to lisa._git

  • lisa.platforms.platinfo: Change cpu-capacities key

  • lisa.proxy: Rename to lisa._proxy

  • lisa.stats: Use holoviews for plots

  • lisa.stats: rename Stats.plot_histogram nbins parameter to bins

  • lisa.tests.base: Add TestConfBase class

  • lisa.tests.base: Add buffer phase in get_rtapp_profile()

  • lisa.tests.base: Remove CannotCreateError

  • lisa.tests: Pass a composed collector to _from_target()

  • lisa.trace: Add text trace parser

  • lisa.trace: Remove barely used Trace.df_events(rename_cols=True) option

  • lisa.trace: Remove big/LITTLE assumption in events sanitization

  • lisa.trace: Remove sanitization for cpu_frequency event

  • lisa.trace: Rename 'func' column to 'ip' for print events

  • lisa.trace: TxtTraceParserBase: rename __time into __timestamp

  • lisa.trace: Use HRTxtTraceParser by default for .txt trace

  • lisa.trace: Use trace-cmd raw report for unknown events

  • lisa.typeclass: Rename to lisa._typeclass

  • lisa.wlgen.rta: Introduce new wlgen rt-app API

  • lisa.wlgen.rta: Port RTA to the new Workload API

  • lisa.wlgen.sysbench: Port Sysbench to the new Workload API

  • lisa.wlgen.workload: Fix long standing Workload API issues

  • lisa.wlgen.workload: Rename required_tools to REQUIRED_TOOLS

  • lisa.wlgen: Make Workload 'name' parameter optional

  • tools: lisa-plot: Rename --platinfo into --plat-info

Breaking changes

Here is a list of commits introducing breaking changes in LISA:

Wed Apr 24 18:57:36 2024 +0100
2e7256f82e31588908e38087f85839986e7c7bb6
lisa.trace: Convert TxtTraceParserBase internals to polars

BREAKING CHANGE

Convert the internals of TxtTraceParserBase to polars.

This unfortunately comes at a loss of feature as polars' regexes (based
on the "regex" Rust crate at the time of writing) does not handle
look-around (look-behind and look-ahead). In turn, that prevents from
parsing correctly string fields with spaces in the values, e.g.

    comm=hello world anotherfield=42

This change is deemed acceptable as this fully removes the dependency on
pandas for lisa.trace (apart from converting legacy pandas inputs), and
the human-readable format of trace-cmd is ambiguous in any case. The
only ftrace format currently really supported is the binary trace.dat
format created by trace-cmd, which does not suffer from these issues.

Wed Apr 24 18:57:35 2024 +0100
195549b63f4c41736ae811a7f930132ce9bf9ebf
lisa.trace: Use Categorical for polars LazyFrame string columns

BREAKING CHANGE

Make all string columns use Categorical rather than just columns that
contain "comm" when using TraceDumpTraceParser.

Wed Mar 27 18:40:37 2024 +0000
8413bca99de43dd235487f519ba5f2dfcde372da
lisa_tests: Split off lisa_tests

BREAKING CHANGE

Split lisa_tests package.

Wed Mar 27 11:47:35 2024 +0000
716d4d29af6196b3e3d7b2d2dcac0a470116256b
lisa.notebook: Do not initialize holoviews

BREAKING CHANGE

Do not initialize holoviews extensions as it is the user's
responsability to choose a backend. This may also simplify debugging of
initializing issues by getting closer to a standard recommended setup.

Tue Mar 26 15:37:04 2024 +0000
769492371286e23d1503c5f471d139cbde094256
lisa.trace: Remove TrappyTraceParser

BREAKING CHANGE

Remove compatibility layer with trappy library. It hasn't been used in
years and was fully replaced by TxtTraceParserBase family of classes.

Tue Mar 26 15:33:43 2024 +0000
02e41c429aea9fc7644e0ff8dacf1cb990a740cc
lisa.datautils: Disallow df_window(clip_window=False)

BREAKING CHANGE

Raise an exception if clip_window=False is passed to df_window(). This
behavior was never really needed and is not possible to support polars
LazyFrame since the actual Time values are not known until collect() is
called.

Tue Mar 26 15:33:43 2024 +0000
8b1d82c4dcee0afb13742e7af071a7069eb6616a
lisa.trace: Remove Trace.df_event(raw=...) parameter

BREAKING CHANGE

Disallow any non-True value for "raw" parameter, as the sanitization is
now the responsibility of the parser.

Tue Mar 26 15:33:43 2024 +0000
1cd321479479825d4b5f913291c312dc88511ffe
lisa.trace: Move event sanitization to parser classes

BREAKING CHANGE

Move the sanitization into the parser classes where they actually
belong.

As a result, remove the sanitization of "print"/"bprint"/"bputs" that
turns the str column into bytes, since the values in-kernel are expected
to be ASCII-encoded strings.

Mon Mar 25 13:46:24 2024 +0000
f56a2d56037e08fd78d820eb91eb06beae19f364
lisa.trace: Make TraceCache private

BREAKING CHANGE

Rename TraceCache into _TraceCache as no value of this type was ever
made public. It's a de-facto implementation detail of Trace, so make it
private.

Tue Feb 13 16:43:01 2024 +0000
02da758c8611fdffa7d57236643c8334997597ef
lisa.trace: Remove cpumask sanitization

BREAKING CHANGE

cpumask fields cannot be handled in a uniform way between all parsers,
so remove any attempts at doing so.  Since the new default trace-dump
parser has direct support for them, drop the sanitization.

The new format for cpumask values is an iterable of booleans where each
position corresponds to one bit, starting from CPU 0, e.g. [True, False,
False] for the cpumask 0b100

Wed Jan 10 12:06:06 2024 +0000
6bdace0fff3a2e87e1642f299e77e58dade1c6c9
lisa.trace: Remove trace_printk@ meta event bputs support

BREAKING CHANGE

trace_printk@... meta events now require a bprint event, bputs is not
supported anymore.

Wed Jan 10 12:06:06 2024 +0000
9cdf0d68078254431a601dec9894222bd0f1c9cf
lisa.trace: Remove optional field support in meta events

BREAKING CHANGE

Remove support for positional field ("__positional") for meta events
such as trace_printk@* or userspace@* events. This feature was not used
and complexifies parsers for little gain.

Wed Oct 11 13:59:29 2023 +0100
7e2cd50bdd5ef9d43de908332c859fe8371b01f1
lisa: Move test application code into its own package

BREAKING CHANGE

This is a breaking change from a setup point of view (separate package)
and when using "exekall run lisa" that now needs to be
"exekall run lisa lisa_tests" since the most class definitions are moved
to the new package.

However, from the API perspective this is not a breaking change, as a
later commit adds import shims (with a deprecation warning) in the old
locations.

Tue Apr 4 15:22:01 2023 +0100
a83f4b32526591af83115b28993220851848599d
lisa.analysis.pixel6: Change df_power_meter() "value" column to "power"

BREAKING CHANGE

Use the more descriptive "power" column name than "value".

Mon Apr 25 12:26:58 2022 +0100
50f179d9fb6cacbfa116e9056d2fe0ac0e9a00eb
lisa.tests.base: Require root for ftrace

BREAKING CHANGE

Previously, we attempted to detect if tracefs was available as non-root
user. Since most distribution do not allow non-root user to use tracefs
and being root is usually not a problem in test setups, force being root
when ftrace is used.

Mon Apr 11 15:08:47 2022 +0100
ef3ecbea05fd2aff901520c808a78125273eb136
lisa.target_script: Remove deprecated module

BREAKING CHANGE

Remove module as it's a leaky abstraction that is not used anymore and
has been deprecated for over a year.

Fri Apr 1 15:06:54 2022 +0100
ebdf75a2a5fafab6be5cc9f084b5fd3daea0c7ff
lisa.trace: Drop df.name attribute

BREAKING CHANGE

Trace.df_event() used to set a .name attribute on dataframes it returns.
This is unfortunately problematic, as it will also set the value of a
column named "name" if it exists.

The replacement is df.attrs['name'] and has been there for a while now.
It's also the recommended way of doing it by pandas documentation
itself.

Thu Mar 31 16:42:43 2022 +0100
7c93c68f28e0802f3da2a934213857d09c53d537
lisa.fuzz: Port lisa.fuzz.Gen over the new lisa.monad API

BREAKING CHANGE

Introduce GenMonad inheriting from lisa.monad.State and make Gen simply
embed a GenMonad action.

Breaking change: retry_until() now needs to be awaited, otherwise it's a
no-op. This will ensure future-proofing of the API, should the
implementation of retry_until need access to the internal state.

Thu Mar 31 16:42:43 2022 +0100
c8c8c6d51ff956bc7554b7dce1f403ebb363653e
lisa.monad: Replace the State monad implementation

BREAKING CHANGE
FEATURE

Replace the original custom-made State monad implementation with a full
featured monad transformer implementation.

This will allow combining multiple independent state monads, as well as
interop with event loops (e.g. asyncio), and allow new effects to be
added along the way such as the short-circuiting Option.

Breaking changes:

* Remove the automatic awaiting of monadic arguments to user-defined
  coroutine functions. The reason is that it was a misfeature that
  worked for the lisa.fuzz.Gen monad, but is actually quite problematic
  for other state monads as the user looses control over the ordering of
  side effects. This can unfortunately be extremely important for some
  monads.

* Re-purpose the "lift" method to act like a monad transformer lift.
  Existing use of "Monad.lift()" need to be converted to "Monad.do()"
  instead.

* Rename StateMonad into State

* StateMonad values can now only be called once.

Thu Sep 30 15:01:32 2021 +0100
f7819202727f8538f4da16deacbd3f810227cc9e
lisa.utils: Replace serialized_whitelist/blacklist/placeholders by ATTRIBUTES_SERIALIZATION

BREAKING CHANGE

Remove:
    * Serializable.serialized_whitelist
    * Serializable.serialized_blacklist
    * Serializable.serialized_placeholders

And replace them by:
    * Serializable.ATTRIBUTES_SERIALIZATION['allowed']
    * Serializable.ATTRIBUTES_SERIALIZATION['ignored']
    * Serializable.ATTRIBUTES_SERIALIZATION['placeholders']

Thu Jul 22 09:53:54 2021 +0100
c5bdd884876a923a291e16e83740eed011a0d38b
lisa.doc: Rename to lisa._doc

BREAKING CHANGE

Thu Jul 22 09:53:54 2021 +0100
0f8be195ba174ec9d31421b8829e822cd1e061a5
lisa.proxy: Rename to lisa._proxy

BREAKING CHANGE

Thu Jul 22 09:53:54 2021 +0100
a5efaaebacb54bf714a2336b6a3199e029030d9b
lisa.typeclass: Rename to lisa._typeclass

BREAKING CHANGE

Thu Jul 22 09:53:54 2021 +0100
f4ae7be508824c8935668648a765adf368311302
lisa.generic: Rename to lisa._generic

BREAKING CHANGE

Thu Jul 22 09:53:54 2021 +0100
23aeb26da9880958d82f880f6c54729eca5e4045
lisa.assets: Rename to lisa._assets

BREAKING CHANGE

Thu Jul 22 09:53:54 2021 +0100
d168e565ab3f49a41cb5ac0a1d03f612bd5ea1d0
lisa.git: Rename to lisa._git

BREAKING CHANGE

Tue Jul 13 18:46:29 2021 +0100
1686a31c095480f8f8faeae08063ef0cdf96e34d
lisa-plot: Update to use refurbished analysis plot backends

Update to use holoviews, using the bokeh backend.

BREAKING CHANGE

--xkcd option is removed as it is not supported by bokeh.
--matplotlib-backend option is removed as it becomes irrelevant.

Tue Jul 13 18:46:29 2021 +0100
a721aa42b060a4b5a8e40124fef7143b700ce1e1
lisa.stats: Use holoviews for plots

BREAKING CHANGE

Plot methods of lisa.stats.Stats used to return matplotlib Axes objects,
they now return holoviews elements.

Tue Jul 13 18:46:29 2021 +0100
c20e0e9be4bff21eec848493413b6093220575a3
lisa.stats: rename Stats.plot_histogram nbins parameter to bins

BREAKING CHANGE

Align all "bins" parameters in lisa to "bins".

Old calls:

    Stats.plot_histogram(nbins=...)

Need to be turned into:

    Stats.plot_histogram(bins=...)

Thu May 13 11:00:22 2021 +0100
5a6613be785e278979f3c760bc4e979e5ff01e75
lisa.tests.base: Remove CannotCreateError

Remove CannotCreateError and turn ResultBundleBase into an Exception.
This allows raising it directly, avoiding the need for a separate
exception type.

BREAKING CHANGE

Replace "raise CannotCreateError(...) from X" by
"ResultBundle.raise_skip(..., from_=X)"

Fri Apr 9 15:44:06 2021 +0100
ebba071bcbddf5171268fdbfde6f619afd974ad9
lisa.colors: Remove lisa.colors as it is unused

BREAKING CHANGE

Thu Apr 8 10:07:33 2021 +0100
4aae9d9c9a071d2c8011fdbf139133d12b6cfc8e
lisa.tests: Pass a composed collector to _from_target()

_from_target() now takes a "collector" parameter, which is an instance
of lisa.trace.ComposedCollector. This collector value is filled-in by
the TestBundleMeta machinery, according to the classes you inherited
from. For example, inheriting from DmesgTestBundle will add a
DmesgCollector to the composed collector.

The "ftrace_coll" parameter is removed since it becomes useless.

BREAKING CHANGE

External API:
TestBundle.from_target(ftrace_coll=...) parameter is removed.

Internal API: Each _from_target() implementation needs to take an
"collector" parameter, which will be filled-in with the
ComposedCollector instance.

Thu Mar 25 12:06:28 2021 +0000
de3e9dbcec4d58e98b772210e027fe5472031325
lisa.wlgen.workload: Rename required_tools to REQUIRED_TOOLS

BREAKING CHANGE

Use an upper case name to reflect the fact that it's a class constant.

Thu Mar 25 12:06:28 2021 +0000
d611979d7a8f8fef717e05ee7e94d62bbf8a373b
lisa.wlgen.sysbench: Port Sysbench to the new Workload API

BREAKING CHANGE

Extra sysbench CLI options passed directly to Sysbench.run() will now
have to be passed inside a dictionary to the "cli_options" parameters:

    # Old code:
    wload = Sysbench(...)
    wload.run(foobar=42)

    # New code:

    # Using deprecated API:
    wload = Sysbench(...)
    wload.run(cli_options=dict(foobar=42))

    # Using the recommended API
    wload = Sysbench(..., cli_options=dict(foobar=42))
    wload.run()

    # Or
    wload = Sysbench(...)
    wload = wload(cli_options=dict(foobar=42))
    wload.run()

This will avoid clashes between CLI options of Sysbench and future
expansion of the Workload API.

Thu Mar 25 12:06:28 2021 +0000
ccc73979bb1bf7986833d368cb5ebc30c396ebb6
lisa.wlgen.rta: Port RTA to the new Workload API

BREAKING CHANGE

The main user-visible change is that some files in res_dir will not be
created as eagerly as they used to be:

  * RTA.local_json is created by RTA.deploy() instead of by RTA.__init__
  * The rt-app log files pulled with log_stats=True are pulled in
    RTA.cleanup(), so they will not be available right after RTA.run()

Thu Mar 25 12:06:28 2021 +0000
20e78a0b0c11ff6484ae6164e50b6ee5780c0b7a
lisa.wlgen.workload: Fix long standing Workload API issues

* Return output from run() instead of mutating an attribute.
* Add a background_run() method
* Provide access to all the settings in workload constructors, rather
  than delayed information through run(). This allows workloads to take
  advantage of settings like "as_root"
* Allow reinitialization of the workload instances with another set of
  constructor parameters, e.g.:
    wload = Workload(...)
    # Create a new instance with other settings
    wload = wload(as_root=True)

BREAKING CHANGE

The user-level API is still backward compatible but subclasses will need
to be ported to the new internal API.

Mon Mar 15 15:53:53 2021 +0000
f950bab5958fcc028a62d2097f143fe32b39bd4b
lisa.tests.base: Add TestConfBase class

Allow hosting all the configurations under a single "test-conf"
top-level key.

BREAKING CHANGE

The DmesgTestConf class top-level key is now test-conf/dmesg instead of
dmesg-test-conf.

Wed Mar 3 11:11:56 2021 +0100
9a6f6b1a0b0a2e1681987edab57d8816c577864f
lisa.tests.base: Add buffer phase in get_rtapp_profile()

BREAKING CHANGE

Subclasses of RTATestBundle will now need to implement
_get_rtapp_profile() rather than directly get_rtapp_profile(), since the
buffer phase will be added by RTATestBundle.get_rtapp_profile().
Also, _get_rtapp_profile() will only be forwarded keyword arguments, not
positional ones.

The change is necessary in order to resolve phase names from the trace,
since the code needs to use the real profile that was used. Otherwise,
the phase IDs are shifted.

Wed Mar 3 11:11:56 2021 +0100
ca8490db86d7edc6b46485b5df483e1a6938b244
lisa.wlgen.rta: Introduce new wlgen rt-app API

BREAKING CHANGE

Implement legacy classes as shims around the new classes.
The only breaking change is loops=-1 not being supported anymore in
RTATask, since rt-app provides no way of correctly composing workloads
with infinite loops (for now).

Also, all the new classes are designed to be immutable, which means that
inplace operations (e.g. +=) will return a new instance instead of
modifying the existing one.

Also split the responsibilities of RTAWorkload:
* running the workload => RTAWorkload
* creating the JSON configuration => RTAConf

This allows creating a configuration file without having to connect to a
target if needed.

Thu Jan 14 19:43:34 2021 +0100
1e3fcafd68392d3720de8240cbe6072986118403
lisa.trace: Use trace-cmd raw report for unknown events

BREAKING CHANGE

Events without a parser already in LISA will now default to parsing the
raw output of trace-cmd, since the human readable one will either be
identical or impossible to parse without a handcrafted parser.

In order to get the old behavior, a custom trace parser needs to be used
(or more likely, use a trace parser with custom event parser). See
Trace(parser=...) parameter documentation for how to do that.

Fri Nov 13 17:52:20 2020 +0000
32782c7b39e18fc9f4ce091687930023f2f001c0
lisa.trace: Rename 'func' column to 'ip' for print events

BREAKING CHANGE

print/bprint/bputs events will now have an "ip" column instead of "func"
to align with kernel names, so it is consistent across parsers.

Note: There is no guarantee on whether the content is an address or
a function name anyway, so filtering it using
FunctionsAnalysis.df_resolve_ksym() is the only reliable way of getting
a function name out of it.

Note 2: This does not affect meta events like trace_printk@... or
userspace@..., which is the typical way of interacting with
print/bprint/bputs.

Wed Nov 11 16:52:13 2020 +0000
1ec18af78bcdca2c762feb3d203844809f27fd0f
lisa.trace: TxtTraceParserBase: rename __time into __timestamp

BREAKING CHANGE

Classes derived from TxtTraceParserBase will have to use "__timestamp"
instead of "__time" as regex group name.

This is done to ease alignment of different parsers.

Wed Nov 11 10:57:30 2020 +0000
357d106ca3008c9ecfa90cb8e8dfa96bd2131668
lisa.trace: Use HRTxtTraceParser by default for .txt trace

BREAKING CHANGE

Since text trace are typically in the human readable format (as opposed
to the semi-machine readable raw output of trace-cmd report), use this
by default for .txt extension.

If the old behaviour is still wanted, traces can be built with:

    from lisa.trace import Trace, TxtTraceParser
    Trace(..., parser=TxtTraceParser.from_txt_file)

The new behaviour is equivalent to:

    from lisa.trace import Trace, HRTxtTraceParser
    Trace(..., parser=HRTxtTraceParser.from_txt_file)

As a side effect, this will enable lisa-plot to work out of the box (to
some extent) with such text trace.

Thu Oct 15 12:17:34 2020 +0100
cfa3dfbbb11e59efbc317666c7aae0ff9a7d4148
lisa.analysis.base: Move COLOR_CYCLES to notebook and rename to COLOR_CYCLE

BREAKING CHANGE

Replace:
from lisa.analysis.base import COLOR_CYCLES

by:
from lisa.notebook import COLOR_CYCLE

Fri Jun 5 17:25:29 2020 +0100
154e994b9ee62abfe418d228cf753bdd8d5b5742
tools: lisa-plot: Rename --platinfo into --plat-info

BREAKING CHANGE

Rename --platinfo into --plat-info to align with other tools.

Wed Jun 3 15:38:28 2020 +0100
66103ec81e257c458735432a17a1bf185aa42d2a
lisa.wlgen: Make Workload 'name' parameter optional

BREAKING CHANGE

The "name" parameter has changed position to make it optional. This code:

    wload = RTA.by_profile(target, "4_50_tasks", profile)

Needs to be rewritten as:

    wload = RTA.by_profile(target, profile, name="4_50_tasks")

Being forced to specify a meaningful name always buggered me, especially
since it is not at all required under the hood - we get uniquely identified
directories and all that.

That plus the qualname ought to be good enough. I was actually tempted to
remove it alltogether, but then there's things like naming the JSON file
for the rt-app calibration that makes sense.

Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>

Wed May 13 13:02:50 2020 +0100
48d761f154d6eab8922ba0adf086a4731f102577
lisa.trace: Remove barely used Trace.df_events(rename_cols=True) option

BREAKING CHANGE

Since the scope of sanitization functions have been reduced to
formatting data so that it matches what is used inside the kernel,
remove column renaming.

This was already done in analysis methods anyway, which are much better
suited to having ad-hoc parameters tailored to a specific event.

Wed May 13 13:02:50 2020 +0100
551ec432a865dbc974456a1829567cfb842bae80
lisa.analysis.functions: Move ksym resolution from event sanitization to analysis

BREAKING CHANGE

Move more advanced data processing to trace analysis for the following
events:

    * funcgraph_entry: trace.analysis.functions.df_funcgraph('entry')
    * funcgraph_exit: trace.analysis.functions.df_funcgraph('exit')

Wed May 13 13:02:50 2020 +0100
9789de3d43e60879abd7cb943f46eea281f71c60
lisa.analysis.functions: Rename FunctionsAnalysis

BREAKING CHANGE

Rename into JSONStatsFunctionsAnalysis to free up the name for an
ftrace-based analysis.

Wed May 13 13:02:50 2020 +0100
e7cb1ccea25b71adbb73226bae45a8a5704cdd4e
lisa.trace: Remove sanitization for cpu_frequency event

BREAKING CHANGE

Consolidate the cleanup logic in FrequencyAnalysis.df_cpus_frequency().
If you want that cleaned up version, please use that:
  trace.analysis.frequency.df_cpus_frequency()
Instead of that:
  trace.df_events('cpu_frequency')

Since the later is the raw version.

Wed May 13 13:02:49 2020 +0100
aa61fcf244a1db5885b2ca6a3d124c2531bea192
lisa.trace: Add text trace parser

BREAKING CHANGE

Parse text traces based on the output of `trace-cmd report -R`
Also add some convenience parser for the human readable format.

Quick & dirty events based on calling trace_printk() in the kernel now
need to be prefixed with "trace_printk@" when parsed in lisa:

 kernel: trace_printk("foo: field1=42 field2=hello world")
 lisa  : trace.df_events('trace_printk@foo')

Matching on calling function name can be achieved with:

 kernel: (called from function foo()) trace_printk("field1=42 field2=hello world")
 lisa  : trace.df_events('trace_printk@func@foo')

Similarly, events generated from userspace by writing to
/sys/kernel/debug/tracing/trace_marker need to be prefixed:

 user: echo "foo: field1=42 field2=hello world" > /sys/kernel/debug/tracing/trace_marker
 lisa: trace.df_events('userspace@foo')

Note: The format is expected to follow the raw format of
`trace-cmd report -R`, such as in the example.

Also, event field names are now reflecting the name they have in the
kernel. Since trappy has non-optional renaming of the fields, client
code update might be needed. The recommended way is to use the high
level df_* functions from the analysis where applicable, since they will
give friendlier names and can usually cope with multiple formats of the
event across kernel versions if necessary.

Wed Apr 22 18:55:34 2020 +0100
22ab4075cdf1d24902f723a5999d09e41cb0044c
lisa.generic: Move generic classes from lisa.conf to lisa.generic

BREAKING CHANGE

Split unrelated classes into a separate module.

Wed Feb 19 14:18:32 2020 +0000
f28c9efcec417a88828cbe899e6fe83b7ea8f2c8
lisa.analysis.tasks: Make df_task_activation() duty_cycle aware of preemption

BREAKING CHANGE

Make sure the duty cycle is not affected by preemption by default.

The former behaviour where preempted == sleeping can be restored with
preempted_is_sleeping=True.

Fri Feb 14 15:29:26 2020 +0000
48af7a276cc0b16d80d6947e24470c8e8358ebb1
lisa.analysis.rta: Rename RTAEventsAnalysis.tasks_window() into task_window()

BREAKING CHANGE

Fix a typo in function name.

Thu Feb 6 17:27:27 2020 +0000
0a2293ae4838001f7103e4461273817051b07a60
lisa.trace: Remove big/LITTLE assumption in events sanitization

BREAKING CHANGE

Remove logic that is big-little specific in events sanitization. If this kind of
processing is still needed, it needs to be added in an analysis as it goes well
beyond sanitization.

Removed features:
* max_capacity column from cpu_capacity event dataframe
* tip_capacity column from cpu_capacity event dataframe
* nrg_diff_pct column from sched_energy_diff event dataframe

* usage_delta_group column from sched_energy_diff event dataframe
* nrg_payoff_group column from sched_energy_diff event dataframe

Thu Feb 6 17:27:19 2020 +0000
f42b49114018a5a2fb1ae89dfe70c6981a70be2a
lisa.analysis.status: Move sched_overutilized synthetic "len" column to analysis

BREAKING CHANGE

Remove the "len" column from the "sanitization" function of sched_overutilized,
and move the feature into StatusAnalysis.df_overutilized().

Code using:
    df = trace.df_events('sched_overutilized')
    df['len']

Should use this instead:
    df = trace.analysis.status.df_overutilized()
    df['len']

Thu Feb 6 17:27:19 2020 +0000
da991ff64e210900f630335afcf2d93d0b8fbb62
lisa.datautils: Update df_window_signals to take a list of signals

BREAKING CHANGE

Rather than taking a list of columns defining the signals, take a list of
SignalDesc. In the future, this could permit defining the end of a signal, which
df_window_signals() would need to take into account.

Thu Jan 16 14:50:22 2020 +0000
692da950dd5ea34466a5016601a68718044f04ae
lisa.platforms.platinfo: Change cpu-capacities key

BREAKING CHANGE: The key has now multiple subkeys:

cpu-capacities:
    orig: {CPU: capa, ...} # Previously "cpu-capacities" key
    rtapp: {CPU: capa, ...} # CPU capacities adjusted to fit rtapp calibration
    writeable: True/False # True if CPU capacities are writeable in sysfs

When using the capacities to create an rtapp workload in implementations of
RTATestBundle.get_rtapp_profile(), "rtapp" subkey must be used in order to work
with the capacities that will be written in sysfs by RTATestBundle.run_rtapp().

If the capacities are not writeable, the "rtapp" subkey is equal to the "orig"
subkey.

Deprecated APIs

Here is a list of deprecated APIs in LISA, sorted by version in which they will be removed:

Deprecated names to be removed in 4.0

Deprecated names to be removed in 4.0

lisa.analysis.base.AnalysisHelpers.cycle_colors()

Made irrelevant by the use of holoviews

Removed in: 4.0

lisa.analysis.base.AnalysisHelpers.do_plot()

Made irrelevant by the use of holoviews

Removed in: 4.0

lisa.analysis.base.AnalysisHelpers.get_next_color()

Made irrelevant by the use of holoviews

Removed in: 4.0

lisa.analysis.base.AnalysisHelpers.set_axis_cycler()

Made irrelevant by the use of holoviews

Removed in: 4.0

lisa.analysis.base.AnalysisHelpers.set_axis_rc_params()

Made irrelevant by the use of holoviews

Removed in: 4.0

lisa.analysis.base.AnalysisHelpers.setup_plot()

Made irrelevant by the use of holoviews

Removed in: 4.0

lisa.analysis.load_tracking.LoadTrackingAnalysis.df_cpus_signals()

Replaced by: lisa.analysis.load_tracking.LoadTrackingAnalysis.df_cpus_signal()

Removed in: 4.0

lisa.analysis.load_tracking.LoadTrackingAnalysis.df_tasks_signals()

Replaced by: lisa.analysis.load_tracking.LoadTrackingAnalysis.df_tasks_signal()

Removed in: 4.0

lisa.analysis.tasks.TasksAnalysis.get_task_by_pid()

This function raises exceptions when faced with ambiguity instead of giving the choice to the user

Replaced by: lisa.analysis.tasks.TasksAnalysis.get_task_pid_names()

Removed in: 4.0

lisa.analysis.tasks.TasksAnalysis.get_task_pid()

Replaced by: lisa.analysis.tasks.TasksAnalysis.get_task_id()

Removed in: 4.0

lisa.analysis.tasks.TasksAnalysis.plot_task_activation()

Deprecated since it does not provide anything more than plot_tasks_activation

Replaced by: lisa.analysis.tasks.TasksAnalysis.plot_tasks_activation()

Removed in: 4.0

lisa.datautils.SignalDesc.from_event()

No new signals will be added to this list, use explicit signal description where appropriate in the Trace API

Removed in: 4.0

lisa.datautils.series_tunnel_mean()

Replaced by: lisa.datautils.series_envelope_mean()

Removed in: 4.0

lisa.energy_meter.ACME

LISA energy meters are deprecated, please use devlib instruments or contribute the instrument to devlib

Removed in: 4.0

lisa.energy_meter.AEP

LISA energy meters are deprecated, please use devlib instruments or contribute the instrument to devlib

Removed in: 4.0

lisa.energy_meter.Gem5EnergyMeter

LISA energy meters are deprecated, please use devlib instruments or contribute the instrument to devlib

Removed in: 4.0

lisa.energy_meter.HWMon

LISA energy meters are deprecated, please use devlib instruments or contribute the instrument to devlib

Removed in: 4.0

lisa.energy_meter.Monsoon

LISA energy meters are deprecated, please use devlib instruments or contribute the instrument to devlib

Removed in: 4.0

lisa.energy_model.EnergyModel.from_debugfsEM_target()

Replaced by: lisa.energy_model.LinuxEnergyModel.from_target

Removed in: 4.0

lisa.energy_model.EnergyModel.from_sd_target()

Replaced by: lisa.energy_model.LegacyEnergyModel.from_target

Removed in: 4.0

lisa.fuzz.Gen.lift()

Note that GenMonad.do() will not automatically await on arguments if they are Gen instances, this must be done manually.

Replaced by: lisa.monad.MonadTrans.do()

Removed in: 4.0

lisa.trace.CollectorBase.get_trace()

Replaced by: lisa.trace.CollectorBase.get_data()

Removed in: 4.0

lisa.trace.TraceBase.df_events()

This method has been deprecated and is an alias

Replaced by: df_event

Removed in: 4.0

lisa.trace.TraceBase.get_task_by_name()

This method has been deprecated and is an alias

Replaced by: lisa.trace.TraceBase.get_task_name_pids()

Removed in: 4.0

lisa.trace.TraceBase.get_task_by_pid()

This method has been deprecated and is an alias

Replaced by: lisa.analysis.tasks.TasksAnalysis.get_task_by_pid

Removed in: 4.0

lisa.trace.TraceBase.get_task_id()

This method has been deprecated and is an alias

Replaced by: lisa.analysis.tasks.TasksAnalysis.get_task_id

Removed in: 4.0

lisa.trace.TraceBase.get_task_ids()

This method has been deprecated and is an alias

Replaced by: lisa.analysis.tasks.TasksAnalysis.get_task_ids

Removed in: 4.0

lisa.trace.TraceBase.get_task_name_pids()

This method has been deprecated and is an alias

Replaced by: lisa.analysis.tasks.TasksAnalysis.get_task_name_pids

Removed in: 4.0

lisa.trace.TraceBase.get_task_pid()

This method has been deprecated and is an alias

Replaced by: lisa.analysis.tasks.TasksAnalysis.get_task_pid

Removed in: 4.0

lisa.trace.TraceBase.get_task_pid_names()

This method has been deprecated and is an alias

Replaced by: lisa.analysis.tasks.TasksAnalysis.get_task_pid_names

Removed in: 4.0

lisa.trace.TraceBase.get_tasks()

This method has been deprecated and is an alias

Replaced by: lisa.analysis.tasks.TasksAnalysis.get_tasks

Removed in: 4.0

lisa.trace.TraceBase.task_ids()

This property has been deprecated and is an alias

Replaced by: lisa.analysis.tasks.TasksAnalysis.task_ids

Removed in: 4.0

lisa.trace._InternalTraceBase.add_events_deltas()

Prefer adding delta once signals have been extracted from the event dataframe for correctness

Replaced by: lisa.datautils.df_add_delta()

Removed in: 4.0

lisa.trace._InternalTraceBase.df_all_events()

This method has been deprecated and is an alias for “trace.ana.notebook.df_all_events()”

Replaced by: lisa.analysis.notebook.NotebookAnalysis.df_all_event

Removed in: 4.0

lisa.wa_results_collector.WaResultsCollector

Replaced by: lisa.wa.WAOutput

Removed in: 4.0

lisa.wlgen.rta.Periodic

Replaced by lisa.wlgen.rta.RTAPhase along with lisa.wlgen.rta.PeriodicWload workload

Replaced by: lisa.wlgen.rta.RTAPhase

Removed in: 4.0

lisa.wlgen.rta.Pulse

Replaced by: lisa.wlgen.rta.RTAPhase

Removed in: 4.0

lisa.wlgen.rta.RTA.by_profile()

Replaced by: lisa.utils.<lisa.utils.PartialInit._PartialFactory object at 0x7ffa5012c190>

Removed in: 4.0

lisa.wlgen.rta.RTA.by_str()

Replaced by: lisa.utils.<lisa.utils.PartialInit._PartialFactory object at 0x7ffa500a2b10>

Removed in: 4.0

lisa.wlgen.rta.RTATask

Replaced by: lisa.wlgen.rta.RTAPhase

Removed in: 4.0

lisa.wlgen.rta.Ramp

Replaced by: lisa.wlgen.rta.DutyCycleSweepPhase

Removed in: 4.0

lisa.wlgen.rta.RunAndSync

Replaced by lisa.wlgen.rta.RTAPhase along with lisa.wlgen.rta.RunWload and lisa.wlgen.rta.BarrierWload workloads

Replaced by: lisa.wlgen.rta.RTAPhase

Removed in: 4.0

lisa.wlgen.rta.Step

Replaced by: lisa.wlgen.rta.DutyCycleSweepPhase

Removed in: 4.0

lisa.wlgen.workload.Workload.output()

Processed output is returned by run() or by the “.output” attribute of the value returned by the run_background() context manager

Removed in: 4.0

Release Process

Making a new release involves the following steps:

  1. Update version_tuple in lisa.version.

  2. Ensure LISA as a whole refers to relevant versions of:

    • Alpine Linux in lisa._kmod

    • Ubuntu in Vagrantfile

    • Binary dependencies in lisa._assets

    • Android SDK installed by install_base.sh

    • Java version used by Android SDK in install_base.sh

  3. Ensure LISA can work with currently published version of devlib.

  4. Create a vX.Y.Z tag.

  5. Make the Python wheel. See tools/make-release.sh for some indications on that part.

  6. Install that wheel in a _fresh_ Vagrant VM. Ensure that the VM is reinstalled from scratch and that the vagrant box in use is up to date.

  7. Run tools/tests.sh in the VM and ensure no deprecated item scheduled for removal in the new version is still present in the sources (should result in import-time exceptions).

  8. Ensure all CIs in use are happy.

  9. Push the vX.Y.Z tag in the main repo

  10. Update the release branch to be at the same commit as the vX.Y.Z tag.

  11. Upload the wheel on PyPI.

Transitioning from LISA legacy

A big refactoring effort was started in mid 2018, which produced a lot of (much needed) changes. If you are used to using LISA before the refactoring came into place, this guide is for you.

Global changes

Project structure

  • $repo/libs/utils is now $repo/lisa/. $repo/libs/wlgen has also been moved to that location.

  • devlib and wa are now under $repo/external/. Git subtrees are now used instead of submodules.

  • All non self-tests have been moved from $repo/tests to $repo/lisa/tests

Updating your tree

Since we no longer use submodules, a git pull is all you need (lisa-update no longer exists). Also, see Kernel testing.

Python 3

With Python 2 end of life drawing near, we decided to bridge the gap and move over to Python 3. Unlike devlib, we didn’t go for compatibility with both Python 2 and Python 3 - LISA is now Python 3 only.

Imports

LISA legacy used implicit relative imports with a bit of dark magic to hold everything together. Say you want to import the lisa.trace.Trace class found in lisa/trace.py, previously you would do it like so:

from trace import Trace

However, implicit relative imports are dangerous - did you know trace.Trace exists in Python’s standard library? This means that with the previous setup, the LISA module would shadow the standard library’s. The above import done in a non-LISA environment would have imported something completely different!

We now mandate the use of absolute imports, which look like this:

from lisa.trace import Trace

Tip

This can help you figure out what you are really importing:

>>> import trace
>>> print(trace.__path__)
/usr/lib/python3.5/trace.py

if that doesn’t work you can try

>>> print(xxx.__file__)

Warning

Do make sure you haven’t kept some PYTHONPATH tweaking in your .bashrc that could lead to an older LISA/devlib being imported.

Logging

Enabling the LISA logger has changed slightly:

LISA legacy:

import logging
from conf import LisaLogging
LisaLogging.setup()

LISA next:

import logging
from lisa.utils import setup_logging
setup_logging()

Notebooks

The LISA shell command to start notebooks has been changed from lisa-ipython to lisa-jupyter (the actual notebooks have been Jupyter for several years now).

We also use the newer Jupyterlab, as the regular Jupyter notebooks will slowly be phased out - see the Official Jupyter roadmap.

Warning

Jupyterlab breaks the TRAPpy plots that use JS injection (e.g. ILinePlot). You can use the “old” notebooks by clicking Help->Launch Classic Notebook, but that is bound to go away eventually.

Furthermore, in LISA legacy notebooks served as documentations and where the main source of examples. We now have a proper documentation (you’re reading it!), so we greatly trimmed down the number of notebooks we had.

We’ve kept older notebooks in ipynb/deprecated, but they have not been ported over to the new APIs (or even to Python3) so they won’t work. They are there in case we find a reason to bring back some of them.

API Changes

TestEnv

Creating a env.TestEnv used to look like this:

target_conf = {
    # Define the kind of target platform to use for the experiments
    "platform"    : 'linux',

    # Preload settings for a specific target
    "board"       : 'juno',  # juno - JUNO board with mainline hwmon

    # Define devlib module to load
    "modules"     : [
        'bl',           # enable big.LITTLE support
        'cpufreq'       # enable CPUFreq support
    ],

    "host"        : '192.168.0.1',
    "username"    : 'root',
    "password"    : 'root',

    "rtapp-calib" : {
        '0': 361, '1': 138, '2': 138, '3': 352, '4': 360, '5': 353
    }
}

te = TestEnv(target_conf)

The equivalent class to use is now lisa.target.Target. It does not require a mapping to be built anymore.

We now have a dedicated class for the target_conf, see lisa.target.TargetConf. The most notable changes are as follows (see the doc for details):

  • "platform" is now "kind"

  • "board" used to load some target-specific settings, which we got rid of. The closest thing to it is "name" which is just a pretty-printing name and has no extra impact.

  • You don’t have to specify devlib modules to load anymore. All (loadable) modules are now loaded. If you find some module too slow to load, you can specify a list of modules to exclude.

  • LISA used to have target.config JSON file at its root. Its equivalent is now target_conf.yml, which is in YAML.

Cool new feature

Target instances can now be easily be created from the configuration file or via the CLI.

Trace

The lisa.trace.Trace class hasn’t changed much in terms of functionality, but we did rename/move things to make them more coherent.

  • Removed last occurences of camelCase

  • Removed big.LITTLE assumptions and made the code only rely on CPU capacities or frequency domains, where relevant.

  • Constructor now only takes trace files as input, not folders anymore.

  • Trace.data_frame is gone:

LISA legacy:

trace.data_frame.trace_event("sched_switch")
# or
trace.df("sched_switch")

LISA next:

trace.df_event("sched_switch")

Analysis

Most of the analysis functionality provided by LISA legacy has made its way into LISA next, although several functionalities were restructured and merged together. Most methods were moved into different modules as well in an attempt to instore some sense of logic - for instance, analysis.latency.df_latency is now df_task_states(). An exact changelog would fill up your screen, so we recommend having a look at Kernel trace analysis.

Note that a new lisa.analysis.load_tracking module has been added to regroup all load-tracking analysis, and provide wrappers to abstract between our different load tracking trace event versions (e.g. df_tasks_signals())

Analysis function calls must now include their respective module:

LISA legacy:

trace.data_frame.cpu_frequency_transitions(0)

LISA next:

trace.ana.frequency.df_cpu_frequency_transitions(0)

To make autocompletion more useful, all methods returning a pandas.DataFrame will start with df_, whereas all methods rendering a plot will start with plot_.

Cool new feature

Trace events required by the analysis methods are now automatically documented, see df_cpu_frequency_residency() for instance.

wlgen

The lisa.wlgen.rta.RTA class has been simplified somewhat:

LISA legacy:

profile = {}
profile["my_task"] = Periodic(duty_cycle_pct=30).get()

wload = RTA(te.target, "foo", calibration)
wload.conf(kind='profile', params=profile)

LISA next:

profile = {
    'my_task': RTAPhase(
        prop_wload=PeriodicWload(
            duty_cycle_pct=30,
            period=16e-3,
            duration=1,
        )
    )
}

wload = RTA.from_profile(te, "foo", profile, res_dir, calibration)

Kernel tests

The Executor from LISA legacy has been entirely removed, and a new test framework has been put in place. Tests are now coded as pure Python classes, which means they can be imported and executed in scripts/notebooks without any additionnal effort. See Kernel testing for more details about using/writing tests.

Energy Meter

Energy meters are all subclasses of lisa.energy_meter.EnergyMeter. They can now be created in two ways. For lisa.energy_meter.HWMon, this would give:

target = Target.from_default_conf()
res_dir = "/foo/bar"

# Directly build an instance
emeter = HWMon(target, channel_map=..., res_dir=res_dir)

# Or using a configuration file
conf = HWMonConf.from_yaml_map('path/to/hwmon_conf.yml')
emeter = HWMon.from_conf(target, conf, res_dir)

with hwmon_conf.yml containing:

hwmon-conf:
     channel-map: ...

All subclasses of lisa.energy_meter.EnergyMeter have a configuration class named *Conf.