Test Suite: v2023dev0 - Test Cases


ade-commands automated

Test ade command and sub-command with default and custom options and arguments.

critical
agl-compositor manual

Test that agl-compositor is able to run, it can show popups and gnome-menus is able to show and launch installed applications.

high
apertis-update-manager-amd64-rollback manual

Test the automatic rollback and blacklist mechanism of apertis-update-manager.

low
apertis-update-manager-ota-check-network-failure manual

Test the apertis-update-manager API for updates over the air when the network is diconnected and reconnected.

medium
apertis-update-manager-ota-api-switch-network manual

Test the apertis-update-manager API for updates over the air when the network is changed from a phone hotspot to wifi.

medium
apertis-update-manager-ota-rollback manual

Test the automatic rollback and blacklist mechanism of apertis-update-manager with network updates.

low
apertis-update-manager-rollback manual

Test the automatic rollback and blacklist mechanism of apertis-update-manager. The automated version of this test: https://qa.apertis.org/aum-rollback-blacklist.html

low
apertis-update-manager-usb-unplug manual

Test the apertis-update-manager automatic update interrupted by mass storage removing.

critical
apparmor-basic-profiles automated

Test basic apparmor profiles. Only parsing and syntax of profiles are tested for now. Security infrastructure: parsing and syntax of basic Apparmor profiles is part of strategic application and services confinement.

critical
apparmor-bluez-avrcp-volume manual

Test AppArmor profile for Bluez AVRCP volume up and down commands.

medium
apparmor-bluez-setup manual

Test AppArmor profiles for setting up of Bluetooth stack by running basic tests using hciconfig. Check if the interface can be brought up/down, can be changed of mode and can discover devices.

high
apparmor-bluez manual

Test AppArmor profile for support for SPP profile (bluez-spp) and core functionalities (bluez-core).

high
apparmor-chaiwala-system automated

Test that AppArmor's system wide rules works according to Apertis security goals (Chaiwala was an older codename for Apertis).

high
apparmor-dbus automated

Security infrastructure: AppArmor D-Bus mediation is part of strategic application and service confinement.

critical
apparmor-functional-demo automated

Show some main AppArmor functionalities, like transitions, net access or resource permissions, under the form of tests.

medium
apparmor-geoclue automated

Test that geoclue apparmor profiles are working properly.

medium
apparmor-gstreamer1-0 automated

Test that AppArmor profiles provided in GStreamer 1.0 works without Auditd complaining.

medium
apparmor-ofono manual

Test that AppArmor profiles provided in ofono package work without unexpected auditd log entries.

medium
apparmor-pipewire automated

Test that AppArmor profiles provided in PipeWire work without apparmor complaining.

medium
apparmor-pulseaudio automated

Test that AppArmor profiles provided in PulseAudio work without apparmor complaining.

medium
apparmor-session-lockdown-no-deny automated

Test that the session lockdown profile is not blocking more than it should.

medium
apparmor-tumbler automated

Tests that the tumbler AppArmor profile doesn't cause false negatives and that it does not allow arbitrary reading of files in the home directory.

medium
apparmor-utils automated

Test apparmor_parser and other tools in apparmor package can be executed

medium
apparmor automated

Security infrastructure: checking that AppArmor is available in the kernel and active is part of strategic application and services confinement.

critical
aum-api automated

Apertis update manager: API test This test ensures that the update manager API behaves as expected.

critical
aum-offline-upgrade-branch automated

Apertis update manager: offline upgrade test between branches. This test ensures that Apertis is capable to upgrade the previos version to the recent. And update manager is able to fix all refs after successful update.

critical
aum-offline-upgrade-collection_id automated

Apertis update manager: test of update failure with invalid update file. This test ensures that the update manager refuses to install an update with improper Collection ID, refs or timestamps.

critical
aum-offline-upgrade-signed automated

Apertis update manager: check signed offline upgrade test This test ensures that the update manager uses static bundle with signed commit only.

critical
aum-offline-upgrade automated

Apertis update manager: Offline upgrade test This test ensures that the update manager is able to install an update provided offline.

critical
aum-ota-api automated

Apertis update manager: OTA API test This test ensures that the update manager API behaves as expected.

critical
aum-ota-auto automated

Apertis update manager: automatic OTA updates test This test ensures that the update manager is able to work in unattended mode.

critical
aum-ota-out-of-space automated

Apertis update manager: Out of space update test during OTA upgrade This test ensures that the update manager behaves appropriately in low disk space conditions.

critical
aum-ota-rollback-blacklist automated

Apertis update manager: blacklist and rollback for OTA updates. This test ensures that the update manager is able to rollback and blacklist an invalid update.

critical
aum-ota-signed automated

Apertis update manager: check signed OTA upgrade test This test ensures that the update manager uses commit signed with known key during OTA upgrades.

critical
aum-out-of-space automated

Apertis update manager: Out of space update test This test ensures that the update manager behaves appropriately in low disk space conditions.

critical
aum-power-cut automated

Apertis update manager: Power cut resiliency test This test ensures that the update manager resists a power cut.

critical
aum-rollback-blacklist automated

Apertis update manager: Blacklist and rollback This test ensures that the update manager is able to rollback and blacklist an invalid update.

critical
bluez-avrcp-volume manual

Test the AVRCP volume up and down commands.

medium
bluez-file-transfer manual

Test all BlueZ features needing interaction with a phone like file transfer of images or audio/video files

critical
bluez-hfp-call-hold manual

Test Hands Free Profile (HFP) BlueZ feature call and hold.

medium
bluez-hfp-incoming-call-notification manual

Test Hands Free Profile (HFP) BlueZ feature for incoming call.

medium
bluez-hfp manual

Test Hands Free Profile (HFP) BlueZ feature.

medium
bluez-phone manual

Test all BlueZ features needing interaction with a phone (Pairing, OBEX, PBAP, AVRCP, A2DP, HSP, HFP, PAN). Strategic component: Bluetooth phone calls is part of smartphone interaction functionality.

critical
bluez-setup manual

Test setup of Bluetooth stack by running basic tests using hciconfig. Check if the interface can be brought up/down, can be changed of mode and can discover devices. Strategic component: Bluetooth setup and basic functionality is part of smartphone interaction functionality.

critical
boot-no-crashes automated

Check that nothing crashed when booting.

high
boot-performance automated

Test boot performance. This test checks boot performance using systemd-analyze statistics.

high
cgroups-resource-control automated

Run cgroup resource control tests

medium
connman-new-supplicant automated

Test the wpa_supplicant interface.

medium
connman-pan-network-access-reconnect manual

Test ConnMan support for bluetooth PAN Network Access Profile. This test will test that after restarting the Bluetooth tethering from the phone side the internet access from Apertis to a bluetooth device sharing its internet connection is maintained.

medium
connman-pan-network-access manual

Test ConnMan support for bluetooth PAN Network Access Profile. This test will test internet access from Apertis to a bluetooth device sharing its internet connection. The underlying connection makes use of Bluetooth Network Access Profile with connman. First the two devices will be paired, then connection will be established. It will then be possible to verify that internet is usable from Apertis.

medium
connman-pan-tethering-reboot manual

Test Connman-pan-tethering support for PAN connectivity after reboot.

medium
connman-pan-tethering manual

Test ConnMan support for PAN connectivity.

medium
connman-services-reboot manual

Tests Network connection management: ensuring that ConnMan can connect to actual networks even after the device under test is rebooted

critical
connman-services-switchover manual

Tests Network connection management: ensuring that ConnMan can connect to actual networks even after the device under test is rebooted

critical
connman-services manual

Tests Network connection management: ensuring that ConnMan can connect to actual networks is part of strategic network management functionality.

critical
connman-usb-tethering manual

Test ConnMan support for USB tethering (connection side).

medium
connman automated

Runs some automated tests using the ConnMan D-Bus API. It guarantees most of the basic features are working. Network connection management: ensuring that ConnMan's APIs are accessible is part of strategic network management functionality.

critical
dbus-dos-reply-time automated

Check the reply time of a few D-Bus method calls while dbus-daemon is loaded by D-Bus traffic. Different patterns of D-Bus loads are tested: 1) A connection calling a D-Bus method repetitively. 2) A process calling a D-Bus method repetitively using a different connection each time to generate traffic related to connections joining and leaving the bus (NameOwnerChanged signals). 3) The same with 100 threads generating traffic. 4) Tens of threads registering hundreds of D-Bus match rules DoS resistance is part of strategic basic services functionality.

high
dbus-installed-tests automated

Regression tests for D-Bus.

high
devroot-manual-arm64 manual

Test cross-compilation support for ARM64 architecture using devroot utility

high
disk-rootfs-fsck automated

Ensure that the root file system check is enabled on boot

high
eclipse-plugins-install-target manual

Test copying files to the remote device from within Eclipse. It will create a dummy project and then create a configuration to copy the binary file to a remote device.

medium
eclipse-plugins-remote-debugging manual

Test remote debugging within Eclipse. It will create a dummy project and then create a configuration to copy the binary file to a remote device and debug it remotely using Eclipse.

medium
eclipse-sysroot-management manual

Test sysroot management and update within Eclipse. It will go through configuring an existing sysroot in Eclipse, and then updating it from the web with the latest version.

medium
evolution-sync-bluetooth manual

Synchronizing Bluetooth-paired phone contacts to Evolution.

medium
fixedfunction-webserver manual

Fixed function Web server.

medium
flatpak-build-mildenhall-helloworld-app manual

Install mildenhall runtime, build helloworld-app using mildenhall runtime and run the helloworld-app.

medium
flatpak-run-demo-cli-app automated

Test a headless Flatpak demo application

medium
flatpak-run-demo-hmi-app manual

Test a graphical Flatpak demo application

medium
flatpak-run-mildenhall-helloworld-app manual

Install helloworld-app from the flatpak repo.

medium
gettext-i18n automated

Check that gettext internationalization works.

medium
glib-gio-fs automated

Run the GLib gio tests that do I/O on the local filesystem.

medium
grilo automated

Check grilo metadata notifications, and filesystem browsing.

medium
gstreamer1-0-decode automated

Test video and audio decoders and demuxers on a list of media. Audio/video decoding through GStreamer is important but not required by any of the currently strategic functionalities.

high
gupnp-services automated

Run a few tests related to service browsing, introspection, etc.

medium
hmi-audio-play-pause manual

Test that the songs present play and can be paused

high
hmi-audio-policy manual

Test that audio policy to mute song application when switching to video application

high
hmi-audio-skip-start-end manual

Test that the songs can be rewind or fast-forwarded

high
hmi-audio-volume-control manual

Test that the volume controls work as expected

high
hmi-flatpak-rhythmbox-app-launch manual

Test that the Rhythmbox app can be launched from the agl-compositor

high
hmi-flatpak-rhythmbox-uninstall manual

Test that the Rhytmbox app can be successfully uninstalled

high
hmi-video-load manual

Tests that the video files are present

high
hmi-video-play-pause manual

Test that the video files play without any glitches

high
image-bootable manual

Test whether the image is bootable. Basic functionality: booting up.

critical
image-gui-start manual

Test whether the image starts a graphical environment. Basic functionality: ensuring that the GUI boots up.

critical
iptables-basic automated

Network connection management: ensuring that iptables is available and usable is part of strategic networking management functionality.

critical
iptables-nmap automated

Test the firewall using nmap from another computer.

high
licensing-summary-availability manual

Test the availability of the licensing summary.

high
newport automated

Run the Newport test suite

medium
nfsroot-simple-boot automated

This test checks that an image that was flashed using the nfsroot properly booted.

high
ofono-sms-receive manual

Test receiving SMS messages with oFono.

medium
ofono-sms-send manual

Test sending of SMS with oFono.

medium
ostree-collection-id automated

OSTree: Collection ID test This test ensures that the system is booted with proper Collection ID.

critical
polkit-parsing automated

Parse polkit policy files

high
rfkill-toggle manual

Test that rfkill is functional.

medium
robotframework manual

This test checks that the Robot Framework can be run in SDK, that the framework can connect to a target via SSH, upload files, run commands and retrieve command output.

critical
sanity-check-manual manual

This test checks that the system bus is up, that a user session is created and that a graphical interface is successfully running in the system, as well as other similar basic services so that a stable and reliable environment can be confirmed before futher tests are executed. Basic functionality: services.

critical
sanity-check automated

This test checks that the system bus is up, that a user session is created and that a graphical interface is successfully running in the system, as well as other similar basic services so that a stable and reliable environment can be confirmed before futher tests are executed. Basic functionality: services.

critical
sdk-ade-remote-debugging manual

Test remote debugging with ADE. Cross-build an example HelloWorld agent, deploy it to the target device and debug it with proper backtraces.

high
sdk-code-analysis-tools-gcov automated

Checks basic gcov functionality by running a sample program a few different ways and ensuring that a sufficient portion of its code paths are executed.

medium
sdk-code-analysis-tools-splint automated

Checks basic splint functionality by checking its output when run on a simple source file.

medium
sdk-cross-compilation automated

Attempt building bash package in the devroot SDK images ship.

medium
sdk-dbus-tools-bustle manual

Test that bustle-dbus-monitor shows services on the session bus.

medium
sdk-dbus-tools-d-feet manual

Test that D-Feet both shows services on the session bus and that new services appear automatically.

medium
sdk-debos-image-building automated

Ensure that the sample image recipes can be built on the SDK and that the commands in the documentation work.

critical
sdk-debug-tools-gdb automated

Checks basic gdb functionality by stopping a program at a set point and checking its backtrace matches expectations.

medium
sdk-debug-tools-strace automated

Checks basic strace functionality by checking its output when run on a simple command line program.

medium
sdk-debug-tools-valgrind automated

Checks basic valgrind functionality by running a program with and without memory leaks, ensuring expected results from valgrind.

medium
sdk-docker automated

Install and run hello-world docker container.

medium
sdk-ide-build-run-program manual

Creates a simple test project and confirms that compiling and executing work.

medium
sdk-import-debian-package automated

Checks basic import-debian-package functionality.

medium
sdk-performance-tools-gprof automated

Checks basic gprof functionality by ensuring a CPU-intensive program shows up in its log as consuming the vast majority of the CPU cycles.

medium
sdk-performance-tools-sysprof automated

Checks basic sysprof functionality by ensuring a CPU-intensive program shows up in its log as consuming the vast majority of the CPU cycles.

medium
sdk-persistent-disk manual

Tests that persistent disk can be used when upgrading the SDK.

critical
sdk-vb-fullscreen manual

Tests that the SDK VM can be fullscreened in VirtualBox. Development environment: ensuring that the VirtualBox SDK provides a seamless experience. Check that when fullscreened the output is not stretched, even for accelerated contents.

critical
secure-boot-imx6 manual

Test that Secure Boot via HAB on the SabreLite board works for bootloader and initial OS image in FIT format.

medium
tiny-container-system-aa-enforcement automated

Test that the AppArmor profile for dbus-daemon is loaded in system-wide privileged container

medium
tiny-container-system-basic automated

Test of Tiny system-wide privileged container using shared networking with the host

medium
tiny-container-system-connectivity-profile automated

Test of AppArmor profile for Tiny system-wide privileged container

medium
tiny-container-system-device-sharing automated

Test of device sharing between host and Tiny system-wide privileged container

medium
tiny-container-system-folder-sharing automated

Test of folder sharing between host and Tiny system-wide privileged container started as user

medium
tiny-container-system-seccomp automated

Test that the custom seccomp profile could be applied for system-wide privileged container

medium
tiny-container-user-aa-enforcement automated

Test that the AppArmor profile for dbus-daemon is loaded in unprivileged container started as user

medium
tiny-container-user-basic automated

Test of Tiny unprivileged container without any networking started as user

medium
tiny-container-user-connectivity-profile automated

Test of AppArmor profile for Tiny unprivileged container started as user

medium
tiny-container-user-device-sharing automated

Test of device sharing between host and Tiny unprivileged container started as user

medium
tiny-container-user-folder-sharing automated

Test of folder sharing between host and Tiny unprivileged container started as user

medium
tiny-container-user-seccomp automated

Test that the custom seccomp profile could be applied for unprivileged container started as user

medium
traprain automated

Traprain library test.

medium
tumbler-thumbnailing automated

Check that all thumbnailing features of tumbler are working properly (large, small, videos, documents, images).

medium
video-animation-on-boot manual

Test whether Plymouth is able to display a graphical theme.

high
webkit2gtk-ac-3d-rendering manual

Test that 3D transforms are applied correctly.

low
webkit2gtk-gstreamer1.0 manual

Tests html5 video playback with GStreamer 1.0 in WebKit2GTK. This test opens an URL to stream video and audio using MiniBrowser.

medium
webkit2gtk-mt-touch-events manual

Test implementation of W3C touch events API in WebKit2GTK.

low
x-hw-accelerated manual

Test whether the correct GLES2 render is used.

medium