diff options
Diffstat (limited to 'doc/develop')
| -rw-r--r-- | doc/develop/index.rst | 25 | ||||
| -rw-r--r-- | doc/develop/menus.rst | 131 | ||||
| -rw-r--r-- | doc/develop/py_testing.rst | 426 | ||||
| -rw-r--r-- | doc/develop/testing.rst | 96 |
4 files changed, 675 insertions, 3 deletions
diff --git a/doc/develop/index.rst b/doc/develop/index.rst index 0a7e204b34..beaa64d8d9 100644 --- a/doc/develop/index.rst +++ b/doc/develop/index.rst @@ -3,13 +3,32 @@ Develop U-Boot ============== +Implementation +-------------- .. toctree:: - :maxdepth: 2 + :maxdepth: 1 - coccinelle commands - crash_dumps global_data logging + menus + +Debugging +--------- + +.. toctree:: + :maxdepth: 1 + + crash_dumps trace + +Testing +------- + +.. toctree:: + :maxdepth: 1 + + coccinelle + testing + py_testing diff --git a/doc/develop/menus.rst b/doc/develop/menus.rst new file mode 100644 index 0000000000..dda8f963fb --- /dev/null +++ b/doc/develop/menus.rst @@ -0,0 +1,131 @@ +.. SPDX-License-Identifier: GPL-2.0+ +.. Copyright 2010-2011 Calxeda, Inc. + +Menus +===== + +U-Boot provides a set of interfaces for creating and using simple, text +based menus. Menus are displayed as lists of labeled entries on the +console, and an entry can be selected by entering its label. + +To use the menu code, enable CONFIG_MENU, and include "menu.h" where +the interfaces should be available. + +Menus are composed of items. Each item has a key used to identify it in +the menu, and an opaque pointer to data controlled by the consumer. + +If you want to show a menu, instead starting the shell, define +CONFIG_AUTOBOOT_MENU_SHOW. You have to code the int menu_show(int bootdelay) +function, which handle your menu. This function returns the remaining +bootdelay. + +Interfaces +---------- + +.. code-block:: c + + #include "menu.h" + + /* + * Consumers of the menu interfaces will use a struct menu * as the + * handle for a menu. struct menu is only fully defined in menu.c, + * preventing consumers of the menu interfaces from accessing its + * contents directly. + */ + struct menu; + + /* + * NOTE: See comments in common/menu.c for more detailed documentation on + * these interfaces. + */ + + /* + * menu_create() - Creates a menu handle with default settings + */ + struct menu *menu_create(char *title, int timeout, int prompt, + void (*item_data_print)(void *), + char *(*item_choice)(void *), + void *item_choice_data); + + /* + * menu_item_add() - Adds or replaces a menu item + */ + int menu_item_add(struct menu *m, char *item_key, void *item_data); + + /* + * menu_default_set() - Sets the default choice for the menu + */ + int menu_default_set(struct menu *m, char *item_key); + + /* + * menu_default_choice() - Set *choice to point to the default item's data + */ + int menu_default_choice(struct menu *m, void **choice); + + /* + * menu_get_choice() - Returns the user's selected menu entry, or the + * default if the menu is set to not prompt or the timeout expires. + */ + int menu_get_choice(struct menu *m, void **choice); + + /* + * menu_destroy() - frees the memory used by a menu and its items. + */ + int menu_destroy(struct menu *m); + + /* + * menu_display_statusline(struct menu *m); + * shows a statusline for every menu_display call. + */ + void menu_display_statusline(struct menu *m); + +Example Code +------------ + +This example creates a menu that always prompts, and allows the user +to pick from a list of tools. The item key and data are the same. + +.. code-block:: c + + #include "menu.h" + + char *tools[] = { + "Hammer", + "Screwdriver", + "Nail gun", + NULL + }; + + char *pick_a_tool(void) + { + struct menu *m; + int i; + char *tool = NULL; + + m = menu_create("Tools", 0, 1, NULL); + + for(i = 0; tools[i]; i++) { + if (menu_item_add(m, tools[i], tools[i]) != 1) { + printf("failed to add item!"); + menu_destroy(m); + return NULL; + } + } + + if (menu_get_choice(m, (void **)&tool) != 1) + printf("Problem picking tool!\n"); + + menu_destroy(m); + + return tool; + } + + void caller(void) + { + char *tool = pick_a_tool(); + + if (tool) { + printf("picked a tool: %s\n", tool); + use_tool(tool); + } + } diff --git a/doc/develop/py_testing.rst b/doc/develop/py_testing.rst new file mode 100644 index 0000000000..f71e837aa9 --- /dev/null +++ b/doc/develop/py_testing.rst @@ -0,0 +1,426 @@ +U-Boot pytest suite +=================== + +Introduction +------------ + +This tool aims to test U-Boot by executing U-Boot shell commands using the +console interface. A single top-level script exists to execute or attach to the +U-Boot console, run the entire script of tests against it, and summarize the +results. Advantages of this approach are: + +- Testing is performed in the same way a user or script would interact with + U-Boot; there can be no disconnect. +- There is no need to write or embed test-related code into U-Boot itself. + It is asserted that writing test-related code in Python is simpler and more + flexible than writing it all in C. +- It is reasonably simple to interact with U-Boot in this way. + +Requirements +------------ + +The test suite is implemented using pytest. Interaction with the U-Boot console +involves executing some binary and interacting with its stdin/stdout. You will +need to implement various "hook" scripts that are called by the test suite at +the appropriate time. + +In order to run the test suite at a minimum we require that both Python 3 and +pip for Python 3 are installed. All of the required python modules are +described in the requirements.txt file in the /test/py/ directory and can be +installed via the command + +.. code-block:: bash + + pip install -r requirements.txt + +In order to execute certain tests on their supported platforms other tools +will be required. The following is an incomplete list: + +* gdisk +* dfu-util +* dtc +* openssl +* sudo OR guestmount +* e2fsprogs +* util-linux +* coreutils +* dosfstools +* efitools +* mount +* mtools +* sbsigntool +* udisks2 + +Please use the appropriate commands for your distribution to match these tools +up with the package that provides them. + +The test script supports either: + +- Executing a sandbox port of U-Boot on the local machine as a sub-process, + and interacting with it over stdin/stdout. +- Executing an external "hook" scripts to flash a U-Boot binary onto a + physical board, attach to the board's console stream, and reset the board. + Further details are described later. + +Using `virtualenv` to provide requirements +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The recommended way to run the test suite, in order to ensure reproducibility +is to use `virtualenv` to set up the necessary environment. This can be done +via the following commands: + + +.. code-block:: console + + $ cd /path/to/u-boot + $ sudo apt-get install python3 python3-virtualenv + $ virtualenv -p /usr/bin/python3 venv + $ . ./venv/bin/activate + $ pip install -r test/py/requirements.txt + +Testing sandbox +--------------- + +To run the test suite on the sandbox port (U-Boot built as a native user-space +application), simply execute: + +.. code-block:: bash + + ./test/py/test.py --bd sandbox --build + +The `--bd` option tells the test suite which board type is being tested. This +lets the test suite know which features the board has, and hence exactly what +can be tested. + +The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may +omit this option and build U-Boot yourself, in whatever way you choose, before +running the test script. + +The test script will attach to U-Boot, execute all valid tests for the board, +then print a summary of the test process. A complete log of the test session +will be written to `${build_dir}/test-log.html`. This is best viewed in a web +browser, but may be read directly as plain text, perhaps with the aid of the +`html2text` utility. + +Testing under a debugger +~~~~~~~~~~~~~~~~~~~~~~~~ + +If you need to run sandbox under a debugger, you may pass the command-line +option `--gdbserver COMM`. This causes two things to happens: + +- Instead of running U-Boot directly, it will be run under gdbserver, with + debug communication via the channel `COMM`. You can attach a debugger to the + sandbox process in order to debug it. See `man gdbserver` and the example + below for details of valid values for `COMM`. +- All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of + time to execute commands. This is useful if U-Boot is stopped at a breakpoint + during debugging. + +A usage example is: + +Window 1: + +.. code-block:: bash + + ./test/py/test.py --bd sandbox --gdbserver localhost:1234 + +Window 2: + +.. code-block:: bash + + gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234' + +Alternatively, you could leave off the `-ex` option and type the command +manually into gdb once it starts. + +You can use any debugger you wish, as long as it speaks the gdb remote +protocol, or any graphical wrapper around gdb. + +Some tests deliberately cause the sandbox process to exit, e.g. to test the +reset command, or sandbox's CTRL-C handling. When this happens, you will need +to attach the debugger to the new sandbox instance. If these tests are not +relevant to your debugging session, you can skip them using pytest's -k +command-line option; see the next section. + +Command-line options +-------------------- + +--board-type, --bd, -B + set the type of the board to be tested. For example, `sandbox` or `seaboard`. + +--board-identity`, --id + sets the identity of the board to be tested. This allows differentiation + between multiple instances of the same type of physical board that are + attached to the same host machine. This parameter is not interpreted by th + test script in any way, but rather is simply passed to the hook scripts + described below, and may be used in any site-specific way deemed necessary. + +--build + indicates that the test script should compile U-Boot itself before running + the tests. If using this option, make sure that any environment variables + required by the build process are already set, such as `$CROSS_COMPILE`. + +--buildman + indicates that `--build` should use buildman to build U-Boot. There is no need + to set $CROSS_COMPILE` in this case since buildman handles it. + +--build-dir + sets the directory containing the compiled U-Boot binaries. If omitted, this + is `${source_dir}/build-${board_type}`. + +--result-dir + sets the directory to write results, such as log files, into. + If omitted, the build directory is used. + +--persistent-data-dir + sets the directory used to store persistent test data. This is test data that + may be re-used across test runs, such as file-system images. + +`pytest` also implements a number of its own command-line options. Commonly used +options are mentioned below. Please see `pytest` documentation for complete +details. Execute `py.test --version` for a brief summary. Note that U-Boot's +test.py script passes all command-line arguments directly to `pytest` for +processing. + +-k + selects which tests to run. The default is to run all known tests. This + option takes a single argument which is used to filter test names. Simple + logical operators are supported. For example: + + - `'-k ums'` runs only tests with "ums" in their name. + - `'-k ut_dm'` runs only tests with "ut_dm" in their name. Note that in this + case, "ut_dm" is a parameter to a test rather than the test name. The full + test name is e.g. "test_ut[ut_dm_leak]". + - `'-k not reset'` runs everything except tests with "reset" in their name. + - `'-k ut or hush'` runs only tests with "ut" or "hush" in their name. + - `'-k not (ut or hush)'` runs everything except tests with "ut" or "hush" in + their name. + +-s + prevents pytest from hiding a test's stdout. This allows you to see + U-Boot's console log in real time on pytest's stdout. + +Testing real hardware +--------------------- + +The tools and techniques used to interact with real hardware will vary +radically between different host and target systems, and the whims of the user. +For this reason, the test suite does not attempt to directly interact with real +hardware in any way. Rather, it executes a standardized set of "hook" scripts +via `$PATH`. These scripts implement certain actions on behalf of the test +suite. This keeps the test suite simple and isolated from system variances +unrelated to U-Boot features. + +Hook scripts +~~~~~~~~~~~~ + +Environment variables +''''''''''''''''''''' + +The following environment variables are set when running hook scripts: + +- `UBOOT_BOARD_TYPE` the board type being tested. +- `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was + specified. +- `UBOOT_SOURCE_DIR` the U-Boot source directory. +- `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory. +- `UBOOT_BUILD_DIR` the U-Boot build directory. +- `UBOOT_RESULT_DIR` the test result directory. +- `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory. + +u-boot-test-console +''''''''''''''''''' + +This script provides access to the U-Boot console. The script's stdin/stdout +should be connected to the board's console. This process should continue to run +indefinitely, until killed. The test suite will run this script in parallel +with all other hooks. + +This script may be implemented e.g. by executing `cu`, `kermit`, `conmux`, etc. +via exec(). + +If you are able to run U-Boot under a hardware simulator such as QEMU, then +you would likely spawn that simulator from this script. However, note that +`u-boot-test-reset` may be called multiple times per test script run, and must +cause U-Boot to start execution from scratch each time. Hopefully your +simulator includes a virtual reset button! If not, you can launch the +simulator from `u-boot-test-reset` instead, while arranging for this console +process to always communicate with the current simulator instance. + +u-boot-test-flash +''''''''''''''''' + +Prior to running the test suite against a board, some arrangement must be made +so that the board executes the particular U-Boot binary to be tested. Often +this involves writing the U-Boot binary to the board's flash ROM. The test +suite calls this hook script for that purpose. + +This script should perform the entire flashing process synchronously; the +script should only exit once flashing is complete, and a board reset will +cause the newly flashed U-Boot binary to be executed. + +It is conceivable that this script will do nothing. This might be useful in +the following cases: + +- Some other process has already written the desired U-Boot binary into the + board's flash prior to running the test suite. +- The board allows U-Boot to be downloaded directly into RAM, and executed + from there. Use of this feature will reduce wear on the board's flash, so + may be preferable if available, and if cold boot testing of U-Boot is not + required. If this feature is used, the `u-boot-test-reset` script should + perform this download, since the board could conceivably be reset multiple + times in a single test run. + +It is up to the user to determine if those situations exist, and to code this +hook script appropriately. + +This script will typically be implemented by calling out to some SoC- or +board-specific vendor flashing utility. + +u-boot-test-reset +''''''''''''''''' + +Whenever the test suite needs to reset the target board, this script is +executed. This is guaranteed to happen at least once, prior to executing the +first test function. If any test fails, the test infra-structure will execute +this script again to restore U-Boot to an operational state before running the +next test function. + +This script will likely be implemented by communicating with some form of +relay or electronic switch attached to the board's reset signal. + +The semantics of this script require that when it is executed, U-Boot will +start running from scratch. If the U-Boot binary to be tested has been written +to flash, pulsing the board's reset signal is likely all this script needs to +do. However, in some scenarios, this script may perform other actions. For +example, it may call out to some SoC- or board-specific vendor utility in order +to download the U-Boot binary directly into RAM and execute it. This would +avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus +saving wear on the flash chip(s). + +Examples +'''''''' + +https://github.com/swarren/uboot-test-hooks contains some working example hook +scripts, and may be useful as a reference when implementing hook scripts for +your platform. These scripts are not considered part of U-Boot itself. + +Board-type-specific configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each board has a different configuration and behaviour. Many of these +differences can be automatically detected by parsing the `.config` file in the +build directory. However, some differences can't yet be handled automatically. + +For each board, an optional Python module `u_boot_board_${board_type}` may exist +to provide board-specific information to the test script. Any global value +defined in these modules is available for use by any test function. The data +contained in these scripts must be purely derived from U-Boot source code. +Hence, these configuration files are part of the U-Boot source tree too. + +Execution environment configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each user's hardware setup may enable testing different subsets of the features +implemented by a particular board's configuration of U-Boot. For example, a +U-Boot configuration may support USB device mode and USB Mass Storage, but this +can only be tested if a USB cable is connected between the board and the host +machine running the test script. + +For each board, optional Python modules `u_boot_boardenv_${board_type}` and +`u_boot_boardenv_${board_type}_${board_identity}` may exist to provide +board-specific and board-identity-specific information to the test script. Any +global value defined in these modules is available for use by any test +function. The data contained in these is specific to a particular user's +hardware configuration. Hence, these configuration files are not part of the +U-Boot source tree, and should be installed outside of the source tree. Users +should set `$PYTHONPATH` prior to running the test script to allow these +modules to be loaded. + +Board module parameter usage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The test scripts rely on the following variables being defined by the board +module: + +- none at present + +U-Boot `.config` feature usage +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The test scripts rely on various U-Boot `.config` features, either directly in +order to test those features, or indirectly in order to query information from +the running U-Boot instance in order to test other features. + +One example is that testing of the `md` command requires knowledge of a RAM +address to use for the test. This data is parsed from the output of the +`bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled. + +For a complete list of dependencies, please search the test scripts for +instances of: + +- `buildconfig.get(...` +- `@pytest.mark.buildconfigspec(...` +- `@pytest.mark.notbuildconfigspec(...` + +Complete invocation example +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and +any required environment configuration Python modules into $HOME/ubtest/py, +then you would likely invoke the test script as follows: + +If U-Boot has already been built: + +.. code-block:: bash + + PATH=$HOME/ubtest/bin:$PATH \ + PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ + ./test/py/test.py --bd seaboard + +If you want the test script to compile U-Boot for you too, then you likely +need to set `$CROSS_COMPILE` to allow this, and invoke the test script as +follows: + +.. code-block:: bash + + CROSS_COMPILE=arm-none-eabi- \ + PATH=$HOME/ubtest/bin:$PATH \ + PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ + ./test/py/test.py --bd seaboard --build + +or, using buildman to handle it: + +.. code-block:: bash + + PATH=$HOME/ubtest/bin:$PATH \ + PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \ + ./test/py/test.py --bd seaboard --build --buildman + +Writing tests +------------- + +Please refer to the pytest documentation for details of writing pytest tests. +Details specific to the U-Boot test suite are described below. + +A test fixture named `u_boot_console` should be used by each test function. This +provides the means to interact with the U-Boot console, and retrieve board and +environment configuration information. + +The function `u_boot_console.run_command()` executes a shell command on the +U-Boot console, and returns all output from that command. This allows +validation or interpretation of the command output. This function validates +that certain strings are not seen on the U-Boot console. These include shell +error messages and the U-Boot sign-on message (in order to detect unexpected +board resets). See the source of `u_boot_console_base.py` for a complete list of +"bad" strings. Some test scenarios are expected to trigger these strings. Use +`u_boot_console.disable_check()` to temporarily disable checking for specific +strings. See `test_unknown_cmd.py` for an example. + +Board- and board-environment configuration values may be accessed as sub-fields +of the `u_boot_console.config` object, for example +`u_boot_console.config.ram_base`. + +Build configuration values (from `.config`) may be accessed via the dictionary +`u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable +names. diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst new file mode 100644 index 0000000000..4bc9ca3a6a --- /dev/null +++ b/doc/develop/testing.rst @@ -0,0 +1,96 @@ +Testing in U-Boot +================= + +U-Boot has a large amount of code. This file describes how this code is +tested and what tests you should write when adding a new feature. + + +Running tests +------------- + +To run most tests on sandbox, type this: + + make check + +in the U-Boot directory. Note that only the pytest suite is run using this +command. + +Some tests take ages to run. To run just the quick ones, type this: + + make qcheck + + +Sandbox +------- +U-Boot can be built as a user-space application (e.g. for Linux). This +allows test to be executed without needing target hardware. The 'sandbox' +target provides this feature and it is widely used in tests. + + +Pytest Suite +------------ + +Many tests are available using the pytest suite, in test/py. This can run +either on sandbox or on real hardware. It relies on the U-Boot console to +inject test commands and check the result. It is slower to run than C code, +but provides the ability to unify lots of tests and summarise their results. + +You can run the tests on sandbox with: + + ./test/py/test.py --bd sandbox --build + +This will produce HTML output in build-sandbox/test-log.html + +See test/py/README.md for more information about the pytest suite. + + +tbot +---- + +Tbot provides a way to execute tests on target hardware. It is intended for +trying out both U-Boot and Linux (and potentially other software) on a +number of boards automatically. It can be used to create a continuous test +environment. See http://www.tbot.tools for more information. + + +Ad-hoc tests +------------ + +There are several ad-hoc tests which run outside the pytest environment: + + test/fs - File system test (shell script) + test/image - FIT and legacy image tests (shell script and Python) + test/stdint - A test that stdint.h can be used in U-Boot (shell script) + trace - Test for the tracing feature (shell script) + +TODO: Move these into pytest. + + +When to write tests +------------------- + +If you add code to U-Boot without a test you are taking a risk. Even if you +perform thorough manual testing at the time of submission, it may break when +future changes are made to U-Boot. It may even break when applied to mainline, +if other changes interact with it. A good mindset is that untested code +probably doesn't work and should be deleted. + +You can assume that the Pytest suite will be run before patches are accepted +to mainline, so this provides protection against future breakage. + +On the other hand there is quite a bit of code that is not covered with tests, +or is covered sparingly. So here are some suggestions: + +- If you are adding a new uclass, add a sandbox driver and a test that uses it +- If you are modifying code covered by an existing test, add a new test case + to cover your changes +- If the code you are modifying has not tests, consider writing one. Even a + very basic test is useful, and may be picked up and enhanced by others. It + is much easier to add onto a test - writing a new large test can seem + daunting to most contributors. + + +Future work +----------- + +Converting existing shell scripts into pytest tests. |