diff --git a/Documentation/applications/testing/nuts/devices.rst b/Documentation/applications/testing/nuts/devices.rst new file mode 100644 index 00000000000..33fd735e95d --- /dev/null +++ b/Documentation/applications/testing/nuts/devices.rst @@ -0,0 +1,15 @@ +======================= +Devices Category (NUTS) +======================= + +This test category tests the behaviour of NuttX devices under `/dev` by +interacting with their character driver interface. The category can be enabled +with ``CONFIG_TESTING_NUTS_DEVICES``. + +The following test suites are included in this category: + +* ``CONFIG_TESTING_NUTS_DEVICES_DEVASCII``: Tests for ``/dev/ascii`` +* ``CONFIG_TESTING_NUTS_DEVICES_DEVCONSOLE``: Tests for ``/dev/console`` +* ``CONFIG_TESTING_NUTS_DEVICES_DEVNULL``: Tests for ``/dev/null`` +* ``CONFIG_TESTING_NUTS_DEVICES_DEVURANDOM``: Tests for ``/dev/urandom`` +* ``CONFIG_TESTING_NUTS_DEVICES_DEVZERO``: Tests for ``/dev/zero`` diff --git a/Documentation/applications/testing/nuts/dstructs.rst b/Documentation/applications/testing/nuts/dstructs.rst new file mode 100644 index 00000000000..2891f0384eb --- /dev/null +++ b/Documentation/applications/testing/nuts/dstructs.rst @@ -0,0 +1,12 @@ +=============================== +Data Structures Category (NUTS) +=============================== + +This test category tests the behaviour of NuttX data structures which are used +in the kernel for several other APIs. The category can be enabled with +``CONFIG_TESTING_NUTS_DSTRUCTS`` + +The following data structures have test suites: + +* ``CONFIG_TESTING_NUTS_DSTRUCTS``: Tests for ``struct circbuf_s`` +* ``CONFIG_TESTING_NUTS_DSTRUCTS``: Tests for ``struct list_node`` diff --git a/Documentation/applications/testing/nuts/index.rst b/Documentation/applications/testing/nuts/index.rst new file mode 100644 index 00000000000..32f00941df9 --- /dev/null +++ b/Documentation/applications/testing/nuts/index.rst @@ -0,0 +1,175 @@ +================================ +NuttX Unit Test Selection (NUTS) +================================ + +NUTS is a selection of unit test suites that can be compiled and run on NuttX +devices to verify correct operation of the OS APIs. NUTS depends on the +:doc:`cmocka ` testing framework in order to +run the unit tests. + +Building & Deploying +-------------------- + +In order to build and deploy NUTS on your target device, you will need to enable +the cmocka library and ``CONFIG_TESTING_NUTS``. Once NUTS is enabled, you will +see a list of available test categories in Kconfig. You may enable the test +categories that you want to run tests for. + +Once a test category is enabled, you will have access to toggles for each test +group within the category. For example, if you enable the 'devices' category, +you will then be able to choose if you want to include the tests for +``/dev/zero`` or ``/dev/null``, etc. By default, all test groups within a +category are enabled and must be opted-out of by disabling their toggle. + +.. note:: + + Some individual tests may have specific dependencies which need to be enabled + first, so do not assume that the visible list of tests in Kconfig is the + complete selection (unless you have enabled showing hidden options in + Kconfig). For example, ``/dev/urandom`` depends on `CONFIG_DEV_URANDOM` to be + enabled, and will not show in the menu unless the dependency is selected. + +Once built and deployed to the target, you can use the following command to +see your test results (example of running ``/dev/zero`` tests only): + +.. code:: console + + nsh> nuts + [==========] /dev/zero: Running 8 test(s). + [ RUN ] open_rdonly + [ OK ] open_rdonly + [ RUN ] open_rdwr + [ OK ] open_rdwr + [ RUN ] open_wronly + [ OK ] open_wronly + [ RUN ] readzero + [ OK ] readzero + [ RUN ] readlarge + [ OK ] readlarge + [ RUN ] writezero + [ OK ] writezero + [ RUN ] writelarge + [ OK ] writelarge + [ RUN ] wrrd + [ OK ] wrrd + [==========] /dev/zero: 8 test(s) run. + [ PASSED ] 8 test(s). + +Included Tests +-------------- + +NUTS tests are split into categories. Within each category there are test +groups/suites which contain all of the individual test cases related to a +specific API. + +.. toctree:: + :caption: Test categories + + devices.rst + dstructs.rst + +Test Cases +---------- + +The output of NUTS displays the pass/fail/skip results of each test case that +has been included in the build. Although the names of the test cases are +intended to describe what the test case is verifying, sometimes more information +is needed. Each test case in NUTS has a corresponding source-code +comment/docstring which includes a description of what the test is verifying. +You can also determine more information by looking at the assertion statements +within the test itself. + +Adding New Tests +---------------- + +In order to add new tests, first take a look at the :doc:`cmocka +` API documentation. + +Then, you should add a new ``.c`` file under the category that you wish to +extend. All of the test cases are static functions that take a ``void **state`` +in order to pass in context from a setup function. Each test case function +should have a docstring comment which describes what the particular test case is +testing (and the name should be as descriptive as possible). Setup and teardown +functions must start with ``setup_`` and ``teardown_`` respectively, where the +suffix is a descriptive name about what is being set up/torn down. + +An example of a test case taken from the circular buffer test suite is: + +.. code:: c + + static int setup_empty_cbuf(void **state) + { + *state = &g_cbuf; + return circbuf_init(&g_cbuf, g_buf, sizeof(g_buf)); + } + + static int teardown_cbuf(void **state) + { + circbuf_uninit(*state); + if (circbuf_is_init(*state)) return -1; + return 0; + } + + /**************************************************************************** + * Name: empty_postinit + * + * Description: + * Tests that a circular buffer is empty right after being initialized. + ****************************************************************************/ + + static void empty_postinit(void **state) + { + struct circbuf_s *cbuf = *state; + + assert_true(circbuf_is_empty(cbuf)); + assert_false(circbuf_is_full(cbuf)); + assert_uint_equal(0, circbuf_used(cbuf)); + assert_uint_equal(CBUF_SIZE, circbuf_space(cbuf)); + } + +Each test group must implement a public function for running all of the test +cases, following the convention: + +.. code:: c + + int nuts_category_group(void) + { + static const struct CMUnitTests tests[] = + { + cmocka_unit_test(this_is_test_one), + cmocka_unit_test_setup_teardown(this_is_test_two, setup_func, + teardown_func), + }; + + return cmocka_run_group_tests_name("group name", tests, NULL, NULL); + } + +where 'category' is the name of the test category, and 'group' is the name of +the test group (i.e. category of 'devices' and group of 'devnull'). + +Inside the ``tests.h`` header file for the category, include the following under +the public function prototypes: + +.. code:: c + + #ifdef CONFIG_TESTING_NUTS_CATEGORY_GROUP + int nuts_category_group(void); + #else + #define nuts_category_group() + #endif /* CONFIG_TESTING_NUTS_CATEGORY_GROUP */ + +where ``CONFIG_TESTING_NUTS_CATEGORY_GROUP`` is the Kconfig option to toggle +on/off the test group. You will have to add this to the NUTS Kconfig file. + +In order to reduce the amount of special switches in the +``Makefile``/``CMakeLists.txt`` files, all source files in each category +directory are included with a wild card. Thus, to avoid compiling unselected +test groups, each test group's C file should be surrounded in the gaurd: + +.. code:: c + + #ifdef CONFIG_TESTING_NUTS_CATEGORY_GROUP + + /* ... your test cases and public test runner function ... */ + + #endif /* CONFIG_TESTING_NUTS_CATEGORY_GROUP */