diff --git a/Documentation/guides/image/syscall-flat-after.png b/Documentation/guides/image/syscall-flat-after.png new file mode 100644 index 00000000000..e274db7f9d5 Binary files /dev/null and b/Documentation/guides/image/syscall-flat-after.png differ diff --git a/Documentation/guides/image/syscall-flat-before.png b/Documentation/guides/image/syscall-flat-before.png new file mode 100644 index 00000000000..1ef26e773b4 Binary files /dev/null and b/Documentation/guides/image/syscall-flat-before.png differ diff --git a/Documentation/guides/image/syscall-protected-after.png b/Documentation/guides/image/syscall-protected-after.png new file mode 100644 index 00000000000..e5fd903871a Binary files /dev/null and b/Documentation/guides/image/syscall-protected-after.png differ diff --git a/Documentation/guides/image/syscall-protected-before.png b/Documentation/guides/image/syscall-protected-before.png new file mode 100644 index 00000000000..64d572463f0 Binary files /dev/null and b/Documentation/guides/image/syscall-protected-before.png differ diff --git a/Documentation/guides/image/task-trace-internal.png b/Documentation/guides/image/task-trace-internal.png new file mode 100644 index 00000000000..e8814c077b1 Binary files /dev/null and b/Documentation/guides/image/task-trace-internal.png differ diff --git a/Documentation/guides/image/task-trace-overview.png b/Documentation/guides/image/task-trace-overview.png new file mode 100644 index 00000000000..4f585b7efab Binary files /dev/null and b/Documentation/guides/image/task-trace-overview.png differ diff --git a/Documentation/guides/image/trace-compass-screenshot.png b/Documentation/guides/image/trace-compass-screenshot.png new file mode 100644 index 00000000000..6e8c841423f Binary files /dev/null and b/Documentation/guides/image/trace-compass-screenshot.png differ diff --git a/Documentation/guides/index.rst b/Documentation/guides/index.rst index 74f7873e732..60862a230e7 100644 --- a/Documentation/guides/index.rst +++ b/Documentation/guides/index.rst @@ -9,3 +9,4 @@ Guides usbtrace.rst simulator.rst drivers.rst + tasktrace.rst diff --git a/Documentation/guides/tasktrace.rst b/Documentation/guides/tasktrace.rst new file mode 100644 index 00000000000..c64ab12b556 --- /dev/null +++ b/Documentation/guides/tasktrace.rst @@ -0,0 +1,20 @@ +========== +Task Trace +========== + +Task Trace is the tool to collect the various events in the NuttX kernel and display the result graphically. + +It can collect the following events. + + - Task execution, termination, switching + - System call enter/leave + - Interrupt handler enter/leave + +.. toctree:: + :maxdepth: 1 + :caption: Contents: + + tasktraceuser.rst + tasktraceinternal.rst + +.. image:: image/task-trace-overview.png diff --git a/Documentation/guides/tasktraceinternal.rst b/Documentation/guides/tasktraceinternal.rst new file mode 100644 index 00000000000..3e015989888 --- /dev/null +++ b/Documentation/guides/tasktraceinternal.rst @@ -0,0 +1,96 @@ +==================== +Task Trace Internals +==================== + +Overview +======== + +.. image:: image/task-trace-internal.png + +The Task Trace is constructed by the following functions. + +NuttX kernel events collection +------------------------------ + +The kernel events are collected by ``sched_note_*()`` API calls embedded in NuttX kernel. + + - For task switch events + + - ``sched_note_start()`` + - ``sched_note_stop()`` + - ``sched_note_suspend()`` + - ``sched_note_resume()`` + + - For system call events + + - ``sched_note_syscall_enter()`` + - ``sched_note_syscall_leave()`` + + - For interrupt event + + - ``sched_note_irqhandler()`` + +Filter logic (``nuttx/sched/sched_note.c``) +------------------------------------------- + +- The ``sched_note_*()`` APIs are implemented here. +- Filter the notes and pass them to noteram driver by ``sched_note_add()`` API. + +Noteram device driver (``nuttx/drivers/note/noteram_driver.c``) +--------------------------------------------------------------- + +- Accumurate incoming note records into the buffer. +- Read the note records from the buffer by user requests. +- The notes are recorded in the binary format of ``struct note_*_s``. +- The detail function is described in :doc:`../reference/os/note`. + +Notectl device driver (``nuttx/drivers/note/notectl_driver.c``) +--------------------------------------------------------------- + +- ``/dev/notectl`` device driver. +- Control the filter logic in ``sched_note.c`` by calling note filter APIs. +- The detail function is described in :doc:`../reference/os/note`. + +"``trace``" Built-In Application (``apps/system/trace/trace.c``) +---------------------------------------------------------------- + +- ``trace`` Built-In Application to control the trace function interactively. +- Read binary note records from ``/dev/note`` and convert into the ftrace text format which is acceptable by `"Trace Compass" `_. +- The command syntax is described in :doc:`tasktraceuser`. + +Getting the system call events +============================== + +To get the system call events, two different methods are used for FLAT build and PROTECTED/KERNEL build. + +FLAT build +---------- + +In FLAT build, a system call is just a function call into the NuttX kernel. + +.. image:: image/syscall-flat-before.png + +To get the system call events, `wrapper function option `_ of the GNU Linker is used. + +The mksyscall tool is fixed to generate the system call wrapper which call system call enter/leave hook. +The wrapper supersedes the system call function call of the NuttX binary by passing ``--wrap`` linker option to the build system. +The wrapper calls the system call hooks before and after calling the real system call function. + +.. image:: image/syscall-flat-after.png + +PROTECTED/KERNEL build +---------------------- + +Different to FLAT build, in PROTECTED and KERNEL build, a system call is issued by an user space is handled as the following steps. + +#. System call issued by an application code is handled by the system call proxy (automatically generated by mksyscall). +#. System call proxy issues the supervisor call instruction to enter into the kernel space. +#. System call handler in the kernel space calls the system call stub (automatically generated by mksyscall). +#. System call stub calls the API implementation in the NuttX kernel. + +.. image:: image/syscall-protected-before.png + +To get the system call events, the mksyscall tool is fixed to generate the system call wrapper which supersedes the system call function call in the system call stub. + +.. image:: image/syscall-protected-after.png + diff --git a/Documentation/guides/tasktraceuser.rst b/Documentation/guides/tasktraceuser.rst new file mode 100644 index 00000000000..82629a6a035 --- /dev/null +++ b/Documentation/guides/tasktraceuser.rst @@ -0,0 +1,348 @@ +===================== +Task Trace User Guide +===================== + +Installation +============ + +Install Trace Compass +--------------------- + +Task Trace uses the external tool `"Trace Compass" `_ to display the trace result. + +Download it from https://www.eclipse.org/tracecompass/ and install into the host environment. +After the installation, execute it and choose ``Tools`` -> ``add-ons`` menu, then select ``Install Extensions`` to install the extension named "Trace Compass ftrace (Incubation)". + +NuttX kernel configuration +-------------------------- + +To enable the task trace function, the NuttX kernel configuration needs to be modified. + +The following configurations must be enabled. + +- ``CONFIG_SCHED_INSTRUMENTATION`` : Enables the feature of scheduler notes. +- ``CONFIG_SCHED_INSTRUMENTATION_FILTER`` : Enables the filter logic of the notes. +- ``CONFIG_SCHED_INSTRUMENTATION_SYSCALL`` : Enable system call instrumentation. +- ``CONFIG_SCHED_INSTRUMENTATION_IRQHANDLER`` : Enables IRQ instrumentation. +- ``CONFIG_DRIVER_NOTE`` : Enables note driver support. +- ``CONFIG_DRIVER_NOTERAM`` : Enables ``/dev/note`` in-memory buffering driver. +- ``CONFIG_DRIVER_NOTECTL`` : Enables ``/dev/notectl`` filter control driver. +- ``CONFIG_SYSTEM_TRACE`` : Enables "``trace``" command +- ``CONFIG_SYSTEM_SYSTEM`` : Enables "``system``" command (required by :ref:`trace_cmd`) + + +The following configurations are configurable parameters for trace. + +- ``CONFIG_SCHED_INSTRUMENTATION_FILTER_DEFAULT_MODE`` + + - Specify the default filter mode. + If the following bits are set, the corresponding instrumentations are enabled on boot. + + - Bit 0 = Enable instrumentation + - Bit 1 = Enable syscall instrumentation + - Bit 2 = Enable IRQ instrumentation + +- ``CONFIG_SCHED_INSTRUMENTATION_NOTERAM_BUFSIZE`` + + - Specify the note buffer size in bytes. + Higher value can hold more note records, but consumes more kernel memory. + +- ``CONFIG_SCHED_INSTRUMENTATION_NOTERAM_DEFAULT_NOOVERWRITE`` + + - If enabled, stop overwriting old notes in the circular buffer when the buffer is full by default. + This is useful to keep instrumentation data of the beginning of a system boot. + +After the configuration, rebuild the NuttX kernel and application. + +If the trace function is enabled, "``trace``" :doc:`../components/nsh/builtin` will be available. + +How to get trace data +===================== + +The trace function can be controlled by "``trace``" command. + +Quick Guide +----------- + +Getting the trace +^^^^^^^^^^^^^^^^^ + +Trace is started by the following command. + + +.. code-block:: + + nsh> trace start + +Trace is stopped by the following command. + +.. code-block:: + + nsh> trace stop + +If you want to get the trace while executing some command, the following command can be used. + +.. code-block:: + + nsh> trace cmd [...] + + +Displaying the trace result +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The trace result is accumulated in the memory. +After getting the trace, the following command displays the accumulated trace data to the console. + +.. code-block:: + + nsh> trace dump + +This will be get the trace results like the followings: + +.. code-block:: + + -1 [0] 7.640000000: sys_close() + -1 [0] 7.640000000: sys_close -> 0 + -1 [0] 7.640000000: sys_sched_lock() + -1 [0] 7.640000000: sys_sched_lock -> 0 + -1 [0] 7.640000000: sys_nxsched_get_stackinfo() + -1 [0] 7.640000000: sys_nxsched_get_stackinfo -> 0 + -1 [0] 7.640000000: sys_sched_unlock() + -1 [0] 7.640000000: sys_sched_unlock -> 0 + -1 [0] 7.640000000: sys_clock_nanosleep() + -1 [0] 7.640000000: sched_switch: prev_comm= prev_pid=1 prev_state=S ==> next_comm= next_pid=0 + -0 [0] 7.640000000: irq_handler_entry: irq=11 + -0 [0] 7.640000000: irq_handler_exit: irq=11 + -0 [0] 7.640000000: irq_handler_entry: irq=15 + -0 [0] 7.650000000: irq_handler_exit: irq=15 + -0 [0] 7.650000000: irq_handler_entry: irq=15 + : + + +By using the logging function of your terminal software, the trace result can be saved into the host environment and it can be used as the input for `"Trace Compass" `_. + +If the target has a storage, the trace result can be stored into the file by using the following command. +It also can be used as the input for "Trace Compass" by transferring the file in the target device to the host. + +.. code-block:: + + nsh> trace dump + + +To display the trace result by `"Trace Compass" `_, choose ``File`` -> ``Open Trace`` menu to specify the trace data file name. + +.. image:: image/trace-compass-screenshot.png + + +Trace command description +========================= + +.. _trace_start: + +trace start +----------- + +Start task tracing + +**Command Syntax:** + +.. code-block:: + + trace start [-c][] + +- ``-c`` : Continue the previous trace. + The trace data is not cleared before starting new trace. + +- ```` : Specify the duration of the trace by seconds. + Task tracing is stopped after the specified period. + If not specified, the tracing continues until stopped by the command. + +.. _trace_stop: + +trace stop +---------- + +Stop task tracing + +**Command Syntax:** + +.. code-block:: + + trace stop + +.. _trace_cmd: + +trace cmd +--------- + +Get the trace while running the specified command. +After the termination of the command, task tracing is stopped. +To use this command, ``CONFIG_SYSTEM_SYSTEM`` needs to be enabled. + +**Command Syntax:** + +.. code-block:: + + trace cmd [-c] [...] + + +- ``-c`` : Continue the previous trace. + The trace data is not cleared before starting new trace. + +- ```` : Specify the command to get the task trace. + +- ```` : Arguments for the command. + +**Example:** + +.. code-block:: + + nsh> trace cmd sleep 1 + +.. _trace_dump: + +trace dump +---------- + +Output the trace result. +If the task trace is running, it is stopped before the output. + +**Command Syntax:** + +.. code-block:: + + trace dump [-c][] + +- ``-c`` : Not stop tracing before the output. + Because dumping trace itself is a task activity and new trace data is added while output, the dump will never stop. + +- ```` : Specify the filename to save the trace result. + If not specified, the trace result is displayed to console. + + +.. _trace_mode: + +trace mode +---------- + +Set the task trace mode options. +The default value is given by the kernel configuration ``CONFIG_SCHED_INSTRUMENTATION_FILTER_DEFAULT_MODE``. + +**Command Syntax:** + +.. code-block:: + + trace mode [{+|-}{o|s|i}...] + +- ``+o`` : Enable overwrite mode. + The trace buffer is a ring buffer and it can overwrite old data if no free space is available in the buffer. + Enables this behavior. + +- ``-o`` : Disable overwrite mode. + The new trace data will be disposed when the buffer is full. + This is useful to keep the data of the beginning of the trace. + +- ``+s`` : Enable system call trace. + It records the event of enter/leave system call which is issued by the application. + All system calls are recorded by default. ``trace syscall`` command can filter the system calls to be recorded. + +- ``-s`` : Disable system call trace. + +- ``+i`` : Enable interrupt trace. + It records the event of enter/leave interrupt handler which is occured while the tracing. + All IRQs are recorded by default. ``trace irq`` command can filter the IRQs to be recorded. + +- ``-i`` : Disable interrupt trace. + +If no command parameters are specified, display the current mode as the follows. + +**Example:** + +.. code-block:: + + nsh> trace mode + Task trace mode: + Trace : enabled + Overwrite : on (+o) + Syscall trace : on (+s) + Filtered Syscalls : 16 + IRQ trace : on (+i) + Filtered IRQs : 2 + +.. _trace_syscall: + +trace syscall +------------- + +Configure the filter of the system call trace. + +**Command Syntax:** + +.. code-block:: + + trace syscall [{+|-}...] + +- ``+`` : Add the specified system call name to the filter. + The execution of the filtered system call is not recorded into the trace data. + +- ``-`` : Remove the specified system call name from the filter. + +Wildcard "``*``" can be used to specify the system call name. +For example, "``trace syscall +sem_*``" filters the system calls begin with "``sem_``", such as ``sem_post()``, ``sem_wait()``,... + +If no command parameters are specified, display the current filter settings as the follows. + +**Example:** + +.. code-block:: console + + nsh> trace syscall + Filtered Syscalls: 16 + getpid + sem_destroy + sem_post + sem_timedwait + sem_trywait + sem_wait + mq_close + mq_getattr + mq_notify + mq_open + mq_receive + mq_send + mq_setattr + mq_timedreceive + mq_timedsend + mq_unlink + +.. _trace_irq: + +trace irq +--------- + +Configure the filter of the interrupt trace. + +**Command Syntax:** + +.. code-block:: + + trace irq [{+|-}...] + +- ``+`` : Add the specified IRQ number to the filter. + The execution of the filtered IRQ handler is not recorded into the trace data. + +- ``-`` : Remove the specified IRQ number from the filter. + +Wildcard "``*``" can be used to specify all IRQs. + +If no command parameters are specified, display the current filter settings as the follows. + +**Example:** + +.. code-block:: console + + nsh> trace irq + Filtered IRQs: 2 + 11 + 15 + diff --git a/Documentation/reference/os/index.rst b/Documentation/reference/os/index.rst index 75f53bad32a..fad70d2b7e1 100644 --- a/Documentation/reference/os/index.rst +++ b/Documentation/reference/os/index.rst @@ -23,3 +23,4 @@ in other header files. paging.rst led.rst iob.rst + note.rst diff --git a/Documentation/reference/os/note.rst b/Documentation/reference/os/note.rst new file mode 100644 index 00000000000..1ce84983bfc --- /dev/null +++ b/Documentation/reference/os/note.rst @@ -0,0 +1,270 @@ +===================== +Note Driver Interface +===================== + +Note driver is the interface to access the instrumentation data. +The following devices are provided. + +- :ref:`notectl` +- :ref:`noteram` + +.. _notectl: + +Notectl Device (``/dev/notectl``) +================================= + + ``/dev/notectl`` is the device to control an instrumentation filter in NuttX kernel. + The device has only ioctl function to control the filter. + +``/dev/notectl`` Header Files +----------------------------- + + The header file ``include/nuttx/note/notectl_driver.h`` provides the interface definitions of the device. + +``/dev/notectl`` Data Structures +-------------------------------- + +.. c:struct:: note_filter_mode_s + + .. code-block:: c + + struct note_filter_mode_s + { + unsigned int flag; /* Filter mode flag */ + #ifdef CONFIG_SMP + unsigned int cpuset; /* The set of monitored CPUs */ + #endif + }; + + - ``flag`` : Filter mode flag. The bitwise OR of the following defines are available. + + .. c:macro:: NOTE_FILTER_MODE_FLAG_ENABLE + + Enable instrumentation + + .. c:macro:: NOTE_FILTER_MODE_FLAG_SYSCALL + + Enable syscall instrumentation + + .. c:macro:: NOTE_FILTER_MODE_FLAG_IRQ + + Enable IRQ instrumentaiton + + - ``cpuset`` : (SMP only) Monitor only CPUs in the bitset. Bit 0=CPU0, Bit1=CPU1, etc. + +.. c:struct:: note_filter_syscall_s + + .. code-block:: c + + struct note_filter_syscall_s + { + uint8_t syscall_mask[]; + }; + + - ``syscall_mask`` : A bitmap array of the syscall filter. If a bit is set, the corresponding syscall is not recorded. + The following helper macros are available: + + .. c:macro:: NOTE_FILTER_SYSCALLMASK_SET(nr, s) + + Set syscall number `nr` as masked. `s` specifies the variable of `struct note_filter_syscall_s` + + .. c:macro:: NOTE_FILTER_SYSCALLMASK_CLR(nr, s) + + Set syscall number `nr` as unmasked. + + .. c:macro:: NOTE_FILTER_SYSCALLMASK_ISSET(nr, s) + + Check whether syscall number `nr` is masked or not. True if masked. + + .. c:macro:: NOTE_FILTER_SYSCALLMASK_ZERO(s) + + Clear all masks. + +.. c:struct:: note_filter_irq_s + + .. code-block:: c + + struct note_filter_irq_s + { + uint8_t irq_mask[]; + }; + + - ``irq_mask`` : A bitmap array of the IRQ filter. If a bit is set, the corresponding IRQ is not recorded. + The following helper macros are available: + + .. c:macro:: NOTE_FILTER_IRQMASK_SET(nr, s) + + Set IRQ number `nr` as masked. `s` specifies the variable of `struct note_filter_irq_s` + + .. c:macro:: NOTE_FILTER_IRQMASK_CLR(nr, s) + + Set IRQ number `nr` as unmasked. + + .. c:macro:: NOTE_FILTER_IRQMASK_ISSET(nr, s) + + Check whether IRQ number `nr` is masked or not. True if masked. + + .. c:macro:: NOTE_FILTER_IRQMASK_ZERO(s) + + Clear all masks. + +``/dev/notectl`` Ioctls +----------------------- + +.. c:macro:: NOTECTL_GETMODE + + Get note filter mode + + :argument: A writable pointer to :c:struct:`note_filter_mode_s` + + :return: If success, 0 (``OK``) is returned and current note filter mode is stored into the given pointer. + If failed, a negated ``errno`` is returned. + +.. c:macro:: NOTECTL_SETMODE + + Set note filter mode + + :argument: A read-only pointer to :c:struct:`note_filter_mode_s` + + :return: If success, 0 (``OK``) is returned and the given filter mode is set as the current settings. + If failed, a negated ``errno`` is returned. + +.. c:macro:: NOTECTL_GETSYSCALLFILTER + + Get syscall filter setting + + :argument: A writable pointer to :c:struct:`note_filter_syscall_s` + + :return: If success, 0 (``OK``) is returned and current syscall filter mode is stored into the given pointer. + If failed, a negated ``errno`` is returned. + +.. c:macro:: NOTECTL_SETSYSCALLFILTER + + Set syscall filter setting + + :argument: A read-only pointer to :c:struct:`note_filter_syscall_s` + + :return: If success, 0 (``OK``) is returned and the given syscall filter mode is set as the current settings. + If failed, a negated ``errno`` is returned. + +.. c:macro:: NOTECTL_GETIRQFILTER + + Get IRQ filter setting + + :argument: A writable pointer to :c:struct:`note_filter_irq_s` + + :return: If success, 0 (``OK``) is returned and current IRQ filter mode is stored into the given pointer. + If failed, a negated ``errno`` is returned. + +.. c:macro:: NOTECTL_SETIRQFILTER + + Set IRQ filter setting + + :argument: A read-only pointer to :c:struct:`note_filter_irq_s` + + :return: If success, 0 (``OK``) is returned and the given IRQ filter mode is set as the current settings. + If failed, a negated ``errno`` is returned. + + +.. _noteram: + +Noteram Device (``/dev/note``) +============================== + + ``/dev/note`` is the device to get the trace (instrumentation) data. + The device has read function to get the data and ioctl function to control the buffer mode. + + +``/dev/note`` Header Files +-------------------------- + + The header file ``include/nuttx/note/noteram_driver.h`` provides the interface definitions of the device. + +``/dev/note`` Ioctls +-------------------- + +.. c:macro:: NOTERAM_CLEAR + + Clear all contents of the circular buffer + + :argument: Ignored + + :return: Always returns 0. + +.. c:macro:: NOTERAM_GETMODE + + Get overwrite mode + + :argument: A writable pointer to ``unsigned int``. + The overwrite mode takes one of the following values. + + .. c:macro:: NOTERAM_MODE_OVERWRITE_DISABLE + + Overwrite mode is disabled. When the buffer is full, accepting the data will be stopped. + + .. c:macro:: NOTERAM_MODE_OVERWRITE_ENABLE + + Overwrite mode is enabled. + + .. c:macro:: NOTERAM_MODE_OVERWRITE_OVERFLOW + + Overwrite mode is disabled and the buffer is already full. + + :return: If success, 0 (``OK``) is returned and current overwrite mode is stored into the given pointer. + If failed, a negated ``errno`` is returned. + +.. c:macro:: NOTERAM_SETMODE + + Set overwrite mode + + :argument: A read-only pointer to ``unsigned int``. + + :return: If success, 0 (``OK``) is returned and the given overwriter mode is set as the current settings. + If failed, a negated ``errno`` is returned. + +Filter control APIs +=================== + +The following APIs are the functions to control note filters directly. +These are kernel APIs and application can use them only in FLAT build. + +The header file ``include/nuttx/sched_note.h`` is needed to use the following APIs. + +API description +--------------- + +.. c:function:: void sched_note_filter_mode(struct note_filter_mode_s *oldm, struct note_filter_mode_s *newm); + + Set and get note filter mode. + (Same as :c:macro:`NOTECTL_GETMODE` / :c:macro:`NOTECTL_SETMODE` ioctls) + + :param oldm: A writable pointer to :c:struct:`note_filter_mode_s` to get current filter mode. + If 0, no data is written. + :param newm: A read-only pointer to :c:struct:`note_filter_mode_s` which holds the new filter mode. + If 0, the filter mode is not updated. + + :return: None + +.. c:function:: void sched_note_filter_syscall(struct note_filter_syscall_s *oldf, struct note_filter_syscall_s *newf); + + Set and get syscall filter setting. + (Same as :c:macro:`NOTECTL_GETSYSCALLFILTER` / :c:macro:`NOTECTL_SETSYSCALLFILTER` ioctls) + + :param oldf: A writable pointer to :c:struct:`note_filter_syscall_s` to get current syscall filter setting. + If 0, no data is written. + :param newf: A read-only pointer to :c:struct:`note_filter_syscall_s` of the new syscall filter setting. + If 0, the setting is not updated. + + :return: None + +.. c:function:: void sched_note_filter_irq(struct note_filter_irq_s *oldf, struct note_filter_irq_s *newf); + + Set and get IRQ filter setting. + (Same as :c:macro:`NOTECTL_GETIRQFILTER` / :c:macro:`NOTECTL_SETIRQFILTER` ioctls) + + :param oldf: A writable pointer to :c:struct:`note_filter_irq_s` to get current IRQ filter setting. + If 0, no data is written. + :param newf: A read-only pointer to :c:struct:`note_filter_irq_s` of the new IRQ filter setting. + If 0, the setting is not updated. + + :return: None