This code owner file is auto-generated by the Python script in tools/
based off the git logs. It will allow new contributors to get better
auto-suggestions for which reviewers to request on their PRs. It also
creates an easy way to track down authors of certain NuttX subsystems
when refactors/changes are to be made.
Signed-off-by: Matteo Golin <matteo.golin@gmail.com>
This PR adds support for the CI system for native Windows as well. It allows you to build NuttX on GitHub and test it locally for Windows users.
With these CI tools with PowerShell scripts, it is possible to build NuttX for Windows Native using (for now only) Cmake + Ninja with the same logic as the CI system with Bash scripts.
This allows the msvc job to be used not only with the simulator (currently only with Visual Studio 17 2022), but also with other architectures using the same Windows runner to get more coverage and avoid future breakage.
As with the other jobs, we use artifacts to save the compilation result at the end of the workflow execution (previously for the simulator it was not done).
The proposed solution is based on the following additions and modified:
Modified Files
buildyml -> only CI Jobs MSVC
New Files in tools/
ci/cibuild.ps1 -> Added Powershell script for Run the CI Builds
ci/platforms/windows.ps1 -> Added Powershell script for installing toolchains and tools.
testlist/windows.dat -> Target (Add sim (msvc), risc-v arm)
tools/testbuild.ps1
We tested the NuttX build on GitHub and locally.
How we build on GitHub and test locally.
Locally
cd .\nuttx\tools\ci\
.\cibuild.ps1 -n -i -A -C -N .\testlist\windows.dat
Signed-off-by: simbit18 <simbit18@gmail.com>
This issue is related to the Arm toolchain for Windows which is available for x86 host architecture only (compatible with x86_64)
Windows (mingw-w64-i686) hosted cross toolchains
AArch32 bare-metal target (arm-none-eabi)
Issue
/bin/sh: line 1: /home/nuttx/nuttxnew/tools/gcc-arm-none-eabi/bin/arm-none-eabi-ar: Argument list too long
On Windows, arm-none-eabi-ar can only accept strings up to a maximum length of 32,768 characters.
We could suppress the 32K include string limitation by setting the CMake variable CMAKE_NINJA_FORCE_RESPONSE_FILE to ON.
This is unfortunately not enough!!! ): In the build phase this error comes out
$ cmake --build build
[2/1025] Building ASM object arch/CMakeFiles/arch.dir/arm/src/armv7-m/arm_exception.S.obj
FAILED: arch/CMakeFiles/arch.dir/arm/src/armv7-m/arm_exception.S.obj
/home/nuttx/nuttxnew/tools/gcc-arm-none-eabi/bin/arm-none-eabi-gcc.exe @arch/CMakeFiles/arch.dir/arm/src/armv7-m/arm_exception.S.obj.rsp -MD -MT arch/CMakeFiles/arch.dir/arm/src/armv7-m/arm_exception.S.obj -MF arch/CMakeFiles/arch.dir/arm/src/armv7-m/arm_exception.S.obj.d -o arch/CMakeFiles/arch.dir/arm/src/armv7-m/arm_exception.S.obj -c /home/nuttx/nxninja/nuttx/arch/arm/src/armv7-m/arm_exception.S
C:/msys64/home/nuttx/nxninja/nuttx/arch/arm/src/armv7-m/arm_exception.S:42:10: fatal error: nuttx/config.h: No such file or directory
42 | #include <nuttx/config.h>
| ^~~~~~~~~~~~~~~~
compilation terminated.
The Workround I found to solve this problem is to overwrite
the responsible file flag CMAKE_${lang}_RESPONSE_FILE_FLAG with $DEFINES $INCLUDES $FLAGS
Maybe there is a better solution but this one it works. :)
CI Test will sometimes run for 6 hours (before getting auto-terminated by GitHub):
- https://github.com/apache/nuttx/issues/14808
- https://github.com/apache/nuttx/issues/14680
This is a problem because:
- It will increase our usage of GitHub Runners. Which may overrun the [GitHub Actions Budget](https://infra.apache.org/github-actions-policy.html) allocated by ASF.
- Suppose right after CI Test there's another build. If CI Test runs for all 6 hours, then the build after CI Test will never run.
For this PR: We assume that Every CI Job (e.g. risc-v-05) will complete normally within 2 hours. If any CI Job exceeds 2 hours: This PR will kill the CI Test Process `pytest` and allow the next build to run.
When compiling the Release Branch of `nuttx-apps`, our CI Workflow `build.yml` checks out the Master Branch of `nuttx` repo, which is incorrect. This happens due to a typo in `build.yml`: https://github.com/apache/nuttx/issues/14513
This PR fixes the typo in `build.yml` to checkout the correct branch of `nuttx` repo.
Due to the [recent cost-cutting](https://github.com/apache/nuttx/issues/14376), we are no longer running PR Merge Jobs in the `nuttx` and `nuttx-apps` repos. For this to happen, I am now running a script on my computer that will cancel any PR Merge Jobs that appear: [kill-push-master.sh](https://github.com/lupyuen/nuttx-release/blob/main/kill-push-master.sh)
This PR disables PR Merge Jobs permanently, so that we no longer need to run the script. This prevents our CI Charges from over-running, in case the script fails to operate properly.
This PR enables the CI Builds for msys2, msvc, arm-08, arm-10, arm-13 for Complex PRs. We disable the CI Builds for arm-01, arm-09, arm-11.
This will help to fix the recent breakage of builds: https://github.com/apache/nuttx/issues/14598
This PR splits the CI Build Job sim-01 and adds sim-03:
Before the Split: Simulator Jobs take up to 1.5 hours to complete
- sim-01 (1 hour 31 mins): adb, citest, lvgl, matter
- sim-02 (28 mins): posix_test, sqlite
After the Split: Simulator Jobs will complete within 1 hour
- sim-01 (58 mins): adb, citest
- sim-02 (35 mins): lvgl, matter
- sim-03 (28 mins): posix_test, sqlite
This will help us comply with the ASF Policy for GitHub Actions, as explained here: https://github.com/apache/nuttx/issues/14376
Not all labels are added because there is no match with the label name.
issue_labeler.yml
os: Linux -> os: linux
001_bug_report.yml
Type: bug -> Type: Bug
Corrected label in links
002_feature_request.yml
Type: enhancement -> Type: Enhancement
Corrected label in links
003_help.yml
Type: question ->Type: Question
Corrected label in links
This PR updates the Build Rules `arch.yml` to build only these Arm32 Targets when we create or update a Complex PR:
- arm-01, arm-03, arm-05, arm-06, arm-07, arm-09, arm-11
No changes for Simple PRs (arm-01 to arm-14) and for Merging PRs (also arm-01 to arm-14).
This will improve our breadth of CI Checks across Arm32 Targets, as explained here: https://github.com/apache/nuttx/issues/14376
This PR creates the new CI Build Jobs `arm64-01` and `x86_64-01`. The new jobs will split and offload the Arm64 and x86_64 Build Targets from `other`. This will reduce our usage of GitHub Runners, to comply with the ASF Policy for GitHub Actions. (Recently we see more PRs for Arm64 and x86_64)
Before the Split: Simple PRs (One Arch and/or One Board) for Arm64 and x86_64 require almost 1 hour for CI Build
- `other` (57 mins): AVR, SPARC, x86, PinePhone, QEMU Arm64, QEMU x86_64
After the Split: Simple PRs for Arm64 and x86_64 will complete under 30 mins
- `other` (24 mins): AVR, SPARC, x86
- `arm64-01` (29 mins): PinePhone, QEMU Arm64
- `x86_64-01` (9 mins): QEMU x86_64
To skip more unnecessary builds: Our Build Rules `arch.yml` shall ignore the label "Area: Documentation", so that a Simple PR + Docs is still a Simple PR. Previously we experienced longer CI Build Times, just because we added docs to our Simple PR. (Now our PR shall be built exactly like a Simple PR)
The updated CI code is explained here: https://github.com/apache/nuttx/issues/13775
When we submit or update a Complex PR that affects All Architectures (Arm, RISC-V, Xtensa, etc): CI Workflow shall run only half the jobs for RISC-V, Xtensa and Simulator:
- `risc-v-01` to `03`
- `xtensa-01`
- `sim-01`
When the Complex PR is Merged: CI Workflow will still run all jobs for RISC-V, Xtensa and Simulator:
- `risc-v-01` to `06`
- `xtensa-01` to `02`
- `sim-01` to `02`
Simple PRs with One Single Arch / Board will build the same way as before:
- `risc-v-01` to `06`
- `xtensa-01` to `02`
- `sim-01` to `02`
We hope to lower drastically our usage of GitHub Runners before the ASF Deadline, as explained here: https://github.com/apache/nuttx/issues/14376
When we submit or update a Complex PR that affects All Architectures (Arm, RISC-V, Xtensa, etc): CI Workflow shall run only half the jobs. Previously CI Workflow will run `arm-01` to `arm-14`, now we will run only `arm-01` to `arm-07`.
When the Complex PR is Merged: CI Workflow will still run all jobs `arm-01` to `arm-14`
Simple PRs with One Single Arch / Board will build the same way as before: `arm-01` to `arm-14`
This is explained here: https://github.com/apache/nuttx/issues/14376
Note that this version of `arch.yml` has diverged from `nuttx-apps`, since we are unable to merge https://github.com/apache/nuttx/pull/14377
To avoid the following CI error:
This environment is externally managed
--> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.12/README.venv for more information.
This PR extends the CI Build Rules to eliminate more unnecessary builds. We use the Board Label for PRs:
- Previously: "Arch: arm" will build only `arm-01` to `arm-14`
- Now: "Board: arm" will also build `arm-01` to `arm-14`
- This applies only to Simple PRs: One Arch Label + One Board Label + One Size Label, like "Arch: arm, Board: arm, Size: L"
- If Arch Label and Board Label are both present: They must be the same. Otherwise All Targets shall be built.
- Works with Arm32, Arm64, RISC-V, Simulator, x86_64 and Xtensa
The updated code is explained here: https://github.com/apache/nuttx/issues/13775
To speed up the CI Workflow, this PR splits the CI Build Jobs for RISC-V into smaller jobs. Each job will now complete within 1 hour.
Before the PR: There are 2 jobs for RISC-V, each requiring more than 1.5 hours
- `risc-v-01` (1 hour 42 mins): BL602, Ox64, ESP32-C3 / C6 / H2
- `risc-v-02` (1 hour 41 mins): K230, Icicle, QEMU, RV32M1-Vega
After the PR: The build is spread across 6 jobs for RISC-V, each job completes within 1 hour
- `risc-v-01` (19 mins): BL602, Ox64
- `risc-v-02` (44 mins): ESP32-C3
- `risc-v-03` (45 mins): ESP32-C6, ESP32-H2
- `risc-v-04` (31 mins): K230, Icicle
- `risc-v-05` (41 mins): QEMU CITest
- `risc-v-06` (38 mins): Rest of QEMU, RV32M1-Vega
Following the same convention as the Arm32 Build Jobs, the above jobs are sorted by Target Name. Performance of the RISC-V Build Jobs is discussed in https://github.com/apache/nuttx/issues/13775
CI Build Job `arm-05` (runtime 2 hours) has become the Performance Bottleneck for CI Workflow. That's because `arm-05` builds too many targets for nRF, RP2040, SAM 3, SAM A and SAM D. This PR splits `arm-05` into multiple smaller jobs, to reduce the CI Build Duration.
Before the PR: `arm-05` is overloaded, build requires 2 hours
- `arm-05` (2 hours): nRF, RP2040, SAM 3, SAM A, SAM D
- `arm-06` (56 mins): STM32 [a-m]*
After the PR: `arm-05` is offloaded (to `arm-06` and `arm-07`), completes within 1 hour
- `arm-05` (47 mins): nRF
- `arm-06` (1 hour): Reserve for RP2040 exclusively
- `arm-07` (1 hour 15 mins): SAM 3, SAM A, SAM D, STM32 [a-m]*
Build Jobs are sorted by Target Name. So we cascade the changes and rename the Build Jobs: `arm-07` becomes `arm-08`, `arm-08` becomes `arm-09` etc. Then `arm-13` becomes a new job `arm-14`. (Which we added to `build.yml`)
Performance of `arm-05` is discussed in https://github.com/apache/nuttx/issues/13775 and https://github.com/apache/nuttx/issues/12773
This PR continues to enhance the CI Workflow, to skip the unnecessary NuttX Builds. The PR will update the CI Build Workflow `build.yml`, to call the Refactored Build Rules in `arch.yml` (which is a Reusable Workflow).
The original rules were migrated from `build.yml` to `arch.yml`:
- We target only the Simple PRs: One Arch Label + One Size Label (e.g. "Arch: risc-v, Size: XS")
- For "Arch: risc-v": Build `risc-v-01`, `risc-v-02`
- For "Arch: xtensa": Build `xtensa-01`, `xtensa-02`
- The above rules apply when the PR is Created or Modified
- When the PR is Merged: All targets shall be built
This PR applies the New and Updated Rules defined in `arch.yml`:
- For "Arch: arm": Build `arm-01`, `arm-02`, ...
- For "Arch: arm64": Build `other`
- For "Arch: simulator": Build `sim-01`, `sim-02`
- For "Arch: x86_64": Build `other`
- For Simple PRs (One Arch Label + One Size Label): Skip the macOS and Windows builds (`macos`, `macos/sim-*`, `msys2`) since these builds are costly and slow
- Except for "Arch: Simulator", which will enable the macOS Builds for `sim-01` and `sim-02`
- If GitHub CLI Fails: Build all targets
The code is explained here: https://github.com/apache/nuttx/issues/13775
This PR continues to enhance the CI Workflow, to skip the unnecessary NuttX Builds. The changes in this PR will not take effect until the next PR, which will switch `build.yml` to use the rules in this PR.
In this PR, we refactor the CI Build Rules into a separate Reusable Workflow `arch.yml`. The original rules were migrated to `arch.yml`:
- We target only the Simple PRs: One Arch Label + One Size Label (e.g. "Arch: risc-v, Size: XS")
- For "Arch: risc-v": Build `risc-v-01`, `risc-v-02`
- For "Arch: xtensa": Build `xtensa-01`, `xtensa-02`
- The above rules apply when the PR is Created or Modified
- When the PR is Merged: All targets shall be built
New and Updated Rules:
- For "Arch: arm": Build `arm-01`, `arm-02`, ...
- For "Arch: arm64": Build `other`
- For "Arch: simulator": Build `sim-01`, `sim-02`
- For "Arch: x86_64": Build `other`
- For Simple PRs (One Arch Label + One Size Label): Skip the macOS and Windows builds (`macos`, `macos/sim-*`, `msys2`) since these builds are costly and slow
- Except for "Arch: Simulator", which will enable the macOS Builds for `sim-01` and `sim-02`
- If GitHub CLI Fails: Build all targets
The code is explained here: https://github.com/apache/nuttx/issues/13775
When building a branch like `releases/12.7`, the CI Workflow incorrectly checks out the `master` branch of `nuttx-apps`, instead of `releases/12.7`. This PR fixes a typo in `apps_ref`, to check out the correct branch.
This PR proposes to enhance the CI Workflow, to skip the unnecessary NuttX Builds. Currently, NuttX Devs wait for the CI Builds to complete across All Architectures (Arm32, Arm64, RISC-V, Xtensa), even though they modified a Single Architecture. With this PR, the CI Workflow will build only the Modified Architecture.
The solution uses the Arch Labels for PRs. We target only the Simple PRs: One Arch Label + One Size Label (e.g. "Arch: risc-v, Size: XS")
- If "Arch: arm / arm64" is the only non-size label, then we build `other`, `arm-01`, `arm-02`, ...
- If "Arch: risc-v" is the only non-size label, then build `risc-v-01`, `risc-v-02`
- If "Arch: xtensa" is the only non-size label, then build `xtensa-01`, `xtensa-02`
- The above rules apply when the PR is Created or Modified
- When the PR is Merged: All targets shall be built
The code is explained here: https://github.com/apache/nuttx/issues/13775
This change adds a specifier to github lables created with "new issue":
- for labels related to architecture specifier is "Arch: xxx"
- for labels related to issue type specifier is "Type: xxx"
- for labels related to host OS type specifier is "OS: xxx"
- for labels related to issue area type specifier is "Area: xxx"
The simple improvement is designed to speed up compilation and reduce download errors on github and local.
Added a folder nxtmpdir for storing third-party packages
nuttxworkspace
|
|- nuttx
|- apps
|- nxtmpdir
tools/Unix.mk:
added export NXTMPDIR := $(WSDIR)/nxtmpdir
tools/configure.sh:
added option -S creates the nxtmpdir folder for third-party packages.
tools/Config.mk:
added macro
CLONE - Git clone repository.
CHECK_COMMITSHA - Check if the branch contains the commit SHA-1.
tools/testbuild.sh:
added option -S
For now I added in the folder this package
ESP_HAL_3RDPARTY_URL = https://github.com/espressif/esp-hal-3rdparty.git
ARCH
arch/xtensa/src/esp32/Make.defs
arch/xtensa/src/esp32s2/Make.defs
arch/xtensa/src/esp32s3/Make.defs
arch/risc-v/src/common/espressif/Make.defs
arch/risc-v/src/esp32c3-legacy/Make.defs
but you can also add other packages (maybe also of apps)
This PR modifies NuttX CI and GitHub Actions, to comply with ASF Policy. Right now, every NuttX Pull Request will trigger 24 Concurrent Jobs (GitHub Runners), executing them in parallel: https://lupyuen.github.io/articles/ci
According to ASF Policy: We should run at most 15 Concurrent Jobs: https://infra.apache.org/github-actions-policy.html
Thus we'll cut down the Concurrent Jobs from 24 down to 15. That's 12 Linux Jobs, 2 macOS, 1 Windows. (Each job takes 30 mins to 2 hours)
(1) Right now our "Linux > Strategy" is a flat list of 20 Linux Jobs, all executed in parallel
(2) We change "Linux > Strategy" to prioritise by Target Architecture, and limit to 12 concurrent jobs
(3) So NuttX CI will initially execute 12 Build Jobs across Arm32, Arm64, RISC-V, Simulator and Xtensa. As they complete, NuttX CI will execute the remaining 8 Build Jobs (for Arm32).
(4) This will extend the Overall Build Duration from [2 hours](https://github.com/apache/nuttx/actions/runs/10817443237) to [2.25 hours](https://github.com/lupyuen4/ci-nuttx/actions/runs/10828246630)
(5) We'll also limit macOS Jobs to 2, Windows Jobs to 1
