Mirrors_BOOKS/LDD-LinuxDeviceDrivers
1
0
Fork 0
mirror of https://github.com/gatieme/LDD-LinuxDeviceDrivers.git synced 2026-08-20 22:04:56 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

description/scheduler: autonuma numa_balancing conflict with load_balancing

Browse Source
This commit is contained in:
gatieme
2022-02-15 23:24:12 +08:00
parent 93fe0cec83
commit a7069fceea
2 changed files with 33 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
study/kernel/00-DESCRIPTION/PERFORMANCE.md
+3
View File
@@ -80,6 +80,9 @@ https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.17-Faster-csum_parti
https://www.phoronix.com/scan.php?page=news_item&px=UDP-IPV6-5P-Optimization
[UDP IPv6 Optimizations Queued Up For Linux 5.18](https://www.phoronix.com/scan.php?page=news_item&px=UDP-IPv6-Opts-5.18)
# 3 IO
-------
study/kernel/00-DESCRIPTION/SCHEDULER.md
+30 -2
View File
@@ -811,6 +811,7 @@ Peter 将 sched/numa 的整体思路上也做了不断的调整和改动, 也开
| 2012/12/07 | Mel Gorman <mgorman@suse.de> | [Automatic NUMA Balancing V11](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?id=d28d433512f4f387e2563c14db45a7bb8a338b1a) | 方案大量借鉴了 Peter sched/numa 的方案 [Latest numa/core patches, v15](https://lore.kernel.org/lkml/1352826834-11774-1-git-send-email-mingo@kernel.org) | v11 ☑ 3.8-rc1 | [LORE v4 00/46](https://lore.kernel.org/lkml/1353493312-8069-1-git-send-email-mgorman@suse.de)<br>*-*-*-*-*-*-*-* <br>[LORE v10,00/49](https://lore.kernel.org/lkml/1354875832-9700-1-git-send-email-mgorman@suse.de), [LKML v10,00/49](https://lkml.org/lkml/2012/12/7/119)<br>*-*-*-*-*-*-*-* <br>[LORE v11,00/50](https://lore.kernel.org/lkml/20121212100338.GS1009@suse.de) |
| 2013/10/07 | Mel Gorman <mgorman@suse.de> | [Basic scheduler support for automatic NUMA balancing V9](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?id=2739d3eef3a93a92c366a3a0bb85a0afe09e8b8c) | | v9 ☑ 3.13-rc1 | [LORE v2,00/13](https://lore.kernel.org/lkml/1372861300-9973-1-git-send-email-mgorman@suse.de), [LKML v8](https://lkml.org/lkml/2013/9/27/211), [LORE 00/63](https://lore.kernel.org/all/1381141781-10992-1-git-send-email-mgorman@suse.de) |
| 2015/06/16 | Srikar Dronamraju <srikar@linux.vnet.ibm.com> | [Improve numa load balancing](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?id=44dcb04f0ea8eaac3b9c9d3172416efc5a950214) | 存在一些情况进程会被移出其首选节点, 但它们最终可能会被 NUMA Balancing 再带回其首选节点. 为了避免上述情况, [实现 migrate_degrades_locality() 替代 migrate_improves_locality()](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2a1ed24ce94036d00a7c5d5e99a77a80f0aa556a) 来处理 NUMA 下 can_migrate_task 的 cache hot. 它还用 NUMA sched_feature 替换了 3 个 sched_feature NUMA、NUMA_Upper 和 NUMA_RESIST_LOWER. 此外[比较 NUMA 域负载的时候使用了 imbalance_pct](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=44dcb04f0ea8eaac3b9c9d3172416efc5a950214). 补丁集只合入了前两个补丁. | v2 ☑✓ | [LORE v2,0/4](https://lore.kernel.org/all/1434455762-30857-1-git-send-email-srikar@linux.vnet.ibm.com) |
| 2015/06/25 | Srikar Dronamraju <srikar@linux.vnet.ibm.com> | [sched/numa: Update numa_balancing stats in /proc](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?id=397f2378f136128623fc237746157aa2564d1082) | 添加 numa_balancing 的统计调试信息.<br>1. `/proc/sched_debug` 中显示进程所在 NUMA node task_node(p) 以及其 numa_group id task_numa_group_id(p).<br>2. `/proc/pid/sched` 中显示进程 numa_balancing 更详细的信息. | v1 ☑✓ 4.2-rc1 | [LORE v1,0/3](https://lore.kernel.org/all/1435252903-1081-1-git-send-email-srikar@linux.vnet.ibm.com) |
### 4.3.2 Automatic NUMA balancing 的优化
@@ -840,13 +841,25 @@ commit [6e5fb223e89d ("mm: sched: numa: Implement constant, per task Working Set
* Limit Scan Rate
| 时间 | 作者 | 特性 | 描述 | 是否合入主线 | 链接 |
|:----:|:----:|:---:|:---:|:----------:|:----:|
| 2015/03/25 | Mel Gorman <mgorman@suse.de> | [mm: numa: slow PTE scan rate if migration failures occur](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=074c238177a75f5e79af3b2cb6a84e54823ef950) | 通常情况下, 扫描速率会根据故障的位置按任务进行调整. 但是, 如果由于任何原因迁移失败, 那么如果故障仍然是远程的, PTE 扫描可能会扫描得更快. 这意味着在我们知道迁移不可能发生的时候, 系统 CPU 开销和 FAULT 的速率获会更高. 因此跟踪迁移失败的时间, 减慢 PTE 扫描的速度. | v1 ☑ 4.0-rc6 | [COMMIT](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=074c238177a75f5e79af3b2cb6a84e54823ef950) |
| 2022/01/28 | Rik van Riel <riel@redhat.com> | [sched,numa: cap pte scanning overhead to 3% of run time](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?id=51170840fe91dfca10fd533b303ea39b2524782a) | task 级别基于运行时的 NUMA 扫描与 mm 级别的 NUMA 扫描之间存在根本性的不匹配. 在一个严重过载、进程非常大的系统上, 这种不匹配可能会导致系统将所有时间都花在 change_prot_numa() 上. 如果任务在 change_prot_numa() 中花费至少两个 ticks 的时间, 并且在 mm 的两个扫描间隔之间进程仅获得两个 ticks 的运行时间, 则可能会发生这种情况.<br>这组补丁确保任务在扫描 PTE 时的花费不会超过运行时的 3%. 它通过确保在 task_numa_work() 运行之间, 任务在其他事情上花费的时间至少是 task_numa_work() 的 32 倍.<br>如果在 task_numa_work() 期间发生 tick, 或者任务被重新调度, 我们会延迟 task_numa_work() 的未来运行, 直到任务在task_numa_work() 内花费的CPU时间至少是其他事情的 32 倍. | v1 ☑ 4.5-rc1 | [LORE 0/2](https://lore.kernel.org/all/1446756983-28173-1-git-send-email-riel@redhat.com) |
| 2015/11/23 | Rik van Riel <riel@redhat.com> | [sched,numa: cap pte scanning overhead to 3% of run time](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?id=51170840fe91dfca10fd533b303ea39b2524782a) | task 级别基于运行时的 NUMA 扫描与 mm 级别的 NUMA 扫描之间存在根本性的不匹配. 在一个严重过载、进程非常大的系统上, 这种不匹配可能会导致系统将所有时间都花在 change_prot_numa() 上. 如果任务在 change_prot_numa() 中花费至少两个 ticks 的时间, 并且在 mm 的两个扫描间隔之间进程仅获得两个 ticks 的运行时间, 则可能会发生这种情况.<br>这组补丁确保任务在扫描 PTE 时的花费不会超过运行时的 3%. 它通过确保在 task_numa_work() 运行之间, 任务在其他事情上花费的时间至少是 task_numa_work() 的 32 倍.<br>如果在 task_numa_work() 期间发生 tick, 或者任务被重新调度, 我们会延迟 task_numa_work() 的未来运行, 直到任务在task_numa_work() 内花费的CPU时间至少是其他事情的 32 倍. | v1 ☑ 4.5-rc1 | [LORE 0/2](https://lore.kernel.org/all/1446756983-28173-1-git-send-email-riel@redhat.com) |
| 2017/07/31 | riel@redhat.com <riel@redhat.com> | [numa,sched: improve performance for multi-threaded workloads](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?id=b5dd77c8bdada7b6262d0cba02a6ed525bf4e6e1) | NUMA Balancing 在运行多线程工作负载时花费了太多的 CPU 时间进行扫描和故障处理. 当存在大量共享故障时, 以及在处理具有大量共享故障的大型 NUMA 组时, 此补丁集会降低 NUMA PTE 扫描的速度. | v1 ☑✓ v4.14-rc1 | [LORE v1,0/2](https://lore.kernel.org/all/20170731192847.23050-1-riel@redhat.com) |
| 2018/05/04 | Mel Gorman <mgorman@techsingularity.net> | [sched/numa: Stagger NUMA balancing scan periods for new threads v2](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1378447598432513d94ce2c607c412dc4f260f31) | 线程共享一个地址空间, 每个线程都可以更改同一地址空间的保护以捕获 NUMA 故障. 这是多余的, 而且可能会适得其反, 因为任何执行更新的线程都已足够. 可能只需要一个线程, 但该线程可能处于空闲状态, 或者可能没有任何局部性问题, 并选择了不合适的扫描速率.<br>该补丁使用独立的扫描周期, 但它们是根据创建线程时地址空间用户的数量错开的. 其目的是, 线程将避免同时扫描, 并有机会在必要时调整其扫描速率. 这会减少线程生命周期早期的总扫描活动.<br>性能测试表明: 总体性能差异不大, 但系统 CPU 使用率和总体扫描活动都有所降低. | v2 ☑ 4.18-rc1 | [LORE v2,0/5](https://lore.kernel.org/all/20180504154109.mvrha2qo5wdl65vr@techsingularity.net) |
* Limit Scan Range
最早的 commit [6e5fb223e89d ("mm: sched: numa: Implement constant, per task Working Set Sampling (WSS) rate")](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6e5fb223e89dbe5cb5c563f8d4a4a0a7d62455a8) 通过 sysctl_numa_balancing_scan_size 来限制每次扫描的大小.
随后 commit [("sched/numa: Set the scan rate proportional to the memory usage of the task being scanned")](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=598f0ec0bc996e90a806ee9564af919ea5aad401) 将扫描划分为 window 去扫描.
| 时间 | 作者 | 特性 | 描述 | 是否合入主线 | 链接 |
|:----:|:----:|:---:|:---:|:----------:|:----:|
| 2015/09/11 | Rik van Riel <tipbot@zytor.com> | [sched/numa: Limit the amount of virtual memory scanned in task_numa_work()](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4620f8c1fda2af4ccbd11e194e2dd785f7d7f279) | 20150911090027.4a7987bd@annuminas.surriel.com | v1 ☑✓ | [LORE](https://lore.kernel.org/all/20150911090027.4a7987bd@annuminas.surriel.com) |
* Process Adaptive
| 时间 | 作者 | 特性 | 描述 | 是否合入主线 | 链接 |
@@ -982,6 +995,21 @@ t/torvalds/linux.git/log/?id=b6a60cf36d497e7fbde9dd5b86fabd96850249f6) 进行了
| 2014/10/17 | Rik van Riel <riel@redhat.com> | [sched,numa: weigh nearby nodes for task placement on complex NUMA topologies (v2)](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?id=9de05d48711cd5314920ed05f873d84eaf66ccf1) | 1413530994-9732-1-git-send-email-riel@redhat.com | v2 ☑ 3.19-rc1 | [PatchWork v2,0/6](https://lore.kernel.org/all/1413530994-9732-1-git-send-email-riel@redhat.com) |
| 2018/06/20 | Srikar Dronamraju <srikar@linux.vnet.ibm.com> | [Fixes for sched/numa_balancing](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/log/?id=b6a60cf36d497e7fbde9dd5b86fabd96850249f6) | NA | v2 ☑ 4.19-rc1 | [LORE v2,00/19](https://lore.kernel.org/all/1529514181-9842-1-git-send-email-srikar@linux.vnet.ibm.com) |
#### 4.3.3.5 处理与 Load Balance 冲突
-------
| 时间 | 作者 | 特性 | 描述 | 是否合入主线 | 链接 |
|:----:|:----:|:---:|:---:|:----------:|:----:|
| 2015/05/14 | Rik van Riel <riel@redhat.com> | [numa,sched: reduce conflict between fbq_classify_rq and migration](https://lore.kernel.org/all/20150514225936.35b91717@annuminas.surriel.com) | 20150514225936.35b91717@annuminas.surriel.com | v1 ☑✓ | [LORE](https://lore.kernel.org/all/20150514225936.35b91717@annuminas.surriel.com) |
| 2015/05/27 | riel@redhat.com <riel@redhat.com> | [numa,sched: resolve conflict between load balancing and NUMA balancing](https://lore.kernel.org/all/1432753468-7785-1-git-send-email-riel@redhat.com) | 20150528095249.3083ade0@annuminas.surriel.com | v2 ☑✓ | [LORE v2,0/2](https://lore.kernel.org/all/1432753468-7785-1-git-send-email-riel@redhat.com) |
| 2018/02/13 | Mel Gorman <mgorman@techsingularity.net> | [Reduce migrations and conflicts with automatic NUMA balancing v2](https://lore.kernel.org/all/20180213133730.24064-1-mgorman@techsingularity.net) | 20180213133730.24064-7-mgorman@techsingularity.net | v2 ☑✓ | [LORE v2,0/6](https://lore.kernel.org/all/20180213133730.24064-1-mgorman@techsingularity.net) |
| 2017/06/23 | riel@redhat.com <riel@redhat.com> | [NUMA improvements with task wakeup and load balancing](https://lore.kernel.org/all/20170623165530.22514-1-riel@redhat.com) | 20170623165530.22514-4-riel@redhat.com | v1 ☑✓ | [LORE v1,0/4](https://lore.kernel.org/all/20170623165530.22514-1-riel@redhat.com) |
| 2018/02/13 | Mel Gorman <mgorman@techsingularity.net> | [Reduce migrations and conflicts with automatic NUMA balancing v2](https://lore.kernel.org/all/20180213133730.24064-1-mgorman@techsingularity.net) | 20180213133730.24064-7-mgorman@techsingularity.net | v2 ☑✓ | [LORE v2,0/6](https://lore.kernel.org/all/20180213133730.24064-1-mgorman@techsingularity.net) |
| 2018/03/26 | Mel Gorman <mgorman@techsingularity.net> | [sched/numa: Avoid trapping faults and attempting migration of file-backed dirty pages](https://lore.kernel.org/all/20180326094334.zserdec62gwmmfqf@techsingularity.net) | 20180326094334.zserdec62gwmmfqf@techsingularity.net | v1 ☑✓ | [LORE](https://lore.kernel.org/all/20180326094334.zserdec62gwmmfqf@techsingularity.net) |
| 2018/09/21 | Srikar Dronamraju <srikar@linux.vnet.ibm.com> | [numabalancing patches](https://lore.kernel.org/all/1537552141-27815-1-git-send-email-srikar@linux.vnet.ibm.com) | 1537552141-27815-7-git-send-email-srikar@linux.vnet.ibm.com | v2 ☑✓ | [LORE v2,0/6](https://lore.kernel.org/all/1537552141-27815-1-git-send-email-srikar@linux.vnet.ibm.com) |
| 2018/10/01 | Mel Gorman <mgorman@techsingularity.net> | [Faster migration for automatic NUMA balancing](https://lore.kernel.org/all/20181001100525.29789-1-mgorman@techsingularity.net) | 20181001100525.29789-3-mgorman@techsingularity.net | v1 ☑✓ | [LORE v1,0/2](https://lore.kernel.org/all/20181001100525.29789-1-mgorman@techsingularity.net) |
| 2020/02/24 | Mel Gorman <mgorman@techsingularity.net> | [Reconcile NUMA balancing decisions with the load balancer v6](https://lore.kernel.org/all/20200224095223.13361-1-mgorman@techsingularity.net) | 20200224095223.13361-14-mgorman@techsingularity.net | v6 ☑✓ | [LORE v6,0/13](https://lore.kernel.org/all/20200224095223.13361-1-mgorman@techsingularity.net) |
## 4.4 rework_load_balance
-------