[core] Soft deprecate heap-allocating string helpers to prevent fragmentation patterns (#13156)

.ai/instructions.md

@@ -293,6 +293,12 @@ This document provides essential context for AI models interacting with this pro
     *   **Configuration Design:** Aim for simplicity with sensible defaults, while allowing for advanced customization.
     *   **Embedded Systems Optimization:** ESPHome targets resource-constrained microcontrollers. Be mindful of flash size and RAM usage.
 
+        **Why Heap Allocation Matters:**
+
+        ESP devices run for months with small heaps shared between Wi-Fi, BLE, LWIP, and application code. Over time, repeated allocations of different sizes fragment the heap. Failures happen when the largest contiguous block shrinks, even if total free heap is still large. We have seen field crashes caused by this.
+
+        **Heap allocation after `setup()` should be avoided unless absolutely unavoidable.** Every allocation/deallocation cycle contributes to fragmentation. ESPHome treats runtime heap allocation as a long-term reliability bug, not a performance issue. Helpers that hide allocation (`std::string`, `std::to_string`, string-returning helpers) are being deprecated and replaced with buffer and view based APIs.
+
         **STL Container Guidelines:**
 
         ESPHome runs on embedded systems with limited resources. Choose containers carefully:
@@ -322,15 +328,15 @@ This document provides essential context for AI models interacting with this pro
            std::array<uint8_t, 256> buffer;
            ```
 
-        2. **Compile-time-known fixed sizes with vector-like API:** Use `StaticVector` from `esphome/core/helpers.h` for fixed-size stack allocation with `push_back()` interface.
+        2. **Compile-time-known fixed sizes with vector-like API:** Use `StaticVector` from `esphome/core/helpers.h` for compile-time fixed size with `push_back()` interface (no dynamic allocation).
            ```cpp
            // Bad - generates STL realloc code (_M_realloc_insert)
            std::vector<ServiceRecord> services;
            services.reserve(5);  // Still includes reallocation machinery
 
-           // Good - compile-time fixed size, stack allocated, no reallocation machinery
+           // Good - compile-time fixed size, no dynamic allocation
-           StaticVector<ServiceRecord, MAX_SERVICES> services;  // Allocates all MAX_SERVICES on stack
+           StaticVector<ServiceRecord, MAX_SERVICES> services;
-           services.push_back(record1);  // Tracks count but all slots allocated
+           services.push_back(record1);
            ```
            Use `cg.add_define("MAX_SERVICES", count)` to set the size from Python configuration.
            Like `std::array` but with vector-like API (`push_back()`, `size()`) and no STL reallocation code.
@@ -372,22 +378,21 @@ This document provides essential context for AI models interacting with this pro
            ```
            Linear search on small datasets (1-16 elements) is often faster than hashing/tree overhead, but this depends on lookup frequency and access patterns. For frequent lookups in hot code paths, the O(1) vs O(n) complexity difference may still matter even for small datasets. `std::vector` with simple structs is usually fine—it's the heavy containers (`map`, `set`, `unordered_map`) that should be avoided for small datasets unless profiling shows otherwise.
 
-        5. **Detection:** Look for these patterns in compiler output:
+        5. **Avoid `std::deque`:** It allocates in 512-byte blocks regardless of element size, guaranteeing at least 512 bytes of RAM usage immediately. This is a major source of crashes on memory-constrained devices.
+
+        6. **Detection:** Look for these patterns in compiler output:
            - Large code sections with STL symbols (vector, map, set)
            - `alloc`, `realloc`, `dealloc` in symbol names
            - `_M_realloc_insert`, `_M_default_append` (vector reallocation)
            - Red-black tree code (`rb_tree`, `_Rb_tree`)
            - Hash table infrastructure (`unordered_map`, `hash`)
 
-        **When to optimize:**
+        **Prioritize optimization effort for:**
         - Core components (API, network, logger)
         - Widely-used components (mdns, wifi, ble)
         - Components causing flash size complaints
 
-        **When not to optimize:**
+        Note: Avoiding heap allocation after `setup()` is always required regardless of component type. The prioritization above is about the effort spent on container optimization (e.g., migrating from `std::vector` to `StaticVector`).
-        - Single-use niche components
-        - Code where readability matters more than bytes
-        - Already using appropriate containers
 
     *   **State Management:** Use `CORE.data` for component state that needs to persist during configuration generation. Avoid module-level mutable globals.
 

esphome/core/helpers.h

@@ -520,6 +520,7 @@ bool str_startswith(const std::string &str, const std::string &start);
 bool str_endswith(const std::string &str, const std::string &end);
 
 /// Truncate a string to a specific length.
+/// @warning Allocates heap memory. Avoid in new code - causes heap fragmentation on long-running devices.
 std::string str_truncate(const std::string &str, size_t length);
 
 /// Extract the part of the string until either the first occurrence of the specified character, or the end
@@ -531,11 +532,13 @@ std::string str_until(const std::string &str, char ch);
 /// Convert the string to lower case.
 std::string str_lower_case(const std::string &str);
 /// Convert the string to upper case.
+/// @warning Allocates heap memory. Avoid in new code - causes heap fragmentation on long-running devices.
 std::string str_upper_case(const std::string &str);
 
 /// Convert a single char to snake_case: lowercase and space to underscore.
 constexpr char to_snake_case_char(char c) { return (c == ' ') ? '_' : (c >= 'A' && c <= 'Z') ? c + ('a' - 'A') : c; }
 /// Convert the string to snake case (lowercase with underscores).
+/// @warning Allocates heap memory. Avoid in new code - causes heap fragmentation on long-running devices.
 std::string str_snake_case(const std::string &str);
 
 /// Sanitize a single char: keep alphanumerics, dashes, underscores; replace others with underscore.
@@ -758,6 +761,16 @@ inline char *format_hex_to(char (&buffer)[N], T val) {
   return format_hex_to(buffer, reinterpret_cast<const uint8_t *>(&val), sizeof(T));
 }
 
+/// Format std::vector<uint8_t> as lowercase hex to buffer.
+template<size_t N> inline char *format_hex_to(char (&buffer)[N], const std::vector<uint8_t> &data) {
+  return format_hex_to(buffer, data.data(), data.size());
+}
+
+/// Format std::array<uint8_t, M> as lowercase hex to buffer.
+template<size_t N, size_t M> inline char *format_hex_to(char (&buffer)[N], const std::array<uint8_t, M> &data) {
+  return format_hex_to(buffer, data.data(), data.size());
+}
+
 /// Calculate buffer size needed for format_hex_to: "XXXXXXXX...\0" = bytes * 2 + 1
 constexpr size_t format_hex_size(size_t byte_count) { return byte_count * 2 + 1; }
 
@@ -807,6 +820,18 @@ inline char *format_hex_pretty_to(char (&buffer)[N], const uint8_t *data, size_t
   return format_hex_pretty_to(buffer, N, data, length, separator);
 }
 
+/// Format std::vector<uint8_t> as uppercase hex with separator to buffer.
+template<size_t N>
+inline char *format_hex_pretty_to(char (&buffer)[N], const std::vector<uint8_t> &data, char separator = ':') {
+  return format_hex_pretty_to(buffer, data.data(), data.size(), separator);
+}
+
+/// Format std::array<uint8_t, M> as uppercase hex with separator to buffer.
+template<size_t N, size_t M>
+inline char *format_hex_pretty_to(char (&buffer)[N], const std::array<uint8_t, M> &data, char separator = ':') {
+  return format_hex_pretty_to(buffer, data.data(), data.size(), separator);
+}
+
 /// Calculate buffer size needed for format_hex_pretty_to with uint16_t data: "XXXX:XXXX:...:XXXX\0"
 constexpr size_t format_hex_pretty_uint16_size(size_t count) { return count * 5; }
 
@@ -840,8 +865,8 @@ static constexpr size_t MAC_ADDRESS_PRETTY_BUFFER_SIZE = format_hex_pretty_size(
 static constexpr size_t MAC_ADDRESS_BUFFER_SIZE = MAC_ADDRESS_SIZE * 2 + 1;
 
 /// Format MAC address as XX:XX:XX:XX:XX:XX (uppercase, colon separators)
-inline void format_mac_addr_upper(const uint8_t *mac, char *output) {
+inline char *format_mac_addr_upper(const uint8_t *mac, char *output) {
-  format_hex_pretty_to(output, MAC_ADDRESS_PRETTY_BUFFER_SIZE, mac, MAC_ADDRESS_SIZE, ':');
+  return format_hex_pretty_to(output, MAC_ADDRESS_PRETTY_BUFFER_SIZE, mac, MAC_ADDRESS_SIZE, ':');
 }
 
 /// Format MAC address as xxxxxxxxxxxxxx (lowercase, no separators)
@@ -850,16 +875,27 @@ inline void format_mac_addr_lower_no_sep(const uint8_t *mac, char *output) {
 }
 
 /// Format the six-byte array \p mac into a MAC address.
+/// @warning Allocates heap memory. Use format_mac_addr_upper() with a stack buffer instead.
+/// Causes heap fragmentation on long-running devices.
 std::string format_mac_address_pretty(const uint8_t mac[6]);
 /// Format the byte array \p data of length \p len in lowercased hex.
+/// @warning Allocates heap memory. Use format_hex_to() with a stack buffer instead.
+/// Causes heap fragmentation on long-running devices.
 std::string format_hex(const uint8_t *data, size_t length);
 /// Format the vector \p data in lowercased hex.
+/// @warning Allocates heap memory. Use format_hex_to() with a stack buffer instead.
+/// Causes heap fragmentation on long-running devices.
 std::string format_hex(const std::vector<uint8_t> &data);
 /// Format an unsigned integer in lowercased hex, starting with the most significant byte.
+/// @warning Allocates heap memory. Use format_hex_to() with a stack buffer instead.
+/// Causes heap fragmentation on long-running devices.
 template<typename T, enable_if_t<std::is_unsigned<T>::value, int> = 0> std::string format_hex(T val) {
   val = convert_big_endian(val);
   return format_hex(reinterpret_cast<uint8_t *>(&val), sizeof(T));
 }
+/// Format the std::array \p data in lowercased hex.
+/// @warning Allocates heap memory. Use format_hex_to() with a stack buffer instead.
+/// Causes heap fragmentation on long-running devices.
 template<std::size_t N> std::string format_hex(const std::array<uint8_t, N> &data) {
   return format_hex(data.data(), data.size());
 }

script/ci-custom.py

@@ -679,6 +679,52 @@ def lint_trailing_whitespace(fname, match):
     return "Trailing whitespace detected"
 
 
+# Heap-allocating helpers that cause fragmentation on long-running embedded devices.
+# These return std::string and should be replaced with stack-based alternatives.
+HEAP_ALLOCATING_HELPERS = {
+    "format_hex": "format_hex_to() with a stack buffer",
+    "format_mac_address_pretty": "format_mac_addr_upper() with a stack buffer",
+    "get_mac_address": "get_mac_address_into_buffer() with a stack buffer",
+    "get_mac_address_pretty": "get_mac_address_pretty_into_buffer() with a stack buffer",
+    "str_truncate": "removal (function is unused)",
+    "str_upper_case": "removal (function is unused)",
+    "str_snake_case": "removal (function is unused)",
+}
+
+
+@lint_re_check(
+    # Use negative lookahead to exclude _to/_into_buffer variants
+    # format_hex(?!_) ensures we don't match format_hex_to, format_hex_pretty_to, etc.
+    # get_mac_address(?!_) ensures we don't match get_mac_address_into_buffer, etc.
+    r"[^\w]("
+    r"format_hex(?!_)|"
+    r"format_mac_address_pretty|"
+    r"get_mac_address_pretty(?!_)|"
+    r"get_mac_address(?!_)|"
+    r"str_truncate|"
+    r"str_upper_case|"
+    r"str_snake_case"
+    r")\s*\(",
+    include=cpp_include,
+    exclude=[
+        # The definitions themselves
+        "esphome/core/helpers.h",
+        "esphome/core/helpers.cpp",
+    ],
+)
+def lint_no_heap_allocating_helpers(fname, match):
+    func = match.group(1)
+    replacement = HEAP_ALLOCATING_HELPERS.get(func, "a stack-based alternative")
+    return (
+        f"{highlight(func + '()')} allocates heap memory. On long-running embedded devices, "
+        f"repeated heap allocations fragment memory over time. Even infrequent allocations "
+        f"become time bombs - the heap eventually cannot satisfy requests even with free "
+        f"memory available.\n"
+        f"Please use {replacement} instead.\n"
+        f"(If strictly necessary, add `// NOLINT` to the end of the line)"
+    )
+
+
 @lint_content_find_check(
     "ESP_LOG",
     include=["*.h", "*.tcc"],