From 46c5f15734c8fe04346913d51ced26aecde6dc0f Mon Sep 17 00:00:00 2001 From: vczh Date: Thu, 18 Jun 2026 17:05:48 -0700 Subject: [PATCH] Sync copilot knowledge base --- .github/KnowledgeBase/Index.md | 2 + .github/KnowledgeBase/Index_Workflow.md | 84 +++ .../KB_Workflow_AstBuildingHelperFunctions.md | 88 +++ ...Workflow_CompilerDriverAndCppGeneration.md | 349 ++++++++++ ...low_Design_CompilerRebuildAndDesugaring.md | 596 ++++++++++++++++++ ...KB_Workflow_Design_GeneratedRpcWrappers.md | 75 +++ ...Workflow_Design_JsonSerializationSchema.md | 188 ++++++ ...KB_Workflow_InterfaceBasedRpcDefinition.md | 283 +++++++++ .../KB_Workflow_JsonRequestRouting.md | 206 ++++++ .github/KnowledgeBase/Learning.md | 2 +- .github/prompts/kb.prompt.md | 2 + 11 files changed, 1874 insertions(+), 1 deletion(-) create mode 100644 .github/KnowledgeBase/KB_Workflow_AstBuildingHelperFunctions.md create mode 100644 .github/KnowledgeBase/KB_Workflow_CompilerDriverAndCppGeneration.md create mode 100644 .github/KnowledgeBase/KB_Workflow_Design_CompilerRebuildAndDesugaring.md create mode 100644 .github/KnowledgeBase/KB_Workflow_Design_GeneratedRpcWrappers.md create mode 100644 .github/KnowledgeBase/KB_Workflow_Design_JsonSerializationSchema.md create mode 100644 .github/KnowledgeBase/KB_Workflow_InterfaceBasedRpcDefinition.md create mode 100644 .github/KnowledgeBase/KB_Workflow_JsonRequestRouting.md diff --git a/.github/KnowledgeBase/Index.md b/.github/KnowledgeBase/Index.md index 214cc25a..fe15f721 100644 --- a/.github/KnowledgeBase/Index.md +++ b/.github/KnowledgeBase/Index.md @@ -43,6 +43,7 @@ Online documentation: https://gaclib.net/doc/current/vlppos/home.html VlppOS provides cross-platform OS abstraction for file system operations, streams, locale support, and multi-threading. Use this when you need to interact with the operating system in a portable way. It offers locale-aware string manipulation, file system access, various stream types with encoding/decoding capabilities, and comprehensive multi-threading support with synchronization primitives. +It offers inter-process communication structure, but actual network protocol implementations are subject for referencing or writing demoes. Detailed project guidance: [Index_VlppOS.md](./Index_VlppOS.md) @@ -113,6 +114,7 @@ Workflow is a script language based on C++ reflection that can execute scripts a Use this when you need scripting capabilities, code generation, or when working with GacUI XML files. It can execute the script if reflection is turned on. It can generate equivalent C++ source files from the the script. +It can run Workflow with RPC on Workflow declared interfaces, but the user is responsible to offere ability of data transmission. Detailed project guidance: [Index_Workflow.md](./Index_Workflow.md) diff --git a/.github/KnowledgeBase/Index_Workflow.md b/.github/KnowledgeBase/Index_Workflow.md index ec615fbc..70ecfe9f 100644 --- a/.github/KnowledgeBase/Index_Workflow.md +++ b/.github/KnowledgeBase/Index_Workflow.md @@ -4,8 +4,70 @@ Project introduction remains in [Index.md](./Index.md#workflow). ### Choosing APIs +#### Interface-Based RPC Definitions + +Workflow interface-based RPC uses `@rpc:*` attributes to decide which Workflow interfaces can cross an RPC boundary, which members are serializable, and how collection values are transported. + +- Use `@rpc:Interface` on non-generic interfaces that are intended to be represented as RPC objects. +- Use `@rpc:Ctor` on service interfaces that can be registered as local singleton services and requested remotely. +- Use `@rpc:Byval` and `@rpc:Byref` on collection-valued properties, methods, and parameters when the default collection transport rule is not the desired rule. +- Use `@rpc:Cached` and `@rpc:Dynamic` on properties to control wrapper-side property caching behavior. +- Use RPC serializable primitive, struct, enum, nullable, strong collection, and `@rpc:Interface` pointer types in RPC signatures; do not expose internal transport structs in user-authored RPC APIs. + +[API Explanation](./KB_Workflow_InterfaceBasedRpcDefinition.md) + +#### JSON Request Routing + +Workflow JSON RPC uses `RpcJsonDispatcher`, `RpcJsonLifecycle`, generated JSON ops, and channel-backed dispatcher helpers to route direct method calls, event broadcasts, service declarations, and byval-return cleanup through JSON envelopes. + +- Use `RpcJsonDispatcher` and `RpcJsonLifecycle` once per endpoint, configured with generated ids, serializers, JSON object ops, JSON object-event ops, event attachers, and wrapper factories before initialization. +- Use `RpcJsonDispatcherClientForTaskQueue` for endpoint-side channel IO when incoming requests should be processed on a shared `TaskQueue`. +- Use `RpcJsonDispatcherServerForTaskQueue` as the coordinator that tracks connected clients, forwards broadcasts, caches service declarations, and consolidates broadcast responses. +- Use `WaitForServer` or `ConnectLocalServer` with required service type names when client startup must wait for remote service declarations. +- Use `FinalizeRpc()` before shutting down the transport or unloading generated Workflow context. + +[API Explanation](./KB_Workflow_JsonRequestRouting.md) + +#### Compiler Driver and C++ Generation + +Workflow compiler driver APIs let callers parse and analyze Workflow source modules, detect when RPC metadata is generated, create RPC wrapper Workflow modules, generate C++ for the final module set, and merge x86/x64 generated C++ output. + +- Use `vl::workflow::emitter::Compile` for ordinary Workflow source strings when a runtime assembly is the only output. +- Use `vl::workflow::analyzer::WfLexicalScopeManager` when compiler state, errors, RPC metadata, generated wrapper modules, or C++ generation are needed. +- Use `vl::workflow::analyzer::GenerateModuleRpc` and `vl::workflow::analyzer::GenerateModuleRpcJson` after a successful RPC-validating rebuild leaves `manager.rpcMetadata->metadataModule` available. +- Use `vl::workflow::WfPrint` to capture generated RPC metadata and wrapper modules as Workflow text. +- Use `vl::workflow::cppcodegen::GenerateCppFiles` to generate the full C++ file set from the final analyzed module set. +- Use `vl::workflow::cppcodegen::MergeCppMultiPlatform` and `vl::workflow::cppcodegen::MergeCppFileContent` to combine x86/x64 generated output and preserve user implementation regions. + +[API Explanation](./KB_Workflow_CompilerDriverAndCppGeneration.md) + +#### AST Building Helper Functions + +Workflow AST building helper functions let code generators construct or reuse `WfModule` AST and child nodes instead of assembling Workflow source text, while still using compiler-owned helpers for reflected types, default-value expressions, complete RPC wrapper modules, and printed output. + +- Use direct `Wf*` node construction when generated code is already structured and no exported helper matches the needed node shape. +- Use `ParseType`, `ParseExpression`, `ParseStatement`, `ParseCoProviderStatement`, `ParseDeclaration`, and `ParseModule` only when the input is already Workflow text. +- Use `GetExpressionFromTypeDescriptor`, `GetTypeFromTypeDescriptor`, and `GetTypeFromTypeInfo` when generated AST should follow reflection metadata. +- Use `CopyType`, `CopyExpression`, `CopyStatement`, `CopyDeclaration`, and `CopyModule` when reusing existing AST without sharing nodes. +- Use `CreateDefaultValue` when a generated initializer should match Workflow compiler conventions for a reflected type. +- Use `CopyAndClearRpcMetadata`, `GenerateModuleRpc`, and `GenerateModuleRpcJson` only for complete generated RPC support modules. +- Use `WfPrint` to inspect or serialize generated AST after construction. + +[API Explanation](./KB_Workflow_AstBuildingHelperFunctions.md) + ### Design Explanation +#### Compiler Rebuild And Desugaring + +Workflow compiler analysis is organized around `WfLexicalScopeManager::Rebuild`, a barriered frontend pipeline that validates modules, builds names and scopes, resolves semantics, and rewrites high-level Workflow syntax into ordinary AST before bytecode or C++ generation. + +- `Rebuild` separates context-free rewrites, structure checks, global-name construction, scope construction, scope completion, semantic validation, and post-semantic metadata population. +- Context-sensitive virtual AST nodes such as `bind`, `$coroutine`, co-provider statements, and state machines are expanded only after their original nodes have enough resolved type and scope information. +- `bind` lowers to an anonymous `IValueSubscription` implementation; co-provider syntax first lowers to a generated raw `WfNewCoroutineExpression`, and that raw coroutine then lowers to a generated `ICoroutine` implementation; state machines lower to generated class members plus a generated coroutine. +- Backends consume `expanded*` ordinary Workflow AST and treat raw high-level nodes as frontend-only constructs. + +[Design Explanation](./KB_Workflow_Design_CompilerRebuildAndDesugaring.md) + #### Attribute System Workflow script attributes (`@category:name`) are translated to reflected struct types following the naming convention `system::workflow_attributes::att_category_name`. The compiler resolves, evaluates, and populates attributes onto type descriptors during assembly generation, and the binary serialization format preserves attribute metadata across assembly load/save cycles. Predefined `@cpp:*` attributes control C++ code generation behavior. @@ -17,3 +79,25 @@ Workflow script attributes (`@category:name`) are translated to reflected struct - C++ code generation emits `ATTRIBUTE_TYPE` / `ATTRIBUTE_MEMBER` macros via `WriteAttributeMacro` in `Source/Cpp/WfCpp_WriteReflection.cpp`. [Design Explanation](./KB_Workflow_Design_AttributeSystem.md) + +#### Generated RPC Wrappers + +Workflow RPC wrapper generation turns RPC metadata into generated ids, serializers, object ops, object-event ops, caller-side ops, listener attachers, and wrapper factories that connect application objects to an `IRpcLifecycle`. + +- Generated `rpc_GetIds()` provides the shared name-to-id map used by lifecycle setup, service registration, and service lookup. +- Generated object ops receive remote method calls, holds, unholds, and event replays, while generated caller-side ops are used by wrappers to send calls and broadcasts through the dispatcher. +- Generated wrapper factories create remote-object wrappers from `RpcObjectReference` values; wrapper construction and destruction send hold and unhold messages to the owner lifecycle. +- Generated JSON ops reuse the same wrapper classes with JSON caller-side ops, adding JSON serialization around arguments, returns, events, exceptions, and byval collection returns. + +[Design Explanation](./KB_Workflow_Design_GeneratedRpcWrappers.md) + +#### JSON Serialization Schema + +Workflow RPC JSON serialization defines the generated TypeScript schema and Workflow serializer behavior for known static types, unknown reflected values, internal transport structs, dispatcher envelopes, and captured test JSON values. + +- Known-type schemas serialize primitives, enums, structs, lists, observable lists, and dictionaries according to the static RPC metadata. +- Unknown-type schemas carry enough type information to reconstruct reflected values for dynamic values and byref collection elements. +- `system::RpcObjectReference`, `system::RpcException`, and `system::RpcByvalReturnValue` are stable internal transport shapes used by generated JSON RPC infrastructure. +- Generated `.d.ts` files expose `KnownTypeSchema`, `UnknownTypeSchema`, concrete enum and struct declarations, and captured `JsonValue_*.ts` validation data. + +[Design Explanation](./KB_Workflow_Design_JsonSerializationSchema.md) diff --git a/.github/KnowledgeBase/KB_Workflow_AstBuildingHelperFunctions.md b/.github/KnowledgeBase/KB_Workflow_AstBuildingHelperFunctions.md new file mode 100644 index 00000000..4176b36b --- /dev/null +++ b/.github/KnowledgeBase/KB_Workflow_AstBuildingHelperFunctions.md @@ -0,0 +1,88 @@ +# AST Building Helper Functions + +Workflow generators should prefer building `WfModule` AST and its child nodes directly instead of composing Workflow text and parsing it back. Direct AST generation keeps structured information in C++, avoids source-formatting concerns, and makes it easier to reuse compiler helpers for reflected types, default values, and safe AST cloning. + +The exported helpers do not cover every possible `Wf*` node shape. When no helper exists for the exact node being generated, create the specific AST node directly and fill its fields. Use the helpers below for the common cases where Workflow already provides a stable API. + +## Build AST From Existing Workflow Text + +Use the parsing helpers only when the input is already Workflow source text: + +- Use `ParseType` to parse a textual type into a `WfType`. +- Use `ParseExpression` to parse a textual expression into a `WfExpression`. +- Use `ParseStatement` to parse a textual statement into a `WfStatement`. +- Use `ParseCoProviderStatement` to parse a textual co-provider statement into a `WfCoProviderStatement`. +- Use `ParseDeclaration` to parse a textual declaration into a `WfDeclaration`. +- Use `ParseModule` to parse a whole textual module into a `WfModule`. + +These functions tokenize the input, invoke the corresponding generated parser entry, and unescape string-related AST content. They are useful at API boundaries where Workflow source text is the input format. They should not be the first choice for generated code when the generator already knows the intended structure. + +## Convert Reflected Types To AST + +Use reflection-to-AST helpers when generated Workflow code needs to refer to C++ reflected types: + +- Use `GetExpressionFromTypeDescriptor` when a generated expression needs a globally qualified type-name expression for an `ITypeDescriptor`. +- Use `GetTypeFromTypeDescriptor` when a generated type annotation should name an `ITypeDescriptor` directly. +- Use `GetTypeFromTypeInfo` when the source of truth is an `ITypeInfo`, including decorated types. + +`GetTypeFromTypeInfo` is the safest choice for generated function arguments, local variables, fields, and return types backed by reflection metadata. It handles raw pointers, shared pointers, nullable values, descriptors, and known generic shapes such as enumerable, map, observable-list, and function types. This avoids manually spelling namespace fragments or accidentally choosing the wrong Workflow type wrapper. + +## Copy Existing AST + +Use copy helpers when a generator needs to reuse an existing AST subtree without aliasing the original node: + +- Use `CopyType` to clone a `WfType`. +- Use `CopyExpression` to clone a `WfExpression`. +- Use `CopyStatement` to clone a `WfStatement`. +- Use `CopyDeclaration` to clone a `WfDeclaration`. +- Use `CopyModule` to clone a `WfModule`. + +Each copy helper has overloads accepting either `Ptr` or raw `T*`, so callers can use whichever form they already have. + +For `CopyExpression`, `CopyStatement`, `CopyDeclaration`, and `CopyModule`, the `expandVirtualExprStat` argument controls whether virtual frontend constructs are copied as written or copied through their expanded ordinary AST: + +- Pass `false` when the generated output should preserve the original AST shape. +- Pass `true` when backend-ready AST is needed and expanded expressions, statements, or declarations should replace virtual nodes when available. + +`CopyType` does not take `expandVirtualExprStat` because Workflow types do not use the same virtual expansion mechanism. + +## Create Default-Value Expressions + +Use `CreateDefaultValue` when generated code needs a default expression for an `ITypeInfo`. + +`CreateDefaultValue` produces the Workflow expression shape expected by compiler-generated code: + +- enum defaults are built as integer `0`, cast through `U8`, then cast to the target enum type; +- strings are built as empty string expressions; +- non-serializable structs are built as `{}` and cast to the struct type; +- booleans are built as `false`; +- numeric primitives are built as `0` inferred as the target type; +- other reference-like values are built as `null` inferred as the target type. + +This helper is the right defaulting API for generated fields, cached values, and temporary variables whose types come from reflection metadata. + +## Generate Complete RPC Wrapper Modules + +Use the RPC module-generation helpers when the goal is a complete generated module rather than individual node construction: + +- Use `CopyAndClearRpcMetadata` to clone RPC metadata input and remove declarations that already exist as reflected types. +- Use `GenerateModuleRpc` after RPC validation has populated metadata and the generator needs the normal RPC wrapper module. +- Use `GenerateModuleRpcJson` after RPC validation has populated metadata and the generator needs the JSON-enabled RPC wrapper and serializer module. + +These functions return `WfModule` instances. They are high-level module generators, not general-purpose factories for arbitrary AST nodes. Choose them when generating the standard Workflow RPC support modules; choose the lower-level helpers or direct AST construction for other generated code. + +## Print Generated AST + +Use `WfPrint` after AST generation when generated Workflow needs to be inspected, logged, compared with baselines, or emitted as text. `WfPrint` has overloads for `WfAttribute`, `WfType`, `WfExpression`, `WfStatement`, `WfDeclaration`, and `WfModule`, and supports both parsing writers and text writers. + +`WfPrint` is not an AST-creation API. It is the matching output and verification tool after code has already been represented as AST. + +## Choosing The Right Helper + +- Use direct `Wf*` node construction when generated code is already structured and no exported helper matches the needed node shape. +- Use `ParseType`, `ParseExpression`, `ParseStatement`, `ParseCoProviderStatement`, `ParseDeclaration`, or `ParseModule` only when the input is Workflow text. +- Use `GetExpressionFromTypeDescriptor`, `GetTypeFromTypeDescriptor`, or `GetTypeFromTypeInfo` when the generated AST should follow reflection metadata. +- Use `CopyType`, `CopyExpression`, `CopyStatement`, `CopyDeclaration`, or `CopyModule` when reusing existing AST without sharing nodes. +- Use `CreateDefaultValue` when a generated initializer should match Workflow compiler conventions for a reflected type. +- Use `CopyAndClearRpcMetadata`, `GenerateModuleRpc`, or `GenerateModuleRpcJson` only for complete generated RPC support modules. +- Use `WfPrint` to inspect or serialize generated AST after construction. diff --git a/.github/KnowledgeBase/KB_Workflow_CompilerDriverAndCppGeneration.md b/.github/KnowledgeBase/KB_Workflow_CompilerDriverAndCppGeneration.md new file mode 100644 index 00000000..a7030618 --- /dev/null +++ b/.github/KnowledgeBase/KB_Workflow_CompilerDriverAndCppGeneration.md @@ -0,0 +1,349 @@ +# Workflow Compiler Driver and C++ Generation + +Workflow source code can be compiled into a runtime `WfAssembly`, converted into generated C++ files, or extended with generated RPC wrapper Workflow modules before either of those outputs is produced. The ordinary compiler path and the RPC compiler path share the same `WfLexicalScopeManager` analyzer, but the RPC path has an additional definition-only pass that creates wrapper modules and metadata before the final link. + +This document focuses on using the compiler APIs. Logging, file layout, and build integration are intentionally left to the caller. + +## Main APIs + +- Use `vl::workflow::emitter::Compile` when all inputs are ordinary Workflow source strings and the caller only needs a `vl::workflow::runtime::WfAssembly`. +- Use `vl::workflow::analyzer::WfLexicalScopeManager` directly when the caller needs intermediate compiler state, generated RPC wrapper modules, generated C++ output, or custom progress reporting through `IWfCompilerCallback`. +- Use `vl::workflow::emitter::GenerateAssembly` after `WfLexicalScopeManager::Rebuild` succeeds. +- Use `vl::workflow::analyzer::GenerateModuleRpc` and `vl::workflow::analyzer::GenerateModuleRpcJson` after RPC validation succeeds and `manager.rpcMetadata->metadataModule` exists. +- Use `vl::workflow::WfPrint` to turn generated `WfModule` values, such as RPC metadata and wrapper modules, back into Workflow text. +- Use `vl::workflow::cppcodegen::GenerateCppFiles` after the final set of modules has been added to a successfully rebuilt `WfLexicalScopeManager`. +- Use `vl::workflow::cppcodegen::MergeCppMultiPlatform` and `vl::workflow::cppcodegen::MergeCppFileContent` when x86 and x64 generated C++ need to become one architecture-neutral output while preserving user implementation regions. + +## Compiler State and Errors + +`WfLexicalScopeManager` owns the parser error handler, the parsed modules, all analyzer state, optional RPC metadata, and the `manager.errors` list. Construct it with a `workflow::Parser` and a `WfCpuArchitecture`. + +`WfCpuArchitecture` controls Workflow's CPU-sized integer mapping: + +- `WfCpuArchitecture::x86` maps CPU-sized signed and unsigned integer types to `vint32_t` and `vuint32_t`. +- `WfCpuArchitecture::x64` maps them to `vint64_t` and `vuint64_t`. +- `WfCpuArchitecture::AsExecutable` maps them to `vint` and `vuint`. + +Compiler errors are `glr::ParsingError` values. `error.codeRange.start.row` and `error.codeRange.start.column` are zero-based positions. A user-facing diagnostic normally adds 1 to both values and prints `error.message`. + +The following code is a reference example for reading compiler errors. It shows the fields that compiler callers usually need; callers can send the text to any log, UI, exception, or diagnostic system. + +```C++ +void WriteWorkflowErrors(stream::TextWriter& writer, const collections::List& errors) +{ + for (auto&& error : errors) + { + writer.WriteLine( + L"Line: " + itow(error.codeRange.start.row + 1) + + L", Column: " + itow(error.codeRange.start.column + 1) + + L", Message: " + error.message); + } +} +``` + +Check `manager.errors` after every phase that can add diagnostics: + +- after `AddModule(const WString&)` or explicit parsing with `ParseModule`; +- after `WfLexicalScopeManager::Rebuild`; +- after `GenerateModuleRpc`; +- after `GenerateModuleRpcJson`; +- after `GenerateCppFiles`. + +The convenience `Compile` API copies `manager.errors` to the caller-provided error list and returns `nullptr` when parsing or analysis fails. + +## Ordinary Workflow Compilation + +For ordinary Workflow source modules, the shortest path is `Compile`. Use the explicit manager path when the caller also needs C++ generation from the same analyzed modules. + +The following reference example shows the explicit manager flow for compiling ordinary modules into a `WfAssembly`. + +```C++ +Ptr CompileWorkflowModules( + workflow::Parser& parser, + analyzer::WfCpuArchitecture architecture, + const collections::List& moduleCodes, + collections::List& errors) +{ + analyzer::WfLexicalScopeManager manager(parser, architecture); + for (auto&& moduleCode : moduleCodes) + { + manager.AddModule(moduleCode); + } + + if (manager.errors.Count() > 0) + { + CopyFrom(errors, manager.errors); + return nullptr; + } + + manager.Rebuild(true); + if (manager.errors.Count() > 0) + { + CopyFrom(errors, manager.errors); + return nullptr; + } + + return emitter::GenerateAssembly(&manager); +} +``` + +`WfLexicalScopeManager::Rebuild(true)` keeps reflected type descriptor names in the manager while clearing compiler state for the current module set. During rebuild, the compiler desugars modules, validates structure, builds scopes, validates semantics, populates attributes, and validates RPC declarations by default. + +## C++ Generation from Workflow Modules + +`GenerateCppFiles` consumes an already analyzed `WfLexicalScopeManager`. It does not parse or rebuild modules by itself. The input object `WfCppInput` controls assembly naming, generated filename stems, file splitting, reflection-file generation, comments, header guard prefix, and includes. + +The result is not a single source file. `WfCppOutput::cppFiles` is the complete generated file set, and the caller decides how to store, log, or pass each entry to the C++ build. `WfCppOutput::entryFileName` names the header stem that downstream code should include, but it is not the only generated file. + +Generated file groups include: + +- the main Workflow translation files: `.h` and `.cpp`, generated for every successful C++ generation call; +- internal split headers: `N.h`, generated when the compiler separates generated classes into additional non-custom headers; +- custom split files: `.h` and `.cpp`, generated when Workflow declarations are assigned to explicit C++ file groups; +- the aggregate include header: `.h`, generated when multi-file output is enabled, and used as `entryFileName` so consumers can include one generated header; +- reflection support files: `.h` and `.cpp`, generated when reflection output is enabled; with `WfCppFileSwitch::OnDemand`, this happens only when the analyzed modules define Workflow types needing reflection registration; +- conditional reflection includes in generated `.cpp` files, so reflection support and caller-provided `reflectionIncludes` can be excluded when `VCZH_DEBUG_NO_REFLECTION` is defined. + +`input->normalIncludes` are added to the main generated header. `input->reflectionIncludes` are added to the generated reflection header when reflection files are emitted, or as conditional includes in generated `.cpp` files when the caller supplies reflection dependencies without generating the reflection source pair. + +`WfCppOutput::containsReflectionInfo` reports whether the analyzed Workflow modules contain reflection metadata. `WfCppOutput::reflection` reports whether reflection files were actually emitted. These can differ when `input->reflection` is disabled: Workflow types may exist, but no reflection source pair is generated. + +The following reference example shows the shape of a C++ generation call after the manager has already been rebuilt successfully. + +```C++ +Ptr GenerateWorkflowCpp( + analyzer::WfLexicalScopeManager& manager, + const WString& assemblyName, + const collections::List& normalIncludes, + collections::List& errors) +{ + auto input = Ptr(new cppcodegen::WfCppInput(assemblyName)); + input->multiFile = cppcodegen::WfCppFileSwitch::OnDemand; + input->reflection = cppcodegen::WfCppFileSwitch::OnDemand; + for (auto&& include : normalIncludes) + { + input->normalIncludes.Add(include); + } + + auto output = cppcodegen::GenerateCppFiles(input, &manager); + if (manager.errors.Count() > 0) + { + CopyFrom(errors, manager.errors); + return nullptr; + } + return output; +} +``` + +`WfCppFileSwitch::OnDemand` lets the generator decide whether to emit multi-file output or reflection files: + +- multi-file output is enabled when generated/custom file grouping requires it; +- reflection files are emitted when the analyzed modules define Workflow types that need reflection registration. + +The caller should consume every `cppFiles` entry, not only the entry header. The following reference example shows the shape of that handoff without assuming where files are written or how generated text is logged. + +```C++ +void VisitGeneratedCppFiles(cppcodegen::WfCppOutput& output) +{ + for (auto [fileName, index] : indexed(output.cppFiles.Keys())) + { + const auto& code = output.cppFiles.Values()[index]; + ConsumeGeneratedCppFile(fileName, code); + } +} +``` + +## RPC Wrapper Generation + +RPC compilation has two stages: + +1. Compile the RPC definition module alone with RPC validation enabled, then generate metadata and wrapper Workflow modules. +2. Add the original definition module, the executable/user modules, and the generated wrapper modules into a fresh final manager, rebuild with RPC validation disabled, and then generate an assembly or C++. + +The definition-only stage is necessary because `GenerateModuleRpc` and `GenerateModuleRpcJson` need `manager.rpcMetadata`, which is produced by `ValidateModuleRPC` during `WfLexicalScopeManager::Rebuild`. + +The compiler signal for "RPC is involved" is `manager.rpcMetadata`, not a caller-side scan for `@rpc:Interface` text. `WfLexicalScopeManager::Rebuild` first clears `rpcMetadata` to `nullptr`, then runs `ValidateModuleRPC` for each module when `validateRpc` is true. `ValidateModuleRPC` creates `manager.rpcMetadata` only when phase 1 collected at least one valid Workflow RPC interface and no errors have been reported. Metadata generation also fills `manager.rpcMetadata->metadataModule`, and both `GenerateModuleRpc` and `GenerateModuleRpcJson` return `nullptr` if either `rpcMetadata` or `metadataModule` is missing. + +After a successful definition-only rebuild with RPC validation enabled, interpret the state as: + +- `manager.errors` is non-empty: report diagnostics first; do not decide the compile flow from missing metadata. +- `manager.rpcMetadata && manager.rpcMetadata->metadataModule`: RPC interfaces were found and metadata was generated, so wrapper generation can run. +- `manager.rpcMetadata == nullptr`: no valid Workflow RPC interface was found during this rebuild, so there are no RPC wrapper modules to generate. + +This signal belongs to the RPC-validating pass. The final RPC link intentionally calls `Rebuild(..., validateRpc = false)`, so it should not be expected to populate `rpcMetadata`. + +`manager.rpcMetadata` contains: + +- `metadataModule`, a generated `WfModule` containing serializable enum, struct, and RPC interface declarations; +- `dts`, generated TypeScript schema text for RPC JSON serialization; +- `typeNames`, `methodNames`, and `eventNames`, mapping RPC full names to generated declarations; +- `typeFullNames`, `methodFullNames`, `eventFullNames`, and deterministic `orderedIds`. + +The following reference example shows the definition-only stage. It captures generated metadata and wrapper Workflow text with `WfPrint`, but the caller decides what to do with the strings. + +```C++ +struct RpcGeneratedWorkflow +{ + Ptr wrapperModule; + Ptr wrapperJsonModule; + WString metadataWorkflow; + WString wrapperWorkflow; + WString wrapperJsonWorkflow; + WString dts; +}; + +bool GenerateRpcWorkflowModules( + workflow::Parser& parser, + analyzer::WfCpuArchitecture architecture, + const WString& definitionCode, + const WString& cppAssemblyName, + RpcGeneratedWorkflow& generated, + collections::List& errors) +{ + analyzer::WfLexicalScopeManager manager(parser, architecture); + + auto definitionModule = ParseModule(definitionCode, parser); + if (!definitionModule || manager.errors.Count() > 0) + { + CopyFrom(errors, manager.errors); + return false; + } + + manager.AddModule(definitionModule); + manager.Rebuild(true); + if (manager.errors.Count() > 0 || !manager.rpcMetadata || !manager.rpcMetadata->metadataModule) + { + CopyFrom(errors, manager.errors); + return false; + } + + generated.wrapperModule = analyzer::GenerateModuleRpc(&manager, cppAssemblyName); + generated.wrapperJsonModule = analyzer::GenerateModuleRpcJson(&manager, cppAssemblyName); + if (manager.errors.Count() > 0 || !generated.wrapperModule || !generated.wrapperJsonModule) + { + CopyFrom(errors, manager.errors); + return false; + } + + generated.metadataWorkflow = GenerateToStream([&](stream::StreamWriter& writer) + { + WfPrint(manager.rpcMetadata->metadataModule, L"", writer); + }); + generated.wrapperWorkflow = GenerateToStream([&](stream::StreamWriter& writer) + { + WfPrint(generated.wrapperModule, L"", writer); + }); + generated.wrapperJsonWorkflow = GenerateToStream([&](stream::StreamWriter& writer) + { + WfPrint(generated.wrapperJsonModule, L"", writer); + }); + generated.dts = manager.rpcMetadata->dts; + return true; +} +``` + +`GenerateModuleRpc` creates flat RPC Workflow infrastructure, including id constants, `rpc_GetIds`, type-id helpers, object ops, object-event ops, caller-side ops, event listeners, wrapper interfaces, wrapper factories, and wrapper dispatch. + +`GenerateModuleRpcJson` creates JSON RPC Workflow infrastructure, including JSON serializers, JSON object ops, JSON object-event ops, and JSON caller-side ops. Flat and JSON generation use the same RPC metadata model. + +## Final RPC Link + +After wrapper modules are generated, the final compilation should parse the definition code and user/executable code again, then add both generated wrapper modules to the final manager. + +The final rebuild should pass `validateRpc = false`. The input definition has already been validated, and the generated wrapper modules are compiler output. Revalidating RPC metadata during the final link is not the intended flow. + +The following reference example produces a final `WfAssembly` from a definition module, an executable module, and generated wrapper modules. + +```C++ +Ptr LinkRpcWorkflowAssembly( + workflow::Parser& parser, + analyzer::WfCpuArchitecture architecture, + const WString& definitionCode, + const WString& executableCode, + Ptr wrapperModule, + Ptr wrapperJsonModule, + collections::List& errors) +{ + analyzer::WfLexicalScopeManager manager(parser, architecture); + + auto definitionModule = ParseModule(definitionCode, parser); + auto executableModule = ParseModule(executableCode, parser); + if (!definitionModule || !executableModule || manager.errors.Count() > 0) + { + CopyFrom(errors, manager.errors); + return nullptr; + } + + manager.AddModule(definitionModule); + manager.AddModule(executableModule); + manager.AddModule(wrapperModule); + manager.AddModule(wrapperJsonModule); + manager.Rebuild(true, nullptr, false); + if (manager.errors.Count() > 0) + { + CopyFrom(errors, manager.errors); + return nullptr; + } + + return emitter::GenerateAssembly(&manager); +} +``` + +The same final manager can be passed to `GenerateCppFiles` to generate C++ for the original Workflow input and both generated RPC wrapper modules. + +## Cross-Architecture C++ Output + +When generated C++ should be architecture-neutral, run the same logical generation twice: + +1. once with `WfCpuArchitecture::x86` and matching x86 reflection metadata; +2. once with `WfCpuArchitecture::x64` and matching x64 reflection metadata. + +The caller should require both runs to produce the same generated filename set. For each matching file, call `MergeCppMultiPlatform(code32, code64)`. If there is already an output file containing user implementation regions, call `MergeCppFileContent(existingCode, mergedCode)` before writing the result. + +The following reference example shows only the text-level merge. File enumeration and storage are caller policy. + +```C++ +WString MergeGeneratedCppFile( + const WString& code32, + const WString& code64, + const WString* existingCode) +{ + auto mergedCode = cppcodegen::MergeCppMultiPlatform(code32, code64); + if (existingCode) + { + mergedCode = cppcodegen::MergeCppFileContent(*existingCode, mergedCode); + } + return mergedCode; +} +``` + +`MergeCppMultiPlatform` accepts only expected architecture differences: + +- `vint32_t` versus `vint64_t`; +- `vuint32_t` versus `vuint64_t`; +- integer literal suffix differences such as 64-bit `L` or `UL`; +- selected generated `static_cast<::vl::vint32_t>` or `static_cast<::vl::vint64_t>` numeric forms. + +Any other difference throws `MergeCppMultiPlatformException` with row and column positions for both inputs. Treat that exception as a generator difference that needs investigation, not as a normal merge conflict. + +`MergeCppFileContent` preserves existing user implementation content in generated `USER_CONTENT_BEGIN` / `USER_CONTENT_END` and `USERIMPL` regions. User content that no longer matches a generated region is kept under `UNUSED_USER_CONTENT`, so it can be inspected instead of silently dropped. + +## Choosing the Flow + +Use the ordinary compile flow when: + +- after a successful rebuild with RPC validation enabled, `manager.rpcMetadata` is `nullptr`; +- no generated RPC wrapper modules are needed for the final manager; +- the caller only needs a `WfAssembly`, or C++ generated directly from those modules. + +Use the RPC flow when: + +- an initial rebuild with RPC validation enabled succeeds and leaves both `manager.rpcMetadata` and `manager.rpcMetadata->metadataModule` non-null; +- generated wrappers, JSON serializers, or `rpc_GetIds` are needed; +- the final assembly or generated C++ must include generated flat or JSON RPC Workflow modules. + +Use the x86/x64 merge flow when: + +- generated C++ should compile as architecture-neutral code using `vint` and `vuint`; +- the Workflow code or reflected metadata contains CPU-sized integer mappings; +- existing generated C++ files can contain preserved user implementation regions. diff --git a/.github/KnowledgeBase/KB_Workflow_Design_CompilerRebuildAndDesugaring.md b/.github/KnowledgeBase/KB_Workflow_Design_CompilerRebuildAndDesugaring.md new file mode 100644 index 00000000..86e5ec4c --- /dev/null +++ b/.github/KnowledgeBase/KB_Workflow_Design_CompilerRebuildAndDesugaring.md @@ -0,0 +1,596 @@ +# Workflow Compiler Rebuild And Desugaring + +## How the Workflow analyzer turns parsed modules and high-level Workflow constructs into ordinary AST for bytecode and C++ generation. + +This document describes the maintenance-facing design of the Workflow compiler frontend. It focuses on `WfLexicalScopeManager::Rebuild` in `Source/Analyzer/WfAnalyzer.cpp`, the analysis passes it coordinates, and how high-level syntax such as `bind`, co-provider operators, raw coroutines, and state machines is lowered before the emitters run. + +The central invariant is that special syntax is a frontend concern. Bytecode generation in `Source/Emitter` and C++ generation in `Source/Cpp` consume ordinary Workflow AST, or virtual nodes whose `expanded*` field points to ordinary Workflow AST. Raw high-level nodes should not be treated as backend concepts. + +## Source Map + +The compiler frontend is spread across these files: + +- `Source/Analyzer/WfAnalyzer.cpp`: owns `WfLexicalScopeManager`, `Clear`, `Rebuild`, and shared manager state. +- `Source/Analyzer/WfAnalyzer_ContextFreeDesugar.cpp`: performs rewrites that do not need type or scope information. +- `Source/Analyzer/WfAnalyzer_ValidateStructure_*.cpp`: checks syntax placement and structural legality before name/type analysis. +- `Source/Analyzer/WfAnalyzer_BuildGlobalNameFromModules.cpp`: creates global names, reflected custom types, and class/type member placeholders. +- `Source/Analyzer/WfAnalyzer_BuildScope.cpp`: creates lexical scopes, symbols, lambda captures, observe scopes, coroutine scopes, and state-machine scopes. +- `Source/Analyzer/WfAnalyzer_CompleteScope.cpp`: fills symbol and member type information after all global names and scopes exist. +- `Source/Analyzer/WfAnalyzer_CheckScopes_*.cpp`: checks duplicate symbols, missing symbol types, and declaration dependency cycles. +- `Source/Analyzer/WfAnalyzer_ValidateSemantic_*.cpp`: resolves expressions, overloads, captures, interface implementations, and context-sensitive expansion. +- `Source/Analyzer/WfAnalyzer_ExpandBindExpression.cpp`: lowers `bind`. +- `Source/Analyzer/WfAnalyzer_ExpandStatement.cpp`: lowers `switch`, `foreach`, co-provider statements, and co-operator statements. +- `Source/Analyzer/WfAnalyzer_ExpandNewCoroutineExpression.cpp`: lowers `$coroutine{}` to `ICoroutine`. +- `Source/Analyzer/WfAnalyzer_ExpandStateMachine.cpp`: lowers state-machine declarations and state-machine-only statements. +- `Source/Emitter/*` and `Source/Cpp/*`: consume expanded AST for bytecode and C++ output. + +Parser syntax and generated AST definitions are in: + +- `Source/Parser/Syntax/Syntax/Expressions.txt`. +- `Source/Parser/Syntax/Syntax/Statements.txt`. +- `Source/Parser/Syntax/Syntax/Decls.txt`. +- `Source/Parser/Syntax/Lexer.txt`. +- `Source/Parser/Generated/WorkflowAst.h`. + +Generated sample outputs under `Test/Generated/Workflow32` and `Test/Generated/Workflow64` are useful for checking the printed shape of expansions. They are generated files and should not be edited directly. + +## Rebuild Pipeline + +`WfLexicalScopeManager::Rebuild` is a barriered compiler pipeline. It captures `errorCount = errors.Count()` after initialization and uses `EXIT_IF_ERRORS_EXIST` after major phases. Because `errorCount` is not reset between phases, any error added by a phase prevents later phases from running. A validation added to an early phase must not depend on state that is only built by a later phase. + +### Environment Load And Reset + +The first phase: + +- calls `IWfCompilerCallback::OnLoadEnvironment`; +- calls `Clear(keepTypeDescriptorNames, false)`; +- creates `globalName` if needed; +- loads reflected type names with `BuildGlobalNameFromTypeDescriptors`; +- calls `IWfCompilerCallback::OnInitialize`. + +`Clear` resets more than modules and errors. It clears semantic side tables such as `expressionResolvings`, coroutine resolution maps, `lambdaCaptures`, `interfaceMethodImpls`, declaration metadata maps, and state-machine maps. Any new analysis cache maintained by `WfLexicalScopeManager` should be cleared here and considered when generated AST is revalidated. + +### Context-Free Desugaring And Structure Validation + +For each module, `Rebuild` calls: + +- `ContextFreeModuleDesugar(this, module)`; +- `ValidateModuleStructure(this, module)`. + +Context-free desugaring rewrites syntax that can be transformed without resolved types or scopes. Existing examples include format expressions, auto properties, and generated helper interfaces used by coroutine result conversion. + +Structure validation checks placement rules that do not require semantic information. Examples include: + +- no nested `bind`; +- no `observe` outside `bind`; +- no explicit `attach` or `detach` inside `bind`; +- no `co-pause` outside `$coroutine{}`; +- no co-operator outside a co-provider statement; +- no state-machine switch or invoke outside a state body; +- state-machine declarations only as class members. + +### Global Names And Initial Scopes + +The next phase creates the name and initial scope model: + +- `BuildGlobalNameFromModules(this)` adds namespaces, Workflow-defined type descriptors, and class/type member placeholders. +- `BuildScopeForModule(this, module)` builds lexical scopes and symbols. +- `ValidateScopeName(this, globalName)` validates the global-name tree. +- `CheckScopes_DuplicatedSymbol(this)` reports duplicate symbols after scopes exist. + +This phase is intentionally before type completion. It gives the compiler a complete declaration graph before signatures and member types are fully filled. + +### Scope Completion And Scope Checks + +The next phase calls: + +- `CompleteScopeForModule(this, module)`; +- `CheckScopes_SymbolType(this)`; +- `CheckScopes_CycleDependency(this)`. + +`CompleteScopeForModule` fills type information that could not be known until all names and scopes were available. State machines rely heavily on this phase because generated input methods, generated cache fields, and the generated `CreateCoroutine` method all need completed signatures and field types. + +### Semantic Validation And Type-Dependent Expansion + +For each module, `Rebuild` then calls: + +- `IWfCompilerCallback::OnValidateModule(module)`; +- `ValidateModuleSemantic(this, module)`. + +This phase resolves expression types into `expressionResolvings`, selects overloads, records lambda captures, records interface method implementations, validates type-dependent rules, and expands context-sensitive virtual AST nodes. + +High-level constructs such as `bind`, `$coroutine{}`, co-provider statements, switch/foreach statement virtual nodes, and state machines are deliberately expanded here because they depend on semantic side tables produced by earlier validation. + +### Post-Semantic Metadata + +After semantic validation: + +- `PopulateAttributesOnTypeDescriptors(this)` transfers validated Workflow attributes onto generated type descriptors. +- `ValidateModuleRPC(this, module)` runs when `validateRpc` is true and can populate `rpcMetadata`. + +The compiler callback is a progress hook, not part of compiler correctness. Logging and external diagnostics can be implemented differently by different callers. + +## Virtual AST Expansion Model + +Workflow has two important virtual-node families: + +- `WfVirtualCfe*`: context-free virtual nodes expanded before scope construction. +- `WfVirtualCse*`: context-sensitive virtual nodes expanded during semantic validation. + +The common context-sensitive recheck pattern is: + +1. Validate the original virtual node enough to collect semantic data. +2. If no new errors were added, create `expandedExpression`, `expandedStatement`, or `expandedDeclarations`. +3. Attach source code ranges from the original node to generated nodes for diagnostics. +4. Run context-free desugaring on the generated subtree. +5. Build scopes for the generated subtree under the original parent scope. +6. Remove stale duplicate-symbol and symbol-type check records for the affected parent scope. +7. Re-run `CheckScopes_DuplicatedSymbol` and `CheckScopes_SymbolType`. +8. Semantically validate the generated subtree. + +This pattern is visible in `ValidateSemantic_Expression.cpp`, `ValidateSemantic_Statement.cpp`, and `ValidateSemantic_Declaration.cpp`. It is the main reason generated code can use the same language features as user-written Workflow code: generated AST goes back through the ordinary compiler checks before backend emission. + +The emitters preserve the invariant: + +- `WfEmitter_Expression.cpp` and `WfCpp_Expression.cpp` dispatch virtual expressions through `expandedExpression`. +- Statement and declaration emitters similarly use expanded statements and declarations. +- Raw coroutine and state-machine-only statements reaching backend generation are internal errors. + +## Bind Lowering + +`bind` is parsed as `WfBindExpression`, a `WfVirtualCseExpression` with an original `expression` and generated `expandedExpression`. `observe` syntax is parsed as `WfObserveExpression`, with `WfObserveType::SimpleObserve` for `parent.observe(...)` and `WfObserveType::ExtendedObserve` for `parent.observe as name(...)`. + +### Validation And Scope Preparation + +Before expansion: + +- `ValidateStructureExpressionVisitor::Visit(WfBindExpression*)` rejects nested `bind`. +- `ValidateStructureExpressionVisitor::Visit(WfObserveExpression*)` requires `observe` to appear under `bind`, rejects nested observe in observe-event expressions, checks simple observe shapes, and requires extended observe to have at least one event. +- `ValidateStructureExpressionVisitor::Visit(WfAttachEventExpression*)` and `Visit(WfDetachEventExpression*)` reject manual attach/detach inside `bind`. +- `BuildScopeForExpression::Visit(WfObserveExpression*)` creates a local scope for extended observe and adds the alias symbol. +- `ValidateSemanticExpressionVisitor::Visit(WfBindExpression*)` validates the bound expression and returns readonly `Ptr`. + +The runtime interface is `IValueSubscription` in `Import/VlppReflection.h`. It exposes `ValueChanged`, `Open`, `Update`, and `Close`. + +### BindContext + +`ExpandBindExpression` in `WfAnalyzer_ExpandBindExpression.cpp` begins by running `CreateBindContextVisitor` over the already-resolved original expression. The visitor builds `BindContext`, which contains: + +- `observeParents`: observed expression to parent expression that must be cached. +- `observeEvents`: observed expression to reflected events that must be attached. +- `orderedObserves`: observes in discovery order. +- `cachedExprs`: parent expressions that become generated `N` fields. +- `renames`: reference expression to expression substituted during dependency discovery and expression rewrite. +- `exprAffects` and `exprCauses`: expression-level dependency graph. +- `observeAffects` and `observeCauses`: observe-to-observe dependency graph. + +`CreateBindContextVisitor::Visit(WfMemberExpression*)` detects implicit observable property access by checking `manager->expressionResolvings[node].propertyInfo`. If the property has `GetValueChangedEvent()` or an event named `Changed`, the member expression becomes an observed expression and its parent is cached. + +`CreateBindContextVisitor::Visit(WfObserveExpression*)` records explicit observes, caches their parent, visits the observed value expression, and records each resolved event expression. + +`renames` is an expression substitution mechanism, not closure capture. It resolves references to `let` variables and extended-observe aliases back to their defining expressions, so dependency discovery can see through source-local names. + +### Dependency Propagation + +`CreateBindContextVisitor::DirectDepend(expr, depended)` propagates all observes that can change `depended` into the causes of `expr`. + +`CreateBindContextVisitor::ObservableDepend(expr, parent)` records the observe, records its parent, ensures the parent is cached, and builds observe-to-observe edges when the parent itself depends on earlier observes. + +The resulting graph drives efficient callback generation. When one event fires, the generated callback can compute the transitive downstream observes that depend on the changed observe, detach only those affected handlers, recompute only the affected cached parents, reattach handlers, and then call the activator. + +### Generated Subscription + +`ExpandBindExpression` replaces the bind expression with a generated `WfNewInterfaceExpression` typed as `Ptr`. + +The generated object contains: + +- `var N : ParentType = default;` for each cached parent expression. +- `var I_J : EventHandler^ = null;` for each observed event. +- `var : bool = false;`. +- `var : bool = false;`. +- `func ()`. +- `func I_J(...)` for each observed event. +- overrides for `Open`, `Update`, and `Close`. + +`CreateWritableVariable` and `CreateDefaultValue` construct generated fields and initial values. `CreateDefaultValue` is also used by coroutine and state-machine lowering, so generated reset behavior is shared. + +### Expression And Control-Flow Rewrite + +`ExpandObserveExpressionVisitor` rewrites the original expression: + +- Cached expressions become references to `N`. +- `let` variables whose values are cached are removed from the generated `let`. +- Simple observe becomes member access on the rewritten parent. +- Extended observe becomes the rewritten inner expression. + +The activator evaluates the rewritten original expression into `` and raises `ValueChanged()`. + +`CreateBindCacheAssignStatement` emits cache assignment using `WfBinaryOperator::FailedThen`, conceptually: + +```workflow +N = expanded-parent-expression ?? default(parent-type); +``` + +This keeps subscription maintenance resilient to failed parent evaluation by resetting the cache to the default type value. + +`CreateBindOpenFunction` generates: + +1. set `` if it was false; +2. assign caches in dependency order; +3. attach all observed event handlers; +4. return true, or return false if already opened. + +`CreateBindUpdateFunction` calls `()` only when opened and not closed. + +`CreateBindCloseFunction` sets ``, detaches non-null handlers, resets caches and handler fields to defaults, and returns whether it actually closed. + +Each generated callback: + +1. starts from the observe whose event fired; +2. walks `observeAffects` to find affected downstream observes; +3. detaches affected downstream handlers in reverse order; +4. recomputes each affected parent cache once; +5. reattaches affected handlers; +6. calls `()`. + +`Test/Generated/Workflow64/Parsing.Codegen.BindLet.txt` demonstrates this printed shape: callbacks detach handlers under old cached child objects, recompute child caches such as `5`, reattach, and invoke the activator. + +### Variables And Captures + +Bind has three variable mechanisms: + +- observed parent expressions become generated cache fields; +- `let` variables and extended-observe aliases are handled through `BindContext::renames` and may disappear from the generated expression if their value is cached; +- outer function variables are captured by ordinary anonymous-interface capture machinery. + +`BuildScopeForExpression::Visit(WfNewInterfaceExpression*)` creates a `WfLexicalCapture` for the generated interface implementation. During semantic resolution, references from generated subscription methods to outer function locals are recorded in `manager->lambdaCaptures`. Bytecode metadata names those as `x`, while bind-owned state uses names such as `0` and `0_0`. + +`ExpandBindExpression` therefore does not convert surrounding locals into bind-specific fields. It relies on the same closure model used by user-written `new interface` expressions. + +## Co-Provider To Raw Coroutine To Trivial Workflow + +Coroutine-related syntax is lowered in two distinct layers: + +1. Co-provider syntax is lowered to raw coroutine syntax. `WfCoProviderStatement` is rewritten to a provider `Create` or `CreateAndRun` call that receives an anonymous function returning `WfNewCoroutineExpression`. +2. Raw coroutine syntax is lowered to trivial Workflow code. `WfNewCoroutineExpression` is compiled into a generated `WfNewInterfaceExpression` implementing `ICoroutine`, with explicit fields, state integers, dispatch loops, and ordinary statements. + +This layering is important for maintenance. Co-provider lowering does not directly build a coroutine flow chart. It generates a raw coroutine body containing `WfCoPauseStatement`, provider method calls, result checks, and normal statements. Then the raw coroutine expander owns all control-flow splitting, variable persistence, pause/resume handling, exception routing, and final `ICoroutine` object generation. + +### Parsed Forms + +The compiler does not have a hardcoded `$yield` node. `Source/Parser/Syntax/Lexer.txt` defines `COROUTINE_OPERATOR` as `$` followed by an uppercase name. Operators such as `$Yield`, `$Await`, and `$Join` all use the same internal shape. + +The parser represents coroutine-related syntax as: + +- `WfCoProviderStatement` for provider bodies such as `$ { ... }` or `$Async { ... }`. +- `WfCoOperatorStatement` for statement operators such as `$Yield i;`. +- `WfCoOperatorStatement` with `varName` for result-producing operators such as `var x = $Await async;`. +- `WfCoOperatorExpression` for context expressions such as `$.Context`. +- `WfNewCoroutineExpression` for raw `$coroutine{}`. + +`WfCoProviderStatement`, `WfCoOperatorStatement`, and `WfCoOperatorExpression` are the high-level provider layer. `WfNewCoroutineExpression` is the raw coroutine layer. + +### Semantic Resolution Before Lowering + +`BuildScopeForStatement::Visit(WfCoProviderStatement*)` creates provider-scope symbols: + +- `$PROVIDER`: selected provider type. +- `$IMPL`: provider implementation type passed to the generated coroutine creator. + +`ValidateSemanticStatementVisitor::Visit(WfCoProviderStatement*)` resolves the provider: + +- unnamed `$ { ... }` starts from the containing function return type and searches for `Coroutine`, including base type descriptors; +- named `$Async { ... }` strips `$`, resolves `Async`, then searches for `AsyncCoroutine` or accepts `AsyncCoroutine` directly; +- void-like provider functions use static `CreateAndRun`; +- value-returning provider functions use static `Create`; +- the selected creator must accept `Ptr, ImplType>>`; +- the selected method is saved in `manager->coProviderResolvings`. + +`ValidateSemanticStatementVisitor::Visit(WfCoOperatorStatement*)` resolves each operator against `$PROVIDER` and `$IMPL`: + +- `$Yield` becomes operator name `Yield`; +- the provider is searched for static `YieldAndRead`; +- if there is no result variable, static `YieldAndPause` is also considered; +- candidates must take `$IMPL` as the first parameter; +- overload selection runs with a dummy `$IMPL` first argument followed by source arguments; +- the selected provider method is saved in `manager->coOperatorResolvings`. + +For result-producing operators such as `var x = $Await async;`, semantic validation also finds a static `CastResult(Value)` helper from argument types. The helper's return type becomes the lexical type of `x`, and the helper is saved in `manager->coCastResultResolvings`. + +Return statements inside provider bodies are provider-specific. `ValidateSemanticStatementVisitor::Visit(WfReturnStatement*)` resolves provider static `ReturnAndExit`, requiring `$IMPL` as the first parameter, and saves the method in `manager->coOperatorResolvings`. + +`WfCoOperatorExpression`, the expression form used for `$.Context`, cannot be expanded at the outer provider level because it needs the generated `` variable. Its expansion is delayed until the generated raw coroutine body is being copied and validated under the scope containing ``. + +### Layer 1: Co-Provider To Raw Coroutine + +`ExpandCoProviderStatement` in `WfAnalyzer_ExpandStatement.cpp` rewrites `WfCoProviderStatement` into ordinary AST that calls the provider creator with a generated function. Conceptually, the generated shape is: + +```workflow +Provider::Create-or-CreateAndRun( + func( : ImplType) : Coroutine^ + { + return $coroutine() { rewritten provider body }; + } +); +``` + +Internally this first layer is represented by: + +- a `WfFunctionExpression`; +- an anonymous `WfFunctionDeclaration`; +- a generated `WfFunctionArgument` named ``, typed as the resolved `$IMPL`; +- a `WfNewCoroutineExpression` named ``; +- a `WfCallExpression` to the selected provider `Create` or `CreateAndRun` method; +- a `WfBlockStatement` stored in `WfCoProviderStatement::expandedStatement`. + +The important result is that the high-level provider statement has become a normal provider method call whose callback returns a raw coroutine. From this point on, the provider body is treated as a coroutine body. + +`ExpandCoProviderStatementVisitor` copies the source provider body into the generated `WfNewCoroutineExpression` and rewrites provider-only statements during the copy. + +For `$Yield i;`, the rewritten raw coroutine body contains: + +- a `WfCoPauseStatement`; +- an inner pause block containing a `WfCallExpression` to the resolved provider method, such as `EnumerableCoroutine::YieldAndPause(, i)`; +- ordinary post-pause checks against ``. + +Conceptually: + +```workflow +$pause +{ + EnumerableCoroutine::YieldAndPause(, i); +} +if ( is not null) +{ + if (.Failure is not null) raise .Failure; +} +``` + +For `var x = $Await async;`, the raw coroutine body is also a pause around the resolved provider call. After resume, it checks `.Failure`, reads `.Result`, converts it through the resolved `CastResult(Value)` method, and emits a normal `WfVariableStatement` for `x`. + +For `return expr;`, the provider body is rewritten to call `Provider::ReturnAndExit(, ...)` and then emit a final `return`. If the provider result type exposes `StoreResult`, the source return expression is wrapped by that helper before being passed to `ReturnAndExit`. + +After this first layer, the remaining provider-specific concepts are expressed through ordinary calls and through `WfCoPauseStatement`. The compiler no longer needs a special co-provider control-flow model. + +### Layer 2: Raw Coroutine To Flow Chart + +`ExpandNewCoroutineExpression` in `WfAnalyzer_ExpandNewCoroutineExpression.cpp` handles raw `WfNewCoroutineExpression`, regardless of whether it came from user-written `$coroutine{}` or from `ExpandCoProviderStatement`. + +The expander builds a flow-chart model: + +- `FlowChart` owns all nodes, `headNode`, `lastNode`, and temporary exception variables. +- `FlowChartNode` holds copied statements, conditional branches, default destination, exception destination, pause destination, action, exception variable, and an `embedInBranch` optimization flag. +- `FlowChartBranch` holds an optional condition and a destination. +- `GenerateFlowChartStatementVisitor::ScopeContext` models structured exits for functions, loops, labeled blocks, and try/finally. + +The flow-chart pipeline is: + +1. `FindCoroutineAwaredStatements` marks statements that affect suspension or control-flow splitting: virtual statements, coroutine statements, `break`, `continue`, `return`, `goto`, `if`, `while`, `try`, and blocks containing aware statements. +2. `FindCoroutineAwaredVariables` finds local variables inside aware regions. +3. `FindCoroutineReferenceRenaming` assigns generated field names for locals and temporary control variables that must survive. +4. `GenerateFlowChart` copies statements into flow-chart nodes and rewrites references to renamed symbols. +5. `RemoveUnnecessaryNodes` merges trivial nodes, removes the default destination after pause sentinels, and marks simple branch targets for embedding. +6. `ExpandNewCoroutineExpression` emits a generated `WfNewInterfaceExpression` implementing `ICoroutine`. + +This second layer is where raw coroutine code stops being structured source code and becomes explicit resumable control flow. + +### Variable Conversion In Raw Coroutine Lowering + +Coroutine variable conversion is symbol-based, not text-based. The renaming map is keyed by `WfLexicalSymbol*`, avoiding collisions when generated or source variables reuse textual names in nested scopes. + +Naming rules include: + +- source local `x` can become `x`; +- generated temporaries such as `i` can become `i`; +- catch variables, `if` expression variables, and try/finally temporary exception variables can also be renamed. + +When a renamed `WfVariableStatement` appears in an aware region, `GenerateFlowChartStatementVisitor::Visit(WfVariableStatement*)` replaces the declaration with an assignment to the generated field. `ExpandNewCoroutineExpression` then emits one generated field declaration for each renamed symbol, using the original symbol type and `CreateDefaultValue`. + +Variables not in the renaming map remain ordinary locals. Outer function variables are handled by normal function/lambda capture analysis around the generated function expression, not by coroutine field conversion. + +For a co-provider body, this means variables introduced by source code and variables introduced by co-operator rewriting both participate in the same raw coroutine persistence algorithm. For example, a variable produced by `var x = $Await ...` is initially emitted as a normal `WfVariableStatement` after the pause. If later coroutine states need `x`, `FindCoroutineReferenceRenaming` can promote it to a generated coroutine field. + +### Pause Conversion In Raw Coroutine Lowering + +`GenerateFlowChartStatementVisitor::Visit(WfCoPauseStatement*)` turns a raw coroutine pause into three flow-chart nodes: + +1. a node with `action = SetPause`; +2. a sentinel node containing an empty `WfCoPauseStatement`; +3. a resume-continuation node stored in `pauseDestination`. + +If the pause has an inner statement, such as `EnumerableCoroutine::YieldAndPause(, i)`, the inner statement is copied into the `SetPause` node before the sentinel pause. + +`ExpandFlowChartNode` converts a `SetPause` node into ordinary code that: + +1. sets coroutine status to waiting; +2. records ``; +3. sets `` to the state after the pause; +4. runs the provider operation or other inner pause statements; +5. returns from `Resume`. + +On the next `Resume`, the generated coroutine: + +- verifies it is waiting; +- sets status to executing; +- examines the resume input when ` != -1`; +- clears the pre-pause state if there is no failure; +- routes failure to a catch node or raises it; +- dispatches through a generated loop using ``. + +In a co-provider-generated coroutine, the resume input is named ``. That is why the first-layer co-provider rewrite emits checks against `` after every operator pause: once the second layer has split the coroutine, those checks execute only after the provider runtime resumes the coroutine with a `CoroutineResult`. + +### Layer 3: Flow Chart To Trivial Workflow Code + +The final output of raw coroutine lowering is generated ordinary Workflow AST, not a special backend coroutine instruction. + +`ExpandNewCoroutineExpression` emits a generated `WfNewInterfaceExpression` implementing `ICoroutine`. The generated object contains: + +- generated fields for renamed locals and temporaries; +- ``; +- ``; +- auto properties for `Failure` and `Status`; +- override `Resume(, )`. + +`Resume` is trivial Workflow control flow: + +- it uses `if` checks and loops to dispatch on ``; +- normal flow-chart edges assign `` and continue; +- branch edges emit ordinary `if` statements, sometimes embedding small destination nodes; +- `break`, `continue`, `goto`, and `return` are represented as jumps to generated flow-chart destinations; +- `InlineScopeExitCode` inserts pending `finally` code while leaving scopes; +- nodes with `exceptionDestination` are wrapped in generated try/catch code that assigns the caught exception to a renamed exception field, sets `` to the catch node, and continues. + +This is the point where both user-written raw `$coroutine{}` and co-provider-generated raw coroutine bodies become the same kind of generated trivial Workflow code. Bytecode and C++ emitters then see ordinary interface implementation, fields, methods, properties, statements, and expressions through `expandedExpression` or `expandedStatement`. + +## State Machine Lowering + +State machines are context-sensitive declarations. The parser represents them as `WfStateMachineDeclaration`, with: + +- `WfStateInput` for `$state_input`; +- `WfStateDeclaration` for `$state`; +- `WfStateSwitchStatement` for `$switch`; +- `WfStateInvokeStatement` for `$goto_state` and `$push_state`. + +### Preparation Before Expansion + +State machines participate in several `Rebuild` phases before `ExpandStateMachine` runs: + +- `BuildGlobalNameFromModules::BuildClassMemberVisitor::Visit(WfStateMachineDeclaration*)` synthesizes class member placeholders for input methods, input argument fields, state argument fields, and `CreateCoroutine`. +- `BuildScopeForDeclarationVisitor::Visit(WfStateMachineDeclaration*)` creates lexical symbols for state inputs, states, state arguments, switch-case arguments, and state bodies. +- `CompleteScopeForClassMemberVisitor::Visit(WfStateMachineDeclaration*)` fills input method signatures, generated field types, and `CreateCoroutine(startState:int):void`. +- `ValidateSemanticDeclarationVisitor::Visit(WfClassDeclaration*)` requires a class containing a state machine to inherit from `system::StateMachine`. +- `ValidateSemanticDeclarationVisitor::Visit(WfStateMachineDeclaration*)` requires a default state and validates each state body. +- `ValidateSemanticStatementVisitor::Visit(WfStateSwitchStatement*)` resolves switch cases to generated input methods and assigns case argument symbol types from input method parameters. +- `ValidateSemanticStatementVisitor::Visit(WfStateInvokeStatement*)` resolves target states and validates invocation arguments against target state argument types. + +`WfStateMachineInfo` stores `createCoroutineMethod`, `inputIds`, and `stateIds`. `inputIds` and `stateIds` are filled by `ExpandStateMachine`. + +### Generated Class Members + +`ExpandStateMachine` turns a state-machine declaration into ordinary class members: + +- private fields named like `ARG` for input arguments; +- private fields named like `ARG` for state arguments; +- generated input methods; +- generated `CreateCoroutine`. + +Generated fields are added to `WfStateMachineDeclaration::expandedDeclarations` as `WfVariableDeclaration` nodes with `@cpp:Private`, and connected back to reflected field metadata through `manager->declarationMemberInfos`. + +This is different from coroutine local conversion. Raw coroutine lowering hoists locals only when the flow chart proves they must survive a pause. State-machine lowering creates class fields up front for semantic state because input arguments and state invocation arguments must survive across input method calls, state transitions, and pushed child states. + +### Generated Input Methods + +For each `$state_input`, `ExpandStateMachine` emits a normal class method using the existing `WfClassMethod` from `stateInputMethods`. + +Each input method: + +1. initializes the state machine on first call by setting `stateMachineInitialized`, creating the default-state coroutine with `CreateCoroutine(0)`, and calling `ResumeStateMachine`; +2. stores the input id in `stateMachineInput`; +3. copies method arguments into `ARG` fields; +4. calls `ResumeStateMachine` again to deliver the input. + +The generated method is ordinary AST: `WfFunctionDeclaration`, `WfIfStatement`, `WfBinaryExpression`, `WfMemberExpression`, `WfCallExpression`, and `WfExpressionStatement`. + +### Generated State Coroutine + +`ExpandStateMachine` emits: + +```workflow +func CreateCoroutine(startState : int) : void +``` + +The method creates a `WfNewCoroutineExpression` and assigns it to `stateMachineCoroutine`. It saves the previous coroutine in `previousCoroutine`, creates a coroutine using `stateMachineObject = this`, and restores the previous coroutine in a `finally` block. + +The generated coroutine body contains: + +- a protected block labeled `OUT_OF_STATE_MACHINE`; +- a `finally` block restoring `stateMachineCoroutine`; +- local `state = startState`; +- `while(true)` labeled with `OUT_OF_CURRENT_STATE`; +- local `currentState = state`; +- assignment `state = -1`; +- a `WfSwitchStatement` over `currentState`; +- one switch case for each declared state. + +Each state case copies `ARG` fields into local state arguments, copies the original state body through `ExpandStateMachineStatementVisitor`, and emits a goto to leave the state machine as default fall-through. + +This generated code intentionally uses the same ordinary control-flow constructs understood by coroutine lowering: labels, gotos, loops, switches, try/finally, and `WfCoPauseStatement`. + +### State Switch Lowering + +`ExpandStateMachineStatementVisitor::Visit(WfStateSwitchStatement*)` rewrites `$switch` into a block containing: + +- an `if` that pauses when `stateMachineInput == -1`; +- a `WfSwitchStatement` over `stateMachineInput`. + +For each explicit case: + +- the case expression is the numeric id from `smInfo->inputIds`; +- the generated case clears `stateMachineInput` to `-1`; +- input arguments are copied from `ARG` fields into local case variables; +- the original case body is copied. + +If the switch type is `WfStateSwitchType::Default`, the expander also creates raising cases for every input not explicitly listed. + +Default branch behavior depends on `WfStateSwitchType`: + +- `Default`: no default branch; missing input cases were generated as raising cases. +- `Pass`: no default branch; unhandled input remains available. +- `PassAndReturn`: `goto OUT_OF_STATE_MACHINE`. +- `Ignore`: clear `stateMachineInput`. +- `IgnoreAndReturn`: clear `stateMachineInput`, then leave the state machine. + +The generated `WfSwitchStatement` is itself another context-sensitive virtual statement and is later expanded by the normal switch expander. + +### State Invoke Lowering + +`ExpandStateMachineStatementVisitor::Visit(WfStateInvokeStatement*)` handles `Goto` and `Push`. + +Both forms first copy invocation arguments into generated state argument fields: + +```workflow +stateMachineObject.ARG = copiedArgument; +``` + +`WfStateInvokeType::Goto` then assigns the target state id to `state` and jumps to `OUT_OF_CURRENT_STATE`. This is a domain-state transition inside the same generated coroutine. + +`WfStateInvokeType::Push` calls `CreateCoroutine(TargetStateId)` and emits `WfCoPauseStatement`. This replaces `stateMachineCoroutine` with a child coroutine and pauses the current coroutine. The stack-like behavior is encoded by the generated `previousCoroutine` local plus the `finally` block that restores `stateMachineCoroutine`. + +`system::StateMachine::ResumeStateMachine` in `Source/Library/WfLibraryPredefined.cpp` drives this protocol. It detects whether a resume pushed a child coroutine, whether a child finished and restored the parent coroutine, and whether a child failure should be delivered back to the parent as a `CoroutineResult`. + +### Shared And Different From Coroutine Lowering + +State-machine lowering shares these constructions with raw coroutine lowering: + +- it generates `WfNewCoroutineExpression`; +- it generates `WfCoPauseStatement`; +- it uses ordinary Workflow control-flow AST; +- it uses `CreateDefaultValue`; +- it relies on the same virtual-node recheck pipeline; +- it ultimately produces an `ICoroutine` implementation with ``, ``, `Failure`, `Status`, and `Resume`; +- locals in the generated coroutine body can be renamed into coroutine fields by `FindCoroutineReferenceRenaming`. + +The important differences are: + +- raw coroutine lowering starts from a statement body and directly builds a `FlowChart`; state-machine lowering starts from a class declaration and first generates fields, methods, and a generated coroutine body; +- raw coroutine state ids are implementation states in `FlowChartNode` order; state-machine ids in `smInfo->stateIds` are domain states, and coroutine lowering later adds another implementation-state layer; +- raw coroutine persistence is discovered by `FindCoroutineAwaredVariables`; state-machine input and state arguments are stored in generated class fields before coroutine lowering; +- raw coroutine resumption receives `CoroutineResult` directly from a `Resume` caller; state-machine resumption goes through generated input methods and `StateMachine::ResumeStateMachine`; +- state-machine switch policies have no raw coroutine equivalent. + +## Backend Contract And Maintenance Rules + +The backend contract is simple: generated code must be ordinary Workflow AST by the time bytecode or C++ generation needs it. + +For maintenance work: + +- decide whether new syntax is context-free or context-sensitive; +- context-free syntax can expand before scopes; +- context-sensitive syntax must validate original nodes first, generate `expanded*`, and re-run desugar, scope building, scope checks, and semantic validation on generated AST; +- generated AST must be compatible with `BuildScope`, `CompleteScope`, semantic validation, bytecode emission, and C++ emission; +- new semantic side tables must be cleared by `WfLexicalScopeManager::Clear`; +- generated nodes should keep source code ranges from the construct that caused them; +- logging and callbacks should not be required for compiler correctness; +- raw coroutine and state-machine statements should remain frontend-only. + +The generated samples under `Test/Generated/Workflow32` and `Test/Generated/Workflow64` are useful for auditing the printed desugared shape. For example, bind samples show generated subscriptions, co-provider samples show generated coroutine state fields, and state-machine samples show generated input methods plus coroutine-backed state loops. diff --git a/.github/KnowledgeBase/KB_Workflow_Design_GeneratedRpcWrappers.md b/.github/KnowledgeBase/KB_Workflow_Design_GeneratedRpcWrappers.md new file mode 100644 index 00000000..d05c57ba --- /dev/null +++ b/.github/KnowledgeBase/KB_Workflow_Design_GeneratedRpcWrappers.md @@ -0,0 +1,75 @@ +# Workflow Generated RPC Wrappers + +This document describes the generated C++ functions produced from Workflow RPC metadata and how an application should wire them into an RPC lifecycle. Attribute rules and serializable type rules are covered by [Workflow Interface-Based RPC Definition](./KB_Workflow_InterfaceBasedRpcDefinition.md). + +## When to Call Generated Functions + +Call `rpc_GetIds()` before a lifecycle starts handling RPC objects. It returns the dictionary from RPC type and member names to numeric ids. The lifecycle uses this dictionary to decide whether a local object can be represented by an RPC type id and to resolve service names; application code also uses it when registering local services. + +Create the generated ops objects once for each lifecycle: + +- `rpcops_IRpcObjectOps(lc)` creates the callee-side object operations. +- `rpcops_IRpcObjectEventOps(lc)` creates the receive-side object event operations. +- `rpcops_IOps_Create(lc)` creates strongly typed caller-side method operations used by generated wrappers. + +Register these objects with the lifecycle's controller, together with the runtime-provided list operations and list event operations. The generated object ops receive remote method calls, object holds, and object unholds. Runtime object-op adapters intercept predefined list method ids and redirect them to the local list operations before delegating all other method ids to the generated object ops. The generated object event ops receive remote object events, while runtime object-event adapters intercept the predefined observable-list event id and redirect it to the local list event operations. The generated caller-side ops are passed to wrappers so wrapper methods can send method calls through the dispatcher. + +Register a wrapper factory that calls `rpcwrapper_Create(ref, lc, ops)`. The lifecycle calls this factory when `RefToPtr(ref)` sees that `ref.clientId` belongs to another client. `rpcwrapper_Create` switches on `ref.typeId` and returns the generated wrapper interface for that remote object. + +Generated wrappers implement the original RPC interfaces and `IRpcWrapperBase`. Call interface methods on them exactly as normal C++ interface methods. A wrapper method checks that it is still connected to a lifecycle, then delegates to the generated caller-side ops. The wrapper destructor sends an object-unhold message to the owner lifecycle unless the wrapper has already been disconnected. + +Call `IRpcLifecycle::RegisterLocalService(rpc_GetIds()[fullName], service)` on the service-owning lifecycle to expose a singleton service for an `@rpc:Ctor` interface before calling `IRpcLifecycle::Initialize()`. The lifecycle stores the local service object and asks the dispatcher to declare the service owner to remote lifecycles. Call `IRpcLifecycle::RequestService(fullName)` from a client lifecycle to get the service as either a local object or a generated wrapper. + +The helper predicates `rpcwrapper_IsInterfaceTypeId(typeId)` and `rpcwrapper_IsCtorInterfaceTypeId(typeId)` are generated for lifecycle and ops validation. Application code usually does not call them directly unless it is implementing custom lifecycle integration. + +## When to Call Generated JSON Serialization Functions + +Use the JSON variants when the RPC payload boundary should carry `JsonNode` values instead of direct reflected `Value` objects. + +Create `rpcops_IRpcSerializer()` and set it on the lifecycle before registering JSON object operations and runtime list adapters. The serializer delegates to generated `rpcjson_Serialize(value)` and `rpcjson_Deserialize(node)` for unknown values. + +Use these JSON ops instead of the non-JSON ops on a JSON lifecycle: + +- `rpcops_IRpcObjectOpsJson(lc)` +- `rpcops_IRpcObjectEventOpsJson(lc)` +- `rpcops_IOps_CreateJson(lc)` + +The wrapper factory still calls `rpcwrapper_Create(ref, lc, ops)`, but the `ops` argument should be the JSON caller-side ops. Generated wrappers do not need a separate JSON wrapper type; the difference is in the ops object they delegate to. + +The generated `rpcjson_Serialize_*` and `rpcjson_Deserialize_*` functions are for known enum, struct, and collection types. The generated `rpcjson_Serialize(value)` and `rpcjson_Deserialize(node)` functions are for unknown values, including collection elements and other values whose exact static type is not known at the call site. `RpcObjectReference` and `RpcException` serialization is handled by the predefined RPC JSON serializer instead of per-assembly generated struct functions. + +## How Generated Functions Finish Their Work + +A wrapper method finishes synchronously. It boxes all arguments according to the RPC byval/byref rules, sends `InvokeMethod(ref, methodId, arguments)` through `IRpcDispatcher::SendToClient_ObjectOps(ref.clientId)`, waits for the target lifecycle to finish the call, then unboxes and returns the result. + +On the receiving lifecycle, generated `IRpcObjectOps::InvokeMethod` switches on `methodId`, converts `ref` back to the target object through `RefToPtr(ref)`, unboxes arguments, calls the real object method, boxes the return value, and returns it to the caller. Unknown method ids raise an exception. + +Generated `IRpcObjectOps::InvokeMethod` catches exceptions raised by the real object method and returns `system::RpcException` with the exception message. Unknown method ids are local dispatch errors and escape directly instead of being encoded as `system::RpcException`. User RPC signatures cannot return or accept `system::RpcException`, so caller-side ops can distinguish this transport value from successful method results. A flat generated caller-side op calls `system::IRpcLifecycle::ReadMethodException` on the raw `InvokeMethod` result before unboxing. A JSON generated caller-side op first deserializes the returned `JsonNode`, then calls `ReadMethodException`; the helper only handles deserialized `system::RpcException` values. + +When the return value is an `@rpc:Byval` collection, generated object ops return `system::RpcByvalReturnValue` instead of the raw boxed collection. Its `value` field contains the boxed transfer value, and its `slot` field identifies a recursive copy of the returned collection cached in the callee-side ops object. Generated caller-side ops cast the result directly to `system::RpcByvalReturnValue^`, unbox `value` into the real return variable, call `EndInvokeMethod(slot)` on the same object ops, and then return that variable. Non-byval return values keep the direct return path. + +Object lifetime is completed through hold and unhold messages. Wrapper creation sends `ObjectHold(..., true)` to the owner lifecycle. Wrapper destruction sends `ObjectHold(..., false)`. The owner lifecycle updates local-object interest counts through `LocalObjectHold` and `LocalObjectUnhold`. + +A generated local event listener first checks the lifecycle controller's event suppression flag. If the event is not suppressed, it calls the generated strong typed caller-side ops event method. The ops method boxes the event arguments, broadcasts them through `IRpcDispatcher::BroadcastFromClient_ObjectEventOps(clientId)`, deserializes JSON return values when needed, and raises returned event exceptions. The returned `system::RpcException[int]` map is an internal transport value keyed by lifecycle client id. + +On the receive side, generated `IRpcObjectEventOps::InvokeEvent` returns `object`. Flat ops return `system::RpcException[int]` or `null`; JSON ops return the serialized JSON value for that same map-or-null result. It sets the suppression flag, raises the local event, catches any exception into a one-entry map keyed by the receiver lifecycle's client id, and clears the flag in a `finally` block. Unknown event ids are local dispatch errors and escape directly instead of being encoded into the returned map. If an RPC definition has no events, generated JSON event ops collapse `InvokeEvent` to a direct unknown-event-id raise. + +Predefined observable-list wrappers use the same exception map for `IRpcListEventOps::OnItemChanged`, also returned as `object`. The send-side list adapter broadcasts `ItemChanged` through `IRpcDispatcher::BroadcastFromClient_ObjectEventOps(clientId)->InvokeEvent(...)` with the predefined observable-list event id. The receive-side object-event adapter redirects that predefined id to the list event operations, which replay the remote list notification under the item-changed suppression flag, catch local handler exceptions into a one-entry map keyed by the receiver lifecycle's client id, and serialize the map when a serializer is configured. The send-side adapter deserializes before calling `IRpcLifecycle::ReadEventException`. + +Service registration finishes in the lifecycle. `RegisterLocalService(typeId, service)` converts the service pointer to a full `RpcObjectReference`, adds the owner hold, stores the service object by type id, and asks the dispatcher to transmit `DeclareLocalService(ref)`. Remote lifecycles receive the same full reference through `DeclareRemoteService(ref)` and store it by `ref.typeId`. Service lookup also finishes through the lifecycle: `RequestService(fullName)` resolves the full type name to a type id with `GetTypeIdFromName`, returns a local registered service first, or converts the stored remote service reference to a local pointer or wrapper with `RefToPtr(ref)`. + +## How Generated JSON Serialization Functions Finish Their Work + +JSON caller-side ops box arguments first, serialize each boxed value to a `JsonNode`, and put those nodes into the argument array. They send the method call through the same dispatcher path, receive a `JsonNode` result, deserialize it, unbox it to the expected return type, and return it. + +JSON callee-side object ops deserialize each incoming `JsonNode` argument before unboxing it to the declared Workflow type. After calling the target method, they box the return value, serialize it to `JsonNode`, and return that node to the caller. + +When a JSON callee-side object op catches an exception raised by the target object method, it serializes `system::RpcException` using the predefined RPC JSON serializer and returns that `JsonNode`. Unknown method ids are local dispatch errors and are not serialized as `system::RpcException`. JSON caller-side ops call `rpcjson_Deserialize` before `system::IRpcLifecycle::ReadMethodException`; if the deserialized result is `system::RpcException`, they raise the contained message. For byval collection returns, this check happens before casting the result to `system::RpcByvalReturnValue^`, so `EndInvokeMethod` is not called on exceptional results. + +For an `@rpc:Byval` collection return, JSON object ops still serialize the transfer value as a `JsonNode`, but that node is stored in `system::RpcByvalReturnValue.value` and the recursive copied collection is cached by `slot` on the callee. JSON caller-side ops cast the `InvokeMethod` result directly to `system::RpcByvalReturnValue^`, deserialize `value`, unbox the real return value, call `EndInvokeMethod(slot)`, and then return the unboxed result. Non-byval JSON returns still receive a direct `JsonNode`. + +JSON event sending follows the same argument rule through generated strong typed caller-side ops. The generated listener does not serialize arguments directly; it calls the strong typed ops method, and the JSON ops method boxes and serializes arguments before broadcasting. JSON event receiving deserializes the argument nodes before raising the local event under the suppression flag. Event exception maps still use `system::RpcException[int]`: JSON receive-side ops serialize the map using existing unknown-value support, and JSON send-side ops deserialize it before deciding whether to raise a normal Workflow exception. + +Known-type JSON serialization finishes by constructing the schema described in [Workflow JSON Serialization Schema](./KB_Workflow_Design_JsonSerializationSchema.md): primitives become JSON primitives or tagged primitive arrays as required, enums become numbers, structs become JSON objects, and collections become arrays or tagged collection objects. Unknown-type serialization includes enough type information in the JSON value for `rpcjson_Deserialize(node)` to reconstruct a reflected value. + +Deserialization validates the JSON shape it expects. Missing struct fields, unknown type tags, or unsupported reflected values raise exceptions. Interface references and byref collection references serialize through `RpcObjectReference`; deserialization converts the reference back through the lifecycle path when the value is unboxed. diff --git a/.github/KnowledgeBase/KB_Workflow_Design_JsonSerializationSchema.md b/.github/KnowledgeBase/KB_Workflow_Design_JsonSerializationSchema.md new file mode 100644 index 00000000..37ea821d --- /dev/null +++ b/.github/KnowledgeBase/KB_Workflow_Design_JsonSerializationSchema.md @@ -0,0 +1,188 @@ +# Workflow JSON Serialization Schema + +There are two kinds of JSON schema for RPC serializable types: +- Schema for known types. They are used when the context gives you enough information therefore you know what type is expected without having to look into actual values, including `object` or `system::JsonNode`. + - Functions in wrapper Workflow Script: + - `func rpcjson_Serialize_Enum_Type_Full_Name(value : enum::type::full::name) : (system::JsonNode^)` + - `func rpcjson_Serialize_Struct_Type_Full_Name(value : struct::type::full::name) : (system::JsonNode^)` + - `rpcjson_Deserialize_*` in reversed form. +- Schema for unknown types. + - Typically needed when access elements in collections transferred using `@rpc:Byref`. + - Functions in wrapper Workflow Script: + - `func rpcjson_Serialize(value : object) : (system::JsonNode^)` + - `func rpcjson_Deserialize(node : system::JsonNode^) : (object)` + +Pay attention to the `RpcObjectReference` and `RpcException` types, they are treated like structs. Serialization for both types is always needed, even when they do not appear in RPC metadata. +Types listed here should include every type appearing in `WfLexicalScopeManager::rpcMetadata`, but when `RpcObjectReference` or `RpcException` also appears in the list, do not duplicate its processing. + +## Schema for Known Types + +1) primitive types + +Use the second element in `UnknownType_PrimitiveSchema` if it is an array, unless it is `null | true | false | string`. + +2) enum types + +Use `number`. + +3) struct types + +`{ field: value, ...}` +- `field` is a string, which is required in JSON anyway. +- `value` is schema for known types. + +4) collection types + +For list: array of schema for known types. +For dictionary: array of `[key, value]`, both are schema for known types. + +## Schema for Unknown Types + +1) primitive types + +See `UnknownType_PrimitiveSchema` in `## Expected format of generated .d.ts files` + +2) enum types + +`["enum::type::full::name", number]`. + +3) struct types + +`{ "$": "struct::type::full::name", strong-type-version-of-this-struct...}` +`system::RpcObjectReference` and `system::RpcException` use this standard unknown-struct schema whenever they cross a JSON RPC boundary as dynamically typed transport values. Unknown RPC method or event ids are local dispatch errors, so they do not use the `system::RpcException` JSON transport schema. + +4) collection types + +- For list and oblist: `{ "$": "list" | "oblist", values: [elements ...]}`. +- For dictionary: `{ "$": "map", values: [[key1, value1], [key2, value2], ...]}`. + +## Implementation of rpcjson_Serialize + +Check if the value is null or bool or string, they can be serialized directly. +And then try to weak cast it to any `PrimitiveType?` type. +And then try to weak cast it to any collection interface type. +And then try to weak cast it to any `InterfaceType^` type. + +Weak casting failure results in `null` instead of an exception. +Throw an exception if all type testings fail. + +## Implementation of rpcjson_Deserialize + +First check if it is `JsonArray` or `JsonObject` or `JsonLiteral` or `JsonString`, +and then check the first element of array or "$" field of object, +and we know the exact type. +Throw an exception if all type testings fail. + +## JSON Return Values for Byval Collections + +When an RPC method returns an `@rpc:Byval` collection, JSON serialization still uses the same known-type or unknown-type schema for the collection value. The generated object ops put the resulting `JsonNode` in `system::RpcByvalReturnValue.value` and use `system::RpcByvalReturnValue.slot` to keep the recursive copied collection alive until the caller calls `EndInvokeMethod(slot)`. Non-byval JSON returns still return the `JsonNode` directly. + +## TypeScript Schema for Dispatcher Messages + +`Release/Rpc.d.ts` describes the JSON message envelopes for the generic RPC ops boundary represented by `IRpcDispatcher`. It does not know the generated RPC metadata for a sample, so every field that contains a serialized `JsonNode` uses a generic type parameter `T`. A concrete test should read `T` as `KnownTypeSchema | UnknownTypeSchema` from the generated `Serialization_*.d.ts` file for that sample. + +The declarations map directly to the C++ ops interfaces in `Source/Library/Rpc/WfLibraryRpc.h`: + +- `IObjectOps_*` maps to `IRpcObjectOps`. +- `IObjectEventOps_*` maps to `IRpcObjectEventOps`. +- `IRpcDispatcher_DeclareRemoteService` maps to dispatcher-level service declaration and is handled by `IRpcLifecycle::DeclareRemoteService(ref)`. + +Request envelopes model how `IRpcDispatcher` chooses an ops object. Calls made through `SendToClient_ObjectOps` include both `sourceClientId` and `targetClientId`. Calls made through `BroadcastFromClient_ObjectEventOps` include only `sourceClientId` because the dispatcher expands the broadcast target list. Response envelopes are always one-to-one from the receiving client back to the requesting client, so they always contain both client ids. `IRpcListOps` and `IRpcListEventOps` are local adapters only: list methods are transported as `IObjectOps_InvokeMethod` with predefined negative method ids, and observable-list `ItemChanged` is transported as `IObjectEventOps_InvokeEvent` with the predefined negative event id. + +The stable internal transport structs are declared in `Rpc.d.ts` itself: `system_RpcObjectReference`, `system_RpcException`, and `system_RpcByvalReturnValue`. The predefined JSON serializer owns `system::RpcObjectReference` and `system::RpcException` serialization; per-RPC generated serializers should not emit dedicated struct functions for them. Void-returning ops still have response envelopes, but no `response` field. Value-returning ops put the serialized value in `response`. `IRpcObjectEventOps::InvokeEvent`, including predefined observable-list `ItemChanged` events, returns the JSON form of `null | [number, system_RpcException][]`, matching `system::RpcException[int]` after deserialization. Request routing, response consolidation, and service declaration replay rules are specified in [Workflow JSON Request Routing](./KB_Workflow_JsonRequestRouting.md). + +## Other Strict Rules + +DO NOT generate any helper function that is not mentioned here, especially which just builds any `system::JsonNode^`. +You are going to repeat JSON AST building code in each function. +For JSON related functions, stick to the list in the beginning of this document. + + +## Expected format of generated .d.ts files + +```TypeScript +export type UnknownType_PrimitiveSchema = + | ["UInt8", number] + | ["UInt16", number] + | ["UInt32", number] + | ["UInt64", number] + | ["Int8", number] + | ["Int16", number] + | ["Int32", number] + | ["Int64", number] + | ["Single", number] + | ["Double", number] + | ["Char", string] + | ["DateTime", string] + | ["Locale", string] + | null + | true + | false + | string + ; + +export type TypeList_Enum = + | "enum::type::full::name" + ... + ; + +export type UnknownType_EnumSchema = [TypeList_Enum, number]; + +export interface UnknownType_List +{ + "$": "list" | "oblist"; + values: UnknownTypeSchema[]; +} + +export interface UnknownType_Map +{ + "$": "map"; + values: [UnknownTypeSchema, UnknownTypeSchema][]; +} + +export interface UnknownType_struct_type_full_name extends struct_type_full_name +{ + "$": "struct::type::full::name"; +} + +export type UnknownTypeSchema = + | UnknownType_PrimitiveSchema + | UnknownType_EnumSchema + | UnknownType_List + | UnknownType_Map + | UnknownType_struct_type_full_name + | ... + ... + +// UnknownType_* interfaces are generated for all structs, including +// system::RpcObjectReference and system::RpcException. + +// below are all known types + +export enum enum_type_full_name +{ + item = number-literal, + ... +} + +export interface struct_type_full_name +{ + field: value_type; +} + +// system::RpcObjectReference and system::RpcException are always generated here. + +// All enum_type_full_name is omitted because in known type enums are just numbers +export type KnownTypeSchema = + | number + | true + | false + | string + | KnownTypeSchema[] + | [KnownTypeSchema, KnownTypeSchema][] + | struct_type_full_name + | ... + ; +``` + +`JsonValue_*.ts` files generated for TypeScript validation contain JSON values captured at the generic RPC ops boundary, so each element is typed as `KnownTypeSchema | UnknownTypeSchema`. diff --git a/.github/KnowledgeBase/KB_Workflow_InterfaceBasedRpcDefinition.md b/.github/KnowledgeBase/KB_Workflow_InterfaceBasedRpcDefinition.md new file mode 100644 index 00000000..53d2d924 --- /dev/null +++ b/.github/KnowledgeBase/KB_Workflow_InterfaceBasedRpcDefinition.md @@ -0,0 +1,283 @@ +# Workflow Interface-Based RPC Definition + +## Attributes + +All user-authored attributes here do not have arguments unless explicitly specified. + +- `@rpc:Interface` can be used on an interface. + - `interface` in this document does not include generic interface: `$interface IDerivbed:IBased;`. +- `@rpc:Ctor` can be used only when `@rpc:Interface` is on that type. +- `@rpc:Byval` and `@rpc:Byref` on a property, method, parameter. +- `@rpc:Cached` and `@rpc:Dynamic` on a property. +- `@rpc:IdString(string)` and `@rpc:IdNumber(int)` on interfaces, events and methods. + - These attributes are generated into RPC Workflow metadata. + - User-authored `@rpc:IdString` and `@rpc:IdNumber` on RPC interfaces, methods and events are ignored when metadata is regenerated. + - `IdString` stores the generated full text id, and `IdNumber` stores the generated numeric id from the sorted RPC id list. + +## Serializable Types + +All types below count as serializable types, but serialization itself is optional: +- Predefined primitive types. +- Struct types. + - If it is reflection serializable, use the serialization. + - Otherwise, trigger the fallback serialization (generated). +- Enum types, including flags. + - Serialized to its interger value. +- Nullable types. + - Nullable could be applied to primitive types, struct types and enum types. + - Also all nullable types should just be serializable. +- Strong typed collection types, its element type, key type, value type should all be serializable. + - Strong typed collection types here mean the type uses Workflow's collection type syntax. + - `T{}`, `T[]`, `V[K]` and `observe T[]`. + - Weak typed collection types, like `system::Enumerable` and all others, are just trivial interfaces, they are not serializable because they do not have `@rpc:Interface`. +- Interface with `@rpc:Interface`. Only shared pointer `T^` is acceptable, `T*` is not. + +`system::RpcObjectReference` and `system::RpcException` are internal RPC transport structs. Generated infrastructure serializes them when needed, but user-authored RPC interfaces cannot use them in function return values, function arguments, or event arguments. + +## Compile Errors: + +- There are multiple types of property definitions, all count as properties. +- `FULL-NAME` means a full name of a type. + - If the target is a member, it becomes `type.member`. + - If the target is a parameter, it becomes `type.method(parameter)`. +- A method is serializable only when all argument types and the return type are serializable. + - If the return type is `void`, it is also serializable. +- When generating helper functions for an error, multiple error messages with the same format can share one helper function. Only text in `xxx` becomes a parameter (or if `xxx` has only one choice in this helper function, it should not be a parameter). + - The error code should be `H\d+`. + - Each helper function will be assigned with a unique error code. +- (AST) does not belong to the error message, it is a hint that this error should be checked by traversing the AST, otherwise it should be checked against `ITypeDescriptor`. + +## @rpc:Interface + +- (AST) `@rpc:Interface` can only apply to an interface definition, but not `FULL-NAME`. + - Triggered when it is applied to anything else. +- `@rpc:Interface` cannot be applied to interface `FULL-NAME` because its base type `FULL-NAME` is not serializable. + - Triggered when any base type does not apply with `@rpc:Interface`. + - Triggered for each inqualify base type. +- `@rpc:Interface` cannot be applied to interface `FULL-NAME` because its member `MEMBER-NAME` is not serializable. + - Triggered when the interface type has unserializable members. + - Triggered for each inqualify member. +- `@rpc:Interface` cannot be applied to interface `FULL-NAME` because its member `MEMBER-NAME` uses reserved RPC type `FULL-NAME` in a return value. + - Triggered when a function return value or property value contains `system::RpcObjectReference` or `system::RpcException`. +- `@rpc:Interface` cannot be applied to interface `FULL-NAME` because its function argument `MEMBER-NAME` uses reserved RPC type `FULL-NAME`. + - Triggered when a function argument contains `system::RpcObjectReference` or `system::RpcException`. +- `@rpc:Interface` cannot be applied to interface `FULL-NAME` because its event argument `MEMBER-NAME` uses reserved RPC type `FULL-NAME`. + - Triggered when an event argument contains `system::RpcObjectReference` or `system::RpcException`. + +## @rpc:Ctor + +- (AST) `@rpc:Ctor` can only apply to an interface definition with `@rpc:Interface`. + - Triggered when it is applied to anything else. + - Triggered when it is applied to an interface without `@rpc:Interface`. + +## @rpc:Byval + +- (AST) `@rpc:Byval` can only apply to a property, a method or a parameter. + - Triggered when it is applied to anything else. +- (AST) `@rpc:Byval` can only be used inside an interface type with `@rpc:Interface`. + - Not triggered any of the above error is triggered on the same attribute. + - Triggered when the member it applies to is not in an interface with `@rpc:Interface`. +- `@rpc:Byval` cannot be used on member `MEMBER-NAME` because it does not have a strong typed collection types. + - For property or parameter, it means the type. + - For method, it means the return type. + - Not triggered any of the above error is triggered on the same attribute. + - Triggered when the member is inqualify. + +## @rpc:Byref + +- (AST) `@rpc:Byref` can only apply to a property, a method or a parameter. + - Triggered when it is applied to anything else. +- (AST) `@rpc:Byref` cannot be used on member `MEMBER-NAME` because it already has `@rpc:Byval`. + - Not triggered any of the above error is triggered on the same attribute. + - Triggered when it is applied to a member with `@rpc:Byval`. +- (AST) `@rpc:Byref` can only be used inside an interface type with `@rpc:Interface`. + - Not triggered any of the above error is triggered on the same attribute. + - Triggered when the member it applies to is not in an interface with `@rpc:Interface`. +- `@rpc:Byref` cannot be used on member `MEMBER-NAME` because it does not have a strong typed collection types. + - For property or parameter, it means the type. + - For method, it means the return type. + - Not triggered any of the above error is triggered on the same attribute. + - Triggered when the member is inqualify. + +## @rpc:Cached + +- (AST) `@rpc:Cached` can only apply to a property. + - Triggered when it is applied to anything else. +- `@rpc:Cached` can only be used inside an interface type with `@rpc:Interface`. + - Not triggered any of the above error is triggered on the same attribute. + - Triggered when the member it applies to is not in an interface with `@rpc:Interface`. + +## @rpc:Dynamic + +- (AST) `@rpc:Dynamic` can only apply to a property. + - Triggered when it is applied to anything else. +- (AST) `@rpc:Dynamic` cannot be used on member `MEMBER-NAME` because it already has `@rpc:Cached`. + - Not triggered any of the above error is triggered on the same attribute. + - Triggered when it is applied to a member with `@rpc:Cached`. +- `@rpc:Dynamic` can only be used inside an interface type with `@rpc:Interface`. + - Not triggered any of the above error is triggered on the same attribute. + - Triggered when the member it applies to is not in an interface with `@rpc:Interface`. + +## Semantic + +## @rpc:Interface + +## @rpc:Ctor + +- Interface with this attribute could have implementation exposed as a singleton. +- RPC interface only allow registering singleton implementing `@rpc:Ctor` marked interfaces. +- RPC interface has dedicated methods for acquiring such singleton. + +## Default options between @rpc:Byval and @rpc:Byref + +- `Compile Errors` already limit these attributes on properties / return values / parameters of strong typed collection. +- If none is offered, the following collection types will by default `@rpc:Byref`, others will by default `@rpc:Byval`: + - `observe T[]`. + - Strong typed collections whose element or value is an interface, directly or through nested strong typed collections. +- During generating RPC Workflow metadata, `@rpc:Byval` and `@rpc:Byref` from the property will be copied to getter's return value and setter's parameter. +- Event arguments will pick the default value, and no customization could apply (because there is no syntax for putting attributes on event arguments). + +## @rpc:Byval + +- The whole collection is sent to a client, and it doesn't keep track on changes from the other side. +- All nested collections in this collection share the same behavior. + +## @rpc:Byref + +- The collection is treated as remote object. +- All nested collections in this collection share the same behavior. + +## @rpc:Cached + +- This is the default option, for any property. +- When property value is not cached, trigger actual RPC action and cache the result. +- When property value is cached, return the result immediately. +- Cached will be cleared if the associate event (if exists) is triggered. +- Client could proactivaly send property values to cache. + +## @rpc:Dynamic + +- The getter will trigger actual RPC action immediately. + +## Message Dispatching and Event Suppression + +The RPC runtime has three local concepts and one cross-client concept: + +- `IRpcLifecycle` owns object/reference conversion for one client. +- `IRpcController` owns local objects, local wrappers, local ops, and local suppression state for one client. +- `IRpcOperations` exposes the local client's list, object, list-event, and object-event operation objects. List operations and list-event operations are local runtime adapters; cross-client list traffic is transported through object operations and object-event operations with predefined negative ids. +- `IRpcDispatcher` is the only object that knows how to send a message from one client to another client. + +Lifecycle and controller implementations should not use another lifecycle's objects directly. When a message needs to leave the current client, it goes through `IRpcDispatcher`. + +### Service Registration and Lookup + +`IRpcLifecycle` owns service lookup state. A local service is stored in the service-owning lifecycle as a map from RPC type id to the local service object. A remote service is stored in each receiving lifecycle as a map from RPC type id to the full `RpcObjectReference` declared by the service owner. + +`IRpcLifecycle::RegisterLocalService(typeId, service)` records a local service before lifecycle initialization. It converts the service object to a local `RpcObjectReference`, adds the owner hold, stores the local service object, and calls `IRpcDispatcher::DeclareLocalService(ref)` so the dispatcher implementation can transmit the declaration. Registering the same local type id twice is a recoverable service-registration error, and registering after `IRpcLifecycle::Initialize()` is also an error. + +`IRpcDispatcher::DeclareLocalService(ref)` is a data-transmission hook, not service storage. The receiving lifecycle handles the declaration through `IRpcLifecycle::DeclareRemoteService(ref)`. Declaring a remote service with the same type id overwrites the earlier remote declaration and stores the full reference, including `objectId`. + +`IRpcObjectOps::RequestService` should not exist. `IRpcLifecycle::GetTypeIdFromName(typeName)` resolves the type name through the lifecycle RPC id map and returns `RpcTypeId_NotFound` when the name is unknown. `IRpcLifecycle::RequestService(typeName)` uses this function, returns the local registered service when that type id exists locally, or finds the stored remote `RpcObjectReference` and calls `RefToPtr(ref)`. + +`RefToPtr(ref)` uses `ref.clientId` to decide whether the ref belongs to the current client: + +- If the ref is local, it returns the local object. +- If the ref is remote, it returns or creates a wrapper for the remote object. + +### Refcounting and Service Hold Semantics + +Refcounting tracks interested clients for local objects. It is not the same concept as `Ptr` or `std::shared_ptr` ownership. A hold means a client has a remote wrapper or an equivalent durable interest in the local object. + +- `IRpcLifecycle::LocalObjectHold(ref, remoteClientId)` records that `remoteClientId` is interested in the local object identified by `ref`. +- `IRpcLifecycle::LocalObjectUnhold(ref, remoteClientId)` removes that interest. +- These functions only track local objects. If `ref` does not belong to the current lifecycle, this is an implementation error. +- The counter should track interested clients, not wrapper instances. Wrapper construction and destruction should happen once for each object/client pair, so duplicate holds or unholds for the same `(ref, remoteClientId)` should not silently create a different ownership meaning. + +`PtrToRef(ptr)` only converts a local object to a `RpcObjectReference`. It should allocate a new local object id when needed and attach the object-to-ref internal property, but it must not increase the interested-client counter. + +For a normal non-service object, the counter becomes 1 only after the first remote client actually receives the ref and creates its wrapper: + +- The remote wrapper constructor calls `IRpcDispatcher::SendToClient_ObjectOps(ref.clientId)->ObjectHold(ref, currentClientId, true)`. +- Generated interface wrappers and predefined container wrappers follow the same rule. +- The owner lifecycle receives this through `IRpcObjectOps::ObjectHold` and calls `IRpcLifecycle::LocalObjectHold(ref, remoteClientId)`. +- The wrapper destructor calls `ObjectHold(ref, currentClientId, false)` if it has not been disconnected from the lifecycle, and the owner lifecycle calls `IRpcLifecycle::LocalObjectUnhold(ref, remoteClientId)`. + +When the interested-client counter decreases to 0, the lifecycle should remove all resources and tracking for that local object reference. After this removal, a later `PtrToRef(ptr)` on the same object should allocate a new object id. + +There is a special case when `PtrToRef(ptr)` is called but no remote client ever receives the returned ref, so no wrapper constructor sends `ObjectHold(..., true)`. The object-to-ref internal property is the final fallback for this case: if the local object is deleted while the lifecycle is still alive, the property cleanup removes the tracking resource. During lifecycle finalization, implementations should remove these internal properties from all tracked local objects, so objects deleted after the lifecycle is gone do not call back into a finalized lifecycle. + +Services use the same local-object tracking, but `RegisterLocalService` adds an owner hold: + +- An object registered by `IRpcLifecycle::RegisterLocalService` has an initial interested-client count of 1. +- This hold represents the owner client being in the interested-client list. It is not created by a remote wrapper. +- Remote clients still send normal hold and unhold messages when they create or destroy wrappers for the service object. +- When all remote clients unhold the service object, the owner hold remains, so the service is not unregistered by ordinary refcounting. +- Ordinary service unregistration is not part of the RPC interface. Finalization clears lifecycle local-object tracking and local/remote service registration state. + +### Point-to-Point Operations + +When a wrapper performs a method call or list operation, the message is sent to the lifecycle that owns `ref.clientId`. + +- Object method calls use `IRpcDispatcher::SendToClient_ObjectOps(ref.clientId)`. +- List operations use `IRpcDispatcher::SendToClient_ObjectOps(ref.clientId)->InvokeMethod(...)` with predefined list method ids through `RpcCallerListOps`. +- Array resize uses its own predefined method id through `IRpcListOps::ArrayResize`; list-only mutation ids such as clear and remove-at should not be accepted as array resize shortcuts. + +The returned ops object is the target client's local operation object. The caller should not know or store the target lifecycle directly. + +### Event Broadcasts + +Events are broadcast from the client that observed or raised the event. The broadcast excludes that caller client and sends to all other clients. + +- Object events use generated strong typed caller-side ops, which call `IRpcDispatcher::BroadcastFromClient_ObjectEventOps(selfClientId)->InvokeEvent(...)`. +- List events use `IRpcDispatcher::BroadcastFromClient_ObjectEventOps(selfClientId)->InvokeEvent(...)` with the predefined observable-list event id through `RpcCallerListEventOps`. + +This rule applies whether the event is raised from a local object or from a wrapper. If a wrapper raises an event, the owner client is still just another client in the broadcast target set unless it is the caller client. + +Event broadcasts return `object` so the same ops interface can carry either direct reflected values or JSON serialized values. After deserialization, the value is `system::RpcException[int]` or `null`, keyed by the lifecycle client id that caught an event handler exception. A broadcast should attempt every target lifecycle even when earlier targets report exceptions, then aggregate all returned maps. The lifecycle that triggered the event receives the aggregate map internally; generated object-event send-side code and predefined container-event send-side code deserialize the returned value and call `system::IRpcLifecycle::ReadEventException`, which raises a normal Workflow exception message when the aggregate is non-empty. + +In the dual-client test implementation, broadcasting can be implemented by returning event ops from the lifecycle whose client id is not `selfClientId`. + +### Event Suppression + +Event suppression is local controller state. It is not dispatcher state. + +Suppression prevents an event replayed from a remote message from being forwarded again by the normal locally attached event handlers. + +`IRpcController` exposes these suppression APIs: + +- `SetEventSuppressedFlag(ref, eventId, bool)` +- `GetEventSuppressedFlag(ref, eventId)` +- `SetItemChangedSuppressedFlag(ref, bool)` +- `GetItemChangedSuppressedFlag(ref)` + +Although the setter takes a `bool`, implementations should store a counter for each suppression key. Setting `true` increments the counter. Setting `false` decrements it. The getter returns true when the counter is greater than zero. Decrementing below zero should be treated as an implementation error. + +Object event suppression is keyed by `(RpcObjectReference, eventId)`. List `ItemChanged` suppression is keyed by `RpcObjectReference`. + +When `IRpcObjectEventOps::InvokeEvent(ref, eventId, arguments)` receives a remote event, generated `rpc_IRpcObjectEventOps` should: + +- Call `lc.Controller.SetEventSuppressedFlag(ref, eventId, true)`. +- Unbox arguments and raise the local event. +- Catch exceptions into a returned `system::RpcException[int]` map keyed by `lc.ClientId`. +- Call `lc.Controller.SetEventSuppressedFlag(ref, eventId, false)` in a `finally` block. +- Treat unknown event ids as local dispatch errors that are raised directly and are not inserted into the returned exception map. + +Generated `rpclistener_*` handlers should check `lc.Controller.GetEventSuppressedFlag(ref, eventId)` before forwarding the event. If the flag is set, the handler returns immediately. Otherwise it calls the generated strong typed caller-side ops event method, letting those ops handle boxing, optional JSON serialization, broadcasting, deserialization, and exception raising. + +List events use the same shape with the predefined observable-list event id. When receive-side `OnItemChanged(ref, index, oldCount, newCount)` replays a remote list notification locally, it should set `SetItemChangedSuppressedFlag(ref, true)`, raise the local list notification, catch exceptions into a `system::RpcException[int]` map keyed by `lc.ClientId`, return that map as `object`, and clear the flag in a `finally` block. The locally attached native list event handler should check `GetItemChangedSuppressedFlag(ref)` before broadcasting through the object-event dispatcher path, deserialize the returned value when a serializer exists, and call `system::IRpcLifecycle::ReadEventException` on the resulting map. + +## Byval Return Collection Lifecycle + +When a method return value is marked with `@rpc:Byval`, the generated `IRpcObjectOps::InvokeMethod` result is not the boxed collection itself. It is a `system::RpcByvalReturnValue` object: + +- `value` stores the boxed returned collection, or the JSON serialized `JsonNode` when JSON object ops are used. +- `slot` stores an incremental index allocated by the callee-side object ops object. + +The generated object ops object owns `_slot : int` and `_byvalReturnValues : object[int]`. For each byval collection return, it first calls `system::IRpcLifecycle::RpcCopyByval` on the real returned collection. This recursively copies every nested collection layer, so later mutation of the original collection cannot affect the value being transported. The copied collection is stored in `_byvalReturnValues[slot]`, and then the copied collection is boxed or serialized for `RpcByvalReturnValue.value`. + +The callee caches the copied collection instead of the boxed or JSON value. This keeps interface elements alive for both non-JSON and JSON transport: boxed byval collections may contain only `RpcObjectReference` values, and JSON values contain no interface objects at all. Holding the recursive copy keeps the actual interface objects alive until the caller has received the result and created the needed wrappers. + +Generated caller-side ops know from metadata whether a return value uses this path. They cast the `InvokeMethod` result directly to `system::RpcByvalReturnValue^`, deserialize and unbox `value` into a local return variable, call `EndInvokeMethod(slot)` on the same object ops object, and then return the local variable. `EndInvokeMethod` removes the cached copied collection from `_byvalReturnValues`. + +Non-byval returns do not use `RpcByvalReturnValue`, `_slot`, `_byvalReturnValues`, or `EndInvokeMethod` for result cleanup. diff --git a/.github/KnowledgeBase/KB_Workflow_JsonRequestRouting.md b/.github/KnowledgeBase/KB_Workflow_JsonRequestRouting.md new file mode 100644 index 00000000..6e2c513a --- /dev/null +++ b/.github/KnowledgeBase/KB_Workflow_JsonRequestRouting.md @@ -0,0 +1,206 @@ +# Workflow JSON Request Routing + +This document describes the runtime meaning of the generic JSON dispatcher envelopes declared in `Release/Rpc.d.ts`. It is independent of any particular generated application or transport. Generated interface and serialization rules are covered by [Workflow Interface-Based RPC Definition](./KB_Workflow_InterfaceBasedRpcDefinition.md), [Workflow Generated RPC Wrappers](./KB_Workflow_Design_GeneratedRpcWrappers.md), and [Workflow JSON Serialization Schema](./KB_Workflow_Design_JsonSerializationSchema.md). + +## Setup + +The reusable JSON RPC setup lives in `Source/Library/RpcJson`. User code normally touches only these pieces: + +- Include `WfLibraryRpcJson.h` when building a custom transport adapter around `vl::rpc_controller::IRpcJsonMessageDispatcher`, `vl::rpc_controller::RpcJsonDispatcher`, and `vl::rpc_controller::RpcJsonLifecycle`. +- Include `WfLibraryRpcJsonDispatcherClient.h` and `WfLibraryRpcJsonDispatcherServer.h` when using the default channel-backed setup in `vl::rpc_controller::channeling`. +- Create one `RpcJsonDispatcher(clientId, messageDispatcher)` and one `RpcJsonLifecycle(clientId, dispatcher)` per RPC endpoint. The `clientId` must be the endpoint id assigned by the transport. +- Configure the lifecycle from generated RPC code before calling `Initialize()`: set the generated id map, pass the generated JSON serializer, JSON object ops, JSON object event ops, type-id callback, and event-attacher callback to `RpcJsonLifecycle::Register`, and register the generated wrapper factory. +- Register local services with `IRpcLifecycle::RegisterLocalService(typeId, service)` before lifecycle initialization. Remote services are discovered through service declaration messages and can be requested by type name after initialization. +- Use `RpcJsonDispatcherClientForTaskQueue` for endpoint-side channel IO when a single `TaskQueue` should process incoming RPC requests. A small generated-app wrapper should derive from it, call `SetRpcObjects`, and expose an app-specific `InitializeRpc(clientId)` function. +- Use `RpcJsonDispatcherServerForTaskQueue` for the transport coordinator. It is not a service owner; it tracks connected client ids, forwards broadcast requests, caches service declarations, and consolidates broadcast responses. +- Pass required remote service type names to `WaitForServer` or `ConnectLocalServer` so client initialization waits until those services have been declared. +- Call `FinalizeRpc()` on endpoint dispatchers before shutting down the transport or unloading generated Workflow context. + +An application-specific dispatcher wrapper normally derives from `RpcJsonDispatcherClientForTaskQueue`, creates the dispatcher/lifecycle pair after the transport assigns a client id, wires generated JSON serializer and ops into the lifecycle, registers the generated wrapper factory, and leaves transport behavior to `Source/Library/RpcJson`. + +### Setup from Workflow Generated Code + +Generated Workflow code is the only source of RPC-specific setup data. After the transport assigns `clientId`, the application-specific dispatcher wrapper creates `RpcJsonDispatcher(clientId, messageDispatcher)` and `RpcJsonLifecycle(clientId, rpcDispatcher.Obj())`, then stores them with `SetRpcObjects`. + +The lifecycle id map comes from generated Workflow code. Setup calls `rpc_GetIds()`, converts the result to `Dictionary` when crossing the C++ reflection boundary, and passes it to `RpcJsonLifecycle::SetIdMap`. Service-owning code resolves a full RPC interface name through `GetTypeIdFromName(fullName)` or reads the same id from `rpc_GetIds()[fullName]`, then calls `RegisterLocalService(typeId, service)` before lifecycle initialization. + +Generated JSON operations are created once per lifecycle: + +- `app.rpcops_IRpcSerializer()` creates the generated serializer object. +- `app.rpcops_IRpcObjectOpsJson(lifecycle.Obj())` creates the generated receive-side object ops. +- `app.rpcops_IRpcObjectEventOpsJson(lifecycle.Obj())` creates the generated receive-side object-event ops. +- `app.rpcops_IOps_CreateJson(lifecycle.Obj())` creates the generated caller-side ops that wrappers and listener attachers reuse. + +`RpcJsonLifecycle::Register` receives those generated ops plus two callbacks. The type-id callback calls generated `rpcwrapper_GetTypeId(BoxValue(obj))`, so local object references can be assigned generated RPC type ids. The event-attacher callback calls generated `rpclistener_Attach(ref.typeId, lifecycle, ref, obj, ops)`, so generated listeners attach to local object events and forward unsuppressed events through the generated caller-side ops. + +The wrapper factory is also generated-code based. `RegisterWrapperFactory` installs a callback that calls generated `rpcwrapper_Create(ref, lc, ops)`. `rpcwrapper_Create` returns a generated wrapper for a remote `RpcObjectReference`, and the same generated wrapper class is used for JSON transport because the JSON caller-side ops object is the `ops` argument passed into the wrapper. + +Endpoint startup then happens through the reusable dispatcher client. A network endpoint calls `WaitForServer(channelClient, rpcChannel, waitingForServices)`, or a local endpoint calls `ConnectLocalServer(channelServer, localClient, rpcChannel, waitingForServices)`. When the channel reports `OnConnected(clientId)`, the application-specific wrapper performs the generated setup described above and then calls `Initialize()`. Client code requests a service with `GetRpcLifecycle()->RequestService(fullName)`; when the service is remote, the lifecycle returns the generated wrapper created by the registered factory, and ordinary interface method calls on that wrapper go through the generated JSON caller-side ops. + +### Implementation + +`IRpcJsonMessageDispatcher` is the transport boundary. `AllocateRequestId()` provides request ids for JSON envelopes, and `OnJsonRequest(message, requestType)` sends a direct, broadcast, or broadcast-and-drop request through the transport. `IRpcJsonMessageDispatcher::DefaultTranslate` is the receiver-side helper that routes JSON envelopes to local object ops, object event ops, or lifecycle service declaration handling. + +`RpcJsonObjectOps` and `RpcJsonObjectEventOps` adapt generated JSON ops to the shared envelopes declared by `Release/Rpc.d.ts`. They build request objects on the caller side, validate matching responses, and translate received requests back to `IRpcObjectOps` or `IRpcObjectEventOps`. + +`RpcJsonDispatcher` is the `IRpcDispatcher` implementation for JSON transport. It creates per-target object ops, one broadcast object-event ops adapter, and sends local service declarations as broadcast-and-drop messages. + +`RpcJsonLifecycle` derives from `RpcLifecycleBase` and installs the generated JSON serializer, object ops, event ops, type-id callback, and event-attacher callback. It also wraps predefined byref collection operations through the reusable list/object ops adapters. + +`WfLibraryRpcJsonDispatcherClient` owns endpoint-side channel details that are not part of the generic RPC lifecycle: nested request processing while waiting for a response, response buffering by request id, pre-initialization service declaration caching, required-service waiting, and server-coordinator login/logout messages. + +`WfLibraryRpcJsonDispatcherServer` owns coordinator-side channel details: connected client tracking, broadcast request redirection, expected response tracking, response consolidation, service declaration replay to future clients, and client disconnect cleanup. The task-queue subclasses keep scheduling policy outside the core translation helpers. + +Every JSON RPC envelope has: + +- `rpcMethod`: a string beginning with `Request:` or `Response:`. +- `rpcRequestId`: the caller-allocated id for matching a response to a request. +- `sourceClientId`: the lifecycle client id that created the envelope. + +Direct requests and ordinary responses also have `targetClientId`. Broadcast requests omit `targetClientId` because the transport coordinator expands the request to multiple clients. + +## Request Kinds + +There are three request kinds at the JSON message dispatcher boundary. + +1. `Direct` + + The request is sent to exactly one `targetClientId`. The receiver translates the request to its local ops object and sends exactly one response with the same `rpcRequestId`. Direct requests are synchronous from the caller's point of view. + +2. `Broadcast` + + The request is sent to the transport coordinator, which sends it to all relevant clients except the originating client. Each receiver translates the request locally and sends a response to the coordinator. The coordinator waits for all expected responses, merges the returned data, and sends one response to the original caller with the original `rpcRequestId`. + +3. `BroadcastAndDrop` + + The request is sent to the transport coordinator, which sends it to all relevant clients except the originating client and stores enough state to replay the declaration to future clients. Receivers translate the request locally but do not send a response. The original caller receives no response; the local `JsonRequest` result is `null`. + +## Requests and Responses + +### `Request:IObjectOps_InvokeMethod` + +Kind: `Direct`. + +The caller sends the request to `targetClientId`, normally the same client id as `ref.clientId`. The receiver calls its local object ops `InvokeMethod(ref, methodId, arguments)` and returns `Response:IObjectOps_InvokeMethod` to the caller. + +The response carries: + +- the same `rpcRequestId`, +- `sourceClientId` equal to the receiver, +- `targetClientId` equal to the original caller, +- `response` containing the serialized method result, serialized `system_RpcException`, or `system_RpcByvalReturnValue` for byval collection returns. + +Unknown method ids and malformed references are local dispatch errors. User-code exceptions are transported as `system_RpcException` according to the generated ops rules. + +### `Request:IObjectOps_EndInvokeMethod` + +Kind: `Direct`. + +The caller sends this to the same client that returned a byval collection slot. The receiver calls local object ops `EndInvokeMethod(slot)` and returns `Response:IObjectOps_EndInvokeMethod` with the same `rpcRequestId`. There is no `response` field. + +### `Request:IObjectOps_ObjectHold` + +Kind: `Direct`. + +The caller sends this to the owner of `ref.clientId` when creating or releasing a wrapper interest. The receiver calls local object ops `ObjectHold(ref, remoteClientId, hold)` and returns `Response:IObjectOps_ObjectHold` with the same `rpcRequestId`. There is no `response` field. + +`remoteClientId` is the lifecycle whose interest changes. The receiver should validate that `ref` belongs to the receiving lifecycle before mutating local-object hold state. + +### `Request:IObjectEventOps_InvokeEvent` + +Kind: `Broadcast`. + +The originating lifecycle has already observed or raised the event locally, so the coordinator must not send the broadcast back to that originating client. Each receiving lifecycle calls local object event ops `InvokeEvent(ref, eventId, arguments)`, which replays the event under event-suppression rules and returns the serialized form of `null | [number, system_RpcException][]`. + +Each receiver sends `Response:Broadcast_Response` to the coordinator with: + +- the redirected broadcast `rpcRequestId` chosen by the coordinator, +- `sourceClientId` equal to that receiver, +- `targetClientId` equal to the coordinator, +- `response` equal to the event exception map or `null`. + +The coordinator sends one `Response:Broadcast_Response` to the original caller with: + +- the original caller's `rpcRequestId`, +- `sourceClientId` equal to the coordinator client id, +- `targetClientId` equal to the original caller, +- `response` equal to `null` if every receiver returned `null`, otherwise a merged event exception map. + +The coordinator must attempt every receiver even when earlier receivers report exceptions. When clients disconnect during an active broadcast, the coordinator removes them from the expected response set and completes the broadcast if all remaining receivers have responded. + +### `Request:IRpcDispatcher_DeclareRemoteService` + +Kind: `BroadcastAndDrop`. + +This request declares that `sourceClientId` owns a service reference. It carries `ref: system_RpcObjectReference`, and `ref.clientId` must equal `sourceClientId`. The receiver calls `IRpcLifecycle::DeclareRemoteService(ref)` and stores the full reference by `ref.typeId`. + +No `Response:*` envelope is created for this request. A caller-side dispatcher returns `null` immediately after sending or caching the message. A receiver-side translator also returns `null`. + +The coordinator caches every service declaration and replays cached declarations to newly connected clients after the new client has learned the coordinator client id. The replayed request keeps the original `sourceClientId` and the original `ref`; it is not rewritten to the coordinator client id. + +## Lifecycle Handling + +`IRpcLifecycle::RegisterLocalService(typeId, service)` is a pre-initialization operation. It creates a full `RpcObjectReference` for the service object, stores the local service by type id, adds the owner hold, and calls `IRpcDispatcher::DeclareLocalService(ref)`. + +For JSON transport, `IRpcDispatcher::DeclareLocalService(ref)` creates `Request:IRpcDispatcher_DeclareRemoteService` and sends it as `BroadcastAndDrop`. A lifecycle receiving this request stores `ref` through `IRpcLifecycle::DeclareRemoteService(ref)`. + +`IRpcLifecycle::GetTypeIdFromName(typeName)` resolves names through the lifecycle id map and returns `RpcTypeId_NotFound` when the name is unknown. `IRpcLifecycle::RequestService(typeName)` uses this function, returns a local registered service first, otherwise looks up the stored remote `RpcObjectReference` by type id and calls `RefToPtr(ref)`. + +Before a lifecycle is initialized, a client may receive service declaration requests. It should cache `Request:IRpcDispatcher_DeclareRemoteService` and reject all other RPC messages. During initialization it processes cached declarations before requesting required remote services. After initialization, later service declarations are processed immediately. + +If a client is waiting for required service type names, it should compare each incoming declaration's `ref.typeId` with `GetTypeIdFromName(typeName)`. When all required services have been declared, the wait completes. + +## Expected Sequences + +### Transport Coordinator Startup + +1. Start the transport layer enough for local endpoints to connect. +2. Connect the coordinator endpoint first and record its client id. +3. Report the coordinator client id to clients as an out-of-band transport login message before replaying RPC declarations. +4. Connect any local service-owning clients. +5. Let local service-owning clients register services and send `Request:IRpcDispatcher_DeclareRemoteService`. +6. Start accepting remote clients only after the coordinator client id is known and local service declarations have been sent. + +The coordinator endpoint is not a service owner. It routes broadcasts, caches service declarations, and consolidates broadcast responses. + +### New Client Startup + +1. Connect to the transport and learn the client's own client id. +2. Register the JSON channel reader before waiting for the coordinator login message. +3. Learn the coordinator client id. +4. Cache any `Request:IRpcDispatcher_DeclareRemoteService` messages that arrive before lifecycle initialization. +5. Initialize the lifecycle, process cached declarations, send local declarations if this client owns services, and wait for required remote service names if needed. +6. Begin ordinary direct method calls and event broadcasts. + +### Direct Method Call + +1. Caller allocates `rpcRequestId` and sends a direct request to `targetClientId`. +2. Receiver translates and executes the local operation. +3. Receiver sends the matching direct response. +4. Caller matches by `rpcRequestId`; while waiting, it may process nested incoming requests and buffer unrelated responses. + +### Event Broadcast + +1. Originating lifecycle raises or observes the event locally. +2. Originating caller sends `Request:IObjectEventOps_InvokeEvent` to the coordinator. +3. Coordinator chooses a redirected request id, sends the request to every expected receiver except the originator, and records the original `(sourceClientId, rpcRequestId)`. +4. Receivers replay the event under suppression and respond to the coordinator. +5. Coordinator merges responses and sends one `Response:Broadcast_Response` to the originator using the original request id. + +### Service Declaration + +1. Service owner registers a local service before lifecycle initialization. +2. Its dispatcher sends or caches `Request:IRpcDispatcher_DeclareRemoteService`. +3. Coordinator caches the declaration and broadcasts it without waiting for responses. +4. Receivers store `ref.typeId -> ref`. +5. Future clients receive the cached declaration during startup. + +## Error Handling Rules + +Unknown `rpcMethod` values are transport or implementation errors. Do not silently drop them. + +A request kind mismatch is an error: object ops are direct, object events are broadcast, and service declarations are broadcast-and-drop. + +Only `Request:IRpcDispatcher_DeclareRemoteService` may be accepted before lifecycle initialization. Other RPC messages before initialization indicate a startup-order violation. + +Broadcast-and-drop requests must not produce responses. A client waiting for a response to this request will deadlock a correct implementation. diff --git a/.github/KnowledgeBase/Learning.md b/.github/KnowledgeBase/Learning.md index 08a72d4f..43bd0d1a 100644 --- a/.github/KnowledgeBase/Learning.md +++ b/.github/KnowledgeBase/Learning.md @@ -19,7 +19,7 @@ - Prefer simple calls before interface casts [2] - Validate expectations against implementation and existing tests [2] - Treat Debug memory leak dumps as required failures [2] -- Keep design documentation aligned with code after refactoring [2] +- Keep design documentation aligned with code after refactoring [3] - Prefer well-defined tests over ambiguous edge cases [1] - Prefer `operator<=> = default` for lexicographic key structs [1] - Prefer two-pointer merge for sorted range maps [1] diff --git a/.github/prompts/kb.prompt.md b/.github/prompts/kb.prompt.md index 1fba49a1..3424dab0 100644 --- a/.github/prompts/kb.prompt.md +++ b/.github/prompts/kb.prompt.md @@ -137,6 +137,8 @@ - The document is for understanding the source code, so you must keep mentioning names instead of using language that is too abstract. - You must use everything before `# DRAFT` with details. Do not just make a summary; that material is already a summary. - Multiple levels of `#` markdown topics containing bullet points are favored. + - Try your best to mention actual file names as other repos might not see the same source structure, mention classes/functions/etc instead. + - Except for naming convention of generated files, or metadata files offered in the release. ## Steps for Improve