GP-0: Fix VarargsFilter doing nothing due to firstVarArgSlot never being set

When the Function Editor assigns parameter storage, PrototypePieces.firstVarArgSlot
was always left at -1, causing VarargsFilter.filter() to unconditionally return false.
Any .cspec rule using <varargs/> (e.g. MSP430's goto_stack for the last fixed arg) had
no effect when storage was (re-)assigned via the UI.

Fix: add a getStorageLocations overload that accepts an isVarArgs flag. When true,
firstVarArgSlot is set to proto.intypes.size() — meaning all supplied parameters are
non-optional and the first vararg slot follows immediately after the last one.
FunctionData.updateParameterAndReturnStorage() now passes hasVarArgs to this overload,
so VarargsFilter-based rules fire correctly.

Fixes #9091

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
vigneshakaviki
2026-04-04 18:43:08 -07:00
committed by James
parent 49155d2384
commit 5259f4b7ae
2 changed files with 29 additions and 1 deletions
@@ -218,7 +218,7 @@ class FunctionData extends FunctionDataView {
}
VariableStorage[] paramStorage =
effectiveCallingConvention.getStorageLocations(getProgram(), dataTypes, true);
effectiveCallingConvention.getStorageLocations(getProgram(), dataTypes, true, hasVarArgs);
returnInfo.setStorage(paramStorage[0]);
@@ -398,6 +398,30 @@ public class PrototypeModel {
*/
public VariableStorage[] getStorageLocations(Program program, DataType[] dataTypes,
boolean addAutoParams) {
return getStorageLocations(program, dataTypes, addAutoParams, false);
}
/**
* Calculate input and output storage locations given a function prototype,
* with optional variable-argument support.
* <p>
* This overload additionally sets {@link PrototypePieces#firstVarArgSlot} when
* {@code isVarArgs} is true, enabling {@code VarargsFilter}-based calling-convention
* rules (e.g. {@code <varargs/>} in a .cspec) to be applied correctly.
*
* @param program is the Program
* @param dataTypes return/parameter datatypes (first element is always the return datatype,
* i.e., minimum array length is 1)
* @param addAutoParams true if auto-parameter storage locations can be generated
* @param isVarArgs true if the function takes variable arguments; when true, all
* data-types supplied in {@code dataTypes} are treated as non-optional and the first
* vararg slot is set to immediately follow the last supplied parameter
* @return dynamic storage locations ordered by ordinal where first element corresponds to
* return storage. The returned array may also include additional auto-parameter storage
* locations.
*/
public VariableStorage[] getStorageLocations(Program program, DataType[] dataTypes,
boolean addAutoParams, boolean isVarArgs) {
DataType injectedThis = null;
if (addAutoParams && hasThis) {
@@ -406,6 +430,10 @@ public class PrototypeModel {
injectedThis = new PointerDataType(program.getDataTypeManager());
}
PrototypePieces proto = new PrototypePieces(this, dataTypes, injectedThis);
if (isVarArgs) {
// All supplied parameters are non-optional; varargs begin immediately after them.
proto.firstVarArgSlot = proto.intypes.size();
}
ArrayList<ParameterPieces> res = new ArrayList<>();
assignParameterStorage(proto, program.getDataTypeManager(), res, addAutoParams);