Decompiler Construction: Chapter 14 Modeling Inlined Functions as Virtual Function

As decompilation progresses into structural recovery, one of the most notoriously difficult hurdles is reversing the compiler’s function inlining pass.

The Inlining Problem

When a compiler inlines a function, it destroys the explicit call boundaries and merges the callee’s intermediate representation directly into the caller. This optimization creates significant challenges for structural recovery.

• Local variables are coalesced into the caller’s stack frame or register allocation.
• Control flow is interwoven with the caller’s basic blocks.
• Parameter passing mechanics and standard ABI conventions are optimized away entirely.
• Stack frames are merged, obscuring where one logical scope ends and another begins.

Decompiled output for heavily inlined code becomes a monolithic, unreadable function that obfuscates the original developer’s intent.

Virtual Function Abstraction

To uninline code, we introduce an abstract modeling technique: treating potential inlined regions as virtual functions during the IR passes.

Instead of immediately attempting to extract an inlined function, we take a table of known heavily optimized IR signatures and match it to the logic within the output IR. This allows us to model complex and known functions as simple calls which implicitly helps other passes perform more aggressive optimizations.

The core idea is to take large code blocks and put it in a call that is univeribly known.

Parameter and Return Recovery using SSA

Once a candidate region is isolated and wrapped as a virtual function, we can utilize SSA form to construct a call from it.

Deducing Inputs

Any SSA variable that is used within the virtual region, but whose definition exists strictly outside and prior to the region, is treated as a virtual parameter.

Deducing Outputs

Any SSA variable defined within the virtual boundary that remains live and is utilized subsequently by the caller is treated as a virtual return value. If any variables are referenced in later code, they can be abstractly referenced.

Example Virtual Boundary

Given the following pseudo-IR:

// Caller environment setup
R1 = -10;

//VIRTUAL INLINE BOUNDARY START
R1 = R1 < 0 ? -R1 : R1;
// VIRTUAL INLINE BOUNDARY END

// Caller uses output
printf("%d\n", R1);

In this scenario, R1 crosses the boundary and is used as both input and output.

We can deduce an inputted hueristic:

ABS(x) = X < 0 ? -X : X;

Which matches are example. The given output will be:

R1 = -10;
R1 = ABS(R1);
printf("%d\n", R1);

Later simplification passes can now make it more readable:

printf("%d\n", 10);

Limitations

Because anything can be represented in any different way this strategy is almost impossible to detect every single logic accurately. But we can detect most regions most of the time meaning that most IR will be simplified making it easier to follow and analyze.

---

Next Chapter: Chapter 15 - Deobfuscation Mixed Boolean Arithmetic and Opaque Predicate Removal

Prev Chapter: Chapter 13 - Safe Page-Level Optimization