Decompiler Construction: Chapter 6 - IR Semantics Edge Cases, Undefined Behavior, and Special Handling
Edge-Cases
Sometimes your IL will contain abstract functions that do not map 1:1 to the IR. These should be handled by flagging them while remaining as descriptive as possible.
The best approach, without bloating IR nodes with extra fields, is to encode the behavior through function arguments.
Ask the following questions about each function:
Does it modify any registers?
If yes, pass those registers as mutable references. They should be excluded during code generation.Does it perform a jump?
If yes, the target label(s) must be represented either as an internal node link (preferred) or as a synthetic argument.
VERY IMPORTANT: All jump targets must be explicitly representable as references, regardless of how abstract the control flow is. The same rule applies to registers: any register read or written must be tracked and represented.
Rules and Handling
Because the IR is abstract, you may encounter behaviors that are not well-defined in isolation. These must be handled according to strict rules to preserve correctness across transformation passes.
Expressions
You may encounter invalid or nonsensical expressions such as:
1
1 + “hello”
Arithmetic or bitwise operations on incompatible types are type violations.
- These must not crash the pipeline.
- They must be recorded as warnings during code generation.
- They must be preserved in the IR until a later validation or type-resolution pass.
- Downstream passes are responsible for enforcing or rejecting type correctness.
Important: Type errors are not execution errors. They are semantic inconsistencies that must remain observable until final validation
Labels
If any statement or expression references a label that does not exist, this is a critical failure.
- All labels must resolve to a valid IR node or block.
- Label resolution must be validated after every transformation pass.
- Unresolved labels invalidate the IR state immediately.
This is a hard correctness invariant: control flow integrity cannot be invalid.
Flag / Environment State
Some transformations depend on external or contextual assumptions (e.g., arithmetic mode, bitwise semantics, CPU flags, or optimization constraints).
To handle this, each transformation pass must accept an explicit environment state, which defines:
- arithmetic mode (signed / unsigned / modular width)
- bitwise behavior rules
- optimization constraints
- architecture-specific assumptions
This environment must be treated as immutable during a pass and explicitly passed forward to subsequent passes.
General Rule
Not all inconsistencies are equal:
- Type violations -> deferred, warning-level, preserved
- Control-flow violations (labels/jumps) -> critical failure
- Missing or invalid environment assumptions -> invalid transformation state
The IR should never silently discard ambiguity unless explicitly defined by a pass rule.
Next Chapter: Chapter 7 - Reconstructing Control Flow using Graph Theory, SSA, and Types