Pandora VM

Pandora VM is a software protection and code virtualization project inspired by tools such as VMProtect. It is designed to help protect applications from unauthorized analysis and modification.

From source to protected execution

Annotated functions are lowered through a compiler-owned intermediate representation before they become authenticated VM images.

Pandora compilation pipeline from annotated C or C++ through LLVM IR, Pandora hardening passes, and Pandora IR to an authenticated VM image and runtime.
The compiler rejects unsupported operations instead of silently leaving protected code native.

Three-level IR architecture

Pandora separates LLVM lowering, VM-neutral optimization, and machine-specific finalization into distinct IR layers. The compiler pipeline is intentionally closed: unsupported input fails during lowering or verification instead of falling back to an unprotected native function.

Layer Role Backend contract
HighIR LLVM-independent semantic IR that still preserves SSA, CFG, abstract memory, imports, and type-sensitive operations. Not consumed directly by VM backends; it is a compiler-side normalization boundary.
MidIR Verified, backend-neutral optimization IR used after semantic lowering and Pandora IR passes. General-purpose VM backends can finalize this layer into their own instruction format and image container.
LowIR Machine-level IR for lower-level execution models; it removes virtual-register opcodes and models explicit machine state. Specialized backends can target this layer when they need a lower-level machine model.

This makes the VM backend replaceable. A customer can add a private backend by registering a backend descriptor, choosing the IR level it consumes, implementing finalization into its image format, and providing the runtime bridge entry used by the protected wrapper. Existing backends include stack, window-register, turing, and dataflow models.

LLVM IR hardening

Pandora can also harden native code that stays outside a VM. The combined pandora-ir-harden pipeline splits basic blocks with opaque predicates, seals direct calls, builds an encrypted continuation dispatcher, and lowers remaining branches through memory-backed predicates and indirectbr target islands.

A simple conditional LLVM control-flow graph transformed by opaque splitting, call sealing, encrypted control-flow flattening, and memory opaque predicates into a dispatcher with runtime guards and indirect target islands.
The transformed graph preserves behavior while making static control-flow recovery more expensive.

Additional standalone passes cover indirect calls and branches, constant encoding, return scattering, self-modifying native code, and light, strong, or ultra anti-debug instrumentation.

Passes and transformations

Indirect calls and branches

Before

A direct call followed by a conditional branch.

After

The call and branch transformed into sealed indirect control flow with validation and target islands.
Targets are sealed with invocation-local state and opened only at the transfer point.

Constant encoding

Before

A literal used directly by an operation.

After

The literal split into masked volatile shares that are recombined at the operation.
Integer literals and strings are reconstructed from scattered, context-bound material at runtime.

Return scattering

Before

A provider returning a plain value to its consumer.

After

The provider returning two context-bound shares through volatile output slots.
Eligible internal providers return two context-bound shares instead of a plain 64-bit value.

Self-modifying jumps

Before

A branch source transferring directly through an indirect branch.

After

A self-modifying block overwriting its inline slot with the selected address before the indirect branch.
The selected destination is written into an inline native jump template before transfer.

Anti-debug instrumentation

Before

An uninstrumented function from entry to return.

After

Timing and platform probes feeding a persistent tamper epoch before the function returns.
Detection persists as a tamper epoch mixed into later keys, control flow, and selected return values.

Real VM nesting

A backend chain creates actual nested images. Each layer keeps a separate instruction format, private state, validation policy, and key boundary.

A host protected call traversing an arbitrary ordered chain of independently authenticated Pandora backend layers through the reserved child-image ABI.
Backend chains may use any order and may repeat a backend.

Three execution models

The backends do not share an instruction format or execution state. Each one uses a different machine model so a single analysis strategy does not transfer directly across every protected image.

Side-by-side comparison of Pandora StackMachine, WindowRegister, and TuringMachine execution models and their main implemented capabilities.
Stack, window-register, and turing can each be terminal or participate in a backend chain.

Demo binary

HMAC demo binaries are available for Linux and Windows. The Linux build is the primary demo target: hmac_demo-linux.

A Windows build is also available, but Windows support is still experimental and may be less complete than the Linux build: hmac_demo.exe.