Experiment with Second Dataset

The second dataset contains 4608 C programs consisting of a main function and another functionf randomly generated by theRandomFunsfeature of Tigress. To generate the programs in this dataset we have also leveraged the program features identified when mapping symbolic execution to our search model, in Section 5.1.2. A set of parameters of theRandomFunsfeature of Tigress, were added – by the author of Tigress – in order to cater to the code features we have identified in Section 5.1.2. The following is a list of parameters and their corresponding values we used to generate this dataset:

• The random seed value:Seed∈ {1,2,4}(3 values). This parameter has no effect on symbolic execution time. However, we use it in order to generate more than one program instance for the same values of all other parameters.

• The data type of variables:RandomFunsTypes∈ {char, short, int, long}(4 values).

This parameter affects the complexity of path constraints because larger data types for symbolic values leads to SMT-/SAT-instance with a larger number of literals.

• The bounds offor-loops:RandomFunsForBound∈ {constant, input, boundedInput} (3 values). This parameter affects the depth of the search tree corresponding to symbolic execution, because in includes variable upper bounds for loops.

• The operators allowed in expressions: RandomFunsOperatorspresented in Ta-ble 5.5 (4 values), which also describes each parameter value. This parameter affects the complexity of path constraints, because different operator types impose various degrees of difficulty for SMT-/SAT-solvers.

• The control structures: RandomFunsControlStructurespresented in Table 5.6 (16 values), which also shows the depth of the control flow. The grammar for this

Code Metric Min Med Avg Max Total LOC 32.00 66.00 78.00 288.00 Average CC 2.67 3.33 3.21 4.00

(a) Before obfuscation

Code Metric Min Med Avg Max

Calculations 22.00 98.00 183.36 870.00 Total LOC 42.00 168.00 578.64 2932.00 Average CC 1.80 5.25 15.73 66.75

(b) After obfuscation Table 5.7.: Overview of randomly generated programs.

parameter allows specifying a nested control-flow structure consisting of if and for statements. Opening a parenthesis increases the nesting level of control flow. The grammar also allows specifying the size of the basic blocks (denoted bb), via an integer numbern. This parameter influences the number of paths, hence queries sent to the SMT-/SAT-solver.

• The number of statements per basic block was changed via the value ofn∈ {1,2}

from Table 5.6. This parameter affects the length of the search tree for symbolic execution as well as the length of path constraints.

The total number of combinations is therefore: 3×4×3×4×16×2 = 4608. All other parameters were kept to their default values, except for the RandomFunsPointTest, which was set to true, meaning that the return value of the randomly generated function is checked against a constant value and if they are equal the program prints a distinctive message, i.e. “You win!” to standard output. We have set this constant value to be equal to the output of the randomly generated function when its input is equal to “12345”.

Therefore, all of the 4608 programs will print “You win!” on the standard output if their input argument is “12345”.

Table 5.7a shows the minimum, median, average and maximum values of various code metrics of only the original (un-obfuscated) set of programs, as computed by the UCC tool [163] and the total number of LOC. Each metric was computed on the entire C file of each program, which includes the randomly generated function and the main function.

Each generated function takes an array of primitive type (e.g. char, int) as input (i.e.in) and outputs another array of primitive type (i.e. out), as shown in Listing 5.3. Each function first expands the input array into a (typically larger) state array via a sequence of assignment statements containing operations (e.g. arithmetic, bitwise, etc.) involving the inputs (lines 3-5). After input expansion, the values in the state array are processed via control flow statements containing various operations on the state variables (lines 6-17).

Finally, the state array is compressed into the (typically smaller) output array via assignment statements (lines 18-19). These three phases represent a generic way to map data from an input domain to an output domain, as a license check would do. Theif-statement on lines

Listing 5.3: Randomly generated program example.

21 void main(int ac, char* av[]) { 22 int out;

23 f(av[1], &out);

24 if (out == 0xa199abd8) 25 printf("You win!");

26 }

24-25 resembles a license check, where the output of the randomly generated functionfis compared against a hard-coded value. Note that the program illustrated in Listing 5.3 is an overly simplified instance of the programs generated by the Tigress. The programs in our dataset are larger and they also contain more complex control-flow structures and boolean conditions (including disjunctions and conjunctions) in control-flow statements. Finding an input value that passes this comparison is harder for a white-box test case generator to find, than an input that would fail the comparison. Hence, this dataset resembles license checking mechanisms, which would be part of larger programs such as games or professional editing and design software, etc. Note that programs containing license checking mechanisms are much larger than our randomly generated programs, however, an attacker would not symbolically execute the entire program. Instead, an attacker would isolate the license checking code and then proceed to symbolically execute only this fraction of the program.

Testbed description: For the experiment described in this subsection we used a machine with more cores to enable running multiple symbolic executions in parallel. The machine uses the Ubuntu 14.04 64-bit operating system and it has an Intel Xeon E5-1650v2 CPU with 12 logical cores each running at 3.50GHz and 64GB of physical memory.

Step 1 - Baseline symbolic execution time before obfuscation: The impact of these code characteristics on the execution time of KLEE given the original (unobfuscated) programs

Data Types char short int long Mean 1.32 9.95 13.41 13.91 StdDev 0.98 6.48 7.86 8.34

Loop Bound Constant Bounded Input Input

Mean 8.45 8.43 10.62

StdDev 8.07 7.28 9.65

Operators Bitwise Simple Arith. Harder Arith. All

Mean 4.74 8.91 9.97 11.23

Mean 6.34 7.74 7.94 11.08 12.4 14.87 StdDev 4.92 5.90 6.51 9.22 9.35 11.97

Total # Loops 0 1 2

Mean 12.27 6.34 6.4

StdDev 9.64 4.57 5.11

Table 5.8.: KLEE execution time (seconds) on original programs w.r.t. code characteristics of 2nd dataset.

can be seen in Table 5.8. As was the case with the input size in Table 5.4, the symbolic execution time increases with the size (ranges) of the data types. This is due to the fact that for higher ranges of values it is more difficult for SAT solvers to find a solution to queries derived from path constraints on symbolic values. Different types of bound conditions placed on loop statements cause a mild difference on symbolic execution time, i.e. if the loop iterates a constant number of times, then the symbolic execution engine will execute faster on average than if the number of loop iterations depends on the program input.

Finally, the type of operators used by the program has an important impact on symbolic execution, because these operators are used by path constraints which are issued as queries to the SMT solver. We notice that bitwise operators are easier to solve than arithmetic operators. The type of arithmetic operators does not cause a large difference in symbolic execution. However, harder arithmetic tends to be slower to solve than simple arithmetic.

More importantly, combining all operators seems to have an additive effect w.r.t. the time taken to solve path constraints.

Step 2 - Compare symbolic execution overhead of different tools after obfuscation: We have obfuscated thef functions with 5 obfuscation transformations from the Tigress tool:

AddO, EncA, EncL, Flat and Virt. We only chose these 5 transformations, due to the fact that Obfuscator LLVM transformations are very similar to AddO, EncL and Flat. Table 5.7b shows the minimum, median, average and maximum values of various code metrics of only the obfuscated set of programs, as computed by the UCC tool [163], and the total number of LOC. Note that the majority of programs have now increased their LOC by one order of magnitude.

For this experiment we used KLEE andangras symbolic execution engines and we let them run until they found the path in the program that prints a distinctive message on the standard output or the timeout of 1 hour is reached. When this path is entered we know that the check guarding that path has been bypassed by the symbolic execution engine. We did not use angr in the previous experiment because angr does not aim to achieve 100%

code coverage, as opposed to KLEE. Note that we have also tried to employ the Triton symbolic execution engine [181] on the obfuscated programs for both datasets. However,

KLEE angr Median Mean StdDev Median Mean StdDev AddO16 0.97 1.03 0.26 1.72 2.25 2.49 EncA 1.14 1.21 0.37 1.39 1.79 1.90 EncL 0.98 0.99 0.22 1.40 2.22 4.60 Flat 1.15 1.22 0.44 3.77 4.45 2.85 Virt 1.53 2.08 1.27 7.32 8.85 5.01

Table 5.9.: Symbolic execution slowdown on programs obfuscated using Tigress, relative to unobfuscated counterparts from 2nd dataset.

Triton crashed when symbolically executing programs obfuscated using Flat and Virt due to insufficient memory. Triton transforms each assembly instruction into a sequence of SMT constraints, which increases directly proportional to the execution trace, which is large for programs obfuscated with Flat and Virt.

Table 5.9 shows the median, mean and standard deviation of symbolic execution slow-down on programs obfuscated from the second dataset w.r.t. their unobfuscated counter-parts. The slowdown is computed as the time needed to symbolically execute an obfuscated program until the path in the program that prints the distinctive message on the standard output is found, divided by the time need to symbolically execute the unobfuscated version of the program to find the corresponding path. The median and standard deviation were taken across 12713 obfuscated programs successfully analyzed by the KLEE and angr within the 1 hour time limit. We make the following observations using Table 5.9.

Observation 11: KLEE incurs a lower slowdown than angr for all of the 5 obfuscation transformations employed in this experiment. This indicates that KLEE is thebest known attackerfor the obfuscation transformations we have employed in this chapter. Note that KLEE also has limitations, e.g. it does not supportgotoinstructions or in-line assembly in C programs. However we see this as a technical, not a fundamental limitation.

Observation 12:The slowdown of finding the path that prints a distinctive message (“win”) is much lower than the slowdown for covering all reachable code (which was the goal of the attacker in Section 5.2.2). This is expected since the symbolic execution engine may discover that particular path before covering all reachable code.