16. Modeling, Computational Complexity

16. Komplexität von Algorithmen und ihre Bestimmung

Ziele dieser Einheit:

- Vor der Implementierung (Programmierung): Analyse des Problems und des Lösungsalgorithmus.

ƒ Elementarbegriffe aus Algorithmenlehre und Formaler Sprachen.

ƒ O-Kalkül.

- Einige Beispiele zur Laufzeitanalyse.

- Zur „Eingewöhnung“: Diese Lehreinheit ist auf Englisch.

- References

ƒ R. Sedgewick, “Algorithmen”, Addison-Wesley, Bonn (1992)

ƒ Th. H. Cormen, et al., "Introduction to Algorithms", The MIT Press, Cambridge (2001)

- The complexity of an algorithm, by means a model, is a measure that estimates how much time or memory (or, space) is required to execute it.

- One can talk about the complexity of a computational problem, expressing the complexity of the best available algorithm for solving it.

- In certain cases, complexity measures can be used to evaluate the hardware that is needed to carry out that task.

- In other cases, they are used to show that a certain algorithm is impractical, or even that a certain problem has no chance of being solved efficiently.

16. Modeling, Computational Complexity

and Computability

H

a b a λ λ

16.1 Turing Machines

- The tape has a fixed left end, and is infinite on the right.

- Cells: The tape is divided into cells that are labeled by a tape symbol from a given alphabet Σ that includes a special blank symbol λ.

- The head can read and (over)write (change) the symbol in the cell that is in the current tape location.

- The head can move one place to the left or to the right of the current cell.

- A Turing machine has a linear tape with cells the content of which can be manipulated by a head.

- Control function of a Turing machine determines for each pair (state, current input symbol)

ƒ the next state,

ƒ a replacement symbol for the current tape location,

ƒ a direction for the move of the head, left or right.

- Thus, at each step, depending on the current machine state and the symbol in the current tape location, the machine

ƒ can change the symbol in the current location,

ƒ move the head,

ƒ and change the current state according to its control table.

16.2 Structure of Turing Machines

- The initial input

ƒ is available at the leftmost tape positions,

ƒ does not include any blank symbols λ, and

ƒ is followed by an infinite contiguous sequence of blank symbols to the right.

- The input size is the number of cells the input occupies.

- Turing machine with multiple tapes

ƒ the input resides on one tape, and cannot be overwritten,

ƒ the other tapes are accounted as the auxiliary memory used.

- Many other variants of Turing machines exist.

- A Turing machine can act as an acceptor

ƒ which accepts a word if the computation terminates in an accepting state and

ƒ rejects a word if the computation either does not terminate or terminates in a rejecting state.

- A Turing machine as a transducer transforms its input to a corresponding output that resides at the end of the

computation on the tape.

- Acceptor -> Recognition; Transducer -> Generation.

- The Turing machine is the utmost abstract model of 16.3 Acceptors and Transducers

- To sum up: Turing machine model is a good mathematical tool because

ƒ it is used as a yardstick to compare different algorithms and computation problems,

ƒ the overall computation that is performed by real computers is the same.

- The computational model of a Turing machine has following advantages:

ƒ It is rather simple and abstract.

ƒ It eliminates the need to consider many details of actual computers.

- Thus, we do not have to analyze a computational problem or an algorithm again when moving from one actual

computer to another.

- This model also allows viewing a computational problem as a language of accepted/transduced input words.

- It is related to other models of computation that are

commonly used in computer science, such as finite state automata.

- A Turing machine can be considered as a finite (!)

automaton (referring to the finite control of the machine) with infinite memory (referring to the infinite tape).

- The efficiency of an algorithm can be measured with respect to

ƒ Time complexity: The number of steps (head moves) executed.

ƒ Space (i.e., memory) complexity: The number of tape symbols (cells) used.

ƒ Space can then be determined as a fraction of the input size when multiple-tape machines are used.

- Complexity measures are given as a function of the size of the input n (key element).

- We denote the worst-case complexity using the ”big-oh” notation O(f(n)) , where f is a function. Examples:

ƒ O(n²): for square (quadratic) complexity

ƒ O(2ⁿ): for exponential complexity where n refers to the input size.

16.4 Time and Space Complexity, “Big-Oh-Notation”

The “Big Oh” notation O(f(n)) is interpreted as follows:

- An algorithm or a computational problem is of complexity O(f(n)) if

ƒ there exist two constants c₀ and c₁, such as for each instance of the input of size n>c₀,

ƒ the time/space needed to execute the algorithm

(solving the problem) on a Turing machine requires no more than c₁*f (n) steps/cells.

- Remark 1: “Big Oh” represents the greek letter Ω (o-mega contrary to o-micron o/ω^).

- Remark 2: “Big Oh” is an order relation and thus an adjective.

- Remark 3: “Big Oh” is not negative.

- One usually does not care if there is a finite number of instances of the problem, smaller than some constant c₀, where the com- plexity does not behave according to f(n).

- The complexity measure is given only up to some constant factor c₁, because we usually do not want to consider the speed of the machine, or its word size. The reasons are

ƒ If we buy later a faster computer, or a computer where each memory element is twice as large as the current one, the complexity will still remain the same.

ƒ The constant c₀ could be eliminated by fixing a big enough constant c₁. However, c₁ could then be unnecessary large.

- There can be instances of size n (even n > c₀), where the

calculation will be faster or require less memory than c_n*f (n). It is only the worst cases instances of size n which count.

The reasons for this interpretation are as follows:

- Thus, we agree to commit only to the worst case in the complexity measure.

- This is because users usually do not complain about

instances of the input where the algorithm runs faster or consumes less memory than anticipated!

- Note that if an algorithm is of complexity O(f(n)) and for each n ≥ 0, (f(n) ≤ g(n) then it is also of complexity measure

O(g(n)).

- The algorithm that is most efficient with respect to the time complexity is not necessarily also the algorithm that is most efficient to space complexity.

- It is sometimes interesting also to provide the average complexity of algorithms that is based on a probability

- Because of the lack of convincing experimental results, the average complexity is often less meaningful than the worst case complexity.

- Let P(n) denote some polynomial function of n of the form P(n) = a_kn^k + a_k-1n^k-1+ a_k-2 n^k-2 + …+ a₀,

with a₀, …, a_{k as} integer constants.

- The degree of a polynomial is the highest power that appears in it.

- Example: P1(n) = 3n⁷ + 9n⁴ + 3 is a polynomial of degree 7.

- A complexity class is a class of computational problems, whose complexity satisfies some given specification.

16.5 Polynomials and Complexity Classes

log – problems of complexity O (log n).

polylog – problems of complexity O (P(log n)).

linear – problems of complexity O (n).

polynomial - problems of complexity O (P(n)).

exponential - problems of complexity O (2^P(n)).

doubly exponential - problems of complexity O (2^{2P (n)}).

Non-elementary – problems for which there is no fixed k, such that

:2n

the complexity is in O(2 ^2. ), i.e., a tower of k exponents.

Some of the main complexity classes:

- For polynomial complexity, the only important factor for the complexity measure is the one with the

highest (fixed) exponent, e.g., O(3n⁷ + 9n⁴ + 3) is the same as O(n⁷).

- Example: Consider problem instances of sizes n = 5, 10, 50,100 and 500.

- For simplicity, assume that the constants c₀ and c₁ are 0 and 1, respectively, in all of the following cases.

ƒ O(n): The time for the calculation is 5, 10, 50, 100 and 500, respectively of some time units, e.g., milliseconds.

ƒ O(n²): The execution time will be 25, 100, 2500, 10 000 and 250 000 milliseconds, respectively.

ƒ O(n³): This is 125, 1000, 125 000, 1 000 000 and 125 000 000 milliseconds, respectively.

ƒ O(2ⁿ): This is 32, 1024 then a number that is greater then 1 125 followed by 26 zeros, then a number that is greater than 3 followed by 150 zeros, etc.

ƒ O(2²ⁿ), even the first instance, where n = 5, will take 4 294 967 296

16.6 Examples for the Interpretation of the Complexity Classes

- Linear complexity: Suppose, we have a O(n) problem and buy a computer that is m times faster than the one we already have.

Then we would

ƒ speed up the computation by m,

ƒ solve the problem in 1/m^th of the time it took us before, or

ƒ we can solve a problem that is m times “bigger” in the same time.

- If our problem is of complexity O(n²), and we buy the m times faster machine, we can solve in the same amount of time a problem that is only sqrt(m) “bigger”.

- Exponential complexity: For the same time it took to solve a

problem with instance m, we will be able to solve instances that are log₂(m) times bigger, e.g.,

ƒ 100-times increase of computing power may perhaps enable to solve problems that are almost eight times as big,

ƒ a 1000-times increase in computing speed would solve within the same time problems whose size is only about 10 times bigger.

- If we use an array of 100 computers instead of one, we can speed up the execution by at most a factor of 100.

- This is provided that we have algorithms that can parallelize

the problem and utilize the array of machines in an optimal way.

- Unfortunately, this is often not the case.

- Consider, in the case where we can parallelize an algorithm only for half of the duration of its execution, our 100 machines will reduce the execution time to

1/2 + 1/200 = 101/200 of the original time.

- That is, in this case the speedup is only about twice the original speed.

- In general, we can benefit from parallelization only if we can parallelize almost all of the algorithm.

16.7 Examples for the Determination of the Complexity Classes - Analyze the following algorithms and determine their

complexity classes.

- Hint:

ƒ Identify the “key element” which characteristically influences the time/memory needs of the

program.

ƒ Identify the loops, especially the nested ones (iterative/recursive ones) that increase the

complexity.

ƒ Form a polynomial function for the time/memory needs to run the program. Trace it manually

void swap (int *zhl1, int *zhl2) {

int temp = *zhl2;

*zhl2 = *zhl1;

*zhl1 = temp;

}

void sort (int vector[], int n) {

int i,j;

for(i=n-1; i>=0; i--) for(j=0; j<i; j++)

if (vector[j] > vector[j+1])

swap(&vector[j], &vector[j+1]);

}

Example 1: „Bubble Sort“

void sort (int vector[], int n) // n: number of elements {

int i,j;

for(i=n-1; i>=0; i--) // n-1 passes for(j=0; j<i; j++) // n-i passes

if (vector[j] > vector[j+1])

swap(&vector[j], &vector[j+1]);

}

„Bubble Sort“ – Runtime Analysis

Count the number of comparisons ( if (vector[j] > vector[j+1])):

For n =5: (5-1) + (5-2) + (5-3) + (5-4) = 4 + 3 + 2 + 1

= 5 (5-1) / 2 = 10

For n=nsum of the comparisons can n (n-1) / 2 comparisons are necessary!

sum of the comparisons can

General: The sum of the comparisons can be determined by the formula:

comparisons comparis

void movetower(int height, int start, int target, int auxil) {

if (height > 0) {

movetower(height-1,start,auxil,target);

printf("Start Tower: %d Target Tower %d \n", start, target);

movetower(height-1,auxil,target,start);

} }

void main() {

int height;

scanf("%d", &height);

if (height > 0)

movetower(height,1,2,3);

}

Example 2: The Towers of Hanoi

Example with height = 2:

Start Tower : 1 Target Tower : 3 Start Tower : 1 Target Tower : 2 Start Tower : 2 Target Tower : 3

3 steps.

Example with height = 3:

Start Tower : 1 Target Tower : 2 Start Tower : 1 Target Tower : 3 Start Tower : 2 Target Tower : 3 Start Tower : 1 Target Tower : 2

Start Tower : 1 Target Tower : 2 Start Tower : 3 Target Tower : 1 Start Tower : 3 Target Tower : 2

Example with height = 4:

Start Tower : 1 Target Tower : 3 Start Tower : 1 Target Tower : 2 Start Tower : 3 Target Tower : 2 Start Tower : 1 Target Tower : 3

Start Tower : 1 Target Tower : 3 Start Tower : 2 Target Tower : 1 Start Tower : 2 Target Tower : 3 Start Tower : 1 Target Tower : 2 Start Tower : 3 Target Tower : 2 Start Tower : 3 Target Tower : 1 Start Tower : 2 Target Tower : 1 Start Tower : 3 Target Tower : 2 Start Tower : 1 Target Tower : 3 Start Tower : 1 Target Tower : 2 Start Tower : 3 Target Tower : 2

Summary

- n =1: 1 transfer - n =2: 3 transfers - n =3: 7 transfers - n =4: 15 transfers

...

- n =n: 2ⁿ-1 transfers

Example

- n =64 leads to 2⁶⁴-1 ~ 2. 10¹⁹ transfers

Remark: If the transfer of a single disk takes 10 sec, this

The Towers of Hanoi – Runtime Analysis

Assumption: A tower of n disks needs T(n) = 2ⁿ-1 disk transfers to be moved from start to target.

Proof by mathematical induction:

Basis step: T(1) =T(n=1)= 1: 2¹-1 = 1 transfer 2nd step: T(2 )=T(n=2)= 3: 2²-1 = 3 transfers

... ...

Inductive step: A tower of n disks needs T(n) = 2ⁿ-1 transfers

Final step: A tower of n+1 disks needs

(Conclusion) T(n+1) = 2ⁿ⁺¹-1 transfers (to be proven)

T(n+1) = T(n) + 1 + T(n) (moving n disks from start to auxil. tower

+ 1 from start to target + n from auxil. to target)

= 2T(n) + 1

= 2 (2ⁿ-1) +1 (by assumption for n)

= 2ⁿ⁺¹ – 2 + 1 (arrange the terms)

- In der Praxis werden Programme immer größer, auch in Ihrer Diplomarbeit.

- PAP nicht günstig für Übersichtlichkeit der Darstellung der Programmlogik und zur Komplexitätsuntersuchung.

- Struktogramme („Nassi-Shneiderman-Diagramme“) besser.