Linear Speed-Up

Linear Speed-Up

c autowp

Linear Speed-Up and Compression Theorems

c 2020 lautosports

Linear Speed-Up and Compression Theorems

The central question in this section is:

How much must a resource be increased in order to be able to compute strictly more?

Consider, for example, the deterministic time classDTIME(t₁), for some resource functiont1. How much stronger thant1must another function,t₂, grow in order to ensure that

DTIME(t₁)6=DTIME(t₂)?

The linear tape-compression and speed-up theorems say that a linear increase of the given resource function doesnotsuffice to get a strictly bigger complexity class.

Linear Compression Theorem

Theorem (Linear Compression) For each total recursive function s,

DSPACE(s) =DSPACE(ILin(s)).

Proof: It is enough to show thatDSPACE(2s)⊆DSPACE(s).

LetMbe a DTM working, on any inputx of lengthn, in space 2s(n).

It is convenient to make, without loss of generality, the following assumptions aboutM:

(a) Mhas only one tape that (b) is infinite in just one direction,

(c) the tape cells are enumerated by 1, 2, etc., and

(d) M’s head makes a left turn only on even-numbered cells.

Linear Compression Theorem

(If the given machine does not have these properties, it is not difficult to replace it by an equivalent one that does have the desired properties.)

Suppose further thatΓis the working alphabet ofM.

The goal is to construct a new DTMN that, on inputx of lengthn, simulates the computation ofM(x)but works in spaces(n).

Idea: N, which has more states thanM and whose working alphabet is Γ×Γ, “delays” the simulation ofM: it will wait and see whatM is going to do next before actually doing it.

To this end, viewM’s tape as being subdivided into blocks of two adjacent cells each, i.e., the blocks are the pairs of cells with numbers (2i−1,2i), fori ≥1.

Linear Compression Theorem

Each such block is now considered to beonetape cell ofN, and every ordered pair of symbols(a,b)∈Γ×Γis now considered to beone symbol ofN.

Then,N(x)simulates the computation ofM(x), except thatN moves its head to the left or to the right only whenM’s head crosses a boundary between two blocks to the left or to the right.

All steps ofMwithin any one block can be simulated byN’s finite control.

That is whyN needs more states thanM.

Clearly,N(x)performs the exact same computation asM(x)and

needs only spaces(n). q

Linear Speed-Up Theorem

Theorem (Linear Speed-Up)

For each total recursive function t with t id, DTIME(t) =DTIME(ILin(t)).

Proof of Linear Speed-Up Theorem: Idea

Proof:

LetA∈DTIME(t), and letM be a DTM such thatL(M) =AandM works in timet(n)on inputs of lengthn.

Goal: Construct a DTMN withL(N) =Abut at leastmtimes as fast asM, for some constantm>1.

That is,msteps ofMare to be simulated within just one step ofN.

Again, the idea is thatpatience will pay off:

Nwill“delay”the simulation ofM, i.e.,Nwill wait and see whatM is going to do within the nextmsteps, then doing it all at once within a single step of its own.

Nwillcompressthe input using a larger alphabet and more states.

Proof of Linear Speed-Up Theorem: Idea

However,Ncan use its compressed encoding not before it has scanned every input bit and has transformed the input into the compressed encoding to be used later on.

In other words, the head moves on the input tape cannot be speeded up.

Thus, the computation ofN, on inputx of lengthn, is done in two phases:

Preparation Phase;

Simulation Phase.

Proof of Linear Speed-Up Theorem: Preparation

Subdivide the input stringx =x₁x₂· · ·x_nof lengthninto blocks of lengthm, where thei^thblock,i ≥1, is represented by the string

β_i =x_1+(i−1)mx_2+(i−1)m· · ·x_i·m.

Then,N writes on its working tape the following redundant encoding of the input string:

(2^m, β1, β2) (β1, β2, β3) (β2, β3, β4) · · · (βk−2, βk−1, βk) (βk−1, βk,2^m)

Every triple of the form(βi−1, βi, βi+1), where 1<i <k, or (2^m, β₁, β₂)or(βk−1, β_k,2^m)is considered to be justonesymbol ofN.

Proof of Linear Speed-Up Theorem: Preparation

AfterN

has copied the input in this compressed (and somewhat redundant) form onto the working tape and

has moved the head back to the leftmost symbol,(2^m, β₁, β₂), this working tape will henceforth be used as the input tape.

The original input tape, which has been erased during the preparation phase, will henceforth be used as a working tape.

The preparation phase requires n+k =

1+ 1

m

n steps.

Proof of Linear Speed-Up Theorem: Simulation

As above,N’s encoding ofa=a₁a₂· · ·a_`is of the form

(2^m, α₁, α₂) (α₁, α₂, α₃) (α₂, α₃, α₄) · · ·(αz−2, αz−1, α_z) (αz−1, α_z,2^m) where

(1) ais subdivided intoz+1 blocks,

a=α₁α₂· · ·α_z+1,

(2) for eachiwith 1≤i ≤z, block

α_i =a1+(i−1)ma2+(i−1)m· · ·ai·m

has lengthm, and

(3) blockα_z+1with|α_z+1|<mis handled byN’s finite control.

Proof of Linear Speed-Up Theorem: Simulation

N(x)simulatesmsteps ofM(x)as follows:

IfM’s head is currently scanningα_j, thenN’s head scans (αj−1, α_j, α_j+1).

Aftermsteps,M’s head has moved by at mostmtape cells.

Hence, it must scan eitherαj−1orα_j orα_j+1, and none of the other blocks has been changed byM.

SinceN’s head scans(αj−1, α_j, α_j+1), it can do all ofM’s changes within a single step of its own, and it moves its head to the symbol:

(αj−2, αj−1, α_j) ifMscansαj−1

(α_j−1, α_j, α_j+1) ifMscansα_j (α_j, α_j+1, α_j+2) ifMscansα_j+1 after thesemsteps.

Proof of Linear Speed-Up Theorem: Simulation

IfMaccepts or rejectsx within thesemsteps, then so doesN.

Hence,L(N) =L(M).

The simulation phase requires at most t(n)

m

steps.

Proof of Linear Speed-Up Theorem: Analysis

Recall thatid≺t, i.e.,n∈o(t(n)). Thus,

(∀c>0) [n<aec·t(n)]. (1) Summing up the time spent in both phases,N(x)needs no more than

1+ 1

m

n+ t(n)

m

<_ae

1+ 1 m

1

m 1+_m¹t(n) + t(n)

m

≤

2t(n) m

+1

steps, where the first inequality follows from (1) for the specific constant

cˆ= 1

m 1+_m¹ = 1 m+1.

Proof of Linear Speed-Up Theorem: Analysis

The finitely many exceptions allowed in the ≤_ae-notation can be handled by table-lookup.

Thus, an arbitrary linear speed-up is possible by suitably

choosingm. q

Proof of Linear Speed-Up Theorem: Illustration

Supposet(n) =d·n, for some constantd >1, andN(x)has running time

T(n) =

1+ 1 m

n+t(n) m

=

1+ 1 m

n+d ·n m

=

1+d+1 m

n,

where we assume for convenience thatmdivides bothnandt(n).

Proof of Linear Speed-Up Theorem: Illustration

Sinced >1, choosing

m> d +1 d −1

impliesT(n)<d·n=t(n)and thus a genuine speed-up.

This example also suggests that the above proof does not work for d =1, i.e., it does not work fort=id.

Existence of Arbitrarily Complex Problems

Fact

For each t ∈IR, there exists a problem At such that At 6∈DTIME(t).

Proof: The proof is by diagonalization. LetM₀,M₁,M₂, . . .be a G ¨odelization (i.e., an effective enumeration) of all DTMs. Define

A_t ={0ⁱ

M_i does not accept 0ⁱ withint(i)steps}.

SupposeAt ∈DTIME(t). Then, there exists aj such thatL(M_j) =At

andtimeM_j(n)≤t(n)for eachn∈N. Hence,

0^j ∈At ⇐⇒ M_j does not accept 0^j withint(j)steps

⇐⇒ 0^j 6∈L(M_j) =At,

which is a contradiction. It follows thatA_t 6∈DTIME(t). q

Existence of Arbitrarily Complex Problems

Since complexity classes such asDTIME(t)are closed under finite invariance,“At ∈DTIME(t)”means:

“For some DTM M, L(M) =A_t andtimeM(n)≤_aet(n).”

Hence,“A_t 6∈DTIME(t)”in the above fact means:

“For each DTM M with L(M) =At,time_M(n)>_iot(n).”

However,“At 6∈DTIME(t)”does not exclude that, for infinitely many othern∈N,time_M(n)≤_iot(n)may nonetheless be true.

In this sense, Rabin’s theorem on the next slide is much stronger than the above fact.

The (omitted) proof uses a clever priority argument in its diagonalization.

Existence of Arbitrarily Complex Problems

Theorem (Rabin)

For each t ∈IR, there exists a decidable set Dt such that for each DTM M deciding D_t, it holds that

time_M(n)>_aet(n).

without proof

Space-Constructibility

Definition (Space-Constructibility)

Letsbe a function inIRmapping fromNtoN.

We say thatsisspace-constructibleif and only if there exists a DTMM such that, for eachn,

M on any input of lengthnuses no more thans(n)tape cells and prints the string

#1^s(n)−2$

on one of its tapes, where#and$are special symbols marking the left and right boundaries.

We then say thatMhasmarked the space s(n).

Time-Constructibility

Definition (Time-Constructibility)

Letf andtbe functions inIRmapping fromNtoN.

We say thatf isconstructible in time t if and only if there exists a DTM M such that, for eachn,

M on any input of lengthnruns for exactlyt(n)steps and prints the string

#1^f(n)−2$

on its tape, where#and$are special symbols marking the left and right boundaries.

We say thatt istime-constructibleif and only ift is constructible in timet.

Space- and Time-Constructibility

Remark: Constructibility of resource functions is necessary to obtain aneffective enumeration of the Turing machinesrepresenting a complexity class. For example, the set

{M

L(M)∈DSPACE(s)}

is decidable ifs is space-constructible;

otherwise it is not even recursively enumerable.

Example: logn, n^k, 2ⁿ, 2^nk, . . . are space- and (exceptlogn, which doesn’t make sense for DTMs) time-constructible.

Effective Enumerations of Turing Machines

Goal: We want to effectively enumerate all DTMs or NTMs that always work within a given time boundt (or space bounds).

For example, forDTIME(t):

Lett be time-constructible via DTMM.

LetM₁,M₂, . . .be a fixed G ¨odelization of all DTMs.

Construct an enumerationM₁⁰,M₂⁰, . . .forDTIME(t)as follows.

M_i⁰ on inputx:

simulatesM_i(x)andM(1^|x|)in parallel;

ifMi(x)stops first or at the same time asM(1^|x|)then:

M_i⁰acceptsx ⇐⇒ M_i acceptsx. ifM(1^|x|)stops first thenM_i⁰ rejectsx.

Space Hierarchy Theorem

Again:

How much must a resource be increased in order to be able to compute strictly more?

Consider, for example, the deterministic space classDSPACE(s₁), for some resource functions₁. How much stronger thans₁must another function,s₂, grow in order to ensure that

DSPACE(s₁)6=DSPACE(s₂)?

Space Hierarchy Theorem

From the linear tape-compression theorem we know that s₂∈ O(s₁) ⇐⇒ ∃c>0:s₂≤_aec·s₁ is not enough.

However, the negation:

s_2ios₁ ⇐⇒ ∀c>0:s₂>_ioc·s₁ does suffice.

Theorem (Space Hierarchy Theorem)

If s₁≺_ios₂and s₂is space-constructible, then DSPACE(s₂)6⊆DSPACE(s₁).

Space Hierarchy Theorem: Proof

Proof: We prove the theorem only for the case ofs₁≥log.

(Using a result of Sipser (TCS 1980), one can get rid of this simplifying assumption.)

To construct a setAin the difference

DSPACE(s₂)−DSPACE(s₁)

by diagonalization, fix a G ¨odelizationM₀,M₁,M₂, . . .of all DTMs having one working tape. (It is easy to see that it is enough to consider, without loss of generality, only one-tape DTMs.)

Define a DTMNwith an input tape and three working tapes.

Space Hierarchy Theorem: Proof

On inputx ∈ {0,1}^∗ of lengthn, DTMNworks as follows:

1 N marks the spaces₂(n)on all three working tapes.

2 Supposex is of the formx =1ⁱy, 0≤i ≤n,y ∈ {ε} ∪0{0,1}^∗. That is,x starts with a (possibly empty) prefix ofiones

followed either by the empty string (in which casex =1ⁿ), or followed by a zero and a (possibly empty) string from{0,1}^∗. DTMNinterpretsias a machine number, and it writes the suitably encoded program ofM_i onto its first working tape.

If this is not possible, sinceM_i’s program is too large to fit in the marked spaces₂(n), thenN aborts the computation and rejectsx.

Space Hierarchy Theorem: Proof

Otherwise,Nproceeds by

simulating the computation ofM_i(x)on the second working tape, using the program ofMi on its first working tape and

reading the symbols ofx from its own input tape.

3 The third working tape contains a binary counter that is initially set to zero and is incremented by one in each step of the simulation ofM_i(x).

If the simulation ofM_i(x)succeeds onN’s second working tape before the counter onN’s third working tape overflows, thenN(x) accepts if and only ifM_i(x)rejects.

Otherwise,Nrejectsx.

Space Hierarchy Theorem: Proof

Some technical explanations are in order:

The counter onN’s third working tape guarantees thatN(x)halts, even ifM_i(x)would never terminate.

There exists a constantc_i such that the simulation ofM_i(x)onN’s second working tape can be done in space at most

c_i·space_Mi(n).

Why?

DTMN must be able to simulateevery DTMM_i,i∈N. If for somei,M_i hasz_i states and`<sub>i</sub> symbols in its working alphabet, thenN can encode these states and symbols in binary, i.e., by strings over{0,1}of lengthdz<sub>i</sub>eandd`_ie, respectively.

Space Hierarchy Theorem: Proof

This encoding causes a constant space overhead for the

simulating machineN, where the constantc_i depends only onM_i. DefineA=L(N). Clearly,A∈DSPACE(s₂).

To prove thatA6∈DSPACE(s₁), suppose for a contradiction that A∈DSPACE(s₁).

Thus, there exists someisuch thatA=L(M_i)and space_M

i(n)≤s₁(n)≺_ios₂(n).

Recall whats₁≺_ios₂means:

(∀c >0) [s₂(n)>_ioc·s₁(n)]. (2)

Space Hierarchy Theorem: Proof

Hence, for each real constantc >0, there exist infinitely many argumentsn₀,n₁,n₂,. . .inNsuch that

s2(n_k)>c·s1(n_k) for eachk.

From this infinite sequence of arguments, choosen_j large enough such that the following three conditions hold:

(a) M_i’s program can be computed and written ontoN’s second working tape in spaces₂(n_j);

(b) the simulation ofM_i(1ⁱ0^nj−i)succeeds in spaces₂(n_j);

(c) time_Mi(n_j)≤2^s2(nj).

Space Hierarchy Theorem: Proof

Condition (a) can be satisfied for a large enoughn_j, since the size of the program ofM_i is a constant not depending on the machine’s input.

Condition (b) can be satisfied for a large enoughn_j, since the simulation ofM_i(1ⁱ0^nj−i)succeeds in space:

c_i·space_Mi(n_j)≤c_i·s₁(n_j)<s₂(n_j),

wherec_i is the above constant that is due toN having to encodeM_i’s states and symbols, and where the last inequality follows from (2).

Condition (c) can be satisfied for a large enoughn_j, since fors₁≥log:

time_Mi(n_j) ≤ 2^{d·spaceMi(nj)} for a suitable constantd

≤ 2^d·s1(nj)

< 2^s2(nj), again by (2).

Space Hierarchy Theorem: Proof

Hence, the simulation ofM_i(1ⁱ0^nj−i)succeeds before the binary counter of lengths₂(n_j)onN’s third working tape is full.

Conditions (a), (b), and (c) and the construction ofN imply that for the stringx =1ⁱ0^nj−i,

x ∈A ⇐⇒ N acceptsx

⇐⇒ M_i rejectsx.

Thus,A6=L(M_i), contradicting our supposition.

Hence,A6∈DSPACE(s₁). q

Time Hierarchy Theorem

Theorem (Time Hierarchy Theorem)

If t₂≥idand t₁≺_iot₂and t₂is constructible in time t₂logt₂, then DTIME(t₂logt₂)6⊆DTIME(t₁).

without proof

Upper Bounds and Lower Bounds

Upper boundfor a problemΠ:There exists some algorithm(of the specified type) that solvesΠwithin the given complexity bound.

Lower boundfor a problemΠ:All algorithms(of the specified type) solvingΠrequire at least / more than the given complexity.

A Lower Bound Proof via Crossing Sequences

Theorem

For each DTM M with only one working tape and no separate input tape that decides the problem

S={x2^|x|x

x ∈ {0,1}^∗} there exists a constant c >0such that

time_M(n)>c·n².

A Lower Bound Proof via Crossing Sequences

Proof: LetMbe a DTM with only one working tape and no separate input such that

L(M) =S={x2^|x|x

x ∈ {0,1}^∗}.

Letw =uv be any input string.

A sequence of states ofM(w), denoted by cs(u|v) = (s1,s2, . . . ,s_k),

is called thecrossing sequence of M(x)at the cell-boundary between u and v ifM’s head crosses this cell-boundary exactlyk times during the computation ofM(x)andM is in states_i during thei^th crossing.

A Lower Bound Proof via Crossing Sequences

Lemma

If uv ∈L(M)andpq ∈L(M)andcs(u|v) =cs(p|q), then uq ∈L(M) andpv ∈L(M).

Proof Sketch of Lemma.

v u

1 2 3

4

s s s s

2 3

4

p q

s

s s s

u

1

q

2 3

4

s s s s

q

A Lower Bound Proof via Crossing Sequences

By this lemma, for stringsx andy withx 6=y, all crossing sequences ofx2^|x|x andy2^|y|y in the block of 2s are pairwise distinct.

Why?

Because otherwise,M would accept stringsnot inS.

For example, consider

w₁=uv =101222101 with u=1012 and v =22101, w2=pq=001222001 with p=0012 and q=22001.

Clearly,x =1016=001=y andw₁,w₂∈S=L(M).

By the lemma, ifcs(u|v) =cs(p|q), thenuq =101222001∈L(M) =S andpv =001222101∈L(M) =S, contradicting the definition ofS.

A Lower Bound Proof via Crossing Sequences

Letz >1 be the number ofM’s states.

Then the number of distinct crossing sequences of length at most`is:

z⁰+z¹+· · ·+z^{`</sup>= z<sup>`+1}−1

z−1 <z^`+1. Consider stringsx2^|x|x inSwithx ∈ {0,1}^∗ and|x|=n.

There are exactly 2ⁿsuch strings of length 3ninS.

We say a crossing sequence isshort with respect to nif its length is shorter than`₀, where

z<sup>`0+1</sup>=2<sup>n</sup>, i.e., `₀= n logz −1.

A Lower Bound Proof via Crossing Sequences

Thus there are fewer short crossing sequences with respect tonthan strings inS∩ {0,1,2}³ⁿ.

Since for distinct strings inS∩ {0,1,2}³ⁿ all crossing sequences in the block of 2s are pairwise distinct, there exists a stringw ∈S,|w|=3n, that has no short crossing sequences with respect tonin the block of 2s, i.e., all its crossing sequences in this block are of length at least`₀. It follows thatMneeds time at least

(n−1)

n

logz −1

. q