If µ issuchthat h ( T )= h ( T ) ,then µ iscalledameasureofmaximalentropy.Wegivesomehistoryrst. h ( T )=sup { h ( T ): µ is T -invariantprobabilitymeasure } µ .Topologicalandmeasure-theoreticalentropyarerelatedbytheVariationPrinciplewhichsaythat Inthisle

Entropy

In this lecture we start discussing the notion of entropy. In dynamical systems, there istopological entropy(which is

independent of invariant measures), and in ergodic theory there is measure-theoretic entropy, also misnamed as metric entropy. It depends on the choice of invariant measureµ.

Topological and measure-theoretical entropy are related by the Variation Principlewhich say that

h_top(T) = sup{h_µ(T) :µ isT-invariant probability measure} Ifµ is such thath_µ(T) =h_top(T), thenµis called a measure of maximal entropy.

We give some history rst.

Entropy

I Entropy was rst used by 19th century physicist Rudolf Clausius and Ludwig Boltzmann as part of thermodynamics.

I Andrej Kolmogorov introduced a mathematical version in probability in the 1950s, and used it as isomorphism invariant.

This is the way we still dene it in ergodic theory.

I Around the same time, Claude Shannon used it in information theory.

I Major work in 1960s by Yakov Sina on measure-theoretic entropy and generators.

I Topological entropy was introduced in 1969 by Roy Adler, Alan Konheim and Harry McAndrew.

I In the early 1970s, Rufus Bowen and Em Dinaburg independently introduced a user-friendlier version of

topological entropy. Around this time, the Variational Principle was proved.

I In 1974 Don Ornstein published his theorem that entropy is completeinvariant for two-sided Bernoulli shifts.

Entropy

The current mathematical denition has very little to do anymore with the original denition from thermodynamics. Only it still expresses the amount of disorder in the system.

I For the circle rotation (S¹,B, µ,R_α) with Lebesgue measure, h_top(R_α) =h_µ(R_α) =0.

I For the doubling map (S¹,B, µ,T)with Lebesgue measure, htop(T) =hµ(T) = log2.

I For the Bernoulli shift ({1, . . . ,N}^NorZ,B, µ_p, σ) with probability vector p= (p₁, . . . ,p_N),

htop(σ) = logN ≥hµp(σ) =−

N

X

i=1

pilogpi.

Jensen's Inequality

A functionf :R→Risconcave if

f(αx+ (1−α)y)≤αf(x) + (1−α)f(y) (1) for allx,y ∈Randα∈[0,1].

Iff :R→Ris C² andf⁰⁰≤0, then f is concave.

Theorem: For every strictly concave function f : [0,∞)→R, and allαi >0,Pn

i=1αi =1 and xi ∈[0,∞)we have

n

X

i=1

αif(xi)≤f(

n

X

i=1

αixi), (2)

withequality if and only if all the x_i are the same.

Jensen's Inequality

Proof of Jensen's Inequality: We prove this by induction onn. Forn=2 it is simply (1). So assume that (2) holds for somen.

Forn+1, take α_i >0 andPn+1

i=1 α_i =1 and write B=Pn i=1α_i. f(

n+1

X

i=1

αixi) = f(B

n

X

i=1

αi

Bxi+αn+1xn+1)

≥ Bf(

n

X

i=1

α_i

Bxi) +αn+1f(xn+1) by (1)

≥ B

n

X

i=1

α_i

Bf(x_i) +α_n+1f(x_n+1) by (2) for n

=

n+1

X

i=1

α_if(x_i)

Equality also carries over by induction. Ifx_i are all equal fori ≤n, then(1) for n+1 is an equality only ifxn+1 =Pn

i=1 αi

Bxi =x₁.

Jensen's Inequality

For measure-theoretic entropy, the functionϕ: [0,1]→Rdened as

ϕ(x) =−xlogx ϕ(0) =ϕ(1) =0 is important. Compute

ϕ⁰(x) =−1−logx, ϕ⁰⁰(x) =−1 x <0, soϕis concave.

By Jensen's Inequality (with allα_i = _N¹)

−

N

X

i=1

p_ilogp_i =N

N

X

i=1

1

Nϕ(p_i)≤Nϕ(1 N

N

X

i=1

p_i) = logN,

withequality if and only ifall pi are the same.

Entropy

Let(X,B, µ) be a probability measure space. We call a collection P ={P_i} Pi measurable

a (measurable)partition if allP_i's are disjoint, and X =∪_iP_i. We say that a partitionP isnerthan Q (written asP Q) if everyP ∈ Pis contained in some Q ∈ Q. For example, the nest possible partition is when ifPi ={i},i ∈X; this is thepoint partition. It is (uncountably) innite ifX is. The coarsest partition P ={X} is called the trivial partition.

Given two partitionsP andQ, the joint is

P ∨ Q:={P ∩Q :P ∈ P,Q ∈ Q};

This joint is ner than bothP andQ.

Entropy

IfT :X →X is measurable, then the n-th joint ofP is dened as

P_n=

(Wn−1

k=−nT^−kP. ifT is invertible, Wn−1

k=0T^−kP. ifT is non-invertible,

For example, ifT :S¹ →S¹ is the doubling map (non-invertible), and

P ={[0,1 2),[1

2,1)}

then

P_n={[i/2ⁿ,(i+1)/2ⁿ),i =0, . . . ,2ⁿ−1} so #P_n=2ⁿ. A partition isgenerating forT if for almost all x6=y ∈X, there is n such that x andy lie in dierent elements ofP_n.

Entropy

Given a nite partitionP of a probability space (X, µ), let H_µ(P) = X

P∈P

ϕ(µ(P)) =−X

P∈P

µ(P) log(µ(P)), (3) where we can ignore the partition elements withµ(P) =0 because ϕ(0) =0. For aT-invariant probability measureµ on (X,B,T), and a partitionP, dene theentropy ofµ w.r.t.P as

hµ(T,P) = lim

n→∞

1 nHµ(

n−1

_

k=0

T^−kP). (4) Finally, themeasure theoretic entropy of µis

h_µ(T) = sup{h_µ(T,P) : P is a nite partition ofX}. (5)

Fekete's Lemma

For this denition to make sense, we need to verify that thelimit in (4) exists. For this we need:

Denition: We call a real sequence (an)n≥1 subadditiveif a_m+n≤a_m+a_n for allm,n∈N. A positive sequence(a_n)n≥1 submultiplicative if

am+n≤am·an for allm,n∈N.

NB: If(an) is submultiplicative, then(logan) is subadditve.

Fekete's Lemma: If (a_n)n≥1 is subadditive, then limn

an

n = inf

r≥1

ar

r .

Fekete's Lemma

Proof of Fekete's Lemma: Every integern can be written uniquely asn=p·q+r for 0≤r <q. Therefore

lim sup

n→∞

a_n

n = lim sup

p→∞

ap·q+r

p·q+r ≤lim sup

p→∞

pa_q+a_r p·q+r = a_q

q .

This holds for all q∈N, so we obtain infq

a_q

q ≤lim inf

n

a_n

n ≤lim sup

n

a_n n ≤inf

q

a_q q , as required.

Fekete's Lemma

Calla_n=H_µ(Wn−1

k=0T^−kP). Then (Prop. 13 of the Class Notes) a_m+n = H_µ(

m+n−1

_

k=0

T^−kP)

≤ Hµ(

m−1

_

k=0

T^−kP) +Hµ(

m+n−1

_

k=m

T^−kP)

byT-invariance ofµ = H_µ(

m−1

_

k=0

T^−kP) +H_µ(

n−1

_

k=0

T^−kP)

= a_m+a_n. ThereforeH_µ(Wn−1

k=0T^−kP) is subadditive, and the existence of the limith_µ(T,P) = lim_n¹_nH_µ(Wn−1

k=0T^−kP) follows.

Entropy

The second natural question about computing entropy:

How can one possibly consider all partitions ofX?

By the next theorem, which we state without proof (see Theorem 22 in the Class Notes), we can reduce all partitions to a single generating partition:

Theorem (Kolmogorov-Sina): Let (X,B,T, µ) be a

measure-preserving dynamical system. If partitionP is such that W∞

j=0T^−kP generatesB ifT is non-invertible, W∞

j=−∞T^−kP generatesB ifT is invertible, thenhµ(T) =hµ(T,P).

Entropy

Now a good property of entropy:

Theorem: Two isomorphic measure preserving systems have the same entropy.

Indeed, let(X,B,T, µ) and(Y,C,S, ν) have full-measured sets X⁰⊂X,Y⁰ ⊂Y and a bi-measurable invertible measure-preserving mapφ:X⁰ →Y⁰ such that

(X⁰,B, µ) −→^T (X⁰,B, µ)

φ↓ ↓φ

(Y⁰,C, ν) −→^S (Y⁰,C, ν) commutes, thenh_µ(T) =h_ν(S).

This holds, because the bi-measurable measure-preserving mapφ preserves all the quantities involved in (3)-(5), including the class of partitions for both systems.

Entropy

For two-sided (i.e., invertible) Bernoulli shifts

(X ={0, . . . ,N}^Z,B, µ_p) based on the probability vector p= (p₁, . . . ,pN}, the cylinder partition

P ={[i] :i =0, . . . ,N} is generating.

Lemma: For the cylinder partitionP hµp(σ,P) =−X

i

pilogpi.

By the Kolmogorov-Sina Theorem, this generating partition suces to compute the entropy.

Theorem (Ornstein 1974): Two two-sided Bernoulli shifts (X, µ_p, σ) and(X⁰, µ_p⁰, σ) are isomorphic if and only if h_µp(σ) =h_µp0(σ).

Entropy

Let us now compute thatWn−1

k=0T^−kP.=−P

ip_ilogp_i, just for two symbols, and using the cylinder partitionP ={[0],[1]}, which can denote head and tail in coin-ips.

P(k heads inn ips) = n

k

p^k(1−p)^n−k, so by full probability:

n

X

k=0

n k

p^k(1−p)^n−k =1.

Here ⁿ_k

= _k!(n−k)!^n! are the binomial coecients, and





 k ⁿ_k

= _{(k−1)!(n−k)!}^n! =n_(k−⁽ⁿ⁻_1)!(n−k)!^1)! =n ⁿ⁻_k−¹₁ (n−k) ⁿ_k

= _{(k)!(n−k−}^n! _1)! =n_k!(n−k⁽ⁿ⁻¹₋^)!_1)! =n ⁿ⁻_k¹ (6)

Entropy

We compute.

Hµ(

n−1

_

k=0

σ^−kP) = −

X1

x₀,...,xn−1=0

µ([x₀, . . . ,xn−1]) logµ([x₀, . . . ,xn−1])

= −

X1

x₀,...,xn−1=0 n−1

Y

j=0

ρ(x_j) log

n−1

Y

j=0

ρ(x_j)

= −

n

X

k=0

n k

p^k(1−p)^n−klog

p^k(1−p)^n−k

= −

n

X

k=0

n k

p^k(1−p)^n−kklogp

−

n

X

k=0

n k

p^k(1−p)^n−k(n−k) log(1−p)

In the rst sum, the termk =0 gives zero, as does the termk =n for the second sum.

Entropy

Thus we leave out these terms and rearrange by (6):

Hµ(

n−1

_

k=0

σ^−kP) = −plogp

n

X

k=1

k

n−1 k

p^k−1(1−p)^n−k

−(1−p) log(1−p)

n−1

X

k=0

(n−k) n

k

p^k(1−p)^n−k−1

= −plogp

n

X

k=1

n n−1

k−1

p^k−1(1−p)^n−k

−(1−p) log(1−p)

n−1

X

k=0

n n−1

k

p^k(1−p)^n−k−1

= n(−plogp−(1−p) log(1−p)). P is generating, so by the Kolomogorov-Sina Theorem, hµ(σ) =hµ(σ,P) = lim

n

1 nHµ(

n−1

_

k=0

σ^−kP) =−plogp−(1−p) log(1−p).