Problem sheet 1, 2005, Oct. 5 MT441 Channels 1 Ex. 1
Verify that the functionH(p1, . . . , pn) =−P
kpklog2pk satisfies all 8 axioms onH.
Ex. 2
(Not to be handed in). List as many of the 8 axioms as you can, (without just looking at the notes).
Ex. 3
Let X be a random variable, taking the values a1 and a2 with probability p1 and p2, respectively. Let Y be a random variable, taking the values b1, b2 and b3 with probabilityq1, q2 and q3, respectively.
Prove that H(X, Y) ≤ H(X) +H(Y), with equality if and only if X and Y are independent. (You should work through the proof here, not just say that this is a special case of a more general theorem stated in the lecture).
Ex. 4
Two fair dice are thrown. X denotes the value obtained by the first, andY the value obtained by the second, and letZ =X+Y be the corresponding random variable.
Evaluate H(X), H(Y), H(X, Y), H(Z), H(X|Y).
From this, verify thatH(X, Y) =H(X) +H(Y) and show that H(Z)< H(X, Y).
Ex. 5
Show that, for any random variableH(X, X2) =H(X).
Show thatH(X2|X) = 0 but thatH(X|X2) is not necessarily zero.
Ex. 6
The random variable X takes the values 1,2, . . . ,2N with equal probability. The random variableY is defined by
Y =
(0 ifX is even, 1 ifX is odd.
Evaluate H(X|Y), and show thatH(Y|X) = 0.
Ex. 7
(Think about this one for a while). Suppose you want to compress, encode and encrypt a message. (Compression for shortening the data if possible, encoding for securing against noise on the channel, encryption to keep the message secret). Does it matter in which order you do this, and if yes, in which order should you do it?
Ex. 8
Make yourself familiar with the restricted bookshelf, there are copies of the recom- mended books, (Welsh, MCKay, Jones and Jones, Hill). (The restricted bookshelf is behind the counter on the left hand side, then inside at the very end close to the copy machine, 001.5*** .. library coding).
To be returned in one week, before the lecture.
My web page contains a collection of related material.
http://www.ma.rhul.ac.uk/∼elsholtz/WWW/lectures/0506mt441/lecture.html
