2.5 Fundamentals of Information Theory
2.5.2 Capacity Limits
2.5.2.1 Shannon’s channel capacity
In Section 2.2, we described the impairments of the communication channel by its transition pdfs p rk|sk=xl
. With the set of transition pdfs and the distribution of transmit symbols, p(xl), the MI I(sk,rk)can be determined according to (2.69). In general, the transmit sym-bols xl∈Xmay be continuous-valued. Shannon introduced the channel capacity CC as the maximum MI I(sk,rk) between channel input sk and output rk among all possible distribu-tions p(xl)of an arbitrary symbol alphabetX.
CC=max
p(xl)I(sk,rk). (2.70)
Shannon’s famous channel coding theorem [4] states that reliable data communication is only possible, if the number of information bits per transmit symbol, η =Rc·M, does not exceed the channel capacity, i.e.,
η≤CC. (2.71)
In other words, the maximum possible bit rate over a given channel of bandwidth B in bits per seconds is CC·B. Thus, the dimension of CC is often given as bits/(s·Hz) or bits/channel usage. The proof of the channel coding theorem, which was later refined by Gallager [93], is not constructive in the sense that it claimes, which channel codes can pro-vide reliable communication. It only considers the ensemble of all randomly chosen codes, and is therefore also called random coding theorem. It is still an ongoing challenge to achieve close to capacity communication systems.
9Another common, yet imprecise, interpretation of I(sk,rk)is that it measures the mutual dependence be-tween skand rk. However, the same vague claim can be stated about crosscorrelation between skand rk.
Examples: For the non-fading AWGN channel, the optimum signal alphabet can be shown to be (complex) Gaussian distributed with zero mean and variance σs2, [4, 67, 93]. The corresponding channel capacity is
CC=log2 1+σs2 σn2
!
, (2.72)
where the SNR σσs22
n is the non-logarithmic version of (2.29). Hence, the AWGN channel capacity is only a function of the SNR and its inverse function is
σs2
σn2 =2CC−1. (2.73)
Setting the spectral efficiency to its maximum possible value, which is according to (2.71) η=CC, and inserting (2.73) into the non-logarithmic version of (2.31), we find
Eb
N0 = 2CC−1
CC . (2.74)
The minimum necessary ENb
0 can be computed with l’Hôpital’s rule as the limit Eb
N0
min
= lim
CC→0
2CC−1
CC = lim
CC→0
2CC·ln(2)
1 =ln(2),−1.59dB, (2.75) which means that the average energy per information bit must not fall below−1.59 dB of the noise power spectral density of the AWGN channel, to allow for any reliable (but low rate) communication at all.
As a second example, we consider the channel capacity of the BEC from Subsection 2.2.4 with erasure probability q. Clearly, the channel is doubly symmetric [93], because the tran-sition pdfs are the same for all input symbols ck and the same is true for the reciprocal channel, where input and output are interchanged. Thus, a uniform input distribution, i.e., P[cκ =0] = 12, yields channel capacity [93, 52], which equals CC=max I(ck,yk) =1−q. (2.76)
2.5.2.2 Signal set capacity
In practical digital communication systems, the signaling alphabet can not be of infinite car-dinality. For digital QAM schemes, we have|X|=L, and usually L=2Mis a power of 2. For such a discrete-valued signal set, we can define the signal set capacity CSin the same manner as in (2.70) as the maximum MI among all input distributions. However, in this thesis we re-strict ourselves to equiprobable signal sets, i.e., P[xl] =L−1. Even though no maximization is involved, we will refer to the MI as signal set capacity, implying that the constraint exists
that equiprobable QAM symbols have to be transmitted over the communication channel.
According to step(4)from (2.69), we have CS=
Z
rk∈C L−1
X
l=0
1
L·p rk|sk=xl
log2 p rk|sk=xl
1 L·LP−1
l1=0
p
rksk=xl1
drk. (2.77)
2.5.2.3 BICM capacity
The signal set capacity CS is the maximum possible signaling rate (in bits / channel usage) under the condition that equiprobable signals from a discrete set are transmitted. If we add as a further constraint on the transmission system that (optimum) demapping is performed only once, then we can compute the BICM capacity CBas the MI between transmit symbols sk and the output of the demapper. As the non-iterative demapper receives no a priori in-formation, its APP output is equal to the extrinsic L-values. It was shown in [13, 33], that a fully10 interleaved BICM schemes corresponds to the transmission of M parallel binary input channels, each of which carries one coded bit b∈ {0,1}, which determines the symbol subsetXm
b. Introducing this fact in the third step of (2.69), we get
CB =
MX−1 m=0
Eb,rk
log2 P
xl∈Xm
b
p rk|sk=xl p(rk)
= Z
rk∈C MX−1 m=0
X1 b=0
X
xl∈Xm
b
1
L·p rk|sk=xl log2
P
xl∈Xm
b
p rk|sk=xl
1 L· P
x′l∈X
p
rk sk=x′l
drk (2.78)
In the second line, expectation is taken over the joint pdf of b and rk. The factor 1L = 21M
comes from averaging over b∈ {0,1}, with P[b] =2−1 and from applying the law of total probability over all xl∈Xm
b, for which P[xl] =2−M+1holds.
Note that in contrast to the signal set capacity, which is determined by the alphabet X, the BICM capacity depends also on the bit labeling functionµ.
2.5.2.4 Comparison of capacity definitions
Figure 2.16 visualizes the previously introduced capacity definitions CC, CS, and CB. For a particular communication channel, which is indicated in the block diagram by input➁and output➂, and which is fully specified by the set of transition pdfs p rk|sk=xl
, the ultimate signaling rate for reliable communication is given by the channel capacity CC. This rate can
10Assuming an infinite depth random interleaver, which yields independent output bits.
only be achieved by optimized distributions of the transmit symbols, which are in general continuous-valued. Signal shaping techniques [32] try to approximate the optimum symbol distributions, even for a finite cardinality of the symbol alphabet.
A more realistic upper bound on the signaling rate is the signal set capacity CS, which in-cludes a certain signaling set as part of the channel. Thus, CS describes the upper limit on the signaling rate for the discrete-input (see➀), continuous-output (see➂) channel. It is well known that achieving the signal set capacity requires the receiver to perform joint demap-ping and decoding. Hence, coded modulation systems as in MLC are able to achieve this capacity, and therefore CSis sometimes also called coded modulation capacity [33]. Instead of the rather complex joint detection, iterative demodulation and decoding, as in BICM-ID, offers a close to optimum alternative. However, one must find mappings that can exploit a priori information most efficiently. These are certain anti-Gray mappings, as will be dis-cussed in this thesis. As an advantage, a rather weak channel code is sufficient for capacity approaching performance.
If the output of an optimum demapper is further included in the overall “channel”, then the BICM capacity CB is the maximum rate from input➀ to output➃. It was shown prag-matically in [13, 33] and theoretically in [94, 95], that Gray mappings achieve the largest CB. Interestingly, Definition 2.1 allows for different classes of Gray mappings. The origi-nal recursive construction according to Gray [11] yields the so called binary reflected Gray labeling, which maximizes CB for both AWGN and Rayleigh fading channel, while other classes of Gray labelings are inferior [94, 95]. In any case, a powerful channel code has to assure a performance close to the BICM capacity.
Figure 2.16: Definitions of channel, signal set, and BICM capacity
Finally, Figure 2.17 depicts the three types of capacities for an AWGN channel. The upper limit is given by Shannon’s channel capacity CCaccording to (2.72). The solid curves with markers belong to the signal set capacities CSfor BPSK, QPSK (cf. Figure 2.6), and 8-PSK, 16-QAM (cf. Figure 2.7), respectively. It is plain to see (and can be proven from (2.77)) that 0≤CS ≤M. The dashed curves correspond to the BICM capacities CB. As we will discuss in Section 3.4, 0≤CB≤CS ≤M, where equality holds only for BPSK and QPSK with Gray labeling. Thus, CB is not plotted for these two mappings. The depicted capacity curves belong to the Gray and anti-Gray labelings from Subsection 2.1.3. As mentioned
above, Gray labeling yields higher values for CB than anti-Gray labelings. But even with Gray labeling, there exist a (small, but positive) difference CS−CBfor M>2, which can be seen for medium SNR values in the range of about[−5,8]dB.
−200 −15 −10 −5 0 5 10 15 20
1 2 3 4 5 6 7
σs2 / σn2 [dB]
capacity [bits/(s⋅ Hz)]
CC
CS: BPSK CS: QPSK CS: 8−PSK CS: 16−QAM CB: QPSK, anti−Gray CB: 8−PSK, Gray CB: 8−PSK, anti−Gray CB: 16−QAM, Gray CB: 16−QAM, anti−Gray
Figure 2.17: Capacities for different mappings over AWGN channel