CS4450 Midterm Exam

CS4450-001: Coding and Information Theory Fall 2019

Midterm Exam

Student Name ___________________

This is an open-book and take-home midterm examination. You are required to complete the examination on your own. By giving your signature below, you allege that you will not get any solutions to the examination questions from others and you will not provide your solutions to any of your classmates.

CS4450 Midterm Exam

Your signature ________________________

1. (18%) (Uniquely Decodable Code)

(1) Construct the sets 𝐶 and 𝐶 for the ternary code 𝐶 = {01, 10, 02, 21, 210}.

(2) Is the code 𝐶 = {01, 10, 02, 21, 210} considered in (1) uniquely decodable? Why? If 𝐶 is not uniquely decodable, find a code-sequence which can be decoded in at least two ways.

2. (8%) (Instantaneous Code) Find an instantaneous ternary code with word-lengths 1, 2, 2, 2, 2, 3, 3, 3, 4.

3. (24%) (Huffman Code)

(1) Construct a binary Huffman code for a source with probabilities 𝑝 = 0.3, 0.2, 0.15, 0.15, 0.1, 0.1.

and find its average word-length.

(2) Find a binary Huffman code, which has the minimal total word-length 𝜎(𝐶) = ∑ 𝑙, for a source with probabilities

𝑝 = 1/3, 1/3, 1/6, 1/6.

(3) Let 𝑆 be a source of two symbols {𝑠, 𝑠} with probabilities 3⁄4, and 1⁄4. Find the probability distribution for 𝑆, and compute the average word-length 𝐿 of binary Huffman code 𝐶 for 𝑆

4. (18%) (Entropy and Shannon-Fano Code) Given a source 𝑆 with probabilities 𝑝 = 0.3, 0.2, 0.15, 0.15, 0.1, 0.1.

(1) Calculate the entropy of 𝑆.

(2) Find the word-lengths, average word-length, and efficiency of a binary Shannon-Fano code for 𝑆.

(3) Let 𝑆 have q equiprobable symbols. Find the average word-length 𝐿 of an 𝑟-ary Shannon- Fano code for 𝑆, and verify that 𝐿 → 𝐻(𝑆) as 𝑛 → ∞.

5. (18%) (Binary Symmetric Cannel)

Let Γ be the BSC. Its input alphabet A = 𝑍 ={0, 1} and its output alphabet B = 𝑍 ={0, 1}. Its probabilities have the form

𝑝 =Pr(𝑎=0)=𝑝and𝑝 =Pr(𝑎=1)=𝑝̅=1−𝑝 for some 𝑝 such that 0 ≤ 𝑝 ≤ 1. Its channel matrix has the form

𝑀=𝑃 𝑃=𝑃 𝑃

𝑃𝑃 𝑃𝑃 for some 𝑃 where 0 ≤ 𝑃 ≤ 1, where 𝑃 is forward probability.

(1) Suppose that 𝑃 = 0.8 and 𝑝 = 0.7, compute the output probability distribution: 𝑞 and 𝑞. Note𝑞 =Pr(𝑏=0)and𝑞 =Pr(𝑏=1).

(2) Suppose that 𝑃 = 0.7 and 𝑝 = 0.8, compute the backward probabilities: 𝑄, 𝑄

(3) Prove that the necessary and sufficient condition for Γ to satisfy 𝑄 > 𝑄 is 𝑝 > 𝑃

6. (14%) (Binary Erasure Cannel)

Suppose that Γ is the binary erasure channel (BEC), the input probabilities of 0 and 1 are 𝑝 and 𝑝̅, and the probability of sending source symbols correctly is 𝑃. Assume 𝑝=0.8 and 𝑃=0.7. Please compute the system entropies H(A), H(B), H(A|B), H(B|A), and H(A,B).CS4450-MidtermExam

CS4450 Midterm Exam