Lecture 1.Samples of possibility and impossibility results in algorithm designing The course is about the ultimate efficiency that algorithms and communication protocols can achieve in various settings.In the first lecture,we'll see a couple ofcleverly designed algorithms/protocols, and also prove several impossibility results to show limits ofalgorithms in various computation settings 1.Communication complexity. 1.1 Setup In information theory,we are usually concerned with transmitting a message through a noisy channel In communication complexity,we consider the problem of computing a function with input variables distributed to two(or more)parties.Since each party only holds part of the input,they need to communicate in order to compute the function value.The communication complexity measures the minimum communication needed. The most common communication models include two-way:two parties,Alice and Bob,holding inputs x and y,respectively,talk to each other back and forth. one-way:two parties,Alice and Bob,holding inputs x andy,respectively,but only Alice sends a message to Bob. Simultaneous Message Passing (SMP):Three parties:Alice,Bob and Referee.Alice and Bob hold inputs x and y,respectively,and they each send one message to Referee,who then outputs an answer. multiparty:k parties,each holding some input variables and they communicate in a certain way. Some example functions are: Equality (EQ):To decide whether x=y,where x.ye(0,1)" Inner Product (IP):To compute (x,y)=xiy1+...+xnyn mod 2,where x.ye(0,1)n Disjointness (Disj):To decide whether 3is.t.x;=y;=1 for x,ye(0,1)". The computation modes include
Lecture 1. Samples of possibility and impossibility results in algorithm designing The course is about the ultimate efficiency that algorithms and communication protocols can achieve in various settings. In the first lecture, we’ll see a couple of cleverly designed algorithms/protocols, and also prove several impossibility results to show limits of algorithms in various computation settings. 1. Communication complexity. 1.1 Setup In information theory, we are usually concerned with transmitting a message through a noisy channel. In communication complexity, we consider the problem of computing a function with input variables distributed to two (or more) parties. Since each party only holds part of the input, they need to communicate in order to compute the function value. The communication complexity measures the minimum communication needed. The most common communication models include two-way: two parties, Alice and Bob, holding inputs x and y, respectively, talk to each other back and forth. one-way: two parties, Alice and Bob, holding inputs x and y, respectively, but only Alice sends a message to Bob. Simultaneous Message Passing (SMP): Three parties: Alice, Bob and Referee. Alice and Bob hold inputs x and y, respectively, and they each send one message to Referee, who then outputs an answer. multiparty: k parties, each holding some input variables and they communicate in a certain way. Some example functions are: Equality (EQn): To decide whether x = y, where x,y{0,1}n . Inner Product (IPn): To compute x,y= x1y1 + … + xnyn mod 2, where x,y{0,1}n . Disjointness (Disjn): To decide whether i s.t. xi = yi = 1 for x,y{0,1}n . The computation modes include
deterministic:all messages are deterministically specified in the protocol.Namely,each message of any party depends only on her input and the previous messages that she saw. randomized:parties can use randomness to decide the messages they send. quantum:the parties have quantum computers and they send quantum messages.(We will not study this mode,or in general,any quantum computing topic,in this course.) In the two-way and one-way model,we say that a protocol computes the function if on all possible input instances,at the end ofthe protocol,both parties know the function value on the input.(In the SMP model,the Referee is required to know the function value.)For randomized protocol,we sometimes allow a small error probability.The cost of a protocol is measured by the total communication bits.The communication complexity of a function f is the minimum cost of any protocol that computes the function.We ignore the issue of local computation cost by assuming that all parties are computationally all-powerful.Though all the computation factors are gone and only communication is focused on,it turns out to be very helpful for communication complexity to be exploited to prove impossibility results for other computational complexity.(More on this later.) 1.2 An efficient deterministic protocol Let's consider the basic case of two parties.Note that ifx and y are both at most n-bit long,then at the very least Alice can send her input to Bob,who then can compute f(x,y)since he has both inputs. Finally he tells Alice the answer by 1 bit communication.Thus the communication complexity ofany 2n-bit function is at most n+1. What makes communication complexity interesting on its own is that some functions enjoy much more efficient protocols. Example 1.1.Max.Alice and Bob hold x,ye(0,1),respectively.Interpret x as a subset of [n]= (1,2,...,n);the subset contains the positions is.t.x:=1.Similarly fory.The task is to compute Max(x,y),the maximal number in x Uy.If Alice sends her entire input,it takes n communication bits. But she can clearly do something better:She merely needs to send the maximal number in her set x, which only takeslog(n)bits.Bob then compares this value with his maximal number in y,and sends the larger one back to Alice as the answer.Altogether the protocol needs only 2l0g(n)bits. This protocol seems trivial,but some others need more thoughts
deterministic: all messages are deterministically specified in the protocol. Namely, each message of any party depends only on her input and the previous messages that she saw. randomized: parties can use randomness to decide the messages they send. quantum: the parties have quantum computers and they send quantum messages. (We will not study this mode, or in general, any quantum computing topic, in this course.) In the two-way and one-way model, we say that a protocol computes the function if on all possible input instances, at the end of the protocol, both parties know the function value on the input. (In the SMP model, the Referee is required to know the function value.) For randomized protocol, we sometimes allow a small error probability. The cost of a protocol is measured by the total communication bits. The communication complexity of a function f is the minimum cost of any protocol that computes the function. We ignore the issue of local computation cost by assuming that all parties are computationally all-powerful. Though all the computation factors are gone and only communication is focused on, it turns out to be very helpful for communication complexity to be exploited to prove impossibility results for other computational complexity. (More on this later.) 1.2 An efficient deterministic protocol Let’s consider the basic case of two parties. Note that if x and y are both at most n-bit long, then at the very least Alice can send her input to Bob, who then can compute f(x,y) since he has both inputs. Finally he tells Alice the answer by 1 bit communication. Thus the communication complexity of any 2n-bit function is at most n+1. What makes communication complexity interesting on its own is that some functions enjoy much more efficient protocols. Example 1.1. Max. Alice and Bob hold x,y{0,1}n , respectively. Interpret x as a subset of [n] = {1,2,…,n}; the subset contains the positions i s.t. xi = 1. Similarly for y. The task is to compute Max(x,y), the maximal number in x∪y. If Alice sends her entire input, it takes n communication bits. But she can clearly do something better: She merely needs to send the maximal number in her set x, which only takes log(n) bits. Bob then compares this value with his maximal number in y, and sends the larger one back to Alice as the answer. Altogether the protocol needs only 2log(n) bits. This protocol seems trivial, but some others need more thoughts
Example 1.2.Median.Alice and Bob again hold subsets x,ye(0,1),respectively.Now they wish to compute the median of the multi-set x Uy.Recall that the median ofk numbers is defined as the Lk/2th smallest number. An efficient protocol is as follows.At any point,they maintain an interval [l,u]which guarantees to contain the median.Initially I=I and u=n.They halve the interval in each round,so it takes only log(n)rounds to finish.How do they halve the interval?Actually very simple:Alice sends to Bob the number ofher elements that are at least m=(1+u)/2,and the number ofelements that are less than m. This information is enough for Bob to know whether the median is above m or not.(Verify this!)Bob uses I bit to indicate the answer,and then the protocol goes to the next round. Since each round hasonly O(log(n))bits,and there are only log(n)rounds,the total communication cost is O(log2(n)),much more efficient than the trivial protocol of cost n+1. 1.3 Hardness for deterministic and rescue by randomness We've seen examples ofefficient protocols.On the other hand,for some problems,there isn't any protocol using communication less than n bits. First,let's observe that a deterministic communication protocol partitions the communication matrix into monochromatic rectangles.Here for a function f(x,y),the communication matrix M is defined by [f(x,y)l),namely the rows are indexed by x and the columns are indexed by y,and the(x,y)-entry is f(x,y).A rectangle is a pair of subsets(S,T)where S is a set of rowsand T is a set of columns.A rectangle (S,T)is monochromatic if Me restricted on it is a submatrix containing only one value. (Namely,all entries in the submatrix are of the same value.)For any value b in range(f),a b-rectangle is a monochromatic rectangle in which all entries of Mr are of value b.By considering how the protocol goes round-by-round,we can observe the following fact. Fact 1.1.A c-bit deterministic communication protocol for fpartitions the communication matrix M into 2 monochromatic rectangles. Example 1.3.Equality. The above fact enables us to show that there is no efficient protocol for solving the Equality function EQ.Indeed,the communication matrix for EQ is IN,the identity matrix of size N=2.Notice that for this matrix,all 1-rectangles are of size 1x1.So no matter how one partitions the matrix into
Example 1.2. Median. Alice and Bob again hold subsets x,y{0,1}n , respectively. Now they wish to compute the median of the multi-set x∪y. Recall that the median of k numbers is defined as the k/2 th smallest number. An efficient protocol is as follows. At any point, they maintain an interval [l,u] which guarantees to contain the median. Initially l = 1 and u = n. They halve the interval in each round, so it takes only log(n) rounds to finish. How do they halve the interval? Actually very simple: Alice sends to Bob the number of her elements that are at least m = (l+u)/2, and the number of elements that are less than m. This information is enough for Bob to know whether the median is above m or not. (Verify this!) Bob uses 1 bit to indicate the answer, and then the protocol goes to the next round. Since each round has only O(log(n)) bits, and there are only log(n) rounds, the total communication cost is O(log2 (n)), much more efficient than the trivial protocol of cost n+1. 1.3 Hardness for deterministic and rescue by randomness We've seen examples of efficient protocols. On the other hand, for some problems, there isn't any protocol using communication less than n bits. First, let’s observe that a deterministic communication protocol partitions the communication matrix into monochromatic rectangles. Here for a function f(x,y), the communication matrix Mf is defined by [f(x,y)](x,y), namely the rows are indexed by x and the columns are indexed by y, and the (x,y)-entry is f(x,y). A rectangle is a pair of subsets (S,T) where S is a set of rows and T is a set of columns. A rectangle (S,T) is monochromatic if Mf restricted on it is a submatrix containing only one value. (Namely, all entries in the submatrix are of the same value.) For any value b in range(f), a b-rectangle is a monochromatic rectangle in which all entries of Mf are of value b. By considering how the protocol goes round-by-round, we can observe the following fact. Fact 1.1. A c-bit deterministic communication protocol for f partitions the communication matrix Mf into 2 c monochromatic rectangles. Example 1.3. Equality. The above fact enables us to show that there is no efficient protocol for solving the Equality function EQn. Indeed, the communication matrix for EQn is IN, the identity matrix of size N=2n . Notice that for this matrix, all 1-rectangles are of size 11. So no matter how one partitions the matrix into
monochromatic rectangles,there are always N 1-rectangles.Considering that there are also 0-rectangles,we have 2>N=2,so c>n.We have thus proved the following theorem. Theorem 1.2.Any deterministic protocol computing EQn needs n+1 bits of communication. Despite the above negative result,next we'll see that the Equality function does enjoy a very efficient (and one-way!)protocol if we are allowed to use some randomness in the protocol.Later in the course, we will see that whether the randomness is shared by the two parties or not doesn't affect the communication complexity much.So let's assume that Alice and Bob have shared randomness;we call such protocols public-coin. Theorem 1.3.There is a one-way public-coin randomized protocol computing EQ with only O(1) bits of communication.(Onany input(x.y),the protocol outputs the correct f(x,y)with probability 99%.) Let's first design a protocol with 50%1-sided error probability,and later boost the success probability. The protocol is as follows. ·Alice: compute (x,r)=xir+...+xr mod 2,where r is a public random string of length n, send the 1-bit result to Bob. ·Bob compute (y,r)=yir+...+yarn mod 2,where r is the same random string that Alice used. Compare(x,r〉andy,r〉.Output“xy”ifx,r）=y,r）and“xy”ifx,ry,r〉. Note that the communication cost of the protocol is only I bit.Let's see what the protocol does.To compare x and y using little communication,Alice tries to give a very short summary ofher input x, and send only this summary to Bob,who computes the same summary of his input y.The summary is so short that it's only I bit,so it contains very very little information of the input string.So...could this possibly work? Let's analyze it case by case.If x=y,then ofcourse (x,r)=(y,r),and the protocol is correct with certainty.Ifxy,we claim that (x,r)(y,r)with probability exactly 1/2!(Namely,if xy,then with half probability,the I-bit summary can catch this distinction.)Actually,when xy,they differ at at least one position.Say it's position i.then
monochromatic rectangles, there are always N 1-rectangles. Considering that there are also 0-rectangles, we have 2 c > N = 2n , so c > n. We have thus proved the following theorem. Theorem 1.2. Any deterministic protocol computing EQn needs n+1 bits of communication. Despite the above negative result, next we’ll see that the Equality function does enjoy a very efficient (and one-way!) protocol if we are allowed to use some randomness in the protocol. Later in the course, we will see that whether the randomness is shared by the two parties or not doesn’t affect the communication complexity much. So let’s assume that Alice and Bob have shared randomness; we call such protocols public-coin. Theorem 1.3. There is a one-way public-coin randomized protocol computing EQn with only O(1) bits of communication. (On any input (x,y), the protocol outputs the correct f(x,y) with probability 99%.) Let’s first design a protocol with 50% 1-sided error probability, and later boost the success probability. The protocol is as follows. Alice: ▪ compute x,r = x1r1 + … + xnrn mod 2, where r is a public random string of length n, ▪ send the 1-bit result to Bob. Bob: ▪ compute y,r = y1r1 + … + ynrn mod 2, where r is the same random string that Alice used. ▪ Compare x,r and y,r Output “x=y” if x,r = y,r and “x≠y” if x,r≠y,r Note that the communication cost of the protocol is only 1 bit. Let’s see what the protocol does. To compare x and y using little communication, Alice tries to give a very short summary of her input x, and send only this summary to Bob, who computes the same summary of his input y. The summary is so short that it’s only 1 bit, so it contains very very little information of the input string. So … could this possibly work? Let’s analyze it case by case. If x = y, then of course x,r= y,r, and the protocol is correct with certainty. If x ≠ y, we claim that x,r ≠ y,r with probability exactly 1/2! (Namely, if x≠y, then with half probability, the 1-bit summary can catch this distinction.) Actually, when x≠y, they differ at at least one position. Say it’s position i. then
k,r)-y,r〉=x-y,r）=(&-y)r1+.+(xnyn)r mod2 =(X-yr:+∑i(-y)mod2 =r+∑j(X-y)加mod2. Now notice that ri takes value 0 and I each with half probability,so regardless of the value of the second item,the summation is I always with probability half.Thus with probability half the protocol detects that x≠y. To make the error probability smaller,one can simply repeat the above protocol k times,dropping the error probability to 1/2k 1.4 Using communication complexity to show computational hardness:a taste Apart from being a very natural and interesting subject on its own,communication complexity is also widely used to prove limitation results for numerous computational models,including data structures, circuit complexity,streaming algorithms,decision tree complexity,VLSI,algorithmic game theory, optimization,pseudo-randomness...Here we just give one such example,with more coming up later in the course. Example 1.4.Lower bound on time-space tradeoff on TM for explicit decision problems. First let's recall the concept of multi-tape Turing machine(TM).Below I just copied the short description in [KN97],which is pretty intuitive without losing much rigor.For detailed definition,you can see standard textbooks such as [Sip02],or the wiki links above
x,r - y,r = x-y,r = (x1-y1)r1 + … + (xn-yn)rn mod 2 = (xi-yi)ri + ∑j≠i (xj-yj)rj mod 2 = ri + ∑j≠i (xj-yj)rj mod 2. Now notice that ri takes value 0 and 1 each with half probability, so regardless of the value of the second item, the summation is 1 always with probability half. Thus with probability half the protocol detects that x ≠ y. To make the error probability smaller, one can simply repeat the above protocol k times, dropping the error probability to 1/2k . 1.4 Using communication complexity to show computational hardness: a taste Apart from being a very natural and interesting subject on its own, communication complexity is also widely used to prove limitation results for numerous computational models, including data structures, circuit complexity, streaming algorithms, decision tree complexity, VLSI, algorithmic game theory, optimization, pseudo-randomness… Here we just give one such example, with more coming up later in the course. Example 1.4. Lower bound on time-space tradeoff on TM for explicit decision problems. First let’s recall the concept of multi-tape Turing machine (TM). Below I just copied the short description in [KN97], which is pretty intuitive without losing much rigor. For detailed definition, you can see standard textbooks such as [Sip02], or the wiki links above