2Complexityand EntropyExercise 2.4Consider a system of N spin-1/2 particles lined up in a row.Distinct states ofthe N-particle system,have different spatial ordering of the spins-up ↑ and spins-down1.ForN=10,onesuch statemightbetutittt/t.Howmanymicroscopicstates (different configurations of the N particle spin system) does this systemhave?Answer:Usethe multiplication principle.Thefirst spin has two configurations,Tand l.After theconfiguration ofthefirst spin is set,the second spin can exist inoneofthesetwoconfigurations,andsoon.Thus,thetotal numberofmicroscopicstates is2×2×.×2=2N.For N=10,the number ofmicroscopic states is1024.For N = 1000 the number of microscopic states is 1.07151 ×x10301.For a smallmagnetic crystal with N = 1023 atoms, the number of microscopic spin states issolargethat itisbeyondcomprehension.Exercise 2.5Takea bag ofN distinct coins (each coin froma different country and each havingone side with thepicture of a head on it)and dump themon thefloor.How manydifferent ways can the coins have n heads facing up?Answer:First ask a different question.Howmany different ways can N distinctcoins be assigned to n pots (one coin per pot)? There are N distinct ways to assigna coin to the first pot and, after that is done, N -1 distinct ways to assign theremainingN-1coinstothesecond pot,..,andN-n+1waystoassigntheremaining coins to the nth pot.Thus,thetotal number of distinct ways to assigntheNcoinstonpotsisNx(N-1)x...x(N-n+1)=N!/(N-n)!.Nownotethat permutation of the coins, among the pots,doesn't give a different answer,so we must divide by n!.Thus, thenumber of distinct ways to assign n heads toNdistinctcoins isN(n)=N!/(n(N-n)).As we will see,these countingrules areextremelyimportantwhen weattempttocount the different microscopic states available to a quantum system containingNparticles.The symmetry properties of the Liouvilleoperator orHamiltonianoperator, under interchangeof the particles,determines whether theparticles areidentical ordistinct.Thenumber of microscopic statesavailabletothesystem,and therefore its physical properties,are very different for these two cases.Con-sider the example discussed in Exercise 2.5. If we have N distinct coins and dropthem on thefloor,the number of distinct waysto assign n"heads"tothecoins(have n"heads"face up)is N(n)=N!/(n!(N-n).However, if the coins areidentical (all US quarters)the number of distinct ways that n“heads"can face upis(n)=1.The question of whether the particles comprising a system are distinct or iden-ticalhasmeasurablephysical consequencesbecausethenumberof microscopicstates available to the system is very different for the two cases.As we have seen,thenumberof microscopic states available to a collection ofN particles is gener-ally huge
8 2 Complexity and Entropy Exercise 2.4 Consider a system of N spin-1∕2 particles lined up in a row. Distinct states of the N-particle system, have different spatial ordering of the spins-up ↑ and spinsdown ↓. For N = 10, one such state might be ↑↓↓↑↓↑↑↑↓↑. How many microscopic states (different configurations of the N particle spin system) does this system have? Answer: Use the multiplication principle. The first spin has two configurations, ↑ and ↓. After the configuration of the first spin is set, the second spin can exist in one of these two configurations, and so on. Thus, the total number of microscopic states is 2 × 2 × ⋯ × 2 = 2N . For N = 10, the number of microscopic states is 1024. For N = 1000 the number of microscopic states is 1.071 51 × 10301. For a small magnetic crystal with N = 1023 atoms, the number of microscopic spin states is so large that it is beyond comprehension. Exercise 2.5 Take a bag of N distinct coins (each coin from a different country and each having one side with the picture of a head on it) and dump them on the floor. How many different ways can the coins have n heads facing up? Answer: First ask a different question. How many different ways can N distinct coins be assigned to n pots (one coin per pot)? There are N distinct ways to assign a coin to the first pot and, after that is done, N − 1 distinct ways to assign the remaining N − 1 coins to the second pot, ., and N − n + 1 ways to assign the remaining coins to the nth pot. Thus, the total number of distinct ways to assign the N coins to n pots is N × (N − 1)×.×(N − n + 1) = N!∕(N − n)!. Now note that permutation of the coins, among the pots, doesn’t give a different answer, so we must divide by n!. Thus, the number of distinct ways to assign n heads to N distinct coins is (n) = N!∕(n!(N − n)!). As we will see, these counting rules are extremely important when we attempt to count the different microscopic states available to a quantum system containing N particles. The symmetry properties of the Liouville operator or Hamiltonian operator, under interchange of the particles, determines whether the particles are identical or distinct. The number of microscopic states available to the system, and therefore its physical properties, are very different for these two cases. Consider the example discussed in Exercise 2.5. If we have N distinct coins and drop them on the floor, the number of distinct ways to assign n “heads” to the coins (have n “heads” face up) is (n) = N!∕(n!(N − n)!). However, if the coins are identical (all US quarters) the number of distinct ways that n “heads” can face up is (n) = 1. The question of whether the particles comprising a system are distinct or identical has measurable physical consequences because the number of microscopic states available to the system is very different for the two cases. As we have seen, the number of microscopic states available to a collection of N particles is generally huge
2.3 Probability2.3ProbabilityOnce wehaveidentifiedthemicroscopicstatesofa system,wecanask whatmightbeobserved inanexperiment.Becausethenumberofmicroscopicstates associat-ed with a macroscopic system is so large,the outcome of an experiment generallywillbedifferenteverytimeitisperformed.However,ifweperformanexperimentmanytimes,wecanbegintoassignquantitativeweights(probabilities)tothevariousoutcomes,consistentwiththeprobabilitiesassociatedwiththemicroscopicstates.This relation between the outcome of experiments and the probabilitiesassigned to those outcomes is the content of the Central Limit Theorem (see Ap-pendix A).The simplest situation (and one very common in nature) is one in which themicroscopic statesareallequallylikelytooccur.Then,ifwehaveNmicroscopicstates,x, (j=1,+*,N),the probabilitythat the state x,appears as aresultofanexperiment isP(x,)=1/N.The entirecollection of microscopic states,withtheirassigned probabilities,forms a sample space S.An event is the outcomeofanexperiment,and itcan involveone or more micro-scopic states. Let us consider two events, A and B, each of which involves severalmicroscopicstates.LetP(A)(P(B))denotetheprobabilitythateventA (B)oc-curs as the outcome of the experiment. The probability P(A) (P(B)) is the sum oftheprobabilities ofall themicroscopicstatesthatcomprisetheeventA (B).If theevent includestheentiresamplespacethenP(S)=1and,iftheeventincludesnoelements of the sample space so A = ( denotes an empty set), then P() = 0.Theunion ofeventsAand B(denotedAUB)contains all microscopicstatesthatparticipate in bothevents.The intersection ofeventsA and B (denoted AnB)con-tains all microscopic states shared bythe two events.Therefore, the probabilitythat both events occur as aresult of an experiment is the probability of theunion,which can bewritten(2.1)P(A U B) = P(A) + P(B) - P(A n B),where P(A n B) is the probability associated with microscopic states in the inter-section.WhenweaddtheprobabilitiesP(A)andP(B),wecountthestatesAnBtwice,sowecorrectthismistakebysubtracting off onefactor of P(AnB)IfthetwoeventsAandBaremutuallyexclusive,thentheyhavenomicroscopicstatesincommonandP(A U B) = P(A) + P(B) .(2.2)We can partition the sample space into a complete set of mutually exclusiveevents Ai,A2**, Am so that A, U A, U ...U Am = S. Then, the probabilities as-sociatedwith the m events satisfy thecondition(2.3)P(A,)+ P(A2) +.. + P(Am) =1 .This partitioning of the sample space will prove extremely useful in subsequentchapters
2.3 Probability 9 2.3 Probability Once we have identified the microscopic states of a system, we can ask what might be observed in an experiment. Because the number of microscopic states associated with a macroscopic system is so large, the outcome of an experiment generally will be different every time it is performed. However, if we perform an experiment many times, we can begin to assign quantitative weights (probabilities) to the various outcomes, consistent with the probabilities associated with the microscopic states. This relation between the outcome of experiments and the probabilities assigned to those outcomes is the content of the Central Limit Theorem (see Appendix A). The simplest situation (and one very common in nature) is one in which the microscopic states are all equally likely to occur. Then, if we have N microscopic states, xj (j = 1, . , N), the probability that the state xj appears as a result of an experiment is P(xj) = 1∕N. The entire collection of microscopic states, with their assigned probabilities, forms a sample space . An event is the outcome of an experiment, and it can involve one or more microscopic states. Let us consider two events, A and B, each of which involves several microscopic states. Let P(A) (P(B)) denote the probability that event A (B) occurs as the outcome of the experiment. The probability P(A) (P(B)) is the sum of the probabilities of all the microscopic states that comprise the event A (B). If the event includes the entire sample space then P() = 1 and, if the event includes no elements of the sample space so A = ∅ (∅ denotes an empty set), then P(∅) = 0. The union of eventsA and B (denoted A∪B) contains all microscopic states that participate in both events. The intersection of eventsA and B (denoted A ∩ B) contains all microscopic states shared by the two events. Therefore, the probability that both events occur as a result of an experiment is the probability of the union, which can be written P(A ∪ B) = P(A) + P(B) − P(A ∩ B) , (2.1) where P(A ∩ B) is the probability associated with microscopic states in the intersection. When we add the probabilities P(A) and P(B), we count the states A ∩ B twice, so we correct this mistake by subtracting off one factor of P(A ∩ B). If the two events A and B are mutually exclusive, then they have no microscopic states in common and P(A ∪ B) = P(A) + P(B) . (2.2) We can partition the sample space into a complete set of mutually exclusive events A1, A2 , . , Am so that A1 ∪ A2 ∪ ⋯ ∪ Am = S. Then, the probabilities associated with the m events satisfy the condition P(A1) + P(A2) + ⋯ + P(Am) = 1 . (2.3) This partitioning of the sample space will prove extremely useful in subsequent chapters
2Complexityand EntropyExercise 2.6Three distinguishable coins (labeled a,b,and c) are tossed. The coins areeach fairso"heads" (h)and"tails" (t) are equally likely. (a) Find the probability of getting noheads.(b)Find the probability ofgetting at least twoheads.(c) Showthattheeventheadson coinaandtheeventtailson coinsband care independent.(d)Showthatthe event only two coins heads and the event three coins heads are dependent andmutually exclusive. (e) Find the conditional probability that, given heads on coina, coinbwillbetails.Constructa sample spaceof thefollowingequallyprobableevents:Answer:(a,b,c) = ((h,h,h), (h, h, t),(h,t,h), (h,t,t),(t, h,h), (t,h,t),(t,t,h),(t,t,t)).(a) The probability of no heads = 1/8. (b) The probability of at least two heads=1/2.(c)DefineeventA="heads on thefirst coin"Define eventB=“tailson the last two coins"Then P(A) = 1/2 and P(B) = 1/4. The union, A U Bhasprobability,P(AUB)=5/8.Thus,theprobabilityoftheintersection isP(A n B) = P(A) + P(B) - P(A U B) = 1/8 = P(A) × P(B). Thus, the events,A and B are independent. (d) Define event C ="only two coins heads"Defineevent D = "three coins heads"Then P(C) = 3/8 and P(D) =1/8. The union,CUDhasprobability,P(CUD)=1/2.Thus,theprobabilityof theintersectionis P(C n D) = P(C) + P(D) - P(C u D) = 0 + P(C) × P(D). Thus, the events CandDaredependent and aremutuallyexclusive.(e)Use as the samplespaceallevents with heads on coin a.This new sample space hasfour states.The condi-tional probability that, given coin a is heads, then coin b will be tails is 1/2.The eventsA and B are independent if(2.4)P(A n B) = P(A)P(B) :Note that independent eventshave some microscopic states in commonbecauseP(A nB)+ O. It is important to note that independent events are not mutuallyexclusive events.Another important quantity is the conditional probability P(B|A),defined asthe probability of event A, using event B as the sample space (rather than S). Theconditional probabilityis defined by the equationP(An B)(2.5)P(B|A) :P(B)SinceP(AnB)=P(BnA),wefindtheuseful relation(2.6)P(A)P(A|B) = P(B)P(B|A) -From Eqs. (2.4)and (2.5), we see that, if A and Bare independent,then(2.7)P(B|A) = P(A) .In Exercise 2.6, we illustrate all these aspects of probability theory for a simplecoin-tossexperiment
10 2 Complexity and Entropy Exercise 2.6 Three distinguishable coins (labeled a, b, and c) are tossed. The coins are each fair so “heads” (h) and “tails” (t) are equally likely. (a) Find the probability of getting no heads. (b) Find the probability of getting at least two heads. (c) Show that the event heads on coin a and the event tails on coins b and c are independent. (d) Show that the event only two coins heads and the event three coins heads are dependent and mutually exclusive. (e) Find the conditional probability that, given heads on coin a, coin b will be tails. Answer: Construct a sample space of the following equally probable events: (a, b, c) = {(h, h, h), (h, h, t), (h, t, h), (h, t, t), (t, h, h), (t, h, t), (t, t, h), (t, t, t)}. (a) The probability of no heads = 1∕8. (b) The probability of at least two heads = 1∕2. (c) Define event A = “heads on the first coin.” Define event B = “tails on the last two coins.” Then P(A) = 1∕2 and P(B) = 1∕4. The union, A ∪ B has probability, P(A ∪ B) = 5∕8. Thus, the probability of the intersection is P(A ∩ B) = P(A) + P(B) − P(A ∪ B) = 1∕8 = P(A) × P(B). Thus, the events, A and B are independent. (d) Define event C = “only two coins heads.” Define event D = “three coins heads.” Then P(C) = 3∕8 and P(D) = 1∕8. The union, C ∪ D has probability, P(C ∪ D) = 1∕2. Thus, the probability of the intersection is P(C ∩ D) = P(C) + P(D) − P(C ∪ D) = 0 ≠ P(C) × P(D). Thus, the events C and D are dependent and are mutually exclusive. (e) Use as the sample space all events with heads on coin a. This new sample space has four states. The conditional probability that, given coin a is heads, then coin b will be tails is 1∕2. The events A and B are independent if P(A ∩ B) = P(A)P(B) . (2.4) Note that independent events have some microscopic states in common because P(A ∩ B) ≠ 0. It is important to note that independent events are not mutually exclusive events. Another important quantity is the conditional probability P(B|A), defined as the probability of event A, using event B as the sample space (rather than S). The conditional probability is defined by the equation P(B|A) = P(A ∩ B) P(B) . (2.5) Since P(A ∩ B) = P(B ∩ A), we find the useful relation P(A)P(A|B) = P(B)P(B|A) . (2.6) From Eqs. (2.4) and (2.5), we see that, if A and B are independent, then P(B|A) = P(A) . (2.7) In Exercise 2.6, we illustrate all these aspects of probability theory for a simple coin-toss experiment
2.4Multiplicity and Entropyof Macroscopic Physical StatesIn the next sections, we consider several different physical systems and determine how the number of microscopic states,and their probability distributions,dependonphysical parametersofthosesystems.2.4Multiplicityand Entropyof MacroscopicPhysical StatesFor a dynamical system with N interacting particles (3N degrees of freedom in 3Dspace),there will bea verylargemultiplicityN (number)ofmicroscopic statesavailable to the system.In addition,a fewconservation laws will allow us to definea set ofmacroscopicstatesthat areparametrized bythevaluesof theconservedquantities.Two of the most important conserved quantities associated with aninteracting many-body system are the particle number (assuming no chemicalreactions occur) and the total energy of the system.However, there can be oth-er conserved quantities. For example, for a lattice of spin-1/2 particles, the spinis a measure of a conserved internal angular momentum of each particle. Spincannot bedestroyedbyinteractions between the particles or with externalforces.Therefore, the spin provides an additional parameter (along with particle numberand total energy)that can be used to specifythe state ofan N-particle spin lattice.We can assign a macroscopic variable,thenumber n of"spinsup,'to the system.EachvalueofthemacroscopicvariablenhasamultiplicityofNr(n)microscopicstates associated to it.The total energy is generally proportional to the number of degrees of freedomofthesystem.Whenwediscussthermodynamics wealsoneed ameasureof themultiplicity ofa system that is proportional to the number ofdegrees offreedom.Thatquantityis theentropy,S.TheentropyofanN-particlesystemwithenergyEand macroscopic states characterized by a parameter n, is defined(2.8)S(N,E,n) = kg In(N(E,n).Thequantity kg =1.38×10-23 J/K is Boltzmann's constant. This expression forthe entropy implicitly assumes that all microscopic states with the same valuesof N,E, and n have the same weight. Another way to say this is that all such mi-croscopic states are equally probable.The fact that all microscopic states with the same energy are equally probable,derives from the ergodic theorem, which has its origins in classical mechanics.A classical mechanical systemis ergodic if it spends equal times in equal areasof the mechanical energy surface. All fully chaotic mechanical systems have thisproperty,and it is thefoundation upon which statistical mechanics is built.It un-derlies everything we talk about in this book.In subsequent sections,wewill compute the multiplicity and entropy of thefourphysical systems; a spin system, a polymer chain, an Einstein solid, and an idealgas
2.4 Multiplicity and Entropy of Macroscopic Physical States 11 In the next sections, we consider several different physical systems and determine how the number of microscopic states, and their probability distributions, depend on physical parameters of those systems. 2.4 Multiplicity and Entropy of Macroscopic Physical States For a dynamical system with N interacting particles (3N degrees of freedom in 3D space), there will be a very large multiplicity N (number) of microscopic states available to the system. In addition, a few conservation laws will allow us to define a set of macroscopic states that are parametrized by the values of the conserved quantities. Two of the most important conserved quantities associated with an interacting many-body system are the particle number (assuming no chemical reactions occur) and the total energy of the system. However, there can be other conserved quantities. For example, for a lattice of spin-1∕2 particles, the spin is a measure of a conserved internal angular momentum of each particle. Spin cannot be destroyed by interactions between the particles or with external forces. Therefore, the spin provides an additional parameter (along with particle number and total energy) that can be used to specify the state of an N-particle spin lattice. We can assign a macroscopic variable, the number n of “spins up,” to the system. Each value of the macroscopic variable n has a multiplicity of N (n) microscopic states associated to it. The total energy is generally proportional to the number of degrees of freedom of the system. When we discuss thermodynamics we also need a measure of the multiplicity of a system that is proportional to the number of degrees of freedom. That quantity is the entropy, S. The entropy of an N-particle system with energy E and macroscopic states characterized by a parameter n, is defined S(N, E, n) = kB ln(N (E, n)) . (2.8) The quantity kB = 1.38 × 10−23 J∕K is Boltzmann’s constant. This expression for the entropy implicitly assumes that all microscopic states with the same values of N, E, and n have the same weight. Another way to say this is that all such microscopic states are equally probable. The fact that all microscopic states with the same energy are equally probable, derives from the ergodic theorem, which has its origins in classical mechanics. A classical mechanical system is ergodic if it spends equal times in equal areas of the mechanical energy surface. All fully chaotic mechanical systems have this property, and it is the foundation upon which statistical mechanics is built. It underlies everything we talk about in this book. In subsequent sections, we will compute the multiplicity and entropy of the four physical systems; a spin system, a polymer chain, an Einstein solid, and an ideal gas
2 Complexity and Entropy2.5MultiplicityandEntropyofaSpinSystemConsider a collectionofN spin-1/2 atoms arranged on a lattice.The spin is a mea-sureof quantizedangularmomentuminternaltotheatom.Spin-1/2atomshavea magnetic moment and magnetic field associated with them due to the intrin-sicchargecurrents thatgive riseto thespin,Generallywhen an arrayof spin-1/2atomsisarrangedonalattice,thevariousatomswill interactwithoneanotherviatheir magnetic fields. These interactions give rise to many interesting propertiesofsuchlattices,includingphasetransitions.We will discuss these in latersectionsof thebook.2.5.1MultiplicityofaSpinSystemSince the atoms arefixed to their respectivelattice sites, they can be distinguishedby their position on thelattice and therefore are distinct. Let n denotethe numberof atoms with spin up (t).Note that for thisproblem,themethod of countingmicroscopic states is the same as that for the bag of N coins in Exercise 2.5. Thenumber ofdistinctwaystoassignn spins"up"isthesameas the numberofdistinctways that N distinct objects can be assigned to n pots, assuming their orderingamongthepotsdoesnotmatter.Thus,themultiplicityof themacroscopicstate"nspinsup"isN!(2.9)Nr(n) =n!(N-n)!This is the number of microscopic states availableto thelatticeforthe givenvalueofn.As a check,letus sum overall possiblevaluesn =O,l,...,N.IfwemakeuseofthebinomialtheoremNN!(a+b)N=N-nbn(2.10)= n(N-n)and seta=b=1we can use Eq.(2.9)to obtain thetotal number ofmicrostatesNNN!Nn(n) =)= 2NNN=(2.11)=0 n(N- n)!N=0Thus, the sum of all the microstates contained in the macrostates gives 2N, as itshould.Notethat our ability to count the number ofmicroscopic states is due tothe fact that the angular momentum intrinsic to the atoms is quantized and isaconsequenceofthequantumnatureofmatterLet us nowfocus on the limit N-→co,and considerthebehaviorofthefractionofmicrostates,F(n)=(Nx(n))/(NN)withnspins"up,N!N!11(2.12)F(n)=n!(N-n)!2N2n!(N-n)
12 2 Complexity and Entropy 2.5 Multiplicity and Entropy of a Spin System Consider a collection of N spin-1∕2 atoms arranged on a lattice. The spin is a measure of quantized angular momentum internal to the atom. Spin-1∕2 atoms have a magnetic moment and magnetic field associated with them due to the intrinsic charge currents that give rise to the spin. Generally when an array of spin-1∕2 atoms is arranged on a lattice, the various atoms will interact with one another via their magnetic fields. These interactions give rise to many interesting properties of such lattices, including phase transitions. We will discuss these in later sections of the book. 2.5.1 Multiplicity of a Spin System Since the atoms are fixed to their respective lattice sites, they can be distinguished by their position on the lattice and therefore are distinct. Let n denote the number of atoms with spin up (↑). Note that for this problem, the method of counting microscopic states is the same as that for the bag of N coins in Exercise 2.5. The number of distinct ways to assign n spins “up” is the same as the number of distinct ways that N distinct objects can be assigned to n pots, assuming their ordering among the pots does not matter. Thus, the multiplicity of the macroscopic state “n spins up” is N (n) = N! n!(N − n)! . (2.9) This is the number of microscopic states available to the lattice for the given value of n. As a check, let us sum over all possible values n = 0, 1, . , N. If we make use of the binomial theorem (a + b) N = ∑N n=0 N! n!(N − n)! aN−n bn , (2.10) and set a = b = 1 we can use Eq. (2.9) to obtain the total number of microstates N = ∑N n=0 N (n) = ∑N n=0 N! n!(N − n)! = 2N . (2.11) Thus, the sum of all the microstates contained in the macrostates gives 2N , as it should. Note that our ability to count the number of microscopic states is due to the fact that the angular momentum intrinsic to the atoms is quantized and is a consequence of the quantum nature of matter. Let us now focus on the limit N → ∞, and consider the behavior of the fraction of microstates, N (n)=(N (n))∕(N ) with n spins “up,” N (n) = N! n!(N − n)! 1 2N = N! n!(N − n)! (1 2 )n (1 2 )N−n . (2.12)