Chapter 1:Entropy and the Second Law 19 and others;its theoretical development is no more circular than that of Lieb and Yngvason [31];furthermore,it claims to encompass broader territory than either by applying both to equilibrium and nonequilibrium systems.It does not,how- ever,provide a microscopic picture of entropy and so is not well-suited to statistical mechanics. (5)Lieb-Yngvason [31]The Lieb-Yngvason entropy SLy is defined through the mathematical ordering of sets of equilibrium states,subject to the constraints of monotonicity,additivity and extensivity.The second law is revealed as a math- ematical theorem on the ordering of these sets.This formalism owes significant debt to work by Caratheodory [30],Giles [52],Buchdahl [53]and others. Starting with a space I of equilibrium states X,Y,Z...,one defines an ordering of this set via the operation denoted pronounced precedes.The various set elements of I can be ordered by a comparison procedure involving the criterion of adiabatic accessibility.For elements X and Y,31] A state Y is adiabatically accessible from a state X,in symbols X <Y, if it is possible to change the state X to Y by means of an interaction with some device (which may consist of mechanical and electrical parts as well as auxiliary thermodynamic systems)and a weight,in such a way that the device returns to its initial state at the end of the process whereas the weight may have changed its position in a gravitation field. This bears resemblance to the GHB weight process above (Figure 1.1).Although superficially this definition seems limited,it is quite general for equilibrium states. It is equivalent to requiring that state X can proceed to state Y by any natural process,from as gentle and mundane as the unfolding of a Double Delight rose in a quiet garden,to as violent and ultramundane as the detonation of a supernova. If X proceeds to Y by an irreversible adiabatic process,this is denoted X Y,and if X Y and Y <X,then X and Y are called adiabatically equivalent, written XAY.If X Y or YX (or both),they are called comparable. The Lieb-Yngvason entropy SLy is defined as 31]: There is a real-valued function on all states of all systems (including compound systems),called entropy and denoted by S such that a)Monotonicity:When X and Y are comparable states then X Y if and only if S(X)<S(Y). b)Additivity and extensivity:If X and Y are states of some (possibly different)systems and if (X,Y)denotes the corresponding state in the composition of the two systems,then the entropy is additive for these states,i.e., S(X,Y)=S(X)+S(Y)
Chapter 1: Entropy and the Second Law 19 and others; its theoretical development is no more circular than that of Lieb and Yngvason [31]; furthermore, it claims to encompass broader territory than either by applying both to equilibrium and nonequilibrium systems. It does not, however, provide a microscopic picture of entropy and so is not well-suited to statistical mechanics. (5) Lieb-Yngvason [31] The Lieb-Yngvason entropy SLY is defined through the mathematical ordering of sets of equilibrium states, subject to the constraints of monotonicity, additivity and extensivity. The second law is revealed as a mathematical theorem on the ordering of these sets. This formalism owes significant debt to work by Carath´eodory [30], Giles [52], Buchdahl [53] and others. Starting with a space Γ of equilibrium states X,Y,Z ..., one defines an ordering of this set via the operation denoted ≺, pronounced precedes. The various set elements of Γ can be ordered by a comparison procedure involving the criterion of adiabatic accessibility. For elements X and Y, [31] A state Y is adiabatically accessible from a state X, in symbols X ≺ Y, if it is possible to change the state X to Y by means of an interaction with some device (which may consist of mechanical and electrical parts as well as auxiliary thermodynamic systems) and a weight, in such a way that the device returns to its initial state at the end of the process whereas the weight may have changed its position in a gravitation field. This bears resemblance to the GHB weight process above (Figure 1.1). Although superficially this definition seems limited, it is quite general for equilibrium states. It is equivalent to requiring that state X can proceed to state Y by any natural process, from as gentle and mundane as the unfolding of a Double Delight rose in a quiet garden, to as violent and ultramundane as the detonation of a supernova. If X proceeds to Y by an irreversible adiabatic process, this is denoted X ≺≺ Y, and if X ≺ Y and Y ≺ X, then X and Y are called adiabatically equivalent, written X A ∼ Y. If X ≺ Y or Y ≺ X (or both), they are called comparable. The Lieb-Yngvason entropy SLY is defined as [31]: There is a real-valued function on all states of all systems (including compound systems), called entropy and denoted by S such that a) Monotonicity: When X and Y are comparable states then X ≺ Y if and only if S(X) ≤ S(Y). b) Additivity and extensivity: If X and Y are states of some (possibly different) systems and if (X,Y) denotes the corresponding state in the composition of the two systems, then the entropy is additive for these states, i.e., S(X,Y) = S(X) + S(Y)
20 Challenges to the Second Law S is also extensive,i.e.,for each t>0 and each state X and its scaled copy tX, S(tx)=tS(X). The monotonicity clause is equivalent to the following: XAY→s(X)=S(Y);and X<<Y→S(X)<S(Y). The second of these says that entropy increases for an irreversible adiabatic pro- cess.This is the Lieb-Yngvason formulation of the second law. The existence and uniqueness of SLy can be shown to follow from assump- tions surrounding adiabatic accessibility and the comparsion process.In this for- malism,temperature is not a primitive concept;rather,it is defined via SLy as =()v,where U is energy and V is volume.The mathematical details of these results are beyond the scope of this discussion;the intrepid reader is directed to[31. (6)Caratheodory Historically preceding SLy,Caratheodory also defined en- tropy in a formal mathematical sense 30,6. For simple12 systems,Caratheodory's principle is equivalent to the proposition that the differential form 6Q:=dU-ow possesses an integrable divisor,i.e.,there exists functions S and T on the state space I such that 6Q =TdS. Thus,for simple systems,every equilibrum state can be assigned values for entropy and absolute temperature.Obviously these functions are not uniquely determined by the relation [6Q =TdS]. Caratheodory's entropy was not widely accepted by working scientists during his lifetime,but it has grown in significance during the last 40 years as thermodynamic foundations have been shored up. (7)Shannon [54]Various information-relevant entropies have been proposed over the last six decades,the most prominent of which are the Shannon entropy and algorithmic randomness [55,56,57].These are especially salient in considerations of sentient Maxwell demons [21],which have helped expose the deep relationships between physics and information theory. Let pj be probabilities of mutually exclusive events,say for instance,the prob- abilties of particular letters in an unknown word.The uncertainty (entropy)of the information about this situation is the Shannon entropy: 12Consult the literature for the requirements of a simple system [6,30]
20 Challenges to the Second Law S is also extensive, i.e., for each t > 0 and each state X and its scaled copy tX, S(tX) = tS(X). The monotonicity clause is equivalent to the following: X A ∼ Y =⇒ S(X) = S(Y); and X ≺≺ Y =⇒ S(X) < S(Y). The second of these says that entropy increases for an irreversible adiabatic process. This is the Lieb-Yngvason formulation of the second law. The existence and uniqueness of SLY can be shown to follow from assumptions surrounding adiabatic accessibility and the comparsion process. In this formalism, temperature is not a primitive concept; rather, it is defined via SLY as 1 T := ( ∂SLY ∂U )V , where U is energy and V is volume. The mathematical details of these results are beyond the scope of this discussion; the intrepid reader is directed to [31]. (6) Carath´eodory Historically preceding SLY , Carath´eodory also defined entropy in a formal mathematical sense [30, 6]. For simple12 systems, Carath´eodory’s principle is equivalent to the proposition that the differential form δQ := dU − δW possesses an integrable divisor, i.e., there exists functions S and T on the state space Γ such that δQ = T dS. Thus, for simple systems, every equilibrum state can be assigned values for entropy and absolute temperature. Obviously these functions are not uniquely determined by the relation [δQ = T dS]. Carath´eodory’s entropy was not widely accepted by working scientists during his lifetime, but it has grown in significance during the last 40 years as thermodynamic foundations have been shored up. (7) Shannon [54] Various information-relevant entropies have been proposed over the last six decades, the most prominent of which are the Shannon entropy and algorithmic randomness [55, 56, 57]. These are especially salient in considerations of sentient Maxwell demons [21], which have helped expose the deep relationships between physics and information theory. Let pj be probabilities of mutually exclusive events, say for instance, the probabilties of particular letters in an unknown word. The uncertainty (entropy) of the information about this situation is the Shannon entropy: 12Consult the literature for the requirements of a simple system [6, 30]
Chapter 1:Entropy and the Second Law 21 Ssh=-∑pjlog(p) (1.25) The logarithm may be taken to any fixed base,but base 2 is standard,giving entropy in bits.Shannon entropy can be seen to be a discrete form of the classical Boltzmann-Gibbs entropy,(1.17) (8)Fisher Shannon entropy is defined over a space of unordered elements,for instance,letters.For a space of ordered elements,for example,a continuous parameter (e.g.,the length or brightness of meteor trails),Fisher information is appropriate.For a probability distribution f(z;)in the random variable x dependent on the unobservable variable o,the Fisher information (entropy)is SE()=KI 02 21ogfz=-K log f(】 (1.26) Clearly,the sharpness of the support curve is proportional to the expection of Sr(),thus high information content (low entropy)corresponds to a sharp distri- bution and a low information content (high entropy)to a broad distribution. (9)Algorithmic Randomness [55,56,57]Algorithmic randomness (algorith- mic complexity,Kolmorgorov complexity)of a string of elements is defined as the minimum size of a program (e.g.,in bits)executed on a universal computer that yields the string.Strings are relatively simple or complex depending on whether its program length is relatively short or long,respectively.For example,the string of 60,000 digits(121223121223121223...)is relatively simple and has relatively low algorithmic randomness since it can be programmed as 10,000 repeating blocks of (121223),whereas a completely random string of 60,000 digits cannot be com- pressed this way and thus has a relatively large algorithmic randomness.Most strings cannot be compressed and,to leading order in binary notation,their al- gorithmic randomness is given by their lengths in bits.By example,a random natural number N,if it can be expressed as N~25,has algorithmic randomness log2N s. Algorithmic complexity,in contrast to other standard definitions of entropy, does not rely on probabilities.However,the randomness of a string is not uniquely determined and there is no general method to discern a simple string from a com- plex one;this is related to Godel's undecidability [59].For example,the sequence (2245915771836104547342715)may appear completely random,but it is easily generated from re.Or,the letter sequence FPURCLK might seem random until it is unscrambled and considered in an appropriate context.Apparently,order can be in the eye of the beholder. Zurek suggests that physical entropy"is the sum of(i)the missing information measured by Shannon's formula and (ii)of the algorithmic content in the avail- able data about the system"[58. (10)Tsallis [60,61]Tsallis entropy is a controversial generalization of Boltzmann- Gibbs entropy and is an heir to the Renyi and Daroczy entropies below.It is
Chapter 1: Entropy and the Second Law 21 SSh = − j pj log(pj ) (1.25) The logarithm may be taken to any fixed base, but base 2 is standard, giving entropy in bits. Shannon entropy can be seen to be a discrete form of the classical Boltzmann-Gibbs entropy, (1.17). (8) Fisher Shannon entropy is defined over a space of unordered elements, for instance, letters. For a space of ordered elements, for example, a continuous parameter (e.g., the length or brightness of meteor trails), Fisher information is appropriate. For a probability distribution f(x; φ) in the random variable x dependent on the unobservable variable φ, the Fisher information (entropy) is SF (φ) = K[ ∂ ∂φ log f(x; φ)]2 = −K[ ∂2 ∂φ2 log f(x; φ)] (1.26) Clearly, the sharpness of the support curve is proportional to the expection of SF (φ), thus high information content (low entropy) corresponds to a sharp distribution and a low information content (high entropy) to a broad distribution. (9) Algorithmic Randomness [55, 56, 57] Algorithmic randomness (algorithmic complexity, Kolmorgorov complexity) of a string of elements is defined as the minimum size of a program (e.g., in bits) executed on a universal computer that yields the string. Strings are relatively simple or complex depending on whether its program length is relatively short or long, respectively. For example, the string of 60,000 digits (121223121223121223...) is relatively simple and has relatively low algorithmic randomness since it can be programmed as 10,000 repeating blocks of (121223), whereas a completely random string of 60,000 digits cannot be compressed this way and thus has a relatively large algorithmic randomness. Most strings cannot be compressed and, to leading order in binary notation, their algorithmic randomness is given by their lengths in bits. By example, a random natural number N, if it can be expressed as N ∼ 2s, has algorithmic randomness ∼ log2N = s. Algorithmic complexity, in contrast to other standard definitions of entropy, does not rely on probabilities. However, the randomness of a string is not uniquely determined and there is no general method to discern a simple string from a complex one; this is related to G¨odel’s undecidability [59]. For example, the sequence (2245915771836104547342715) may appear completely random, but it is easily generated from πe. Or, the letter sequence FPURCLK might seem random until it is unscrambled and considered in an appropriate context. Apparently, order can be in the eye of the beholder. Zurek suggests that physical entropy “is the sum of (i) the missing information measured by Shannon’s formula and (ii) of the [algorithmic content] in the available data about the system” [58]. (10) Tsallis [60, 61] Tsallis entropy is a controversial generalization of BoltzmannGibbs entropy and is an heir to the R´enyi and Dar´oczy entropies below. It is
22 Challenges to the Second Law defined as s=g-/e (1.27) where g is a real number entropic inder and f(r)is a probability distribution function.For g=1,Srs reduces to the Boltzmann-Gibbs entropy. Primary virtues of the Tsallis entropy include its mathematical simplicity and descriptiveness of nonextensive systems.A physical quantity is extensive if its value scales linearly with the size of the system 13.The extensive Boltzmann-Gibbs entropy of two independent systems A and B is SBG(A+B)=SBG(A)+SBG(B), while for the Tsallis entropy it is STs(A+B)=STs+STs(B)+(1-q)STs(A)STs(B) The parameter q can be taken as a measure of nonextensivity14. Tsallis entropy has been applied to numerous disparate physical phenomena that are deemed beyond the reach of equilibrium thermodynamics.Notably,these include systems with long-range nonextensive fields (e.g.,gravitational,electro- static)such as plasmas and multi-particle self-gravitating systems (e.g.,galaxies, globular clusters).It has been applied to the behaviors of self-organizing and low- dimensional chaotic systems and processes far from equilibrium;examples include financial markets,crowds,traffic,locomotion of microorganisms,subatomic par- ticle collisions,and tornados.Unfortunately,its underlying physical basis has not been well established,leading critics to label it ad hoc and its successes little more than "curve fitting."Its elegant simplicity and adaptability,however,cannot be denied. The entropic index(nonextensivity parameter)g is taken to be a measure of the fractal nature of a system's path in phase space.Whereas under Boltzmann-Gibbs formalism,a system on average spends equal time in all accessible,equal-sized vol- umes of phase space (equal a priori probability),under the Tsallis formalism the phase space path is fractal,thereby allowing it to model chaotic,nonequilibrium systems,and display rapid and radical changes in behavior and phase. (11-21)Other Entropies There are a number of other entropy and entropy-like quantities that are beyond the scope of this discussion.These include (with p the density matrix,unless otherwise noted): Daroczy entropy [62]: 1 SD=2-a-Tr(p)-10, (1.28) with a>0anda≠1. Renyi entropy [63]: SR=1-0 lnTr(p小， (1.29) again with a>0anda≠l. 13Extensivity is a traditional requirement for thermodynamic quantities like energy and entropy. 14Notice that if g-1,then STs-SBG
22 Challenges to the Second Law defined as ST s = 1 q − 1 1 − f q(x)dx , (1.27) where q is a real number entropic index and f(x) is a probability distribution function. For q = 1, ST s reduces to the Boltzmann-Gibbs entropy. Primary virtues of the Tsallis entropy include its mathematical simplicity and descriptiveness of nonextensive systems. A physical quantity is extensive if its value scales linearly with the size of the system 13. The extensive Boltzmann-Gibbs entropy of two independent systems A and B is SBG(A+B) = SBG(A)+SBG(B), while for the Tsallis entropy it is ST s(A+B) = ST s+ST s(B)+(1−q)ST s(A)ST s(B). The parameter q can be taken as a measure of nonextensivity14. Tsallis entropy has been applied to numerous disparate physical phenomena that are deemed beyond the reach of equilibrium thermodynamics. Notably, these include systems with long-range nonextensive fields (e.g., gravitational, electrostatic) such as plasmas and multi-particle self-gravitating systems (e.g., galaxies, globular clusters). It has been applied to the behaviors of self-organizing and lowdimensional chaotic systems and processes far from equilibrium; examples include financial markets, crowds, traffic, locomotion of microorganisms, subatomic particle collisions, and tornados. Unfortunately, its underlying physical basis has not been well established, leading critics to label it ad hoc and its successes little more than “curve fitting.” Its elegant simplicity and adaptability, however, cannot be denied. The entropic index (nonextensivity parameter) q is taken to be a measure of the fractal nature of a system’s path in phase space. Whereas under Boltzmann-Gibbs formalism, a system on average spends equal time in all accessible, equal-sized volumes of phase space (equal a priori probability), under the Tsallis formalism the phase space path is fractal, thereby allowing it to model chaotic, nonequilibrium systems, and display rapid and radical changes in behavior and phase. (11-21) Other Entropies There are a number of other entropy and entropy-like quantities that are beyond the scope of this discussion. These include (with ρ the density matrix, unless otherwise noted): Dar´oczy entropy [62]: SD = 1 21−α − 1 (T r(ρα) − 1), (1.28) with α > 0 and α = 1. R´enyi entropy [63]: SR = k 1 − α ln[T r(ρα)], (1.29) again with α > 0 and α = 1. 13Extensivity is a traditional requirement for thermodynamic quantities like energy and entropy. 14Notice that if q → 1, then ST s → SBG.
Chapter 1:Entropy and the Second Law 23 Hartley entropy 64]: SH=kIn[N(p)], (1.30) where N(p)is the number of positive eigenvalues of p. Infinite norm entropy: SIn =-k In lplloo, (1.31) where llplloo=Pmaz is the largest eigenvalue of p. Relative entropy (classical mechanics)[65,66]: SRel.c=- p(Inp-Ina)dr, (1.32) where p and o are probability distributions and r is the phase space coordinate. Relative entropy (quantum mechanics): SRet.(olp)=Tr[p(Inp-Ina)], (1.33) where p and o are distinct density matrices.It is non-negative [67. In addition to these,there is Segal entropy [68,which subsumes many of the quantum mechanical entropies mentioned above;Kolmogorov-Sinai(KS)entropy, which describes dynamical systems undergoing discrete time evolution;Kouch- nirenko A entropies,close relatives to KS entropy;skew entropy 69;Ingarden- Urbanik entropy [70;Macdonald entropy [18.For completeness,you may add your own personal favorite here: 1.4 Nonequilibrium Entropy There is no completely satisfactory definition of entropy.To some degree,every definition is predicated on physical ignorance of the system it describes and,there- fore,must rely on powerful ad hoc assumptions to close the explanatory gap.These limit their scopes of validity.Let us review a few examples.The Boltzmann-Gibbs entropy assumes equal a priori probability either of phase space or ensemble space. While this is a reasonable assumption for simple equilibrium systems like the ideal gas and Lorentz gas,it is known to fail for large classes of systems,especially at disequilibrium;the molecular chaos ansatz(Boltzmann's Stosszahlansatz)is sim- ilarly suspect.It is not known what the necessary conditions are for ergodicity. The thermodynamic limit,which is presumed or necessary for most quantum and classical thermodynamic formalisms,on its face cannot be completely realistic, particularly since it ignores boundary conditions that are known to be pivotal for many thermodynamic behaviors.Extensivity,also presumed for most entropies, is ostensibly violated by systems that exhibit long-range order and fields-these include systems from nuclei up to the largest scale structures of the universe [38. Information entropies are hobbled by lack of general definitions of order,disorder
Chapter 1: Entropy and the Second Law 23 Hartley entropy [64]: SH = k ln[N(ρ)], (1.30) where N(ρ) is the number of positive eigenvalues of ρ. Infinite norm entropy: SIn = −k ln ρ∞, (1.31) where ρ∞ = pmax is the largest eigenvalue of ρ. Relative entropy (classical mechanics) [65, 66]: SRel,c = − ρ(ln ρ − ln σ)dτ, (1.32) where ρ and σ are probability distributions and τ is the phase space coordinate. Relative entropy (quantum mechanics): SRel,q(σ|ρ) = T r[ρ(ln ρ − ln σ)], (1.33) where ρ and σ are distinct density matrices. It is non-negative [67]. In addition to these, there is Segal entropy [68], which subsumes many of the quantum mechanical entropies mentioned above; Kolmogorov-Sinai (KS) entropy, which describes dynamical systems undergoing discrete time evolution; Kouchnirenko A entropies, close relatives to KS entropy; skew entropy [69]; IngardenUrbanik entropy [70]; Macdonald entropy [18]. For completeness, you may add your own personal favorite here: . 1.4 Nonequilibrium Entropy There is no completely satisfactory definition of entropy. To some degree, every definition is predicated on physical ignorance of the system it describes and, therefore, must rely on powerful ad hoc assumptions to close the explanatory gap. These limit their scopes of validity. Let us review a few examples. The Boltzmann-Gibbs entropy assumes equal a priori probability either of phase space or ensemble space. While this is a reasonable assumption for simple equilibrium systems like the ideal gas and Lorentz gas, it is known to fail for large classes of systems, especially at disequilibrium; the molecular chaos ansatz (Boltzmann’s Stosszahlansatz) is similarly suspect. It is not known what the necessary conditions are for ergodicity. The thermodynamic limit, which is presumed or necessary for most quantum and classical thermodynamic formalisms, on its face cannot be completely realistic, particularly since it ignores boundary conditions that are known to be pivotal for many thermodynamic behaviors. Extensivity, also presumed for most entropies, is ostensibly violated by systems that exhibit long-range order and fields — these include systems from nuclei up to the largest scale structures of the universe [38]. Information entropies are hobbled by lack of general definitions of order, disorder