§5 · independent configurations · Light quanta: from entropy to an energy scale

§5 · independent configurations

Which probability the entropy uses

What does Einstein mean by the probability of a state, and why does he insist on it?

§5 opens with a warning about a word. When entropy is calculated from molecular theory, Einstein writes, 'probability' is often used in a sense that does not match its definition in the theory of probability. In particular, which cases count as equally probable is often fixed by hypothesis, even where the theoretical picture is definite enough to deduce them instead.

He announces that he will show in a separate paper that the statistical probability is all that thermal questions need, and hopes this removes a logical difficulty that still stands in the way of applying Boltzmann's principle. Here he states the principle only in general and applies it to special cases. The first special case shows what statistical probability means: the probability that, at a moment chosen at random, the system is found in the state considered.

Return to the probability of a state. (included in this file)

S-S_0=\frac{R}{N}\lg W

The entropy difference S minus S zero equals R over N times the logarithm of W.

Here W is the probability of the state with entropy S relative to a starting state with entropy $S_0$ , R is the gas constant, N the number of real molecules in a gram-equivalent, and lg is the natural logarithm as the paper writes it. Read with a statistical W, the formula needs no list of equally likely cases: a state the system is found in at 1 moment in 16, compared with a state it is always in, has W = 1/16, and its entropy is lower by R/N times ln 16.

The reading matters for everything after it. The probability that n independent points are all in part of a volume, and in §6 the probability that all the energy of monochromatic radiation is in part of its volume, are statistical probabilities in exactly this sense. The step from entropy to energy quanta is only as good as that reading of W.

Letters: the paper writes W for the probability of a state, S for its entropy, R for the gas constant, N for the number of real molecules in a gram-equivalent, and lg for the natural logarithm, which this explanation also writes ln.

The warning: in molecular calculations of entropy, the word 'probability' is often used in a sense that differs from its definition in the theory of probability.
The shortcut Einstein objects to: decide by hypothesis which cases are equally probable, then count cases. He objects where the model is definite enough to deduce those probabilities instead of assuming them.
His alternative, the statistical probability: watch the system over a long time and ask what fraction of the moments it spends in the state. Equivalently, pick a moment at random and ask how likely it is that the system is in the state then.
Worked example: four points that move independently in a box, each equally likely to be anywhere in it. At a random moment one point is in the left half with probability 1/2, so all four are there with probability 1/2 × 1/2 × 1/2 × 1/2 = 1/16. Watched for a long time, the box spends about 1 moment in 16 with all four points on the left.
Boltzmann's principle, as Einstein uses it: the entropy of a system is a function of the probability of its state. The next paragraphs of §5 show that the function has to be a logarithm, because entropies of separate systems add while their probabilities multiply.

S-S_0=\frac{R}{N}\lg W

The entropy difference S minus S zero equals R over N times the logarithm of W.

Reading the formula: S − $S_0$ is the entropy of the state minus the entropy of the starting state, and W is the probability of the state relative to the starting state.
In the worked example, relative to the box with its points anywhere, W = 1/16. So S − $S_0$ = (R/N) ln(1/16) = −(R/N) ln 16. Since ln 16 = 4 ln 2 ≈ 4 × 0.693 ≈ 2.77, the state with all four points on the left has an entropy lower by about 2.77 × R/N.
Nothing in this used a list of equally likely cases decided in advance: the 1/16 came from the independence of the points and the fraction of the volume, both properties of the model.
Einstein promises a separate paper to show that the statistical probability suffices for thermal questions; this paper does not contain that proof, so here it remains his announced position.

Return to the chance of finding the state at a random moment. (included in this file)

Assumptions

The probability of a state is its statistical probability: the chance of finding the system in that state at a moment chosen at random.
Boltzmann's principle, as Einstein uses it: the entropy of a system is a function of the probability of its state.
The argument compares entropy differences between states; it needs no absolute entropy.

Limits

Einstein says he will show in a separate paper that statistical probability is enough; this paper does not give that proof, and this passage does not supply it.
A moment chosen at random presupposes a system in a steady condition, watched over a long time; the passage does not treat systems that are not.

Independence supplies the exponent

Why is the probability f to the power n, rather than just f?

Each independent point has probability f of being inside the chosen fraction. For all n to be inside, multiply n independent probabilities. Taking the logarithm makes the exponent n a coefficient.

Return to why the entropy is the logarithm of the probability. (included in this file)

W=f^n,\qquad\Delta S=k_B\ln W=nk_B\ln f

The probability is f to the n; the entropy difference is Boltzmann's constant times its logarithm, or n times Boltzmann's constant times the logarithm of f.

With four independent points and half the volume, the probability is 1/16. If all four positions are exactly the same uniformly distributed position, the probability is 1/2 instead. Having four labels is not enough to justify four independent factors.

The concentration from V₀ to V smaller than V₀ has a negative entropy difference. Reversing the constrained comparison reverses the sign. The independent-configurations laboratory makes the probability and entropy readouts separate.

Return to the independence assumption behind the volume exponent. (included in this file)

Return to the chance that all n points are in the chosen part. (included in this file)

Letters: the paper writes W for the probability of a state and $S-S_0=\frac{R}{N}\lg W$ for Boltzmann's principle, and for the case below $W=(v/v_0)^n$ . This explanation writes f for $v/v_0$ and $k_B$ for R/N.

Boltzmann's principle, as the paper states it: the entropy of a system is a function of the probability of its state. For two systems that do not act on each other, the entropies add and the probabilities multiply. A function that turns products into sums is a logarithm, so the entropy is C times ln W plus a constant, and the paper takes C = R/N from the kinetic theory of gases.
The setting: n movable points in a volume $v_0$ , so few that they do not act on each other, with no part of the space and no direction preferred. They could be the molecules of a dilute gas or of a dissolved substance.
The question: at a moment chosen at random, what is the probability that all n points are in a chosen part v of $v_0$ ?
One point is equally likely to be anywhere, so the chance that it is in v is the fraction of the volume, $f=v/v_0$ .
Two independent points are both inside with probability f × f. For n independent points, multiply n times: $W=f^n$ .
Worked case: n = 4 and f = 1/2. Mark each point as in or out: there are 2 × 2 × 2 × 2 = 16 equally likely patterns, and exactly one has all four in, so W = 1/16, which is $(1/2)^4$ .
Take the logarithm: $\ln W=n\ln f$ . In the worked case, $\ln(1/16)=4\ln(1/2)$ , about -2.77.
Multiply by $k_B$ , the paper's R/N, to get the entropy difference.

Return to why the entropy is the logarithm of the probability. (included in this file)

W=f^n,\qquad\Delta S=k_B\ln W=nk_B\ln f

The probability is f to the n; the entropy difference is Boltzmann's constant times its logarithm, or n times Boltzmann's constant times the logarithm of f.

In the paper's letters this is $S-S_0=R\frac{n}{N}\lg\frac{v}{v_0}$ . Here n/N is the number of gram-molecules of points, so R times it is the gas constant for that amount.
A footnote draws the gas law from it: the work $p\,dv$ equals $T\,dS=R\frac{n}{N}\frac{dv}{v}$ , so $pv=R\frac{n}{N}T$ , the law of Boyle and Gay-Lussac and the same law for osmotic pressure. The paper remarks that no assumption about how the molecules move was needed.
Independence is doing the work. If the four points were locked to one shared position, the chance of finding all of them in half the volume would be 1/2, not 1/16. Four labels do not by themselves give four independent factors.
The sign: concentrating the points into a smaller part (f less than 1) makes ln f negative, so the concentrated state has the lower entropy. Reversing the comparison reverses the sign.

Return to the independence assumption behind the volume exponent. (included in this file)

The count comes down as a coefficient

Modern teaching equation, not the equation as printed.

\Delta S_{\mathrm{gas}} = k_B\,\ln\left(W\right) = n\,k_B\,\ln\left(f\right)

The entropy change is k B times the logarithm of W, which is n times k B times the logarithm of f.

Boltzmann's principle turns the probability into an entropy difference. The logarithm of f to the n is n times the logarithm of f, so the number of independent points becomes the coefficient, and that coefficient is what section 6 compares with the radiation's entropy.

Multiply one chance per independent point

Modern teaching equation, not the equation as printed.

W = f^{n}

The probability W is f to the power n.

Each of n independent points is inside the fraction f of the volume with probability f, so the chance that all of them are inside at once is f multiplied by itself n times. Independence does the work: four labels on one shared position would give f, not f to the fourth.

Assumptions

Each point is uniformly distributed over the initial volume.
The point positions are statistically independent for the independent model.
Entropy differences are related to logarithms of relative configuration probabilities.

Limits

The locked-position example is an authored mathematical counterexample, not an established alternative theory of radiation.
Spontaneous concentration probability and constrained-state entropy are related but are not the same observable.

Foundations included in this file

A sign records direction

How can movement add up to zero?

On the ruler below, the starting point is zero. A final mark three units to the right has displacement +3; one three units to the left has displacement −3. Adding the signed displacements gives zero. Adding the distances from the start gives six.

Displacement compares where something ends with where it started, whatever it did in between. A walker who goes three units right and comes back has displacement zero after walking six, which is the lower pair of arrows in the drawing.

Worked example: Two walkers, opposite directions, average zero

Two walkers both finish one unit from their start, one at −1 and the other at +1. The signed average is zero. The average distance is one.

Where this lesson stops

The sign names a direction relative to a chosen axis; it does not mean a negative distance.

This lesson builds on

Adding and averaging

Return to the chapter

A probability, a frequency and a single trial

What does a probability of 1/4 say about one try, and about a hundred?

Drop two tokens into a box, each landing in the left or the right half with equal chance. There are four outcomes: left and left, left and right, right and left, right and right. One of the four is both left, so the model probability of both left is 1/4.

Now drop them 100 times and count. Both might land left 23 times: a frequency of 23/100. Over more and more trials the frequency tends to settle near 1/4, but any particular count can differ from 25.

A single drop either lands both left or it does not. A probability of 1/4 is not a quarter of an outcome. It says how often, over many drops.

The light-quanta paper counts in exactly this way in §5. The chance that n independent points are all in a part v of a volume v₀ is (v/v₀)ⁿ, and the paper calls it a statistical probability: a frequency over many moments.

Worked example: Counting outcomes for two tokens and for four

Two tokens: 4 equally likely outcomes, 1 of them both left, so the probability is 1/4.
Four tokens: 2⁴ = 16 outcomes, 1 of them all left, so the probability is 1/16, which is (½)⁴.
A run of 100 drops of four tokens might show all left 5 times: a frequency of 5/100, near 1/16 (about 6 in 100) but not equal to it.
Any one drop shows all left or it does not. The 1/16 is about the long run.

Where this lesson stops

A probability counts favoured outcomes among equally likely ones; a frequency counts what happened. How a probability becomes an entropy is the lesson on entropy and the number of ways.

This lesson builds on

bridge-fractions-ratios

Return to the chapter

Adding and averaging

What does an average keep, and what does it lose?

Four jars hold 3, 1, 1 and 3 marbles. Pour them together and there are 8. Share the 8 equally between the four jars and each holds 2. Two is the average, although no jar held 2 to begin with.

The average keeps the total and the count and nothing else, so 3, 1, 1, 3 and 2, 2, 2, 2 have the same average. Listing every value twice doubles both the total and the count, and the average stays at 2.

Worked example: Three mornings with the same average

One morning the four jars hold 3, 1, 1 and 3 marbles. Add them: 8. Divide by the four jars: an average of 2.
Another morning they hold 8, 0, 0 and 0. Add them: 8 again. Divide by four: 2 again, and now three jars are empty.
A third morning every jar holds 2, which is the average itself. Add them: 8. Divide by four: 2.
Write any of those three lists out twice over. The total doubles to 16 and the count doubles to 8 jars, and 16 divided by 8 is still 2.
Four descriptions and one average. It keeps the total and the count, and the differences between the mornings are what it leaves behind.

Where this lesson stops

An average is the total shared equally among the observations, and two very different lists can have the same one.

Return to the chapter

Entropy and the number of ways

Why is entropy the logarithm of a probability?

Four particles move independently in a box. The chance that all four happen to be in the left half at a given moment is ½ × ½ × ½ × ½ = 1/16. For a hundred particles it is (½)¹⁰⁰, about 8 × 10⁻³¹: possible, but never seen. A state is more probable when more of the ways the particles can be arranged produce it.

Entropy measures this probability on a logarithmic scale. Boltzmann's principle, as §5 of the light-quanta paper states it, makes the entropy of a system a function of the probability W of its state, and that function must be a logarithm:

S - S_0 = k_B \ln W

S minus S nought equals k B times the natural logarithm of W.

Why a logarithm: two independent systems have a combined probability W = W₁ × W₂, while their entropies add, S = S₁ + S₂. Only a logarithm turns the product into the sum, and that is the argument §5 gives. The constant $k_B$ is R/N, which is how the paper writes it, and the paper prints the natural logarithm as lg.

Apply it to the box. Squeezing n independent particles from a volume v₀ into a part v of it has probability W = (v/v₀)ⁿ, so the entropy changes by S − S₀ = n $k_B$ ln(v/v₀), a decrease, since v/v₀ is less than 1. §5 derives exactly this, and remarks that it needs no assumption about the law by which the molecules move.

The probability here is a frequency: the fraction of moments at which the state is found. §5 insists on this, and criticizes calculations in which the equally probable cases are simply postulated.

Worked example: Squeezing a gas into half its volume

Take one gram-molecule of gas, n = N = 6.02 × 10²³ particles, and ask for all of them in the left half: v/v₀ = ½.
The probability is (½) to the power 6.02 × 10²³, far too small to write out. Its logarithm is N ln ½, about −4.2 × 10²³.
The entropy change is $k_B$ × N × ln ½ = R ln ½ = 8.314 × (−0.693), about −5.76 joules per kelvin.
Thermodynamics gives the same number for compressing an ideal gas to half its volume at fixed temperature, R ln ½. Counting ways and measuring heat agree.

§6 of the paper finds the same form for dilute radiation, with E/(βν) standing where the gas has Rn/N, and reads it as radiation behaving, in this respect, like independent quanta.

Where this lesson stops

This lesson stops at entropy as the logarithm of a probability. What the comparison with radiation shows, and what it does not, is §6 of the light-quanta paper.

Return to the chapter

Logarithms: turning products into sums

Why does entropy need a logarithm?

Adding is easier than multiplying, and a logarithm turns one into the other. The natural logarithm of 2 is about 0.693, and of 4 about 1.386. Add them and you get 2.079, which is the natural logarithm of 8, the product of 2 and 4.

The natural logarithm ln x answers one question: to what power must e, about 2.718, be raised to give x? Raised to 0.693, e gives 2, and raised to 1.386 it gives 4. Multiplying two powers of e adds the powers, so raised to 2.079 it gives 8. Written for any numbers, the logarithm of a product is the sum of the logarithms, and a power becomes a multiple:

\ln(AB) = \ln A + \ln B, \qquad \ln\left(f^{\,n}\right) = n \ln f

The log of A times B is log A plus log B, and the log of f to the power n is n times log f.

Entropy needs exactly this. In §5 of the light-quanta paper, Einstein takes two independent systems. Their entropies add, S = S₁ + S₂, while the probabilities of their states multiply, W = W₁W₂. If entropy is some function φ of probability, then φ(W₁W₂) = φ(W₁) + φ(W₂), and he concludes that φ is a logarithm:

S - S_0 = \frac{R}{N} \ln W

S minus S nought equals R over N times the natural log of W.

R is the gas constant and N the number of molecules in a mole, so R/N is what is now called Boltzmann's constant. The step from φ(W₁W₂) = φ(W₁) + φ(W₂) to the logarithm needs φ to be a smooth function, which Einstein takes for granted.

Einstein prints lg for the natural logarithm, as German physics did in 1905. Today lg usually means the base-10 logarithm and ln the natural one. So a printed lg 2 means ln 2, about 0.693, not log₁₀ 2, about 0.301.

Worked example: Ten molecules found in half the volume

In §5 Einstein asks how probable it is that n molecules, each moving independently through a volume v₀, are all found at one moment in a part v of it. Each molecule is in that part with probability v/v₀, and independent probabilities multiply:

W = \left(\frac{v}{v_0}\right)^{n}

W equals v over v nought, to the power n.

Take n = 10 molecules and v = v₀/2. Then W = (1/2)¹⁰ = 1/1024, about 0.000977.
The logarithm turns the tenth power into ten times: ln W = 10 × ln(1/2) = 10 × (−0.693) = −6.93.
So the entropy of the state with every molecule in one half is lower by 6.93 × R/N: 6.93 times Boltzmann's constant.

For a mole of gas, n = N, and the same formula gives S − S₀ = R ln(v/v₀). That is the entropy change thermodynamics gives for an ideal gas compressed from v₀ to v at constant temperature.

Where this lesson stops

A logarithm turns products into sums, which is why it connects probabilities that multiply to entropies that add. In the 1905 papers, lg means the natural logarithm.

This lesson builds on

exponentials

Return to the chapter

Probability and independence

When you square a sum of random steps, why do the mixed terms average to zero?

Flip a coin twice and take one step for each flip: a metre to the right for heads, a metre to the left for tails. How far from the start do you end up, on average?

The four outcomes, right-right, right-left, left-right and left-left, are equally likely: one chance in four each.
They leave you 2 m to the right, back at the start, back at the start, or 2 m to the left.
Square those distances so that left and right count alike: 4, 0, 0 and 4. Their average is 2, one square metre for each step.

Combining the two steps added nothing beyond one square metre each. In symbols, with Δ₁ and Δ₂ for the two steps, squaring the sum gives each step squared and a mixed term, twice their product:

(\Delta_1 + \Delta_2)^2 = \Delta_1^2 + 2\Delta_1\Delta_2 + \Delta_2^2

Delta one plus delta two, squared, equals delta one squared plus two delta one delta two plus delta two squared.

In the coin walk the product of the two steps is +1 for right-right and left-left and −1 for the other two, so on average it is zero and the mixed term adds nothing. Two conditions make that happen. Independence: learning the first step does not change the chances for the second, so the chance of a pair is the product of the two chances. Centring: each step averages zero, left as likely as right. Then the average of the product is the product of the averages, zero times zero:

\langle \Delta_1 \Delta_2 \rangle = \langle \Delta_1 \rangle \langle \Delta_2 \rangle = 0

The average of delta one times delta two equals the average of delta one times the average of delta two, which is zero.

Both conditions matter. If each step drifts, averaging +1, the product of the averages is 1, not 0. If the second step tends to copy the first, the product is positive more often than negative. The Brownian paper's §4 assumes that a particle's motions in successive intervals are independent, as long as the intervals are not chosen too small.

Worked example: What changes when the second step copies the first

Now let the second flip always copy the first. Only right-right and left-left can happen, one chance in two each.
The product of the two steps is +1 every time, so the mixed term no longer averages zero.
You always end 2 m from the start, so the squared distance is 4 every time: the average is 4, not 2. Independence removed that extra 2.

Where this lesson stops

Before dropping a mixed term, ask two things: does learning one step change the chances for the next, and does each step average zero?

This lesson builds on

Adding and averaging
A sign records direction

Return to the chapter

Static laboratory examples

lq-05

This laboratory is not included in this file. The chapter's authored foundation examples remain available above. Open lq-05 online

Source references

A. Einstein, Über einen die Erzeugung und Verwandlung des Lichtes betreffenden heuristischen Gesichtspunkt. Annalen der Physik (4), 17, 132–148 (1905).