Axiomatic Relation between Thermodynamic and Information-Theoretic Entropies

Thermodynamic entropy, as defined by Clausius, characterizes macroscopic observations of a system based on phenomenological quantities such as temperature and heat. In contrast, information-theoretic entropy, introduced by Shannon, is a measure of uncertainty. In this Letter, we connect these two notions of entropy, using an axiomatic framework for thermodynamics [E. H. Lieb and J. Yngvason Proc. R. Soc. 469, 20130408 (2013)]. In particular, we obtain a direct relation between the Clausius entropy and the Shannon entropy, or its generalization to quantum systems, the von Neumann entropy. More generally, we find that entropy measures relevant in nonequilibrium thermodynamics correspond to entropies used in one-shot information theory.

Figure 1

The contribution of this Letter is to draw a connection between thermodynamic and information-theoretic entropy based on formal axioms and general principles such as additivity and monotonicity that are satisfied within both theories (Proposition 1). This replaces the textbook identification of entropy in thermodynamics and statistical mechanics based merely on analogies, such as common behavior of the two entropy functions in specific models like the ideal gas. The entropy functions in information theory and statistical mechanics are already formally equivalent, and conceptual links have been established through Jaynes’ maximum entropy principle [22], as well as through the works of Landauer [5] and Bennett [6] (based on explicit protocols that convert one into the other in an underlying microscopic model).