Information processing and the second law of thermodynamics: an inclusive, Hamiltonian approach

We obtain generalizations of the Kelvin-Planck, Clausius, and Carnot statements of the second law of thermodynamics, for situations involving information processing. To this end, we consider an information reservoir (representing, e.g. a memory device) alongside the heat and work reservoirs that appear in traditional thermodynamic analyses. We derive our results within an inclusive framework in which all participating elements -- the system or device of interest, together with the heat, work and information reservoirs -- are modeled explicitly by a time-independent, classical Hamiltonian. We place particular emphasis on the limits and assumptions under which cyclic motion of the device of interest emerges from its interactions with work, heat, and information reservoirs.


I. INTRODUCTION
Three classic expressions of the second law of thermodynamics are formulated in terms of cyclic processes. The Kelvin-Planck statement asserts that [1,2] no process is possible whose sole result is the extraction of energy from a heat bath, and the conversion of all that energy into work.
The Clausius statement reads [3], no process is possible whose sole result is the transfer of heat from a body of lower temperature to a body of higher temperature.
Finally, the Carnot statement declares that [4] no engine operating between two heat reservoirs can be more efficient than a Carnot engine operating between those same reservoirs.
These formulations refer to processes involving the exchange of energy among idealized subsystems: one or more heat reservoirs; a work source-for example, a mass that can be raised or lowered against gravity; and a device that operates in cycles and affects the transfer of energy among the other subsystems. All three statements follow from simple entropy-balance analyses and offer useful, logically transparent reference points as one navigates the application of the laws of thermodynamics to real systems.
This paper concerns extensions of these classic statements to situations involving information processing. In addition to the above-mentioned elements, we will consider an information reservoir-a system that exchanges information but not energy with the device. As we will show, the Kelvin-Planck, Clausius, and Carnot statements are each generalized in a natural way in the presence of such a reservoir. Although these generalized statements can be derived ad hoc, simply by including the Shannon entropy of the information reservoir in the entropy-balance analysis, our aim is to obtain these results directly from microscopic, Hamiltonian dynamics, highlighting the assumptions and approximations that are made along common idealizations.
Among the various connections that exist between information theory and thermodynamics, two are relevant in the present context. The first involves the relationship between the thermodynamic entropy defined via the Clausius relation R }Q=T ¼ ÁS and the Shannon entropy of information theory [5] H ¼ Àtrf lng À Z ln; where R denotes an integral over phase space. Since these definitions coincide for a system in canonical equilibrium with a heat reservoir [6], it is highly tempting to use Eq. (1) to define the entropy of a nonequilibrium state. Indeed, if a system in contact with one or more thermal reservoirs evolves from an initial statistical state i to a final state f -neither of which is assumed to correspond to thermal equilibrium-then the Clausius-like inequality Z f i }Q T Àtrf f ln f g þ trf i ln i g ÁH can be established from microscopic principles, as shown in Ref. [7] under assumptions similar to those we will make in the present paper; see also Refs. [8][9][10][11][12][13][14][15][16][17][18][19][20][21] for related results and alternative derivations. On the other hand, for an isolated classical system, the Shannon entropy H remains constant with time (by Liouville's theorem), which conflicts with the observation that the entropy of an isolated physical system increases until equilibrium is attained. Such considerations show that, at the very least, one must be careful when identifying Shannon entropy with thermodynamic entropy, away from thermal equilibrium. The second connection involves the question of whether information about molecular-scale motions, gained by external observation, can be used to subvert the second law, in the sense suggested by Maxwell's famous thought experiment [22]. In an illuminating refinement of the Maxwell demon framework, Szilárd described a hypothetical scenario in which an intelligent being takes advantage of microscopic observations to manipulate a single-particle gas, so as to extract energy systematically from a reservoir and convert it to work [23]. Szilárd explicitly raised the possibility that this humanlike intelligence could be replaced by a purely physical device, in apparent violation of the Kelvin-Planck statement. By current consensus, the resolution of this paradox resides in Landauer's principle [24], which assigns a minimal thermodynamic cost to the erasure of the information gathered by the device; see Refs. [25][26][27] for details, Ref. [28] for an experimental treatment, Refs. [19,[29][30][31][32][33][34][35] for illustrative models, and Refs. [36][37][38][39] for dissenting perspectives.
These topics have gained recent prominence in the context of microscopic feedback control. Sagawa and Ueda have analyzed the amount of work that can be delivered by the measurement and manipulation of small, fluctuating systems [40][41][42][43][44]. Their predictions have been verified experimentally using trapped colloidal particles [45] and mathematically illustrated for a solvable system with a linear feedback protocol with a Kalman filter [46]. Their analyses are based on an approach considering an integral fluctuation theorem. The corresponding detailed fluctuation theorems are discussed in Refs. [47][48][49][50]. These results have been extended to systems prepared in initial nonequilibrium stationary states [51][52][53][54] and to quantum systems [55][56][57]. Alternative treatments of feedback control, which do not rely on fluctuation theorems, can be found in Refs. [58][59][60][61][62] with applications to theoretic models [63][64][65][66] and experimental systems [67,68].
In the feedback control paradigm, the microscopic state of the system of interest (or of a measurement device [69]) is observed, and on the basis of those observations, a protocol is adapted to manipulate the system. Implicit in this paradigm is an external agent or apparatus-the demon or feedback controller [19]-who makes these observations and implements the feedback. The results derived within this approach are thus expressed as relationships between thermodynamic quantities such as work and information-theoretic quantities that measure the quality of the observations. In this paper, we aim at a treatment that does not involve an external agent. Instead, we consider a self-contained universe, a composite system containing the elements mentioned earlier: a device, one or more thermal reservoirs, a work source, and an information reservoir; cf. the illustration in Fig. 1. This composite system evolves autonomously under Hamilton's equations of motion, and any effective feedback control arises entirely from the interplay of the subsystems. Within this inclusive framework, we will obtain inequalities that generalize the Kelvin-Planck, Clausius, and Carnot statements to processes involving the exchange of information.
We will begin in Sec. II by specifying our theoretical framework and terminology. In Sec. III, we will obtain formal inequalities, which will then be combined in Sec. IV with physical interpretations, to obtain generalized statements of the second law for the cyclic process. In order to complete the analysis, we will derive a generalized maximum work theorem for the noncyclic process in Sec. V. Finally, we will conclude in Sec. VI. In obtaining these results, we will make a number of assumptions and approximations, reflecting idealizations that commonly arise in analyses of thermodynamic principles, and we will discuss the roles of these assumptions in our treatment.

II. THERMODYNAMICS WITHIN A HAMILTONIAN FRAMEWORK
In this section, we describe our framework, beginning with concepts and terminology. For our purposes, systems are categorized as devices, heat sources, work sources, and information sources. It is important to understand this categorization as an idealization of real physical systems with one dominant behavior.
A heat source (or sink) is a system that exchanges energy with other systems, in the form of heat but not work. Relaxation processes within a heat source are generally assumed to occur rapidly, implying that its temperature remains well defined throughout any process under consideration [70]. Moreover, if its heat capacity is sufficiently large, then that FIG. 1. Thermodynamic setup: A device exchanges heat with thermal reservoirs and work with a work source. The process is observed by a Maxwell demon. temperature remains effectively constant and the heat source can be viewed as a heat reservoir. Because we wish to make contact with the usual formulations of the Kelvin-Planck, Clausius, and Carnot statements, we will use the term heat reservoir in the analysis that follows. In particular, we will see that the assumption of a large heat capacity is crucial for the emergence of cyclic motion of the device.
Analogously, a work source (or sink) can exchange energy in the form of work but not heat, and its internal relaxation processes are again assumed to be rapid [70]. As a result, the entropy of the work source remains constant and can be neglected. The assumption of rapid relaxation will appear implicitly in our treatment: We will model the work source as a single degree of freedom-effectively, a collective coordinate such as the center of mass of a macroscopic system-while ignoring its internal degrees of freedom; this assumption prevents the energy of the work source from being lost to internal dissipation. If the inertia of a work source is sufficiently large, then its motion is largely unaffected by interactions with other systems and it can be viewed as a work reservoir. In order to model cyclic processes, we will assume a work source with an effectively infinite inertia; we will return to this point in greater detail shortly.
Heat and work sources are convenient conceptual idealizations, familiar from classic thermodynamic treatments [2,71]. To this list, we add an information source, a system that exchanges information but not energy with other systems. The information source can exist in a number of physically distinct accessible states with identical free energies. A useful example is a memory register with N bits, hence 2 N energetically degenerate states. The capacity of the information source is measured by the natural logarithm of the number of accessible states, i.e., N ln2 in the case of the memory register. When this capacity is sufficiently large, the information source becomes an information reservoir.
Finally, we will consider a device, or (sub)system, of interest, which interacts with the above-mentioned elements. These interactions give rise to the exchange of energy with the heat and work sources, and they influence the dynamics of the information source among its degenerate states. As a result, data relating to the evolution of the device may become encoded in the information source. It is precisely this possibility that adds a new element to the standard analyses of the Kelvin-Planck, Clausius, and Carnot statements.

A. First law of thermodynamics
We proceed by formulating the first law of thermodynamics within our classical, Hamiltonian framework. To begin, we restrict the analysis to include only a device, a work source, and a single heat source (later, we will add multiple heat sources and an information source), and we model these elements with a Hamiltonian with ¼ ðx; p; ; '; X; PÞ, and where H uni is our notation for the Hamiltonian of the universe. Here, ðx; pÞ denotes the microstate of the device, and ð; 'Þ is that of the heat reservoir. The bold letters indicate vectors in the configuration and momentum spaces of these subsystems. We use ðX; PÞ to specify the microstate of the work reservoir, which we model with a single degree of freedom. Finally, denotes a point in the full phase space, describing the combined microstate of all three subsystems. We view the first term on the right side of Eq. (3) H 0 as the bare Hamiltonian for the device, parametrized by the configuration of the work source X. The second term h gives the interaction between the device and the heat source, and the third and fourth terms are the bare Hamiltonians for the work and heat sources. Defining H dev ðx; p; ; '; XÞ H 0 ðx; p; XÞ þ hðx; p; ; 'Þ; (4) we have We interpret the three terms on the right side of Eq. (5) to be the instantaneous energies of the device, heat source, and work source, respectively. In our accounting, all interaction terms contribute to the energy of the device. The microscopic evolution of our composite system is described by a Hamiltonian trajectory ðtÞ, along which the value of H uni remains constant: The three subsystems exchange energy among themselves, with the total energy remaining fixed. The heat and work sources are not directly coupled to one another, but each is coupled to the device. Therefore, the rate at which the work source loses energy is interpreted as the rate at which work is performed on the device: Similarly, energy lost by the heat source is equated with heat absorbed by the device: Combining Eqs. (6)-(8), we arrive at where the dots indicate derivatives with respect to time. Equation (9) constitutes the first law of thermodynamics in our framework. By direct evaluation-using Hamilton's equation for the work source _ Thus, with Eq. (7), the work performed on the device from time t 1 to t 2 is given by which is the familiar integral of displacement Â force used in thermodynamics [72].

B. Work reservoir
As mentioned earlier, there are two assumptions one might make about the properties of the work source: rapid self-equilibration and large inertia. We have built the first assumption into our framework by modeling the work source with a single degree of freedom X. We will now also make the second assumption, which will allow us to address cyclic processes.
To formalize the assumption of large inertia, let us consider a specific example, in which a massive piston (the work source) confines a rarefied gas (the device) within a cylinder. The bare Hamiltonian for the work source is where the potential energy term models an ideal spring attached to the piston, as illustrated in Fig. 2. The piston begins in a microstate ðX 0 ; P 0 Þ, then evolves together with the gas and the surrounding thermal reservoir over a time interval 0 < t < t f , where t f specifies the duration of the process in which we are interested. In the limit M ! 1, with initial conditions ðx 0 ; p 0 ; 0 ; ' 0 ; X 0 ; P 0 =MÞ held fixed, the motion of the massive piston becomes unaffected by the remaining degrees of freedom and is given by its free dynamics where V ¼ P=M is the piston speed. (See Appendix A for details.) Thus, for sufficiently large M, we can treat the motion of the piston as fully prescribed, given the initial conditions. This observation allows us to simplify the description of the total system. The piston now evolves independently, and the device and heat source evolve under a Hamiltonian with an externally imposed time dependence determined by XðtÞ.
where ¼ ðx; p; ; 'Þ specifies a point in the reduced phase space of the device and heat source. The Hamiltonian H tot gives the combined energy of these two subsystems and generates their motion via Hamilton's equations. Because it is explicitly time dependent, its value is not preserved. Rather, the net change in H tot along a trajectory ðtÞ corresponds to the work performed on the device. This conclusion follows from energy conservation in the full phase space (6) and (7)] as well as directly from Eq. (11): Here, we have made use of the Hamiltonian identity dH tot =dt ¼ @H tot =@t [73]. By construction, H tot ð; tÞ is a periodic function of time, with period ¼ 2=!. The limit of large inertia [Eq. (13)] thus takes us from an inclusive description involving three subsystems (the device, the heat source, and the work source) to a reduced description in which the device, coupled to the heat source, is subjected to time-periodic external driving. We will continue our analysis within the reduced framework, making use of time-periodic Hamiltonians of the form given by Eq. (15). However, we emphasize that the explicit time dependence of H tot is entirely induced by the dynamics of the massive work source.
We have used the piston and spring as an illustrative example, but the work source can equally well be modeled using a generic one-dimensional potential, provided the limit M ! 1 is taken (as above) with P 0 =M held fixed. Thus, the time dependence of the coordinate XðtÞ, while periodic, need not be sinusoidal. In the remainder of the paper, we will use to denote the period of the motion of the coordinate X (in the large-inertia limit), whether it is harmonic or not.

C. Heat reservoir
The limit of large work-source inertia gives us timeperiodic driving, as we have just argued, but does not yet guarantee that the device itself relaxes to a time-periodic FIG. 2. Example of a work reservoir: The spring is attached to a piston with large mass M and confining a rarefied gas in a cylinder. The gas is in thermal contact with a heat bath of temperature À1 . steady state. For that, we will require two assumptions about the heat source, namely, that it is self-equilibrating and has a large heat capacity. In classical, macroscopic thermodynamics, these properties are among the defining properties of an idealized heat reservoir [70]. We now discuss these assumptions in the context of our explicitly microscopic setup and we formulate a plausibility argument for the emergence of a periodic steady state [Eq. (19)].
A large heat capacity implies that the number of degrees of freedom of our heat source N heat far exceeds that of the device. This statement can be formalized by considering the thermodynamic limit N heat ! 1 while holding fixed the intensive properties of the heat source-its temperature, density, and chemical composition. In this limit, the characteristic energy exchanged between the device and the heat source, during the process in question, becomes a negligible fraction of the total energy of the heat source. Therefore, its intensive properties, and particularly its temperature, remain unchanged. We take the assumption of self-equilibration to mean the following: From a generic initial microstate and in the absence of external influences, the heat source evolves to a microstate that-for the purpose of subsequent calculations-can be treated as a random sample from an equilibrium probability distribution [74]. A first-principles justification of this assumption involves issues that are well beyond the scope of this paper [75,76]. Empirically, however, macroscopic systems do relax to equilibrium when left undisturbed (leaving aside special cases such as glassy systems), and these equilibrium states are accurately modeled by the standard probability distributions of classical statistical mechanics. We will therefore assume that the heat source satisfies the property of selfequilibration, and we will investigate the consequences of this assumption.
Let us first consider the extreme limit, in which relaxation to equilibrium occurs on a time scale that is much faster than any other relevant time scale in our problem. In this case, even when the heat source interacts and exchanges energy with the device, its microstate at any instant can be treated as a random sample from an equilibrium ensemble. Effectively, then, the heat source evolves through a sequence of equilibrium states, as it absorbs or releases energy. Moreover, in the limit of infinite heat capacity, its temperature remains constant ! N heat !1 0 , as discussed in a previous paragraph. Now, let z n ¼ ½xðnÞ; pðnÞ denote the microstate of the device at the start of the nth period, and similarly define Z n ¼ ½ðnÞ; 'ðnÞ for the heat source. The evolution of the combined system from one period to the next is given by the iteration of a deterministic mapping: ÁÁÁ ! ðz nÀ1 ;Z nÀ1 Þ ! ðz n ;Z n Þ ! ðz nþ1 ;Z nþ1 Þ ! ÁÁÁ: (17) Each microstate n in this sequence is reached from the previous one, by evolving under Hamilton's equations for one period of the time-dependent Hamiltonian H tot ð; tÞ. In the limit of extremely rapid self-equilibration of the heat source, the Z n 's effectively become uncorrelated random samples from a fixed equilibrium distribution. Abstractly, we can view Z n as a set of freshly generated random numbers that collectively determine the value of z nþ1 , given z n ; in the next iteration, a new set of random numbers Z nþ1 determines the transition from z nþ1 to z nþ2 , and so forth. Adopting this perspective, the stroboscopic evolution of the device from one period to the next is given by the iteration of a stationary, stochastic, Markovian mapping. This Markov chain relaxes to a unique stationary state described by a fixed, generally nonequilibrium distribution " dev ðx; pÞ. (This result is a consequence of the Perron-Frobenius theorem [77], under standard assumptions.) Therefore, the time-dependent probability distribution for the device relaxes to a periodic steady state dev ðx; p; t þ Þ ¼ dev ðx; p; tÞ; where dev ðx; p; nÞ ¼ " dev ðx; pÞ. To reach Eq. (19), we have assumed that the selfequilibration of the heat source occurs, in effect, infinitely rapidly. Now, we loosen this assumption by allowing the relaxation time scale of the heat source to be comparable to other time scales in the problem. In this case, Z nþ1 in Eq. (17) may be statistically correlated with Z n and with z n . Nevertheless, it is reasonable to assume that there exists some integer K > 0, such that Z nþK is statistically uncorrelated with Z n and z n . In other words, a time interval of duration K is sufficient for the heat source to ''forget'' its microstate. Then, the stroboscopic evolution of the device in time increments K is a Markov chain. The Z n 's are no longer necessarily sampled from equilibrium. However, if the heat source itself reaches a stationary state, in which the energy exchanged with the device is transported at a fixed rate to more distant regions of the heat source, then Eq. (20) becomes a stationary Markov chain, and the final arguments of the previous paragraph continue to apply: The device eventually relaxes to a periodic steady state. As mentioned, the reasoning of the preceding paragraphs is intended as a plausibility argument for the emergence of cyclic motion of the device, under the conditions and limits we have discussed: The large inertia of the work source induces a time-periodic Hamiltonian for the device, and the large heat capacity and self-equilibration of the heat source cause the relaxation of the device into a timeperiodic steady state. For the remainder of this paper, we will assume that these arguments apply-hence, the device reaches a periodic steady state-and we will explore their INFORMATION PROCESSING AND THE SECOND LAW OF . . . PHYS. REV. X 3, 041003 (2013) 041003-5 consequences. As suggested at the beginning of Sec. II, we will henceforth use the terms work reservoir and heat reservoir. We note that once the device has reached a periodic steady state, both its internal energy and its Shannon entropy become time periodic as well: We will make use of this observation in our later analysis.

D. Information reservoir
In the preceding subsections, work and heat reservoirs have been discussed within a classical, Hamiltonian framework. We now complete this framework by introducing the possibility of information processing. In effect, we aim to describe thermodynamic processes in the presence of a physical device capable of acting like Maxwell's demon, performing microscopic measurements and feedback on the other subsystems in our picture. The key feature that we wish to capture is the demon's memory, where it stores information that it has gathered. To this end, we introduce an idealized information reservoir, representing the demon's memory. All other components of the mechanical demon are implicitly treated as belonging to the device of interest.
To describe the complete system consisting of device, heat reservoir, and information reservoir, the total Hamiltonian (15) is extended to read where ðÄ; ÈÞ is the microstate of the information reservoir and H info is its bare Hamiltonian. The term hð; Ä; ÈÞ describes the interaction between the information reservoir and the device and thermal reservoir. The assumption that the information and thermal reservoirs are coupled is important for the following discussion.
Let us first describe the information reservoir in the presence of a single thermal reservoir, at inverse temperature , before discussing its interaction with the device.
For specificity, we will take the information reservoir to be a memory register consisting of N bits [78]. A single bit is physically implemented using a large collection of atoms or molecules, whose total magnetization (or some other collective observable) acts as a binary order parameter. We will distinguish between the microstate of the information reservoir c ðÄ; ÈÞ and its informational state . The microstate c is a point in the phase space of the entire collection of atoms and molecules comprising the memory register, whereas the informational state is a given sequence of bit values, e.g., 0110 Á Á Á 10. We assume that each microstate c corresponds to a particular informational state , and we will use the functionðc Þ to specify the informational state associated with the microstate c .
The variables c and thus represent fine-grained and coarse-grained descriptions of the state of the information reservoir.
The functionðc Þ partitions the phase space of the information reservoir into 2 N distinct regions, each corresponding to one informational state. To guarantee a stable and reliable memory register, we assume these regions are separated by large free-energetic barriers, so that over the time scales that concern us, the probability of a spontaneous, thermally driven transition from one informational state to another is negligible. It then becomes useful to consider a constrained equilibrium state, described by a conditional probability distribution p eq ðc jÞ ¼ ;ðc Þ expðÀ½H info ðc Þ À F info Þ: (23) Here, the Kronecker function acts as an indicator variable; hence, p eq ðc jÞ is simply a canonical probability distribution, restricted to the region of phase space corresponding to the information state . The free energy F info is determined by normalization R dc p eq ðc jÞ ¼ 1, and in the usual manner, we can define an equilibrium internal energy and entropy hH info i eq; ¼ Z dc p eq ðc jÞH info ðc Þ; H eq; info ¼ À Z dc p eq ðc jÞ lnp eq ðc jÞ: Equation (23) represents the statistical state of the information reservoir, when it has been left undisturbed in the informational state , in the presence of a thermal reservoir. Following Bennett [79], we will refer to the N bits as information-bearing degrees of freedom, or IBD, and the remaining microscopic variables as non-informationbearing degrees of freedom, or NBD. Using this terminology, Eq. (23) represents an equilibrium state of the NBD (c j) for a given state of the IBD ().
Let us now consider the behavior of the information reservoir in the presence of the device of interest. We explicitly assume that interactions with the device can give rise to transitions among the informational states. In this manner, information about the evolution of the device of interest becomes encoded in the IBD. Let us further assume that (1) the 2 N informational states have the same equilibrium energies and entropies and (2) after a transition from one informational state to another, thermal equilibration of the NBD occurs rapidly. Under these assumptions, the energy of the information reservoir effectively remains constant, aside from equilibrium thermal fluctuations. In the presence of the device of interest and thermal reservoir, the evolution of the information reservoir is a sequence of transitions from one equilibrated informational state to another.
The total information encoded in the reservoir is quantified by its Shannon entropy, which can formally be decomposed into contributions from the informationbearing and non-information-bearing degrees of freedom: as we show in Appendix B. Moreover, under the assumptions of the previous paragraph, H NBD info ðtÞ does not vary with time and is given simply by the equilibrium entropy of the microscopic, non-information-bearing degrees of freedom (again, see Appendix B for details). As a result, any change in the Shannon entropy of the information reservoir, resulting from its interactions with the device over an interval of time, is entirely captured by the net change in the probability distribution of the mesoscopic, information-bearing degrees of freedom: In Secs. III, IV, and V below, we will use the notation ÁH info rather than ÁH IBD info , to avoid clutter, but it will be understood that the net change in the Shannon entropy of the information reservoir refers to the change in its information-bearing degrees of freedom.

III. NON-NEGATIVITY OF INFORMATION EXCHANGE
The rest of this paper is devoted to investigating specific thermodynamic processes within the framework introduced above. To this end, we begin by obtaining an inequality for the sum of changes of the Shannon entropy for the individual subsystems [Eq. (33)] from which we derive an inequality related to the behavior of our system in the periodic steady state [Eq. (43)]. The latter result and its generalization [Eq. (44)] will then be exploited in Sec. IV. As in the recent work of Hasegawa et al. [15,17] and Esposito et al. [16,18], our approach in this section will draw on properties of the canonical distribution, the Shannon entropy, and the Kullback-Leibler divergence [80], as well as assumptions about the initial state of the system.
We adopt an explicitly statistical perspective, in which we consider an ensemble representing different possible microscopic realizations of the process. The probability distribution in the full phase space at an initial time t ¼ 0 reflects the preparation of the system prior to this time, and we now spell out the assumptions that we make regarding this preparation. As in Refs. [7,8,[15][16][17][18], we assume that the total system begins in a product state tot ð; Ä; È; 0Þ ¼ dev ðx; p; 0Þ Â heat ð; '; 0Þ info ðÄ; È; 0Þ: (27) This assumption does not substantially restrict the generality of the following discussion, as we expect the device to relax into a time-periodic steady state that is independent of its initial preparation (see Sec. II). For the time being, we restrict ourselves to considering only a single thermal reservoir, but as discussed below, the results generalize easily to multiple reservoirs; see, e.g., Eq. (44).
We take the initial state of the heat reservoir to be given by the canonical distribution heat ð; '; 0Þ ¼ 1 Z heat exp½ÀH heat ð; 'Þ eq heat ð; 'Þ; (28) with free energy F heat ¼ À À1 lnZ heat . For the distribution info ðÄ; È; 0Þ, we assume that the microscopic, non-information-bearing degrees of freedom are in equilibrium with the thermal reservoir (see Sec. II D), whereas the distribution of the mesoscopic, information-bearing degrees of freedom reflects the manner in which the information reservoir was prepared. For instance, the memory register might be initialized in a blank state 000 Á Á Á 0, in which case H IBD info ¼ 0. At the other extreme, it may be prepared so that every possible N-bit sequence (informational state) is equally likely; hence, H IBD info ¼ N ln2. We will not place any restrictions on the initial statistical state of the device dev ðx; p; 0Þ.
After the initial preparation, the full system evolves under the time-periodic Hamiltonian given by Eq. (22). In general, the total density at time t > 0, tot ð; Ä; È; tÞ, will not be a product state, and the reduced densities for device, heat reservoir, and memory are obtained by integrating out the other subsystems. Thus, for the device, we have dev ðx; p; tÞ ¼ Z dd' Z dÄdÈ tot ð; Ä; È; tÞ; (29) and similarly for the heat and information reservoir. We can use these reduced densities to define the Shannon entropy [81] of each subsystem, e.g., H dev ðtÞ ¼ Àtrf dev ðtÞ ln dev ðtÞg À Z dxdp dev ðx; p; tÞ ln dev ðx; p; tÞ; (30) and analogously for the heat and information reservoir. By Liouville's theorem, the Shannon entropy of the total system remains constant under Hamiltonian dynamics: H tot ðtÞ ¼ H tot ð0Þ. Moreover, since the system is prepared in a product state (27) (32) due to the subadditivity of the Shannon entropy [81]. Subtracting Eq. (31) from Eq. (32), we obtain where ÁH dev denotes the net change in the Shannon entropy of the device of interest, and similarly for the other subsystems.
For the heat reservoir, we can write H heat ðtÞ ¼ Àtrf heat ðtÞ ln heat ðtÞg ¼ E heat ðtÞ À F heat À Dð heat ðtÞ k eq heat Þ; where E heat ðtÞ trfH heat heat ðtÞg is the average energy of the reservoir, and DðÁ k ÁÞ denotes the Kullback-Leibler divergence [80] Dð k Þ ¼ trf lng À trf lng ! 0: Combining Eq. (34) with our assumption that the heat reservoir is prepared in equilibrium [Eq. (28)], we get ÁH heat ¼ E heat ðtÞ À E heat ð0Þ À Dð heat ðtÞ k eq heat Þ ðE heat ðtÞ À E heat ð0ÞÞ ¼ ÁE heat ; using Eq. (36). This result in turn combines with Eq. (33) to give Note that we have taken two distinct steps to arrive at Eq. (38). First, we have obtained Eq. (33) from our assumption that the subsystems are statistically uncorrelated at the initial time [Eq. (27)]. In fact, the left side of Eq. (33) quantifies the degree to which correlations develop between the subsystems because of their mutual interactions; Esposito et al. [16] have explicitly interpreted this buildup of correlations as representing entropy production. Next, to get to Eq. (38), we have used the assumption that the reservoir is initialized in the canonical distribution [Eq. (28)], together with the non-negativity of the Kullback-Leibler divergence. Similar manipulations appear in Refs. [15][16][17][18]. We will now use Eq. (38) to arrive at inequalities that characterize the behavior of our system in the periodic steady state.
A natural time scale for our process is given by the driving period (Sec. II B). Let us set t ¼ n > n 0 , where n 0 is the number of periods needed for the device to relax into its periodic steady state. Then, the process in question can be divided into a transient interval (0 ! n 0 ) followed by an interval of time-periodic behavior (n 0 ! n). Expressing each term in Eq. (38) as a sum of contributions from these two intervals, we get ÁH 0!n 0 dev þ ÁE 0!n 0 heat þ ÁH 0!n 0 info þ ÁH n 0 !n dev þ ÁE n 0 !n heat þ ÁH n 0 !n info ! 0; which can further be rewritten as ÁH 0!n 0 dev þ ÁE 0!n 0 heat þ ÁH 0!n 0 info þ ðn À n 0 ÞðÁE cyc heat þ ÁH cyc info Þ ! 0; where ÁE cyc heat ¼ is the average heat absorbed by the heat reservoir, per cycle, in the periodic steady state, and is the average change in the Shannon entropy of the information reservoir, per cycle, in the periodic steady state. Note that the similarly defined quantity ÁH cyc dev vanishes, by Eq. (21). Dividing both sides of the inequality in Eq. (40) by ðn À n 0 Þ, then taking the limit n ! 1, we finally obtain ÁE cyc heat þ ÁH cyc info ! 0: In Sec. IV, we will exploit this result [or its generalization, Eq. (44)] to obtain generalized versions of the Kelvin-Planck, the Clausius, and the Carnot statements of the second law. Equation (43) has a simple interpretation: The first term on the left represents the net change in the thermodynamic entropy of the heat reservoir, and the second term is the net change in the Shannon entropy of the information reservoir, specifically, its information-bearing degrees of freedom. Either term can be positive, negative, or zero, but their sum must be non-negative.
In the preceding analysis, for convenience, we have restricted ourselves to a single heat reservoir. The arguments are readily generalized to the case of multiple heat reservoirs by replacing the change of Shannon entropy for one reservoir by a sum over all reservoirs, in Eq. (33), and by assuming that each heat reservoir is independently prepared in a canonical distribution corresponding to a particular temperature. In particular, for one hot and one cold reservoir, Eq. (43) becomes hot ÁE cyc hot þ cold ÁE cyc cold þ ÁH cyc info ! 0: In the case of multiple reservoirs, we will assume that the information reservoir is coupled only to a single thermal reservoir, and its microscopic degrees of freedom remain in equilibrium at the corresponding temperature.
The results that we derive in the following section do not depend on which reservoir is selected for this role.

IV. THE SECOND LAW AND INFORMATION PROCESSING
In the periodic steady state, hH dev ðt þ Þi ¼ hH dev ðtÞi [Eq. (21)]. Therefore, by the first law of thermodynamics, integrating Eq. (9) over a single cycle, we have hW cyc i þ hQ cyc i ¼ 0; (45) where W cyc is the net work performed on the device, and Q cyc is the net heat absorbed by the device (from one or more heat reservoirs) over one cycle in the periodic steady state, and angular brackets denote averages over many realizations. By Eq. (8), the heat absorbed by the system is defined as the net decrease in the bare energies of the heat reservoir(s); hence, Eq. (45) becomes Kelvin-Planck statement.-The Kelvin-Planck statement [1] expresses the observation that in cyclic, isothermal processes, the average work is always non-negative hW cyc i ! 0. To generalize this statement, we consider a device of interest, coupled to a single heat reservoir at inverse temperature , a work reservoir, and an information reservoir. Equation (46) then combines with Eq. (43) to give which constitutes a generalized version of the Kelvin-Planck statement. For processes during which information is written to the information reservoir (ÁH cyc info > 0), the net work over one cycle can be negative. In other words, there can be a systematic transfer of energy from the heat reservoir to the work reservoir, provided the Shannon entropy of the information reservoir increases. This conclusion is consistent with the current consensus regarding the Maxwell demon paradox [25][26][27]31]. For processes during which information is erased (ÁH cyc info <0), Eq. (47) becomes equivalent to Landauer's principle [8,15,17,18,24], placing a lower limit on the amount of work that must be expended in order to accomplish this erasure.
Clausius statement.-To generalize the Clausius statement, we consider a device interacting with two heat reservoirs, one hot and one cold, as well as a work reservoir and an information reservoir. As above, these interactions produce exchanges of both energy and information. In general, the net work performed on the device over one cycle can have either sign, and the device may be able to operate as either a heat engine or a refrigerator. For the Clausius statement, we restrict our attention to processes for which hW cyc i ¼ 0. In this case, Eq. (46) becomes Consequently, Eq. (44) can be written as which generalizes the Clausius statement. Note that the left side of Eq. (49) represents classical thermodynamic entropy, i.e., the heat exchanged over temperature, whereas the rights side quantifies the internal information gain in the memory. Since cold > hot , Eq. (49) allows for processes during which heat flows systematically from cold to hot (hQ hot i < 0 < hQ cold i), provided information is written to the memory, as illustrated schematically in Fig. 3. Conversely, if information is to be erased, then the right side of the inequality is positive, and we get a lower bound on the amount of heat that must flow from the hot to the cold reservoir. For the erasure of one bit of information per cycle ÁH cyc info ¼ À ln2, the average heat flow must satisfy hQ cyc hot i ! ð cold À hot Þ À1 ln2; which represents a modified version of Landauer's principle [35].
Carnot statement.-The Carnot statement asserts that the efficiency of a heat engine is always less than the Carnot efficiency C 1 À hot = cold [4]. To generalize this result, we again consider a device interacting with two heat reservoirs, a work reservoir, and an information reservoir, but now we consider processes for which the work performed on the device over one cycle is negative (in other words, the device delivers work) and the heat absorbed from the hot reservoir is positive; hence, the device operates as a heat engine, with efficiency ¼ ÀhW cyc i=hQ cyc hot i > 0. Equation (44) takes the form À hot hQ cyc hot i À cold hQ cyc cold i þ ÁH cyc info ! 0; and Eq. (45) can be written as hQ cyc cold i ¼ ÀhW cyc i À hQ cyc hot i: Combining these equations, we obtain À hot hQ cyc hot i þ cold hW cyc i þ cold hQ cyc hot iþ ÁH cyc info ! 0: After rearrangement of terms, we find that the efficiency must satisfy Thus, for cyclic processes in which information is systematically written to the memory, the efficiency can exceed the Carnot limit. Note that Eq. (54) does not depend on whether the information reservoir is coupled to the hot or the cold heat reservoir.

V. MAXIMUM WORK THEOREM
In Sec. IV, we considered only cyclic processes. Now, let us briefly consider what happens when we relax this restriction. For noncyclic processes in the presence of a single heat reservoir, the second law is formulated in terms of the Helmholtz free energy F ¼ E À À1 S, where E ¼ hHi is the mean internal energy, F the free energy, and S the thermodynamic entropy of the system in question, in a state of thermal equilibrium. If the system begins in one equilibrium state and ends in another, then the average work performed on the system during the process satisfies hWi ! ÁF, where the equality holds for reversible processes. Equivalently, the decrease in free energy gives the maximum usable, i.e., extractable, work during such a process.
To generalize this result, we consider a device of interest, coupled to a single heat reservoir, a work reservoir, and an information reservoir, without assuming cyclic motion. As before, we assume an initial product state [Eq. (27)] without imposing any restrictions on the initial state of the device and we imagine observing the entire system over some interval of time. Integrating Eq. (9) over this interval, we get averaging over many realizations of the process. Combining this result with Eq. (38), which was derived without assuming cyclic processes, we obtain ÁH dev À ÁE dev þ hWi þ ÁH info ! 0: (56) In order to further simplify Eq. (56), we introduce the information free energy F ¼ E dev À À1 H dev ¼ F þ Dð k eq Þ; (57) which generalizes the equilibrium free energy to an arbitrary nonequilibrium state characterized by a probability distribution . This nonequilibrium free energy has previously appeared in both Hamiltonian treatments [15][16][17], for instance, to derive Landauer's principle [18], as well as stochastic treatments [19,[82][83][84][85]. A generalized free energy of this form has also appeared in the thermodynamic description of open system dynamics [86]. More recently, it was shown that F is a Lyapunov function for nonequilibrium stationary states [54]. In terms of this quantity, Eq. (56) becomes hWi ! ÁF À ÁH info ; which is a generalized version of the maximum work theorem. If the device begins and ends in equilibrium, ÁF is replaced by the equilibrium free energy difference ÁF. Equation (58) is similar to a version of the maximum work theorem applicable to systems with external feedback control [42,52,54,66,69]: hWi ! ÁF À hIi: Here, hIi denotes the mutual information that quantifies the quality of the measurements that are performed by an external agent. In Eq. (58), by contrast, ÁH info is the change of Shannon entropy of an explicitly modeled subsystem (our information reservoir), without reference to feedback control. See Ref. [19] for a treatment that combines both perspectives.

VI. CONCLUDING REMARKS
By categorizing thermodynamics systems as devices, thermal reservoirs, work reservoirs, and information reservoirs, we have developed an inclusive approach for investigating the thermodynamics of information processing, in which all participating subsystems are explicitly modeled. This approach is based on autonomous evolution under a time-independent Hamiltonian, supplemented by a number of limits, approximations, and assumptions, spelled out in Sec. II. Our main results in Sec. IV generalize the Kelvin-Planck, Clausius, and Carnot statements for cyclic thermodynamic processes and they support the consensus view [24][25][26][27] that the Shannon entropy in a random data set (as encoded by a memory register's information-bearing degrees of freedom, for instance) should be placed on the same footing as the Clausius entropy when analyzing the second law of thermodynamics. Thus, for example, work can systematically be extracted from a single heat bath, heat can flow from cold to hot, and the Carnot efficiency can be exceeded, provided these entropy-decreasing consequences are compensated by the writing of information to a memory register. Section V extends these results to noncyclic processes in the form of a generalized maximum work principle.
As mentioned, our derivations have elements in common with previous treatments, particularly those of Refs. [15][16][17][18][19]. However, our focus on a fully autonomous, inclusive framework, on cyclic processes, and on the designation of an information reservoir as a separate element in thermodynamic analyses distinguishes our approach. In the spirit of a fully inclusive framework, Maes and Tasaki [21] have derived the maximum work statement of the second law of thermodynamics using a time-independent Hamiltonian. Their emphasis is on a mathematically rigorous treatment and does not focus on information processing.
Very recently, Tasaki [87] has analyzed a Hamiltonian model of Maxwell's demon, involving an engine and a memory that interact by the exchange of information, and Barato and Seifert [88] have investigated feedback control with an explicit information reservoir, within the framework of stochastic thermodynamics.
Finally, it is worth mentioning that, at least formally, the present analysis can be extended to quantum-mechanical systems. In place of Hamiltonian dynamics, one would use the unitary dynamics of the ''universe'' under consideration, the Shannon entropy would be replaced by the von Neumann entropy, and classical ensemble averages