Second law for active heat engines

Macroscopic cyclic heat engines have been a major motivation for the emergence of thermodynamics. In the last decade, cyclic heat engines that have large fluctuations and operate at finite time were studied within the more modern framework of stochastic thermodynamics. The second law for such heat engines states that the efficiency cannot be larger than the Carnot efficiency. The concept of cyclic active heat engines for a system in the presence of hidden dissipative degrees of freedom, also known as a nonequilibrium or active reservoir, has also been studied in theory and experiment. Such active engines show rather interesting behavior such as an ``efficiency'' larger than the Carnot bound. They are also likely to play an important role in future developments, given the ubiquitous presence of active media. However, a general second law for cyclic active heat engines has been lacking so far. Here, upon using a known inequality in stochastic stochastic thermodynamics for the excess entropy, we obtain a general second law for active heat engines, which does not involve the energy dissipation of the hidden degrees of freedom and is expressed in terms of quantities that can be measured directly from the observable degrees of freedom. Besides heat and work, our second law contains an information-theoretic term, which allows an active heat engine to extract work beyond the limits valid for a passive heat engine. To obtain a second law expressed in terms of observable variables in the presence of hidden degrees of freedom we introduce a coarse-grained excess entropy and prove a fluctuation theorem for this quantity.


I. INTRODUCTION
Thermodynamics [1] is a major theory in physics that deals with the transfer of heat into other forms of energy. This theory provides universal bounds through its second law on processes that involve heat transfer. Cyclic engines that transform part of the heat taken from a hot reservoir into useful work are central objects in thermodynamics. One of the most prominent statements of the second law of thermodynamics is the Carnot bound on the efficiency of heat engines. Yet standard thermodynamics has considerable limitations: only heat engines with negligible fluctuations and that operate in the quasistatic limit can be analyzed quantitatively.
Real engines, however, operate at finite time. Furthermore, it is now possible to realize in experiments heat engines made of a small number of constituents, for which fluctuations cannot be neglected. The modern theory of stochastic thermodynamics [2,3] does not suffer from the same limitations as standard thermodynamics and is able to deal with such heat engines. Within stochastic thermodynamics a model for a cyclic heat engine with a colloidal particle in a harmonic potential has been proposed in [4]. This model has been realized in an experiment in [5]. Much theoretical [6][7][8][9][10][11][12] and experimental [13][14][15][16][17] work has been done on such engines. For such heat engines, the Carnot bound on efficiency remains valid.
A novel perspective was put forward in an experiment of a so-called active heat engine [18]. This cyclic heat engine is likewise composed of a colloidal particle in a harmonic trap. However, in contrast to a passive heat engine the colloidal particle is not immersed in a solution that simply acts as an equilibrium heat bath. Rather the solution has bacteria that collide with the particle.
Such a bacterial bath may be called an nonequilibrium or active reservoir. Key observations made in this work were that an active heat engine can extract more work than its passive counter-part and that the pseudo-efficiency (work divided by heat taken from hot reservoir) of the engine can be larger than the Carnot efficiency, which might be called a "violation" of the second law.
A standard explanation that can be given to the observation of a pseudo-efficiency larger than the Carnot efficiency is that if we properly account for the energy dissipated by the bacteria then there is no violation of the second law. This explanation is in principle correct, if we also consider the dissipation of the bacteria then the total entropy production must be positive. However, such statement of the second law is rather limited. First, the energy dissipation of the bacteria is probably much larger than the extracted work and would be the dominating term is such second law. Hence, this second law would not represent a meaningful bound on the amount of extracted work. Second, this energy dissipation is related to degrees of freedom that are not accessible in the experiment since only the position of the colloidal particle particle is measured.
Beyond this particular experiment, an active heat engine is an interesting concept. We can imagine a cyclic heat engine with a system composed of a few degrees of freedom. This system interacts with, possibly hidden, dissipative degrees freedom that make the external medium active. A second law that contains the energy dissipation due to this activity has two main issues. First, the energy dissipated per cycle period due to the activity increases indefinitely if we increase the cycle period, since a nonequilibrium system dissipates energy even after reaching a steady state. However, the extracted work per period saturates and becomes a constant for large period. Second, this energy dissipation might be related to degrees of freedom that cannot be observed in an experiment and their measurement is not necessary to calculate extracted work.
What is the correct statement of the second law for active heat engines? Such statement must fulfill the following requirements. First, the bound on extracted work can only contain terms that have the same scaling with the period of the engine. Second, it must accommodate the observations made in the experiment from [18], such as a pseudo-efficiency larger than the Carnot efficiency. Third, for a passive heat engine, it must become the known standard second law. Fourth, it must contain terms that depend only on the observable degrees of freedom.
In this paper we show that the appropriate statement of the second law for generic cyclic active heat engines, which fulfills all the aforementioned requirements, is obtained from a well known quantity in stochastic thermodynamics called excess entropy [3,[41][42][43][44][45][46] (or the related non-adiabatic entropy [47][48][49]). Crucially, we obtain this result with a suitable decomposition of the excess entropy. Beyond the known excess entropy, in order to fulfill the fourth requirement, we introduce a coarse-grained excess entropy and prove a fluctuation theorem for it.
This decomposition of the excess entropy for cyclic active heat engines leads to three terms, the heat taken from the hot reservoir, the extracted work and a novel term that has an information theoretical interpretation. Interestingly, this term relates the non-equilibrium steady state distribution associated with the active heat engine to the equilibrium Boltzmann distribution associated with its passive counterpart. Our results are illustrated with three models that allow for analytical calculations and one numerical model inspired by the experiment in [18].
The paper is organized as follows. In Sec. II we use a two-state model to illustrate our main result. The general formalism of our second law for active heat engines is introduced in Sec. III. In Sec. IV we introduce the coarse-grained second law, which is expressed in terms of quantities that only depend on an observable degree of freedom. In Sec. V we discuss three simple models for active heat engines that allow for analytical calculations. Sec. VI contains a more complex model that is analyzed with numerics. We conclude in Sec. VII. The fluctuation theorem for the coarse-grained excess entropy is proved in the appendix. 1. (a) Depiction of the protocol for the two-state heat engine with a period divided into four parts labeled by l. The four changes are as follows: from l = 1 to l = 2 the energy changes from E to E + δE, from l = 2 to l = 3 the inverse temperature changes from βc to β h , from l = 3 to l = 4 the energy changes from from E + δE to E, and from l = 4 to l = 1 the inverse temperature changes from β h to βc. (b) Possible reactions for the two-state system. The transition rate for the forward (backward) phosphorylation reaction in Eq. (3) is κ+ (κ−) and the transition rate for the forward (backward) dephosphorylation reaction in Eq. (4) is ω+ (ω−). We parametrize the transition rates as κ + (t) = ω − (t) = k, κ − (t) = ke β(t)[E(t)+∆µ] , and ω + (t) = ke β(t)E(t) .

II. SECOND LAW FOR TWO-STATE HEAT ENGINE
Consider the two-state heat engine shown in Fig. 1. It has a down state with energy zero and an up state with energy larger than zero. The protocol of this engine with period τ is as follows. It has four parts, each with duration τ l , with τ 1 + τ 2 + τ 3 + τ 4 = τ . There are two energy changes: at time τ 1 the energy changes from E to E + δE and at time τ 1 + τ 2 + τ 3 the energy changes back from E + δE to E. Moreover, there are two temperature changes: at time τ 1 +τ 2 the temperature changes from β c to β h and at time τ the temperature changes back from β h to β c .
Work can be extracted in this cycle in the following way. If the system finishes the third part at the up state then an amount δE of work is extracted. Moreover, if the system finishes the first part at the up state then an amount δE of work is spent on the system (−δE of extracted work). Hence, the maximum amount of work that can be extracted in a cycle is δE. A similar passive two-state heat engine has been studied in [6]. The known second law for this passive heat engine reads where Q h is the average heat taken from the hot reservoir during a period and W is the average extracted work during a period. This second law implies the well known bound on the efficiency of a passive engine, where η C is the Carnot efficiency. Now we consider an active two-state heat engine. The simplest way to make the engine active is to include two different transition paths between the two states. A possible physical interpretation shown in Fig. 1(b) would be an enzyme that can be in two states, X and XP , where XP is the phosphorylated form of the enzyme. The state X has energy 0 and the state XP has non-zero energy, which can be either E or E + δE. One transition path from X to XP is a phosphorylation reaction The other transition path is a dephosphorylation reaction During any of the four steps of the cycle the enzyme burns ATP. The free energy of one ATP hydrolysis is written as ∆µ = µ AT P − µ ADP − µ P , where µ is the chemical potential. This exactly solvable model for an active heat engine is fully defined in Sec. V B. In this section we use the results obtained with this model for illustrative purposes. If ∆µ = 0 the heat engine is a standard passive heat engine and the second law in Eq. (1) applies. For ∆µ = 0 the heat engine is an active heat engine. As shown in Fig.  2, this simple active heat engine produces remarkable behavior. First, making the engine active with a non-zero ∆µ can increase the amount of extracted work. Second, the pseudo-efficiency can be larger than the Carnot efficiency η C . The heat Q h requires a word of caution. This quantity is not exactly the heat taken from the hot reservoir for an active heat engine. This passive heat Q h only accounts for the contributions coming from the energy changes of the system associated with transitions between the two states during the part of the period for which the inverse temperature is β h . This Q h does not account for dissipated heat associated with ATP consumption. The first solution for the issue η ps > η C is to account for the average entropy production associated with the consumption of ATP during a cycle ∆S act . The standard second law from stochastic thermodynamics for this model reads The condition η ps > η C is then fully consistent with this second law since η ps can be larger than η C if ∆S act > 0. However, it is often the case that this second law does not produce a relevant bound on the rate of extracted work W . Let us consider the period of the engine τ . For large τ , the amount of entropy produced due to the ATP consumption ∆S act increases linearly with τ . This scaling with the period τ is different from the scaling of Q h and W . For instance, the stochastic work that can be extracted in a cycle is at most δE. Hence, for large τ the average work extracted per cycle cannot be more than the constant δE. This difference in scaling is illustrated in Fig. 3. We conclude that in the regime of large τ the standard second law in Eq. (6) does not produce a relevant bound, ∆S act dominates the inequality, which simply states that ∆S act is positive.
What is the appropriate statement of the second law that does not suffer from the limitation discussed above? Before we answer this question let us consider the following question. How can an active heat engine deliver more work than a passive heat engine? If we introduce a non-zero ∆µ in our model the probability of a state is shifted. If this shift favors work extraction, then the active heat engine will deliver more work than its passive counterpart.
Our main finding is that the appropriate statement of the second law for active heat engines come from a quantity known as excess entropy. A main interest in this quantity in stochastic thermodynamics comes from the fact that it fulfills a fluctuation theorem [3]. Here we show that it has a central role for cyclic active heat engines. For the present model this second law derived from the excess entropy reads The term I is the novel term compared to Eq. (6). It has an information theoretic expression and it quantifies the shift in the probability of a state due to a non-zero ∆µ. A positive I allows an active heat engine to operate beyond the limits that are valid for a passive heat engine, as illustrated in Fig. 2.
Concerning the scaling with the period τ , I scales in the same way as W and Q h . Furthermore, we show that ∆S act ≥ I independently of the value of τ , as illustrated in Fig. 3. The second law above is consistent with η ps > η C . For a passive heat engine I is zero and Eq. (7) becomes the known second law for passive heat engines in Eq. (1).
This minimal two-state model does not allow for the discussion of the issue of coarse-graining. For the case with possible hidden dissipative degrees of freedom we obtain a second law that has the exact same structure of Eq. (7), as shown in Sec. IV.

A. Invariant distribution and accompanying distribution
We consider Markov processes with discrete states. A transition rate from state i to state j is denoted k ij (t).
Since we want to model cyclic engines, the transition rate is periodic in time with a period τ , i.e., k ij (t) = k ij (t+τ ). The master equation for the probability to be in state i at time t is The following condition is assumed to hold, if k ij (t) = 0 then k ji (t) = 0. In the long time limit, P i (t) becomes an invariant time-periodic distribution with period τ . We denote this invariant distribution simply by P i (t).
The accompanying distribution [50,51] P S i (t) is generally different from P i (t). It is defined as the stationary distribution the system would reach if the transition rates were fixed and given by k ij (t). Hence, P S i (t) is also timeperiodic with period τ . Specifically, P S i (t) is the solution of the equation which is the stationary form of Eq. (8) for fixed t. If each term in the sum in j is zero, i.e., if detailed balance is fulfilled, then this accompanying distribution P S i (t) is an equilibrium distribution denoted by P eq i (t). Otherwise, the accompanying distribution corresponds to a nonequilibrium stationary distribution denoted by P S i (t). We reiterate the difference between three probability distributions we consider here. First, P i (t) is the timeperiodic distribution of the system, corresponding to the long time solution of Eq. (8). Second, P S i (t) is the accompanying distribution corresponding to a nonequilibrium steady state. Third, P eq i (t) is the accompanying distribution corresponding to an equilibrium steady state.

B. Generalized detailed balance
The periodicity of the transition rates comes from the fact that energies and temperature are periodic. The energy of state i at time t is denoted E i (t) = E i (t + τ ). The inverse temperature remains between β h and β c and is written as where h(t) is a dimensionless time-periodic function bounded by the inequalities 0 ≤ h(t) ≤ 1. If h(t) = 0 then β(t) = β c and if h(t) = 1 then β(t) = β h . An active heat engine must contain internal thermodynamic forces or affinities. An internal affinity is denoted by A α . We use the index α since the system can have more than one internal affinity. The generalized distance associated with this affinity for a transition from i to j is d α ij (t) = d α ij (t + τ ). This distance is antisymmetric, i.e., d α ij (t) = −d α ji (t). For example, if A α is a force applied to a colloid then d α ij (t) is the spatial distance between i and j.
The generalized detailed balance relation [3] is a postulate of stochastic thermodynamics that relates transition rates to physical parameters such as temperature, free energy, and affinities. This relation is written in the form There is no loss of generality in assuming a constant A α since a time-dependence of A α can be absorbed by the timedependence of the distance d α ij (t). We can now give a clear mathematical definition for active and passive heat engines. If A α = 0 for all α then the heat engine is a passive heat engine. In this case the stationary distribution the system would have if the protocol was frozen, i.e., the accompanying distribution, which corresponds to the solution of Eq. (9), is the Boltzmann distribution P eq is the partition function. Note that a passive heat engine is a nonequilibrium system due to the periodic variation of temperature and energies. For nonzero A α the accompanying distribution P S i (t) is not the equilibrium distribution and the heat engine is active.

C. Entropy production
The elementary (probability) current from state i to state j is defined as The mathematical form of the average entropy production during a period is [3] ∆S where the sum i<j is over all pairs of states. Using the generalized detailed balance relation in Eq. (11) we obtain (14) where The second term in the expression for ∆S in Eq. (14) can be written as where The physical interpretation for this term is as follows.
The energy change of the system in a jump from i to j is ∆E ij (t). The probability current J ij (t) is the average number of jumps from i to j minus the average number of jumps from j to i per unit of time. By summing over all pair of states we have the average energy gain of the system at time t. The multiplication by β c means that energies are measured in units of β −1 c . If the engine is passive the term J q is a generalized heat flux during a period, since the energy change in a jump is the energy taken from the thermal reservoir.
The generalized heat flux J q for an active heat engine is not the sole contribution to heat flux since the internal affinities A α can also contribute to heat dissipation. However, as we show next there is a refined first law that also holds true for an active heat engine and does not involve heat dissipation due to the internal affinities A α . We refer to J q as passive generalized heat flux.
For the models analyzed here, the inverse temperatures only takes the two values β c and β h during the period τ . The function h(t) only takes the values h(t) = 0 for where Q h is the heat taken from the hot reservoir during a period.
We point out that even though J q is not the only contribution to heat this quantity can be measured in an experiment. One way to measure it would be to observe the trajectory of the system and sum up h(t)∆E ij (t) whenever there is a jump from i to j. An alternative way to measure it would be to evaluate the total heat flux and the heat flux due to the activity. Hypothetically, the first quantity could be measured by monitoring small temperature differences in a large but finite reservoir and the second one by monitoring changes in the concentration of a chemical fuel such as ATP. The difference between both gives the passive contribution.

D. Refined first law and standard second law
The entropy production in Eq. (14) can be written as where we used Eq. (16). The time-derivative of the average energy Since The term on the right hand is the extracted work W from stochastic thermodynamics [3]. This equation constitutes the refined first law for active heat engines. If the inverse temperature only take the values β c and β h , i.e., h(t) is either 0 or 1, this equation becomes where is the average passive heat delivered to the cold reservoir during a period. We reiterate that this refined first law for active heat engines does not involve energy dissipation associated with the affinities A α . The entropy production in Eq. (18) then becomes where ∆S act ≡ α A α J α is the contribution due to the internal affinities. This contribution to the entropy production is the one that makes the heat engine active. This standard form of the second law has been obtained in [11].
A key issue is the scaling with the period τ of the different terms in this entropy production. A cycle in the network of states is a sequence of jumps that finishes in the state it started. Consider a cycle that happens while the external protocol is fixed. The total change of energy associated with this cycle, i.e., the net exchange of passive heat, is zero. The entropy change during the cycle associated with A α is not necessarily zero since the distance d α ij (t) cannot be written as a difference such as ∆E ij (t) = E j (t) − E i (t). The larger period τ , the larger the number of times cycles are performed. Therefore, the active part of the entropy change increases indefinitely with τ , unlike passive heat and work. This difference in scaling is an important limitation of the second law in Eq. (23). For large enough τ this second law does not provide a relevant bound on W since the term ∆S act dominates the inequality. A relevant bound on extracted work can only contain terms that scale with τ is the same way that W does. We point out that it is possible to make ∆S act scale in the same way as heat and work with τ by using infinite energy barriers that blocks the system from performing cycles in the space of states during the period. Models for molecular pumps do have such energy barriers [11,52].

E. Excess entropy and second law for active heat engines
The entropy change ∆S can be written as where ∆S ex is the excess entropy and ∆S hk is the house keeping entropy. A main interest in these quantities within stochastic thermodynamics is related to the fact that each of these terms is positive and, their fluctuating versions, each fulfill a separate fluctuation theorem [3].
Here we are interested in the mathematical expression of the excess entropy for periodically driven systems. As shown in the appendix, the average excess entropy change during a period is given by We now show that this inequality provides the appropriate statement of the second law for active heat engines. The main step is to define The excess entropy in Eq. (25) can be written as The relation ln P eq j (t) P eq i (t) = β(t)∆E ji (t) together with Eq. (17) and Eq. (20), leads to our second law for active heat engines This inequality involves the extracted work W , the generalized heat flux J q , and the new term I. Our second law allows for the following insight about active heat engines. For an active heat engine there is a shift from P eq to P S , this shift can allow for a larger amount of work extraction, beyond the bound given by the second law for a passive heat engine. The amount of energy dissipated by the active process is not directly relevant, rather the way the probability P S is shifted in relation to P eq , as quantified by I, is the relevant limiting factor on W .
For an active heat engine the pseudo-efficiency η ps can go beyond the Carnot bound η C . If we consider the efficiency which accounts for the term I, then the second law for active heat engines in Eq. (28) becomes the traditional statement of the second law η ≤ η C . We note that an active heat engine is able to extract work even if the inverse temperature is fixed and equal to β c , as we show with specific examples in Sec. V. In this case, W ≤ β −1 c I. Our second law in Eq. (28) has the correct scaling with the period τ and becomes the second law for passive heat engines for the case P S = P eq during the whole period. First, for a cycle in the space of states the change of Hence, for large τ , the scaling with τ of the term I is the same as the scaling of W and J q . Second, for a passive heat engine I = 0 and Eq.(28) becomes the standard second law for passive heat engines.
From the second law for the housekeeping entropy [3], we obtain This inequality, together with Eq. (23) and Eq. (28), leads to ∆S act ≥ I.
Hence, the second law in Eq. (28) provides a tighter bound on the extracted work W than the one in Eq. (23), for any period τ . The term I is the information theoretic contribution to the second law for an active heat engine and ∆S act can be viewed as the total thermodynamic cost to "generate" this term. This total thermodynamic cost to generate an information-theoretic term shows up in many examples in the relation between information and thermodynamics [53][54][55][56]. We here assumed that there is only one transition path between a pair of states i, j. However, it is possible to have more than one transition path between a pair of states, which is the case for the two-sate model in Sec. V B. The generalization of the expressions obtained here for the case that includes this possibility of more than one link between a pair of states is straightforward. If we use the label ν to represent the different links, the transition rate from i to j through link ν is k ν ij . For the generalized detailed balance relation in Eq. (11), the distances d α ij (t) depend on the index ν. The expression for quantities that show up in the second law inequalities remain similar, with the sum i<j substituted by ν i<j .
The second law obtained here is relevant for generic periodically driven active heat engines. Another important class of heat engines are those driven by constant time-independent forces that operate in a steady state. Since the average excess entropy in Eq. (25) is zero in a steady state, it provides no relevant information about the performance of steady state heat engines, which can be quantified with the standard entropy production [28].
This term can be written as Taking a a derivative of this term we obtain Since ln P S P eq (t) = ln P S P eq (t + τ ), integration of the equation above yields This equality provides a different expression for I in Eq. (26), which is Hence, the term I is part of the time-derivative of the difference of Kullback-Leibler distances in Eq. (33). Furthermore, the expression in Eq. (36) can be more convenient to calculate I.

A. Observable variable
We now address the issue of coarse-graining. The first step is to divide the state i into two variables i = (x, a).
The variable x can be observed and the variable a is hidden. If we consider the experiment from Ref. [18], x would be the position of the colloidal particle and a would be related to the degrees of freedom that determine the state of the bacteria. If we consider the variable a as part of the reservoir then we can say that the system labeled by the variable x is in contact with a nonequilibrium or active reservoir.
The master equation in Eq. (8) implies an equation for the time evolution of P x (t) ≡ a P x,a (t). Writing k ij as k a,x;a ,x and with the definition K xx (t) = a,a P a|x (t)k x,a;x ,a (t), where P a|x (t) ≡ P x,a (t)/P x (t) is a conditional probability, this equation becomes We point out that the dynamics of the variable x is non-Markovian since K xx depends on the dynamics of the hidden degree of freedom represented by a.
The transition rates k x,a;x ,a (t) fulfill the generalized detailed balance relation from Eq. (11). The affinities A α are related to the variable a and they are responsible for the active nature of the engine. The accompanying distribution is still the solution of Eq. (9). For the coarsegrained variable x we have P S x = a P S x,a .

B. Passive counterpart
We assume that the time-dependence of the energy E i (t) = E x,a (t) is contained only in the observable variable x. If this assumption does not hold we would also need access to the variable a to evaluate the work. This energy can then be written as where E int x,a represents the interaction energy between x and a.
The passive counterpart for our active heat engine is defined as follows. The equilibrium accompanying distribution associated with this passive counterpart is P eq where Z(t) = x e −βEx(t) . Such a passive heat engine corresponds to the complete absence of the degrees of freedom represented by the variable a. Physically, if we consider the experiment in [18], this passive counterpart corresponds to an experiment where all bacteria are removed from the solution.
A different passive counterpart would be to keep the variable a but set all affinities A α to zero. In this case, the accompanying equilibrium distribution for the passive counterpart would be where E eff x,a +Ea ) . The difference is that this effective energy also contains the interaction energy with the degrees of freedom represented by a. If we consider the experiment in [18], this would correspond to an experiment with all bacteria present but with their internal driving forces set to zero.
We proceed with the elimination of the variable a as the passive counterpart of the engine. However, this choice does not represent a limitation in our framework. As we show below, our formalism also applies if we choose this second type of passive counterpart of the engine, with modified expressions for the terms in the second law for active heat engines.

C. Work, heat and entropy
The work in Eq. (20) can be written as This expression follows from the fact that the timedependence of E x,a (t) is fully contained in E x (t). The coarse-grained generalized heat flux is defined as where . This expression is in general different from Eq. (17). The coarse-grained variable x also has a refined first law, which is obtained in the following way. The timederivative of the average of the part of the energy that depends on x is (43) From the fact that this average energy is periodic, we obtain which is the refined first law for the variable x.
For the particular case where the inverse temperature only takes two values β c and β h , i.e., h(t) is either 0 or 1, the coarse-grained generalized heat on Eq. (42) becomes the coarse-grained heat taken from the hot reservoir Q cg h . The first law in Eq. (44) becomes where is the coarse-grained heat released to the cold reservoir.
The coarse-grained excess entropy, as shown in the appendix, follows the second law inequality Analogously to the procedure used for ∆S ex , the coarsegrained excess entropy can be written as where (48) Similar to I, the coarse-grained term I cg has the same information theoretical expression as a part of the timederivative of x P x (t) ln P S x (t) P eq x (t) , which leads to the alternative formula for I cg , Remarkably, the coarse-grained second law in Eq. (47) has the exact same form as compared to the full second law in Eq. (28), in spite of the different values of the coarse-grained quantities. The physical interpretation of I cg as an extra resource that allows for more extracted work in an active heat engine also applies to this coarsegrained second law. The standard entropy production of stochastic thermodynamics also has this feature of invariance under coarse-graining that we found here for the excess entropy [57].
The structure of the second law for active heat engines is independent of whether one has full access to the state i or only access to x. For the first kind of active heat engine the system represented by i is in contact with an equilibrium heat bath and for the second kind the system represented by x is in contact with an nonequilibrium heat bath.
As proved in the appendix the excess entropy is larger than the coarse-grained excess entropy, i.e., Hence, even if we do have access to the hidden variable a the coarse-grained excess entropy provides a tighter bound on extracted work.
The passive counterpart of the engine was so far chosen as the absence of the variable a. For a passive counterpart that corresponds to setting all affinities A α to zero, the expressions above change as follows. The work W remains the same. The coarse-grained generalized heat J cg q has the expression in Eq. (42) with E eff x (t) instead of E x (t). The term I cg has the expression in Eq. (48) (or the alternative expression in Eq. (49)) with p eq x (t) in Eq. (40) instead of P eq x (t) in Eq. (39). The coarse-grained excess entropy ∆S cg ex is independent of the choice of the passive counterpart.
For all the models we consider here, either there are no interactions between x and a, i.e., E int x,a = 0 or, the condition that a e −β(t)(E int x,a +Ea) is independent of x holds. For these cases p eq x (t) = P eq x (t) and both passive counterparts correspond to the same expression for the terms that show up in ∆S cg ex . In our framework we have considered a master equation with discrete states. Hence, our results are also valid for a continuous state space that follows overdamped Langevin dynamics, since an overdamped Langevin equation can be obtained as a limit of a master equation. In fact, we do consider two models with continuous variables in the next two sections. The generalization of our framework to underdamped Langevin dynamics is left for future work.

V. ANALYTICAL CASE STUDIES
A. Expressions for W , Q h , I, and I cg For the models analyzed in this section the period has four parts illustrated in Fig. 4. Each part has duration τ l , with l = 1, 2, 3, 4 and 4 l=1 τ l = τ . There are two energy changes and two temperature changes. The timedependent temperature is β(t) = β c for 0 ≤ t ≤ τ 1 + τ 2 and β(t) = β h for τ 1 + τ 2 < t ≤ τ . The time-dependent energy is E x for 0 ≤ t ≤ τ 1 and for τ 1 + τ 2 + τ 3 ≤ t ≤ τ , and E x +δE x for τ 1 ≤ t ≤ τ 1 +τ 2 +τ 3 . Since the accompanying distributions P S x (t) and P eq x (t) are also divided into four parts, we write them as P S,l x and P eq,l x , respectively. The affinities A α can also change for different parts of the period.
The times are chosen such that τ 1 = τ 3 and τ 2 = τ 4 . Furthermore, the temperature changes are assumed to be instantaneous, which corresponds to the limit τ 2 → 0. Physically, the temperature changes have to be much shorter than the typical time for a transition between states. In this limit, τ 1 = τ /2. For the two-state model we do obtain analytical expressions for finite time τ . For all other models we consider the limit of large τ , which means that at the end of the first and third parts the system has reached the respective stationary state. This assumption simplifies calculations since the distribution P x (t) at the end of these two parts is the same as the ac-companying distribution P S x (t). We point out that this case does not correspond to a quasi-static limit since the temperature changes are instantaneous. Within this particular limit the expressions for W , Q h = (β c ) −1 J q , I, and I cg acquire the following forms.
First the work in Eq. (41) is This expression can be directly understood from Fig. 4 as follows. From the first part to the second part the energy increases by δE x , which gives the contribution −P S,1 x δE x . From the third to the fourth part the energy decreases by δE x , which gives the contribution P S,3 x δE x . The heat taken from the hot reservoir in Eq. (17) is x,a P eq,3 x,a − ln P S,1 x,a P eq,1 x,a .
The coarse-grained term I cg in Eq (49) is given by x P eq,3 x − ln P S,1 x P eq,1 x .
Since J q = J cg q for the models in this section, the inequality in Eq. (50) implies I ≥ I cg .

B. Two-state model
The two-state model is defined as follows. The transition rates of the model are given by κ + (t) = ω − (t) = k, κ − (t) = ke β(t)[E(t)+∆µ] , and ω + (t) = ke β(t)E(t) , where the rates are represented in Fig. 1(b). The timedependent energy E(t) is E during the first and fourth parts of the period and E + δE during the second and third parts of the period. The affinity A α is the free energy of one ATP hydrolysis ∆µ.
Using the methods from Ref. [58], which contains expressions for generic currents in two-state systems, we can evaluate Q h , W , I, and ∆S act analytically for finite τ . The expressions are too long to show here. The results for this model are shown in Sec. II.

C. Active Particle under the action of a harmonic potential
We now consider a model of an active Brownian particle in one dimension under the action of a harmonic potential [59][60][61]. Similar models for cyclic active heat engines have been analyzed in [19,20,22,24]. Let us first consider the model for the case of fixed temperature and energy illustrated in Fig. 5(a). The position of the particle is labeled x. For this model x is a continuous variable, we then write the energy E x as E(x). The harmonic potential is written as E(x) = κx 2 /2. This model can be described by an overdamped Langevin equation that can be obtained as the continuous limit of the master equation for discrete x. A force of magnitude F acts on the active Brownian particle. The sign of the force is a Poisson process with a rate γ. The variable a then takes two values, one a = +1 corresponding to a force +F and the other a = −1 corresponding to the force −F . The force F multiplied by distance is the affinity A α for this model.
We consider the limit for which γ is much smaller than the inverse relaxation time of the particle, the particle reaches a stationary state with a given sign for the force F before changing the sign of the force. The stationary probability density for the particle to be at position x with a = ±1 is then Furthermore, the probability density to be at position x is given by (56) The equilibrium distribution that corresponds to F = 0 is given by During the first part of the period the stiffness of the harmonic potential is κ 1 and the force is F 1 . During the third part of the period the stiffness is κ 2 and the force is F 2 . The second and fourth parts of the period are instantaneous. In terms of the notation in Sec. V A, The work W in Eq. (51), with an integral over x instead the summation over x, is obtained with Eq. (56), which leads to the expression (58) The heat taken from the hot reservoir in Eq. (52) for the present model is The term I in Eq. (53) is obtained with Eq. (57) and Eq. (55), which lead to The coarse-grained term I cg in Eq. (54) follows from Eq. (56) and Eq. (57), For this quantity the expression is in terms of an integral that has to be performed numerically.
We now restrict to the case F 1 = 0 and F 2 = F . In Fig. 5(b) we show that β c η C Q h − β c W can be negative and that the inclusion of the term I cg recovers the appropriate statement of the second law for active heat engines. We also illustrate the inequality I ≥ I cg valid for the models analyzed here. In Fig. 5(c) we illustrate the second law for active heat engines from a different perspective. We compare the pseudo efficiency η ps = w/Q h and the efficiency η ≡ W Q h +I cg /(βc−β h ) with the Carnot efficiency η C . Our second law does become the standard statement η ≤ η C if we also account for the term I cg , while the pseudo-efficiency does cross the Carnot bound.
This active heat engine can extract work even if the temperature is constant, i.e., β h = β c , as shown in Fig.  5(d), for which the extracted work becomes positive for F > 1.4 approximately. In this case, the second law becomes W ≤ (β c ) −1 I cg .

D. Two-interacting particles
To illustrate the applicability of our second law to the case with interactions between x and a, we now consider a simple model for two interacting particles illustrated in Fig. 6(a). The particles can be in three different positions. The observable particle has its position labeled by x = −1, 0, 1. The energy for this particle is E(t) for x = ±1 and zero for x = 0. The time-dependent energy is E for the first part of the period and E + δE for the third part of the period. The position of the hidden particle is labeled by a = −1, 0, 1. The energy E a is zero for all a. This particle is driven in the clockwise direction by an affinity A. Finally, the particles interact with each other through a hardcore potential.
There are six possible states i = a, x. We label these states in the following way. State i = 1 corresponds to x = −1 and a = 0. State i = 2 corresponds to x = −1 and a = 1. State i = 3 corresponds to x = 0 and a = −1. State i = 4 corresponds to x = 0 and a = 1. State i = 5 corresponds to x = 1 and a = −1. State i = 6 corresponds to x = 1 and a = 0. Using this labeling of the states, the transpose of the stochastic matrix for this model reads where k (γ) sets the time-scale for transitions of the observable (hidden) particle. The escape rates r i are the sum of non-diagonal elements of a row. To shorten the matrix we have omitted the time-dependence of A, E, and β. For the first part of the period they are A 1 , E, and β c and for the third part of the period they are A 2 , E + δ E , and β h . The stationary distribution P S is the eigenvactor associated with the eigenvalue 0 of this matrix. With this stationary distribution we obtain analytical expressions for all desired quantities defined in Sec. V A. The analytical expressions are too long to be displayed here.
This model displays all the same features from the previous model, as illustrated in Fig. 6, which reiterates the fact that the second law for active heat engines is independent of the particular model. The efficiency in Fig.  6

A. Relation to experiment
The models of the active particle in a harmonic potential and of the two interacting particles have some similarities with the experiment in [18]. There is an observable particle labeled by x. Furthermore, heat and work can be evaluated by an expression that only depends on the observation of the variable x. The difference is in the hidden degrees of freedom. For the first model, this hidden degree of freedom is the sign of the force F . For the second model, the hidden degree of freedom is the position of the driven particle. For the experiment involving a colloidal particle in a bacterial bath, the hidden degrees of freedom are more complex. They correspond to the positions and other variables that determine the state of the bacteria.
However, the fact that the hidden degrees of freedom in the experiment are more complex does not present a challenge for our second law. The new term I cg can also be calculated by the sole observation of the variable x, no matter how complex the hidden degrees of freedom are. Our second law should be directly applicable to the experiment in [18] and to future experiments with cyclic active heat engines. To illustrate the generality of our second law we consider a more complex model than our analytical case studies. This model has several hidden degrees of freedom and is inspired by the experiment in [18].

B. Model definition
The two-dimensional model depicted in Fig. 7 has a passive particle and many active particles that make the reservoir active. These active particles correspond to the bacteria in the experiment. The passive particle has its position labeled by x = (x 1 , x 2 ) and is subjected to a harmonic potential where κ is the stiffness of the potential and x ≡ |x|. The active particles are not influenced by this harmonic potential and they do not interact with each other. They all interact with the passive particle via the repulsive potential V (|r|), where r is the distance to the passive particle. This potential is given by where R is the radius of the passive particle, and κ rep > 0 such that the interaction is repulsive. The passive particle obeys the Langevin equatioṅ where µ p is the mobility of the passive particle. The term ξ(t) corresponds to Gaussian white noise with intensity µ p /β. The sum over m is over all active particles and a m is the position of the active paticle indexed by m. The active particle labelled by m obeys the Langevin equatioṅ where µ a is the mobility of the active particles and ζ m (t) is a Gaussian white noise with intensity µ a /β. For the first term on the left hand side, u is the active speed, n(φ m ) is a unit vector pointing in the direction of angle φ m . The angle φ m undergoes Brownian diffusion with diffusion coefficient D r [62,63].
For the active particles the vector a m is constrained to the box shown in Fig. 7 with periodic boundary conditions. The passive particle is unconstrained. However, due to the harmonic potential E(x) the passive particle rarely leaves this box.
The protocol has the same four parts as the generic one shown in Fig. 4. Each part of the protocol takes the time τ /4. The temperature changes are the same as in all models analyzed here. The stiffness of the potential E(x) switches from κ 1 to κ 2 from first to the second part and changes back from κ 2 to κ 1 from the third to the fourth part. The velocity u of the active particles also changes with the protocol. During the parts of the protocol where the inverse temperature is β c (β h ) this velocity is u c (u h ).
We here have set the parameters to the following values: the period is τ = 20; there are 30 active particles; the stiffness takes the values κ 1 = 10, κ 2 = 15; the inverse temperatures are β −1 c = 0.1, β −1 h = 0.12; the radius of the passive particle is R = 0.2; the mobilities are µ p = 0.05 and µ a = 0.1; the parameter of the repulsive potential is κ rep = 1000; the diffusion coefficient associated with the direction of the active speed is D r = 1. The active speed is set as u h = u 0 and u c = u 0 /100, where u 0 is parameter that we vary. For u 0 = 0 the heat engine is passive.

C. Simulations and observables
Simulations are performed by integrating the Langevin equations (65) and (66) using a simple stochastic Euler scheme with a sufficiently small time step. We have used a time-step of 10 −4 . For each trajectory, we write stochastic work and heat as w and q h , respectively. The heat Q h and work W are related to the average of these quantities over stochastic trajectories.
From Eq. (41), the increments to the stochastic work w happen at the two times of the period that the stiffness changes. For the change in the stiffness from first to the second part the increment in the work is (κ 1 − κ 2 )x 2 /2, where x is the position of the passive particle at the moment of the change. For the second change in the stiffness from the third to the fourth part, the increment in the work is (κ 2 − κ 1 )x 2 /2.
From Eq. (42), the increments to the stochastic heat q h happen when the passive particle changes position during the third and fourth parts of the period, for which the inverse temperature is β h . This increment can be formally written as dq h = h(t)k(t)d(x 2 )/2, where the function h(t) in Eq. (10) is 1 if β(t) = β h and 0 if β(t) = β c .
Estimates for the average heat Q h and work W per cycle then follow from the final value of q h and w divided by the number of cycles performed in a long trajectory (we average over 5000 cycles).
The calculation of I cg is more involved. From Eq. (49), to calculate the increment associated with this quantity, we need the equilibrium distribution and the accompanying distribution P S . Since the protocol has four parts, there are four stationary distributions P S,l (x). To sample these distributions we run four separate simulations where κ, β and u remain constant. The results for these distributions are shown in Fig. 8. This method of sampling P S,l (x) from realizations with fixed parameters that reach a steady state can also be used in experiments for which this probability is not known.
The stochastic quantity corresponding to I cg can now be calculated in similar way to the stochastic work w. Whenever there is a change from part l to part l + 1 of a cycle and the passive particle is in position x, we add up the increment ln P S,l (x)P eq,l+1 (x) P eq,l (x)P S,l+1 (x) . To get an estimate of the average I cg we divide the total sum of all increments during the trajectory by the number of cycles.
We reiterate that our estimates of W , Q h and I cg only depend on the monitoring of the position of the passive particle x. This scheme can also be applied to an experiment where one does not have access to the hidden active degrees of freedom, such as the experiment in [18].

D. Results
The results for for the numerical model are shown in Fig. 9. They are plotted as a function of u 0 , which parameterize the active speed. For u 0 = 0 the heat engine is passive. We observe that for increasing u 0 the active heat engine becomes more efficient. In fact, for our chosen parameters, activity allows the system to operate as a heat engine since for u 0 = 0 the extracted work W is negative. The pseudo efficiency η ps ≡ W/Q h does go beyond the Carnot bound while the efficiency η ≡ stays below the bound, as predicted by our second law.
As shown in Fig. 8, with increasing active speed u 0 the stationary distribution departs further from the equilibrium distribution. In Fig. 9 we demonstrate that the term I cg , which quantifies the difference between these distributions, increases, from about 30 % of the value of (β c − β h )Q h for u 0 = 10 to 80 % for u 0 = 40. When the engine is highly active, I cg acts as an important resource for the extraction of work.
Related models have been analyzed in Refs. [19,22], using Langevin equations for the passive particle with a noise that is not the standard Gaussian white noise. This noise effectively captures the influence of the active particles that interact with the passive particle. Both these studies use the concept of an effective temperature. An advantage of our approach is that we have no need to invoke an effective temperature, our second law inequality contains bona fide temperatures.

VII. CONCLUSION
The second law of thermodynamics determines which processes are possible. For well known passive heat engines, this statement is expressed as the Carnot bound on the efficiency. We here derived the appropriate statement of the second law for cyclic active heat engines. Unlike a passive heat engine, the second law for an active heat engine includes an information-theoretic term, which allows active heat engines to perform tasks beyond the ones that can be performed by passive heat engines, such as work extraction at constant temperature.
Our second law for cyclic active heat engines has three main merits. First, it does not include the dissipation due to the hidden degrees of freedom. Second, it is expressed in terms of quantities that can be evaluated by the sole monitoring of the observable degrees of freedom. Third, it is general, as it only requires the mild assumption that at some level of description the observable degrees of freedom together with the hidden degrees of freedom follow Markovian dynamics. Furthermore, the system corresponding to the observable degrees of freedom does not have to be a single colloidal particle, which is the case for most models analyzed in the literature, but it can be any arbitrary collection of possibly interacting degrees of freedom.
From the perspective of stochastic thermodynamics, we have unveiled a central application of excess entropy in periodically driven systems. While this quantity has been known to fulfill a fluctuation theorem we here show that it provides the correct statement of the second law for cyclic active heat engines. Beyond the known excess entropy we introduced a coarse-grained excess entropy that was key for the derivation of the second law in terms of quantities that only depend on the observable degrees of freedom. Interestingly, the second law for active heat engines retains its structure under coarsegraining. This feature highlights the power of stochastic thermodynamics, which also applies to an active reservoir with hidden dissipative degrees of freedom that leads to non-Markovian dynamics.
Active matter is an important modern concept in physics. Quite generally, if we drive a system immersed in such active matter with an external periodic protocol with the objective of completing some task such as work extraction, the second law derived here constitutes a general bound on whether the completion of this task is possible. Therefore, our results are relevant for future experiments with active heat engines. From the theoretical side, the second law derived here opens up the possibility to develop a linear response theory for active heat engines, it is central for problems such as the optimization of work and efficiency, and it should be useful for future models of, possibly interacting, active heat engines. To prove the second law for coarse grained excess entropy we must define quantities as functionals of stochastic trajectories from time t = 0 up to time t = T . The time t is discretized with time-intervals δt to simplify the expression for the probability of a trajectory, the continuous time limit is recovered for δt → 0. For discrete time, the transition probability from i to j is M ij (t) = δtk ij (t) for i = j and M ii (t) = 1 − j =i δtk ij (t). For the proof in this appendix the time-dependence of the transition rates is arbitrary, there is no need to assume a periodic timedependence. For the calculations below, a key quantity is the dual transition rate k † ij (t) = k ji (t)P S j (t)/P S i (t) (the same equation applies for transition probabilities M † ij (t)), where P S i (t) is the stationary probability for fixed rates at time t, i.e., they are the solution of Eq. (9).
The probability of a trajectory Γ = i 0 → i 1 . . . i N −1 → i N , where N = T /δt, is given by The probability of the reversed trajectory with dual dynamics (without reversing the time-dependence of the transition rates) is The trajectory dependent excess entropy change is defined as .
(A3) Furthermore, the system entropy change is Therefore, the sum of these two terms gives From this formula it is easy to show that the excess entropy fulfills a fluctuation theorem [3]. A useful equation is where the brackets indicate an average over stochastic trajectories and the Kullback-Leibler distance here is over the space of trajectories Γ. If the transition rates are time-periodic with period τ , the final time is T = τ , and at the starting time t = 0 the system has already reached the asymptotic time-invariant probability, this inequality becomes the second law used in Eq. (25). Note that the average change of system entropy is zero if the system entropy is time-periodic.
To define the coarse-grained excess entropy we write i = (x, a), where x is the observable variable and a is hidden. The dynamics for the coarse-grained variable x is not Markovian. The probability of a trajectory in the space of trajectories of x is defined as where Γ cg = x 0 → x 1 . . . x N −1 → x N . In order to express this probability in a form similar to Eq. (A1) we use the quantity M xx (t) = a,a P a|x (t)M x,a;x ,a (t), which leads to the expression Prob(Γ cg ) = P x0 (0) For this expression we used P x0,a0 (0) = P x0 (0)P a0|x0 (0) and an P an|xn (nδt)M xn,an;xn+1,an+1 (nδt) = M xnxn+1 (nδt)P an+1|xn+1 ((n + 1)δt).
The same formula is valid for the reversed trajectory with dual rates, i.e., where M † xx (t) = a,a P a|x (t)M † x,a;x ,a (t) . The coarse-grained excess entropy is defined as .
(A11) The coarse-grained system entropy change is The sum of both terms gives ∆S cg ex (Γ) + ∆S cg sys (Γ) = ln Hence, where we have used the superscript cg in the Kullback-Leibler distance to indicate that we are considering probabilities in the space of trajectories Γ cg . For transition rates that are time-periodic with period τ , if the time interval is T = τ and if at the starting time t = 0 the system has already reached the asymptotic time-invariant probability, the inequality in Eq. (A14) becomes the coarse-grained second law used in Eq. (47). The average system entropy is periodic, and, therefore, the average system entropy change in a period is zero.
The average excess entropy is larger than the average coarse-grained excess entropy, since which implies the inequality in Eq. (50). A similar inequality for the standard entropy production has been used in [64]. Finally, we can prove a fluctuation theorem for the coarse-grained excess entropy using relation (A13),   4. Depiction of the protocol for the case studies. The inner box represents the observable degree of freedom x and the respective energy Ex. The particles with arrows represent hidden active degrees of freedom labeled by a. The four steps are as follows: from l = 1 to l = 2 energy changes from Ex to Ex +δEx, from l = 2 to l = 3 the inverse temperature changes from βc to β h , from l = 3 to l = 4 the energy changes from from Ex + δEx to Ex, and from l = 4 to l = 1 the inverse temperature changes from β h to βc.   Setup for our numerical case study. A passive particle (grey) trapped in a harmonic potential (indicated in red) interacts with many active Brownian particles (white) that are self-propelled in randomly varying directions (orange arrows).