Free spin 2 duality invariance cannot be extended to general relativity

We show by explicit computation that the recently discovered duality invariance of $D=4$ linearized gravity is lost, already at first self-interacting, cubic approximation of general relativity. In contrast, the cubic Yang-Mills extension of Maxwell does admit a simple deformed duality.