Ignoring performance for but a moment, I'm not sure your initial observation is so. There is no reason to think that a processor could not maintain a register-private key used for decryption. If we used crypto techniques how does this change your rationale? I'm not suggesting this is the way to go, just possible within the design space.
-JMS