A cluster of crucial arguments here, launched by an exotic question:
What if artificial intelligence is so unfamiliar that we have a hard time recognising it? Could our machines have become self-aware without our even knowing it? The huge obstacle to addressing such questions is that no one is really sure what consciousness is, let alone whether we’d know it if we saw it. …
Despite decades of focused effort, computer scientists haven’t managed to build an AI system intentionally, so it can’t be easy. For this reason, even those who fret the most about artificial intelligence, such as University of Oxford philosopher Nick Bostrom, doubt that AI will catch us completely unawares. And yet, there is reason to think that conscious machines might be a byproduct of some other effort altogether. …
Extracted from a consistently fascinating post on quantum computation and consciousness by Scott Aaronson:
Yes, consciousness is a property of any suitably-organized chunk of matter. But, in addition to performing complex computations, or passing the Turing Test, or other information-theoretic conditions that I don’t know (and don’t claim to know), there’s at least one crucial further thing that a chunk of matter has to do before we should consider it conscious. Namely, it has to participate fully in the Arrow of Time. More specifically, it has to produce irreversible decoherence as an intrinsic part of its operation. It has to be continually taking microscopic fluctuations, and irreversibly amplifying them into stable, copyable, macroscopic classical records.
The immediately subsequent clarification is also crucial: “let me be extremely clear about what this view is not saying. Firstly, it’s not saying that the brain is a quantum computer, in any interesting sense — let alone a quantum-gravitational computer, like Roger Penrose wants! Indeed, I see no evidence, from neuroscience or any other field, that the cognitive information processing done by the brain is anything but classical.”
Once seen, this is an argument that cannot be unseen. (It’s almost an instance of consciousness-production through ratcheted decoherence in itself.)