Some might argue that in fact we do know what we want, in part because we do have the examples of information processing found in organic brains. So for example, we want ``intelligent'' software. Something that could exhibit a human-like intelligence. But this kind of thinking reveals the burden of our preconceptions, derived from our prior experience with organic life. The ``Turing Test'' epitomizes this problem.
This test was proposed by Alan Turing ([39]) as a method for recognizing when a machine has become intelligent. The test involves communicating through a teletype device, with some humans and some machines. If the user can not distinguish between the human and the machine, then we can consider the machine to be intelligent.
Proposals such as the Turing Test are always carefully presented as ``sufficient'' not ``necessary'' tests. Yet I consider them to be deeply misguided, because they have a tendency to focus the debate and even the research on what amounts to a red herring. I consider it to be ludicrous to suggest that an intelligent machine might be indistinguishable from a human (actually, the Turing Test should be able to distinguish even between humans from different cultures).
I believe that a true machine intelligence is likely to be fundamentally different from a human (or any organic) intelligence. A machine intelligence would be even more alien than an intelligence from another planet, because such an extra-terrestrial creature would probably be carbon based, whereas a machine intelligence is not.
Imagine a machine intelligence living in the internet. Scanning a terabyte of data distributed globally over the net for instances of foolish predictions, and doing the job as a distributed process in a few minutes, would probably be very exciting for such an intelligence. It can perform enormously complex numerical calculations and process huge volumes of numerical data at phenomenal speeds. It could transport itself physically to any point on the planet's surface in milliseconds. At any instant of time, it might actually be distributed widely around the planet. The data flow would be a direct sensory experience for this creature, not something happening in a separate information processing tool. This creature lives in a digital informational universe, not the material one we live in. It's pleasures and pains will be completely alien to us. We will never mistake if for human. Forget the Turing test.