This entry is a bit of analysis into John R. Searle’s paper “Minds, Brains and Programs” published in 1980, and one of its objections. It only seems fitting to explore these important ideas in light of increasing fervour (and optimism, depending on your vantage point) about endowing human-level intelligence in machines. The developments of improved text generation and language comprehension, observing emergent behaviours in AI systems, roadmaps towards higher-level reasoning utilizing similar technologies and the unforeseen limits of further progress certainly bring into question the underpinnings of what we conceive to be intelligence, and if we’d realize it should we succeed in building it. It’s been 40 years since the publication of Searle’s famous “Chinese Room” thought experiment and the issues he contemplates are only more salient today.
In the Chinese Room thought experiment, John Searle sets out to refute the plausibility of strong AI (the claim that a suitably programmed computer is a literal instantiation of intelligent human-level understanding). His argument begins by considering that he, an English speaker, is locked in a room and given a set of Chinese characters. He is provided with a rulebook, written in English, that maps an arbitrary set of Chinese input characters to another set of output characters. A set of input characters could, for example, carry a linguistic interpretation such as a ‘story’, ‘question’ or a ‘script’. However, this is unknown to Searle as he is only equipped to interpret these Chinese characters according to the prescriptions of the rulebook, or ‘program’. In other words, Searle’s Chinese language processing capability is limited to symbolic manipulation. There are no instructions for him to extract denotative or connotative meanings from the input characters. Yet, Searle is still allowed to accept English inputs and provide English outputs that he, of course, understands without consulting the rulebook. The setup of the argument is concluded by saying that the program and Searle’s ability to execute it become so advanced that any native Chinese speaker outside the room would believe that he is indeed fluent in Chinese. That is, when Searle receives Chinese inputs he becomes an instantiation of a computer program exhibiting intelligent behaviour indistinguishable from a Chinese speaking human. When Searle receives English inputs he behaves like himself, an English speaking human. In this way, Searle’s Chinese Room can be said to pass the Turing Test.
Despite this, Searle disagrees with the claims of strong AI proponents that (1) the Room actually understands Chinese and that it (2) provides a basis to explain human understanding. Blindly following the rulebook does not equate to any intelligent, human-like interpretation of the inputs. As an English speaker, he is completely oblivious to the Chinese meaning contained within the characters and therefore cannot be said to have any understanding of Chinese. Second, he posits that if the program is built only from computational operations on formally specified elements, the Chinese Room can display no serious connection to human understanding.
In response to his example, Searle considers a variety of responses, the most interesting of which is ‘the systems reply’. The objection is that while it’s true Searle himself cannot be said to understand Chinese, the sum of the program and Searle himself demonstrates comprehension. That is, the system as whole can be said to understand the story, question or script. From the Room emerges a mind that understands Chinese.
Searle addresses this objection by considering the program internalized within himself (he memorizes every routine and instruction). In this way, there still exist two separate subsystems: one that understands English (and understands that it understands English) and one that responds in Chinese (but still does not understand it). From here, he presses the point that any form of program that exclusively performs symbol manipulation cannot be said to understand Chinese. No matter what form the program realizes and expresses itself, whether it is recited as a rulebook or contained within the memories of a human, Searle contends that it still cannot be said to truly understand. In essence,he claims that any arbitrarily complex sequence of functional operations that exclusively maps input to outputs in a believable fashion is not a sufficient condition to be a ‘mind’—the manner in which it’s executed has no bearing on the end result.
Searle clarifies his position by removing the room and the physical instantiation of the program as a rulebook and reduces the scenario to a single mind with the rulebook ‘loaded’ onto it. He identifies that the two subsystems still remain completely separate entities and operate independently of each other. He outlines that the motivation behind the objection as the belief that understanding can be said to be achieved so long as the inputs and outputs of the program match that of a native Chinese speaker. This is misguided in Searle’s view, as the very point of his example is to demonstrate that a ‘mapping’ procedure is not a sufficient condition for ‘understanding’. Importantly, Searle emphasizes the inadequacy of the Turing Test in measuring a program’s ability to understand. While a program that passes the Turing Test may be indistinguishable from a human on the outside, it has nothing to say about whether or not the program exhibits any meaningful cognition on the inside.
Proponents of ‘the systems response’ objection claim that understanding is achieved by expressing the knowledge of Chinese embodied within the program by way of its execution. That is, an undeniable understanding of the Chinese language is demonstrated when the interpreter (Searle) executes it to produce an acceptable output. The problem here is a misguided understanding of the concept of ‘understanding’. While it is possible to accept that the Chinese Room exhibits a kind of objective, global understanding of the rules of Chinese (otherwise it wouldn’t be able to produce viable outputs), it cannot be said that Chinese Room is aware of its understanding (and by extension, its potential misunderstandings). This is the key distinction between Searle’s understanding of English inputs and his understanding of Chinese inputs. In the case of the former, Searle understands not only the rules that govern the manipulation of English symbols, but also the subjective meaning that is ascribed to each symbol. It is this ability to form a sense of meaning that Searle refers to as true understanding. It is unclear how the Chinese Room as a system demonstrates any correspondence between an understanding of logical operations to an awareness, or consciousness, of the operations it performs.
The problem then lies with the fact that the program is not aware of its expression. The mysterious ability for human brains to be aware of themselves, and henceforth develop a sense of Self, is the essence of true understanding. It’s not inconceivable that one could design a program that achieves this, perhaps by providing instructions for Searle to extract meaning from inputs (for example, meta-rules that map a meaning symbol to a group of symbols). Indeed, this is how Searle may have come to learn the English language in the first place. However, in Searle’s view the Room does not exhibit these causal powers that enable understanding. In this way, Searle packages together understanding (or intelligence) with a notion of ‘intentionality’, ‘causality’, and ‘awareness of understanding’—attributes that must be seriously considered when seeking to build human-level intelligence.
An interesting distinction to make here is that the Chinese Room’s rulebook could be interpreted to resemble a massive lookup table. By most, the existence of such a table wouldn’t be mistaken with anything that could be “intelligent”. But what about its expert utilization by Searle? What about the design and development of the rulebook? Searle artfully keeps the exact implementation of the rulebook ambiguous, and doesn’t consider the process of developing the rulebook within his scope. In some sense, modern machine learning algorithms create, and utilize their own rulebooks. Does that make them intelligent?