How can it possibly be that we can exhaustively search an infinite space in a finite time?
There has been a bit of talk recently about how it is possible to exhaustively search certain infinite spaces in a finite time. Unfortunately the explanations seem to be couched in topological language making them a little inaccessible. So I thought it'd be good to attempt a rigorous, but elementary, proof so that is possible to see exactly where the 'magic' happens.
Consider the type of infinite streams of booleans:
type Natural = Integer
Stream :: Natural -> Bool
Can we write a function search, that takes as argument a boolean predicate on Stream, and tells us whether or not there is a stream that satisfies it?
search :: (Stream -> Bool) -> Bool
This sounds like an impossible task. There are uncountable many infinite streams of booleans, so how can we hope to search them all? There's one important thing in our favour: we're restricting ourselves to predicates that are computable and total. In other words, whatever stream we pass to the predicate, they have to return a boolean in a finite time.
First, let's deal with an objection that make search seem impossible to implement. Given a 4-tuple of integers, we can encode these integers in a stream. For example, the integers (written in binary) (11,10,1,111) could be encoded as
where we code the binary digit b as 1b, we separate the integers using 00 and terminate the sequence using 01. We could then write a predicate that checks to see if the 4-tuple gives a solution to xn+yn=zn with x,y,z>0 and n>=2. Plugging that into search we apparently have a way to decide the Fermat-Wiles theorem without doing any difficult number theory. Surely, therefore, search can't exist? But the predicate we've just described isn't total. Suppose we feed it the string 111111... as input. It's going to think that these represent an x with leading digits 111... and it's going to sit there collecting digits forever. It's never going to terminate. So the requirement that the predicate be total is such a strong condition that we can't actually implement the Fermat predicate, even though it uses nothing more than simple arithmetic. So maybe search is beginning to look a little more plausible.
Here's the key thing: any predicate on boolean streams must terminate after a finite time, meaning it only reads a finite number of booleans. So given any predicate p, there is a (mathematical) function dp on the boolean streams with the property that dp(s) is the index of the last bit p looks at when given s as input. dp(s) is obviously always finite.
Now at this point you might immediately jump to the opposite conclusion that of course search is possible because you only need to check a finite number of bits. But even though dp(s) is always finite, it might be unbounded as s varies over the infinite set of streams. So the finiteness of dp on its own is not enough to bound our search. For example, consider predicates on 4-tuples of integers again. Each of these predicates reads only a finite number of bits, and yet we don't expect a general purpose terminating search algorithm can tell us whether or not Fermat's equation has a solution. So to prove termination for the boolean stream case we need to exploit some more structure.
Here's a first hint at what that structure might be. Suppose we try to make dp unbounded by arranging that
You might notice that we immediately have a contradiction. If dp(00000...)=1 then the predicate must stop reading bits after it has read an initial 0. So actually, the sequence dp(01000...) is also 1. Lots of streams share prefixes in this way, so the question is, does the infinitude of the set of boolean streams win out, or does the amount of sharing of prefixes eventually tame the infinity?
Consider the following infinite binary tree:
Each boolean stream corresponds to a single downward path through this tree. Fix a predicate p. Suppose, then, that this predicate, when fed the stream 011..., requires only 3 bits to be read, ie. dp(011...) = 3. Then we can cut off everything below that point on the tree because no predicate need read that far. We've removed every infinite path passing through 011:
Every stream s has a finite cutoff after dp(s) steps. And after we've hacked away all of the bits that our predicate never looks at what we're left with is a finitely branching tree with no infinite paths. Now comes the magic: we invoke König's Lemma which says "every tree that contains infinitely many vertices, each having finite degree, has at least one infinite simple path.". Conversely, if there are no infinite paths, the tree must be finite. In other words, for any predicate p we only need to search a finite tree.
The rest is just details, such as the precise method by which we determine when we've finished searching. In the case of Martín Escardó's paper the details are some of the most ingenious coding I've ever seen.
Update: I fixed the type of search above. Thanks to Cale for pointing this out. Unfortunately I deleted his comment by accident so I'm crediting him here.
Consider the type of infinite streams of booleans:
type Natural = Integer
Stream :: Natural -> Bool
Can we write a function search, that takes as argument a boolean predicate on Stream, and tells us whether or not there is a stream that satisfies it?
search :: (Stream -> Bool) -> Bool
This sounds like an impossible task. There are uncountable many infinite streams of booleans, so how can we hope to search them all? There's one important thing in our favour: we're restricting ourselves to predicates that are computable and total. In other words, whatever stream we pass to the predicate, they have to return a boolean in a finite time.
First, let's deal with an objection that make search seem impossible to implement. Given a 4-tuple of integers, we can encode these integers in a stream. For example, the integers (written in binary) (11,10,1,111) could be encoded as
11110011100011001111110100000000000000000...
where we code the binary digit b as 1b, we separate the integers using 00 and terminate the sequence using 01. We could then write a predicate that checks to see if the 4-tuple gives a solution to xn+yn=zn with x,y,z>0 and n>=2. Plugging that into search we apparently have a way to decide the Fermat-Wiles theorem without doing any difficult number theory. Surely, therefore, search can't exist? But the predicate we've just described isn't total. Suppose we feed it the string 111111... as input. It's going to think that these represent an x with leading digits 111... and it's going to sit there collecting digits forever. It's never going to terminate. So the requirement that the predicate be total is such a strong condition that we can't actually implement the Fermat predicate, even though it uses nothing more than simple arithmetic. So maybe search is beginning to look a little more plausible.
Here's the key thing: any predicate on boolean streams must terminate after a finite time, meaning it only reads a finite number of booleans. So given any predicate p, there is a (mathematical) function dp on the boolean streams with the property that dp(s) is the index of the last bit p looks at when given s as input. dp(s) is obviously always finite.
Now at this point you might immediately jump to the opposite conclusion that of course search is possible because you only need to check a finite number of bits. But even though dp(s) is always finite, it might be unbounded as s varies over the infinite set of streams. So the finiteness of dp on its own is not enough to bound our search. For example, consider predicates on 4-tuples of integers again. Each of these predicates reads only a finite number of bits, and yet we don't expect a general purpose terminating search algorithm can tell us whether or not Fermat's equation has a solution. So to prove termination for the boolean stream case we need to exploit some more structure.
Here's a first hint at what that structure might be. Suppose we try to make dp unbounded by arranging that
dp(00000....) = 1
dp(10000....) = 2
dp(01000....) = 3
dp(11000....) = 4
and so on.
You might notice that we immediately have a contradiction. If dp(00000...)=1 then the predicate must stop reading bits after it has read an initial 0. So actually, the sequence dp(01000...) is also 1. Lots of streams share prefixes in this way, so the question is, does the infinitude of the set of boolean streams win out, or does the amount of sharing of prefixes eventually tame the infinity?
Consider the following infinite binary tree:
Each boolean stream corresponds to a single downward path through this tree. Fix a predicate p. Suppose, then, that this predicate, when fed the stream 011..., requires only 3 bits to be read, ie. dp(011...) = 3. Then we can cut off everything below that point on the tree because no predicate need read that far. We've removed every infinite path passing through 011:
Every stream s has a finite cutoff after dp(s) steps. And after we've hacked away all of the bits that our predicate never looks at what we're left with is a finitely branching tree with no infinite paths. Now comes the magic: we invoke König's Lemma which says "every tree that contains infinitely many vertices, each having finite degree, has at least one infinite simple path.". Conversely, if there are no infinite paths, the tree must be finite. In other words, for any predicate p we only need to search a finite tree.
The rest is just details, such as the precise method by which we determine when we've finished searching. In the case of Martín Escardó's paper the details are some of the most ingenious coding I've ever seen.
Update: I fixed the type of search above. Thanks to Cale for pointing this out. Unfortunately I deleted his comment by accident so I'm crediting him here.
posted by sigfpe at Saturday, October 06, 2007
15 Comments:
You wrote:
But even though dp(s) is always finite, it might be unbounded as s varies over the infinite set of streams.
Although you come to the right conclusion in the end, what you wrote above is not true: in fact you end up proving that for a given predicate there is a definite depth that you don't need to look beyond.
dthurston,
That's an epistemic "might".
Ie. "might" = "I do not yet know that it is impossible that" rather than "there exists an example such that".
Thanks for the article, it was very informative. As an unrelated question, may I ask what program you used to produce those binary trees?
The trees were drawn with Inkscape. To misquote Chruchill, Inkscape is the worst free drawing application, apart from all the rest.
Fascinating stuff. And thanks for your explanation, I can't follow that paper.
I'm having a little trouble while exploring this problem more deeply. I "know" that there must not be a total function from infinite sequences of booleans onto the integers. I have devised this proof:
If there were, then we could construct a function from booleans into booleans answering the question "such and such Turing machine halts in n steps", and then we could use the algorithm you just presented to solve the halting problem.
But that doesn't give me any intuition for any deeper reasons. Could you explain a little more why there's no way to construct a total function from infinite sequences of booleans onto the integers?
Thanks,
Luke
http://luqui.org/blog
It's interesting to note that one can't implement "search" as a pure recursive function of the type "(Stream -> Bool) -> Bool". The problem is that its argument is a "black box", an opaque predicate. Although we know that the search tree (as described in the post) is finite, we don't know what it is and so we can't search it in finite time.
However, you can implement "search", if you allow side effects, e.g. using "unsafePerformIO". This is left as an exercise for the reader :)
I suspect that this "inevitability of impurity" is not accidental.
Anonymous,
See here for a pure implementation: http://conway.rutgers.edu/~ccshan/wiki/blog/posts/Exhaustive/
It seems that the complexity of your "search" program is not computable because the proof of its termination is (necessarily) not constructive. The proof tells you that the program will stop but it can't tell you when it will stop. So you believe that the program will terminate in all cases but in some cases you would never see the end of its execution (even if you were an eternal beeing): each day you could think that the program will stop tomorrow. There is no difference between the "search" program and any looping program with an undecidable termination.
However, if the predicates eated by "search" are primitive recursive for example, then the complexity of "search" can probably be bounded by an Ackermann-like function (the predicates given as examples here are primitive recursive, although there is no stream in primitive recursion :o).
> There is no difference between the "search" program and any looping program with an undecidable termination.
Not so. For any input, this program is guaranteed to terminate. For any input, there is a time t after which you will get a 'yes' or a 'no'. This is different from, say, doing a brute force search for a solution to Fermat's last equation. There is no time t at which such a loop would terminate and say "no solution". It can only answer "yes, here's a solution" or loop indefinitely. These are distinct classes of program.
> For any input, this program is guaranteed to terminate. For any input, there is a time t after which you will get a 'yes' or a 'no'.
Your proof says that "this program is guaranteed to terminate". It is provable, but is it true in the real world? Can you compute this t? In some metasystems of classical logic there is some decidable properties A(N) on integers such that: whe can prove "not(A(0))", "not(A(1))", "not(A(2))", "not(A(3))"... for all integers, and we can also prove "there is N such that A(N)", without incoherence. Despite the apparent contradiction, it's all coherent. Intuitively, we might chech the property A(N) for all integers to reach a contradiction, but no one can check an infinity of properties, even an eternal beeing, so we cannot reach a contradiction. Such a property A(N) is similar to the property "the program (search p) ends after N steps" with a well chosen p.
This kind of example is similar to the coherent system S, defined by Gödel, which says that "there is an incoherent subsystem of S". So the "search" program is not garanteed to terminate, but we can believe in its termination without fear of contradiction, even if it's false in the real world. We don't have such paradoxes with constructive proofs because each proof generates a witness of reality: if we can prove that "there is a N such that A(N)", then we can prove "A(N)" for some N. This property does not hold in classical systems (because of the König's lemma in the case of "seach").
Anonymous,
Reminds me of Goodstein's theorem.
I think we're members of different mathematical cults.
In my cult, an axiom system in which "not(A(0))", "not(A(1))", "not(A(2))", "not(A(3))"... for all integers, and we can also prove "there is N such that A(N)" is an interesting curiosity, but not an axiom system I'd use to describe the real world.
:-)
Unfortunately, the pathological "A(N)" is not a vacuous statement: it states that the "search" program will terminate. It is more vicious than the Goodstein's theorem because it remains pathological in all coherent classical systems (the pathology arises because the Kolmogorov complexity of the predicates is not bounded). Moreover, the axiom system in which "not(A(0))", "not(A(1))", "not(A(2))", "not(A(3))"... and "there is N such that A(N)" are true is much more than a formal curiosity: it's a model of your axiomatic. That means it could be the true essence of the real world. Nobody can choose what is the real world and how it works. This is a valid instance of the real world: you coherently prove that "search" terminates in all cases and "search p" coherently loops with infinitely many p. It's even worse: it's validity is a consequence of your axiomatic, so you necessarily use it to describe the real world. This incongruity exists in all (big enough) classical systems, in all axiom systems you might believe in: there is no way to circumvent it... except being constructive.
There is no cult in constructions. Constructions are facts. Classical statements are beliefs. You can believe in a model, but it's not logic, the "good" model is beyond logic (it's not based on a recursive axiomatic), it's just a sweat dream (and we can prove that).
Anonymous. Can you suggests some books on the flavour of mathematics you're espousing? It's very interesting.
Re: pure implementation of "search"
Oh, I must tie that knot on my too quick tongue. Thank you, Dan!
Anonymous, correct me if I'm wrong, but in a constructive setting shouldn't the argument of "search" be (the code of) a recursive function together with a constructive proof of its totality? Then, I guess, we can prove the termination of "search" in a constructive way, too.
Post a Comment
<< Home