Absence of Evidence is Evidence of Absence
Suppose that a priori X has probability p of being true. We now look for evidence for X of a certain type. Suppose that there is a probablity q that we find this evidence if X is true and probability q' that we find this evidence if X is false. We will assume p<1 (otherwise we wouldn't bother looking for evidence) and that q>q' (otherwise it couldn't be said that the evidence we're looking for is evidence for X).
So we have four possibilities:
- X is true and we find evidence for X: probability pq
- X is true and we don't find evidence for X: probability p(1-q)
- X is false and we find evidence for X: probability (1-p)q'
- X is false and we don't find evidence for X: probability (1-p)(1-q')
Theorem
Under the hypotheses above, the conditional probability that X is true given that we failed to find the evidence is p(1-q)/(p(q'-q)+1-q').
Proof
Use Bayes' Theorem.
Some elementary rearrangement shows this is always less than p given the above hypotheses. It doesn't matter if we are unable to assign an a priori probability, this holds whatever value p has as long as it's less than 1. And if we don't know that q>q' then we shouldn't be in the business of looking for evidence. If the experiment we're doing is any good then q'=0 but as I have shown, the result holds even if we relax this condition.
So clearly failing to find evidence for X should lower our estimate of the probability that X is true.
I wonder what made Sagan say this. I think that maybe he meant to say "absence of evidence is not proof of absence". The theorem shows that under the original hypotheses the conditional probability is never 1, and so while we have evidence of absence, we don't have a proof. But if we can look for enough independent types of evidence it's quite possible for the conditional probability to get close to 1.
Update: The first paragraph was badly written and so I've edited it slightly. The main argument is unchanged. The argument stands or falls regardless of what Sagan actually said, but read the comments below on some context for this quotation. (02 Jan 09)
posted by sigfpe at Thursday, August 11, 2005
18 Comments:
If you do a google news search on absence of evidence you can see it's been used quite a bit by politicians lately - especially with reference to WMD. Again I think we have to interpret what they say as "absence of evidence isn't proof of absence" otherwise it makes no sense.
I think Sagan was definitely thinking of it from a deductive logic framework; one wouldn't be surprised if Sagan cautioned one against using ad hominem arguments, but couldn't we also show that a speaker's honesty and other personal characteristics should affect our priors about that speaker's statements?
"Suppose that there is a probability q that we find this evidence if X is true and probability q' that we don't find this evidence if it is false." - surely the "don't" should be deleted?
I'm also not convinced about your maths. Unpicking your rearrangements and stating it in the simplest form, your core assertion is that
p(1-q)/(p(1-q) + (1-p)(1-q')) < p
But cancelling p gives us
(1-q)/(p(1-q) + (1-p)(1-q')) < 1
and rearranging to gather p terms
(1-q)/(p((1-q)-(1-q')) + (1-q')) < 1
which can be expressed as
(1-q) < p((1-q)-(1-q')) + (1-q')
or
(1-q)-(1-q') < p((1-q)-(1-q'))
or, cancelling for commons,
1 < p
which contradicts our initial condition of p < 1.
(Admittedly my maths is very rusty, and I could well have screwed up something basic; if so, please don't spare my blushes!)
Damn it, I didn't realise this post was two years old... thank you reddit.
I think your list is mislabeled.
You define q' as the probability of not finding evidence, given that X is false. So, q' = P( not E | not X). So, the probability of X being false and not finding evidence for X (4) is:
P( not X and not E) = p( not X ) * P( not E | not X ) = (1-p) * q'.
You seem to mix up numbers 3 and 4. That definitely changes your denominator, though I'd guess your conclusion still works out to be correct.
On the other hand, I always assumed that "evidence" referred more closely to absolute "proof", rather than probabilities as you suggest. Otherwise, as you say, the concept of "evidence" as a means to work toward a fact, is meaningless.
Gwenhwyfaer: I ended up with 1-p < q-q'.
However, if q' is the probability that you don't find evidence for false X, isn't it always 1?
It's not clear what kind of things we're talking about here. So, I assume X is a logical formula and in this system logical axioms are randomly generated initially. Then, we can say that X has a probability p of being true.
The evidence we're looking for is a deduction. We don't know the axioms, and we're going to carry out the same steps with some assumptions. However, we have a machine that can halt our deduction sequence if at a given step our assumption is not satisfied by the axiom. Then we can speak of a probability q that we complete the proof. q could be 0 for some true X due to Godel's incompleteness, but q' has to be 1 because our deduction just can't terminate with X as the result, or else our logic is inconsistent.
Furthermore, I don't think P(X is true and we find evidence for it) is just pq because they're not independent: an axiom that invalidates X will not let you find evidence for a false X.
It is certainly interesting to see a mathematical take on the phrase, but I've always heard it said "the absence of proof is not proof of absence." Cheers
I read Demon Haunted World. In context, Sagan was saying absence of evidence is not proof of evidence of absence. The original quote is taken out of context.
> "absence of evidence is not proof of evidence of absence"
That can't be right. I doubt he was saying anything about "proof of evidence". It'd be cool if someone posted the exact quotation and context.
Here's the excerpt.
Carl Sagan used it as an example of bad logic or fallacy. Here is the complete context:
http://www-static.cc.gatech.edu/people/home/idris/Essays/Sagan_The_Demon_Haunted_World_Excerpt.htm
"I know Carl Sagan said the opposite, but he was clearly wrong."
No shit, Sherlock. As previously mentioned, he was using it as an example of poor logic.
"Suppose that a priori X has probability p of being true."
Sorry but the truth of a statement doesn't have a probability. From that critique, everything else falls down.
And no, Carl Sagan meant exactly, "absence of evidence is not evidence of absence" and he was not wrong. He is a scientist, and so wouldn't talk about proof anyway. No, he meant evidence, which is what he said, and what he said makes sense. It's called "Argument from ignorance". Look it up.
Curious: If one does not find evidence, then one cannot assign a probability to something...zero exists, while nothing does not. (1 - nothing) does not make sense.
You may find the following wikipedia page interesting, as it describes the (not yet fully worked out) statistics you are starting down the path towards.
http://en.wikipedia.org/wiki/Dempster-Shafer_theory
Anonymous writes, “And no, Carl Sagan meant exactly, "absence of evidence is not evidence of absence" and he was not wrong. He is a scientist, and so wouldn't talk about proof anyway. No, he meant evidence, which is what he said, and what he said makes sense. It's called "Argument from ignorance". Look it up.”
OK, let’s look it up. The Demon-Haunted World, chapter 12 “The Fine Art of Baloney Detection”:
• appeal to ignorance — the claim that whatever has not been proved false must be true, and vice versa (e.g. There is no compelling evidence that UFOs are not visiting the Earth; therefore UFOs exist — and there is intelligent life elsewhere in the universe. Or: There may be seventy kazillion other worlds, but not one is known to have the moral advancement of the Earth, so we're still central to the Universe.) This impatience with ambiguity can be criticized in the phrase: absence of evidence is not evidence of absence.
Carl Sagan criticizes the impatience with ambiguity in the phrase “Absence of evidence is not evidence of absence” as an example of an appeal from ignorance.
Robin,
Ultimately this is a semantic issue.
Wikipedia uses the example of cancerous cells in a body. Finding no cancerous cells in a biopsy makes it less likely you have cancer. Some people might call the lack of cancer cells a lack of evidence (for cancer), some might call it evidence (that cancer is not present).
As for scientists not talking about proof, have you ever talked to a scientist? I guess they tone the language down when it comes to actual publications.
Right, the absence of evidence (of cancer) is evidence of absence (of cancer).
Post a Comment
<< Home