A Drunkard’s Walk…with Drift

I just finished Leonard Mlodinow’s book, A Drunkard’s Walk:  How Randomness Rules Our Lives.  It’s a superb guide to properly interpreting seemingly improbable events and should be a mandatory read for anyone in Business, Sports, Politics, Economics…pretty much everybody.  A part of me enthusiastically devoured the myriad examples of long shots and how when taken in the proper context are easily explained by random chance.  A different part of me struggled to reconcile my belief in the laws of chance that can explain remarkably well chaotic events (from Roger Maris to Warren Buffet) and my certain knowledge of Divine Providence.

First, a thought experiment that captures how randomness can explain that 1 in a million shot.  What’s the probability of flipping a fair coin and getting 20 heads in a row?  It’s about 1 in a million for a specific run.  Now, imagine a million people flipping coins; those whose coin lands on heads get to flip again.  After the first round you’d expect about half to be eliminated and half again after the next round and so forth.  You would expect 1 person on average to end up making it 20 rounds.

Most of life’s experiences are of the sort where there are multiple players with a distribution of possible outcomes.  Take the stock market.  There are only a few Warren Buffets.  Those few believe they are able to leverage market inefficiencies and we all bow down in deference at their knowledge.  Or it could just be that there are millions of traders and hedge fund managers and a few of them would be expected to perform a great deal better than the market just by chance.

My favorite thing about this book is its wealth of examples.  It’s a very compelling case.  So how do I reconcile a belief in Randomness and a perfect knowledge of Divine Providence?  The answer came as I turned the final digital pages of the book.  Mlodinow’s  allusion to a Random Walk is incomplete.  We are not particles suspended in a liquid being bombarded by molecules which, in aggregate, move us along in a Brownian motion.  Our life is not a random walk…it’s a random walk with drift!  The drift is guided by an active God who is able to influence our lives and His work in ways that can only be perceived by those with adequate methods of estimation.

This explains how God can build a church, a nation, or a life when everything looks like chaos. He can move things along despite free will, natural disasters, and market turmoil.  The drift also explains how God can influence human events without being responsible for the random walk component, the devastating tsunamis, earthquakes, murder, war, hunger, etc.  He can use these events to guide nations and lives without being the cause of evil.

Here’s a graph of a random walk and a random walk with drift.  History clearly follows the latter as does evolution.  World Wars, depressions, global recessions, none have extinguished technological progress.  And as for biology, random permutations of DNA all tend toward improvement.

The noise and randomness may increase, and it may get harder to see the signal through the noise, but those with eyes to see will have the power (in the statistical sense) to estimate the drift parameter.


1 Comment

Filed under Ordinary

Personal Experience Shows…

The film critic Pauline Kael allegedly said, in response to Richard Nixon’s landslide victory, “How is that possible!  I don’t know a single person who voted for him.”  While she may not have actually said it, this kind of reasoning is not as rare as you might hope.

I had a friend in high school who wasn’t allowed to drive a car.  His parents gave him a scooter because they had known more people who died in car crashes than motorcycle accidents.  Apparently 2-wheeled death traps are much safer than cars.  And take this recent article.  A presumably educated and credentialed psychiatrist actually heralds the death of marriage as an institution with this gem:

I would venture that 90 percent of the married patients I speak with would rank their marriages in the top two stressors in their lives, while only 10 percent would rank their marriages as one of the top two sources of strength in their lives.

Besides being incredibly vague (rated in the top two stressors?  out of how many?) the good doctor doesn’t even pretend to have real summary statistics about his biased sample (“I would venture that…”). If there’s one thing worse than the “studies show” argument, it’s conversations that begin, at least conceptually, with “my own experience with self selecting groups shows….”    It’s this type of thinking that leads people to believe that figures lie and liars figure.

Leave a comment

Filed under Studies Show...

Little Green Men

Last St. Patrick’s Day my wife played a few fun tricks on our little ones.  For breakfast she dropped a splash of green food coloring in the frying pan before adding the eggs.  My four-year-old caught a glimpse of her sneaking about.  When the eggs started to turn green my wife said it must be the leprechauns playing tricks.  My son thoughtfully commented, “Leprechaun’s have red hair.  You don’t have red hair.  So it must not have been a leprechaun.”  I was stunned that a four-year-old could turn such a textbook logical argument.  (I’ll allow that he omitted a minor premise.)  I was proud of my little guy.  After all, isn’t it cold Reason and its prized pupil Logic that set us apart from lesser beasts?

On the other hand that same four-year-old is deadly convinced that there is a “gunny monster” that will eat his toys if they’re left out.  Even after we’ve told him that there is no such monster, he’s meticulous about picking up every night…just in case.

It turns out wild imagination and logic are really two sides of the same coin:  They are both uniquely human attributes.  In a Cartesian world there’s no question which is more prized, but it is imagination/passion/faith that is by far the more powerful.  Think of it this way, all technological, scientific, and artistic achievements  are a product of faith.  Faith is not stepping to the edge of the light and leaping blindly into the dark hoping there’s something worthwhile there.  True faith is exercising every ounce of strength we possess in pursuit of something that may exist nowhere but in our own minds, but is true.  If what you believe turns out to be false despite a rational degree of belief in its truthfulness, then it was not true faith.  Putting a man on the moon was an act of pure faith.  It had never been done.  Many were convinced of it’s possibility and worked their guts out until it was accomplished.

So what does this have to do with probability and statistics?  This post is already too long, so I’ll leave that for another time.

Leave a comment

Filed under Ordinary

Friday Fun

Because I can’t resist a rap that mentions battling regressions. You may recall an earlier post by Mr. Keynes.

Leave a comment

Filed under Ordinary

Ghost Stories

The human brain is remarkable at recognizing patterns.  In fact, humans are so adept at connecting dots and filling in blanks that we quite often see patterns that don’t exist.  We like to put ideas, stories, images, etc. into nice, tidy and complete packets; it helps us sleep better at night.  But we’re prone to filling in details skewed by our preconceptions.  That’s why conspiracy theories will always exist, ghosts will always be seen in moonlit rooms and why you see a complete triangle that doesn’t exist in the Kanizsa image below.

For statisticians, it’s why we’ll always have to explain how large variations and “trends” can be meaningless.  I got roped into being an unbiased arbiter in a heated dispute about a manufacturing process improvement.  Half the group wanted the project dead and the other half had $50 million a year riding on its success.  I showed a graph that I thought told a pretty compelling story in favor of advancing the project.  It ended up being a perfect Rorschach.  One person went so far as to say, “any rational person looking at that would see an overwhelming significance!”  Others were more than underwhelmed.

While statisticians seek new ways to visualize data we ought to keep in mind the brain’s uncanny ability to identify spurious patterns.

Leave a comment

Filed under Ordinary

99.9% Effective

How comfortable are you with 99.9% effective?  That’s the number we like to use to show near certainty.  But in a lot of cases 99.9% effective is totally unacceptable.   It turns out to be about 3 bad takeoffs from LaGuardia airport per week, 8 failed open heart surgeries a year, or two typos per page in a typical novel.  Six Sigma in the strict sense is about getting to a point where you have only 3.4 mistakes per million opportunities.  That translates to about 99.999% effective.  What a difference a couple of decimal points can make.

Leave a comment

Filed under Ordinary

Nature vs. Nurture

The bane of agricultural statistics is what biologists call Genotype by Environment interaction, or GxE for short.  Broadly speaking it is the old problem of nature vs. nurture.  But for some reason when we’re talking about a stalk of corn there’s really no debate about which dominates.  Environmental effects can easily swamp genetic expression.  This is why it’s so difficult  to predict which varieties are superior.  So why is it that so many people insist that when we’re talking about humans, gene’s dominate?  Humans are organisms that are not only affected by environmental factors like nutrition and sleep, but we can also be influenced by ideas, by imagination and by a sense of wonder.  At our best we can even be persuaded by reason.

At least with corn you can produce genetically identical varieties in hundreds of locations to estimate how robust the genes are.  It would be impossible, or at least unethical, to clone a few dozen individuals and raise them in diverse, controlled environments.  We can talk about twin studies till we’re blue in the face, but they are not well designed experimental trials and are prone to all the problems of observational studies.  And that’s the bane of the social scientists.

Leave a comment

Filed under Ordinary