Yesterday a colleague came by to see if I could help her do some simple stats to look at defect trends. She said she was shocked, SHOCKED that there had been a dramatic spike in costly defects in a certain process. Compared with the same quarter last year defects went from 6 to 14, MORE THAN DOUBLED! she said. After probing a little we discovered that the number of batches processed had also nearly doubled. In fact, on a per batch basis the defect rate was just a hair higher. “It still went up,” she insisted. But was it up because things were actually worse, or was it just process noise?
This is a simple example of one of the most perilous traps of “data driven” decisions: seeking meaning in natural variation. In other words, be sure you’re not making big, costly decisions based on noise. Think of your friendly neighborhood statistician as your own pair of noise-canceling headphones like these: