When Things Look Too Good to Be True….
Validate Your Assumptions to Avoid Confirmational Bias
by Dave Righthouse – Senior Project Manager, Plant Operations, NAES Corporation
As plant operators, we often initiate projects aimed at improving performance – but when the results appear to show more improvement than we expected, it’s time to look that gift horse in the mouth. As a first step toward critical thinking, we need to validate our assumptions.
At one of the solid-fuel facilities we operate, the staff recently made a number of improvements to the boiler, monitoring equipment and controls. As one of these, they changed out the Continuous Emissions Monitor System (CEMS) probe heater, but did so using an unconventional approach. When the plant came back on line, they noted lower levels of CO2 and NH3 (ammonia) in the flue gas and a reduced rate of limestone consumption.
They were glad to see these positive outcomes, which they naturally attributed to the improvements they’d made and which happily exceeded their expectations. Several skeptics, however, suggested they validate these readings. This idea met with resistance at first, but eventually the others agreed to have a portable meter brought in to check the original numbers. After some difficulty in setting it up, the meter showed numbers that were not nearly as impressive as those originally recorded. The ‘believers’ refused to accept the new measurements, however, because of the trouble involved in setting up the meter.
A month later, the plant performed a required RATA (Relative Accuracy Test Audit), which again showed the original post-improvement readings to be inaccurate. An investigation revealed a problem with the CEMS probe heater that had been replaced during the outage. An air leak was diluting the flue gas samplings, which thus showed much lower concentrations of CO2 and NH3 and thus reduced the mix of limestone. Corrective action was taken to restore accuracy. Consistent with NAES’ values, the plant self-reported the issue to the state’s department of environmental protection. The two parties agreed on a settlement of just over $100,000 for the infraction.
At another facility, the NAES staff attempted to solve an issue with a critical seal that had shown a recent pattern of wearing out after only six months to a year of operation. In years past, the same seal had lasted two to three years, but other equipment revisions over time had prompted the OEM to change the seal’s design and material in a way that shortened its service life. In keeping with the NAES philosophy of continuous improvement, the staff made modifications that allowed use of a different seal in hopes of restoring the two-to-three year service life.
They put the machine back together and restarted, watching for signs of wear in the new seal. Over the weeks that followed, they were excited to find no indication of wear at all. The modification looked like an unqualified success, but the crew wanted to wait and watch a bit longer before opening the champagne. A month, two months, three months went by, still with no signs of wear. When six months had gone by with zero wear showing, they thought they’d finally resolved the issue.
Then suddenly the seal failed, flooding the area and causing a forced outage. What could have happened that abruptly broke the seal with no prior indication of wear? NAES launched an investigation and found that the seal had worn just as before. However, the seal’s wear indicator had failed because critical components had been left out the last time the machine had been reassembled. This caused the wear indicator’s connecting rod to disconnect, ultimately freezing the wear indicator in place. Thus, the seal stayed in normal service until it failed completely, surprising everyone and forcing a costly unplanned outage.
The lesson from both of these scenarios: If things appear too good to be true, then it’s time to validate your assumptions. In particular, look for ‘confirmational bias’ – the tendency to search for, interpret or recall information in a way that confirms one’s beliefs or hypotheses.
If the results turn out in fact to be valid, then take a bow. If not, you’re in a position to solve the problem before it results in a costly outage or regulatory violation.