Tweet
When Bad Things Happen to Good Projects
CIO Magazine
HP's project managers knew what could go wrong with their ERP rollout. They just didn't plan for so much of it to happen at once.
BY CHRISTOPHER KOCH
IT GOES AGAINST
The Limits of Project Management When contingency planning prevents a disaster, it's nearly impossible to tell whether everything that was done was necessary. Remember Y2K? Many business leaders continue to suspect that the billions spent on Y2K was a waste of money because nothing happened. CIOs who try to warn their CEOs that programming glitches could cost millions will almost invariably be met with this response: Then make sure the glitches don't happen!
That's why IT project management has become high art while business contingency planning remains in the Dark Ages. But the problems that can affect an enterprise software project increase all the time as the projects encompass more code and more business processes. And there are nearly infinite combinations of small problems that together can have devastating effects. In the end, it is riskier for companies to try to protect themselves from glitches in enterprise software projects through ever better IT project management techniques than it is to plan for manual workarounds to get products to customers in case of failure.
Other companies besides HP have faced similar business disasters from relatively small IT errors. Nike, for example, had a problem with a demand-planning application when it switched to a centralized SAP system in 2001. The problem was tamed within a few weeks. But because the company did not have an adequate business contingency plan, the small glitch in IT cost Nike $100 million in revenue. (See "Nike Rebounds.")
In both Nike's and HP's cases, the IT problems were due to a combination of factors that would have been difficult to eliminate in the project-planning process. At HP, Hanger says her team tested the connections between the legacy front-end ordering system and the SAP system. And the connections worked fine for orders that did not involve any custom configuration, as well as for some custom orders.
But Hanger's team was unable to adequately test orders that could be configured by customers because the product marketing team had not fully scoped the breadth of configurations that customers would want. When the system went live, some of these custom configurations went through and others did not. Those that didn't got spat out into a dead zone, sitting idle until they could be entered manually. "We had customers wanting a little more flexibility in how they purchased and configured their servers," she says. "And we did not always have the data modeled correctly [for that to happen]." Could Hanger have tested more configurations prior to going live? Yes. Could the team have envisioned all of them? Probably not.
But that wasn't the only problem. Hanger's team trained the customer service representatives who would be using the new system two weeks prior to going live. All representatives were required to pass a test showing that they could enter orders without making errors. But when the system went live, the representatives had trouble remembering all the training and were flustered, compounding the number of dropped orders. Hanger's team offered refresher training two weeks into the rollout, but by then, too many orders had fallen out to catch up. "We might have improved things if we had started giving them refresher training a week into the rollout instead of two weeks," she says.
Things only got worse when customer demand for configure-to-order systems spiked by 35 percent in June, beyond what HP's demand forecast models predicted. Instead, Hanger's business contingency plan called for normal demand through the summer. "We had planned for three weeks of extra inventory at a 50/50 split between standard servers and configure-to-order servers," she says. The factory in Omaha that had set aside some of its lines to handle the three weeks of orders quickly became overwhelmed. HP rushed to get a second factory in Europe online to help take up the slack two-to-three weeks later, but by then, it was too late.
Once the orders backed up, the only way HP could respond was by speeding up deliveries. Depending on the size of the product and distance shipped, the extra cost ranged between 35 percent and 40 percent, according to HP, and gained the company a few days on orders that had already been delayed by weeks.
The Contingency Plan That Wasn't By traditional IT contingency planning standards, HP had already gone beyond the call of duty. Most IT contingency planning focuses on preparing extra code rather than extra products. Convention says that to ensure a problem-free transition to a new system, there should be a redundant system ready to handle things if the rollout goes sour. At the very least, there should be a "rollback" strategy to go back to the old system if there is an issue.
Next Time, Imagine the Worst
Here's what Bouchard would have done differently. He would have expanded the contingency plan to cover five or six weeks. Instead of trying to prevent IT problems that were too small and rolled up in too many strange combinations, it would have been easier to bring the backup factory in Europe online earlier and stockpile more generic servers. In his interview with CIO, he did not address whether additional manual workarounds, such as a manual order-entry process, would have helped.
CIO Magazine
HP's project managers knew what could go wrong with their ERP rollout. They just didn't plan for so much of it to happen at once.
BY CHRISTOPHER KOCH
IT GOES AGAINST
The Limits of Project Management When contingency planning prevents a disaster, it's nearly impossible to tell whether everything that was done was necessary. Remember Y2K? Many business leaders continue to suspect that the billions spent on Y2K was a waste of money because nothing happened. CIOs who try to warn their CEOs that programming glitches could cost millions will almost invariably be met with this response: Then make sure the glitches don't happen!
That's why IT project management has become high art while business contingency planning remains in the Dark Ages. But the problems that can affect an enterprise software project increase all the time as the projects encompass more code and more business processes. And there are nearly infinite combinations of small problems that together can have devastating effects. In the end, it is riskier for companies to try to protect themselves from glitches in enterprise software projects through ever better IT project management techniques than it is to plan for manual workarounds to get products to customers in case of failure.
Other companies besides HP have faced similar business disasters from relatively small IT errors. Nike, for example, had a problem with a demand-planning application when it switched to a centralized SAP system in 2001. The problem was tamed within a few weeks. But because the company did not have an adequate business contingency plan, the small glitch in IT cost Nike $100 million in revenue. (See "Nike Rebounds.")
In both Nike's and HP's cases, the IT problems were due to a combination of factors that would have been difficult to eliminate in the project-planning process. At HP, Hanger says her team tested the connections between the legacy front-end ordering system and the SAP system. And the connections worked fine for orders that did not involve any custom configuration, as well as for some custom orders.
But Hanger's team was unable to adequately test orders that could be configured by customers because the product marketing team had not fully scoped the breadth of configurations that customers would want. When the system went live, some of these custom configurations went through and others did not. Those that didn't got spat out into a dead zone, sitting idle until they could be entered manually. "We had customers wanting a little more flexibility in how they purchased and configured their servers," she says. "And we did not always have the data modeled correctly [for that to happen]." Could Hanger have tested more configurations prior to going live? Yes. Could the team have envisioned all of them? Probably not.
But that wasn't the only problem. Hanger's team trained the customer service representatives who would be using the new system two weeks prior to going live. All representatives were required to pass a test showing that they could enter orders without making errors. But when the system went live, the representatives had trouble remembering all the training and were flustered, compounding the number of dropped orders. Hanger's team offered refresher training two weeks into the rollout, but by then, too many orders had fallen out to catch up. "We might have improved things if we had started giving them refresher training a week into the rollout instead of two weeks," she says.
Things only got worse when customer demand for configure-to-order systems spiked by 35 percent in June, beyond what HP's demand forecast models predicted. Instead, Hanger's business contingency plan called for normal demand through the summer. "We had planned for three weeks of extra inventory at a 50/50 split between standard servers and configure-to-order servers," she says. The factory in Omaha that had set aside some of its lines to handle the three weeks of orders quickly became overwhelmed. HP rushed to get a second factory in Europe online to help take up the slack two-to-three weeks later, but by then, it was too late.
Once the orders backed up, the only way HP could respond was by speeding up deliveries. Depending on the size of the product and distance shipped, the extra cost ranged between 35 percent and 40 percent, according to HP, and gained the company a few days on orders that had already been delayed by weeks.
The Contingency Plan That Wasn't By traditional IT contingency planning standards, HP had already gone beyond the call of duty. Most IT contingency planning focuses on preparing extra code rather than extra products. Convention says that to ensure a problem-free transition to a new system, there should be a redundant system ready to handle things if the rollout goes sour. At the very least, there should be a "rollback" strategy to go back to the old system if there is an issue.
Next Time, Imagine the Worst
Here's what Bouchard would have done differently. He would have expanded the contingency plan to cover five or six weeks. Instead of trying to prevent IT problems that were too small and rolled up in too many strange combinations, it would have been easier to bring the backup factory in Europe online earlier and stockpile more generic servers. In his interview with CIO, he did not address whether additional manual workarounds, such as a manual order-entry process, would have helped.
Comment