Management | Marshall Tarley

January 26, 2018January 26, 2018

Hawaii Five-No. This Is Not a Drill! Check & Balance or Bust.

A Good Leader Must also be a Good Manager

October 2000. I got into the office early that morning and already had a voicemail to get to an 8:30 meeting. I quickly found out that a distribution system that was running for about a year had maxed out the bank account that drew funds to make these payments. I was put in charge of a task force to find out what happened and get it fixed. It didn’t take long to determine that data from one system was being transferred to another system that was paying the money out. The problem: when the payment system took the records in, there was no confirmation to determine whether the output equaled the input in the number of records or amount of money…nor were there any other validations. While one might expect a check and balance that went to a more granular level, at the very least, there should have been a systematic check at the macro level. And, that was only the beginning. The distribution system was designed so badly, it was making duplicate payments – and sometimes to people who shouldn’t have been paid at all. I came to call it the Chernobyl System*, after the shoddy Soviet-era nuclear power plant that had no containment system. It blew up, contaminating millions of people with radioactivity and creating a forbidden zone with a radius of at least thirty miles that will remain uninhabitable for at least 180 years, and some scientists say 3,000 years.

Hawaii Incoming
Fast forward to Monday, January 13, 2018. Vern Miyagi, the man in charge of the Hawaii Emergency Management Agency (HI-EMA), standing side by side with the Governor of Hawaii, had the courage to say, “This was my fault.” The procedures in place allowed one single individual to “push the button” to issue a live alert that there was an incoming missile minutes away from impact to Hawaii. What!?!?!! Yes, you heard me right. No checks or balances or confirmations. One person can make this error, and it can go through to strike fear and panic in the hearts of 1.4 million Hawaiians and tens of thousands of tourists.

Hacked to Death
C’mon, stuff like that is one in a million. Really, let’s go back into the ancient history of
September 2017. Equifax, the giant credit agency, announces it has had a data breach of
143 million Americans – including their Social Security Numbers, driver’s license data, security questions and answers and more. Equifax CEO, Richard Smith, who was forced to resign, testified before the House of Representatives Energy and Commerce Committee and said, “The human error was that the individual who was responsible for communication in the organization to apply the patch, did not.” One person – no checks, no balances, only errors…one huge error.

These are examples of inexcusable malfeasance and incompetence, yet it seems to go on more often than we could imagine in the most important organizations. Most managers are competent, but when they are not and when processes are put in place that lacks the appropriate necessary checks and balances, there should be a leader at the top and leaders up and down the line who are asking the right questions.

Check, Check and Double-Check…Really?
So, let’s get a few things straight. Checks and balances cost money. You don’t need to check everything all the time. You need to have the appropriate checks and balances in place. So, how do you know when to check and how often? There is no black and white answer to that, but let’s set down some sound rational ground rules.

When the distribution system I described at the top, the so-dubbed Chernobyl distribution system, was reprogrammed to have the appropriate checks and balances programmed to work systematically, we went forward to use it for the first time. When we did, I put in place manual stops, where we downloaded data and checked it manually in Excel. Each step of the process was manually audited in this way. It was tedious, laborious and time-consuming, but that was the first time we were using the revamped system, and after all, we had already lost millions of dollars, most of which could not be recouped. Think of the financial and political exposure of repeating the mistakes that were already made. We conducted the same validations the second time we ran it, confirming each step of the process, and we were verifying that the system itself was performing the checks and balances correctly. Now, we could have a reasonable expectation that it would run right and pay accurately. We pulled back many of these laborious manual audits and, as we went forward, we did spot checks and a reasonability analysis at the end of a run.

What are the ground rules?
When a process, an operation or a system is in place, here are some criteria to follow:
Criticality, Newness, Reliability
How critical is this?

Criticality
What are the stakes? The more that’s at stake, the more checks and balances you need.
Legal – Are there legal considerations that could make you, your organization or others legally liable if something goes wrong?
Financial – Is there significant financial risk to the organization or are you dealing with monies that amount to a virtual rounding error?
Political – Will there be fireworks in the boardroom? Will a key client, a key board member or the public be negatively impacted in a significant way?
You can add to the list, but these are the main events, and each of these have a scale. You have to balance the scale between the risks and costs.

Newness
How new is your process/operation/system?
When something is new, it needs the checks and balances to ensure that it is running right. This may mean a number of checks along the way. Over time, as you have ensured that it is running correctly and accurately, you can pull back on some, even many of these checks, but you always need some verifications. AND, you must always keep criticality in mind.

Reliability
We’ve run this over and over again without a single problem.
Don’t get lulled to sleep by this. Things change. All of sudden, they’ve replaced the server, or someone new is on the job. If something has been humming along, that’s great, but you still need to beware of the unexpected and still need to have appropriate checks and balances in place. AND, you must always keep criticality in mind.

*Note: In the wisdom of the senior management of this organization, the software designer who built what I called the Chernobyl distribution system was later promoted and went on to build another system with equally disastrous results.

December 12, 2017December 13, 2017

The Pit and the Pendulum – Lessons in Executive Leadership

Some executives believe that pitting managers or teams against each other will create vigorous competition, with the best leader(s) emerging on top. They’re wrong! The result will likely be the most divisive manager or team at the top, ready to sow seeds of distrust, suspicion, hostility and stress. Those ingredients kill-off creativity, innovation, collaboration and growth. In that Pit of conflict, the Pendulum may come to a complete halt. I had the experience of being called in to this kind of environment to conduct what can only be called “an intervention.”

The multi-year, enterprise-wide IT project that already cost many tens of millions of dollars, was way overdue and way over budget. The board of directors was more than a little concerned. It had reached a crescendo. The COO and CIO brought me in to work with the IT management teams. There were two teams and they were at war. I interviewed each manager individually and met with each team separately before daring to bring them together. The stress, anguish and frustration was palpable and poured forth in outbursts of rage and finger-pointing. The Delivery Team was responsible for the system design, software development, coding and delivery of the system. The Testing Team was responsible for every level of testing and for identifying bugs in the system. The Testing Team was dependent on the Delivery Team to understand every element of the system, in order to design the appropriate testing to identify flaws.

There was a powerful interdependency between the teams. That’s usually good.
But, the two management teams had two different sets of goals, and the goals of one team were in direct conflict with the goals of the other team. To add heat and fire to that situation, the teams’ incentive compensation was based on reaching their goals. The results – these management teams were in constant conflict and the project was in a quagmire. The Pendulum had stopped, but the clock and related budget dollars had been ticking on.

The first meeting with both teams had the feel and fireworks of marriage counselling.
Once they had the opportunity to air their differences, I pointed something out that they already knew – the status quo was unacceptable. The board was ready to step in and act. That’s why the COO and CIO sent me in and that’s why I had their full support. Now,
“What do you want to do about it?”

Unfortunately, their solutions were down in the weeds of the existing conflicts.
I asked them, “What is the overall goal for the company as a whole?”
It took a while, and they were able to articulate it and come to consensus on that goal.
Then came the tricky part. As I asked the next question, I felt a little like the guy on the high wire over Niagara Falls, because everything hinged on this.

“Now that we have identified the overall goal for the company, and since both teams have dependencies on each other that must be met in order for the system to be completed, would it make sense to have one set of goals that both teams are mutually judged on?”

There was silence, then grumbling. Finally, a couple of voices perked up. The logic was so compelling, so obvious, they slowly emerged to a consensus and embraced the idea. It was the crucial next step.

(I actually wanted them to be one team, but there was so much individual team identification and team pride – a good thing in most situations but not in this one. I left that as the next evolution.)

I took them a little by surprise at that point. I told them, “Stay right here. I’m going to try to get the COO and the CIO into the room right now to endorse this change.”

I scurried around the building fishing them both out of meetings, explaining the situation and prepping them for the need to approve this change right now. I brought them both into the room and asked one manager from each team to present the new joint goals the two teams had agreed to and the new direction they wanted to take. The CIO and COO approved it on the spot. The CIO immediately followed up with the Director of HR to change their incentive compensation to align both teams to the same goals and metrics.

There was one more small but important change we made. When testing showed up a bug, it was being called a “bug” or an “error.” The design and development managers felt highly insulted every time that happened and it destroyed trust and cooperation. We came up with a euphemism, “item to investigate.” It didn’t completely quell the issue, but it did take a lot of the hurt out of it and made it more objective. It turned out to be an important step.

I went on to meet with the management teams weekly for several months, which they came to refer to as their therapy sessions. The quagmire had been broken. They were clearly making strides forward. It wasn’t always happiness and joy, but sometimes it was, and the bite and battles had simmered down and mostly disappeared. At that point, my job was done. They went on to meet their new goals and their newly appointed deadlines. They delivered this huge system – a revolutionary change for the entire enterprise.

In retrospect, it all seems so obvious. But, it took a completely objective observer, one who could articulate the issues and, more importantly, gain the trust of the management teams and the senior executives, to allow them to embrace the change they needed to move forward.