Making IT Work: Lessons in Leadership in IT Infrastructure
Several years ago I found myself in a position analogous to Chief Technology Officer and Chief Information Officer of a large organization - at the same time. (The military terms were Assistant Chief of Staff Intelligence, Assistant Chief of Staff Combat Systems, and Command and Control Warfare Coordinator). As such, I was responsible for the installation, and operation of all information systems as well as the use of that information for keeping our boss (the Admiral) knowledgeable as to what was going on in the world around him.
There was only one minor problem: none of the systems were installed.
The task at hand was to install all the information technology gear necessary to handle all the requirements of a modern, nuclear powered aircraft as well as all the ships accompanying her. What are those specific requirements?
First, the aircraft carrier is a small town of 5000, with (as this was still in the early days of networks) 1400 LAN drops (for reasons of both security and issues of RFI WANs are more problematic on a ship then on dry land, and they are also a problem when there are classified networks in some of the same rooms as the internet systems). This was the first aircraft carrier fitted with this many PCs, almost an order of magnitude more PCs than previous aircraft carriers. Similar LANs were being installed on most of our other ships, as well, also for the first time.
Second, the ship is an active airfield with roughly 70 to 80 aircraft operating from it. Additionally, those aircraft require their own classified networks to support planning, maintenance and operations. That too was part of the IT installation.
Third, the aircraft carrier was the flagship of a 14-ship group – known in the Navy as a battle group or strike group, and known during any specific operation as a task group.
The 14 ships included two Aegis cruisers and two Aegis destroyers, similar to the ship that recently shot down the NRO satellite. The Aegis ships were all receiving extensive software upgrades to both their radars and their weapons control systems, upgrades that were substantially more complex then any that had been attempted in a number of years. There were also three amphibious assault ships, which included 2500 Marines and another 25 aircraft.
Fourth, the ship/network needed to be able to handle unclassified information, secret information, top secret information, special intelligence information as well as information that could be shared with allied navies – requiring the ability to accurately handle information from multiple classifications through the same communications systems, while assuring both the correct movement of information between systems when allowed, but protecting that information when not allowed.
Fifth, the ship had to be able to function as both the command ship for a multi-carrier group – known as a ‘battle force’ – that might include as many as four aircraft carriers, 300 aircraft, as many as 60 ships and as many as 60,000 personnel, as well as its own Network Operations Center for the extended networks of the battle force – another ‘first ever.’ As it turned out, we did act as the command or flag ship, as well as network control, for a ‘task force’ that consisted of three aircraft carriers, 25 ships, two submarines, more than 200 aircraft and roughly 30,000 people.
All told, these installations, to include, hardware, software, necessary modifications to the ships to support the upgrades, plus training and maintenance for the total period amounted to something in the neighborhood of $100 million.
When we began this process the carrier was brand new. (Brand new is a bit difficult to define; an aircraft carrier takes more than 5 years to build and after it is ‘finished’ and commissioned, it will be another year and a half of drills before the major fitting out and training of the ship begins). At that start point the systems installed on the carrier are limited to those systems necessary to conduct basic flight operations around the ship – the flight control radars for landing aircraft, one air search radar, several basic navigation radars and the radios for navigation and for flight control.
All of the complex, advanced Satellite Communications systems, advanced air defense radars, and sophisticated data links had yet to be installed, the network internal to ship had not been installed, the network handling system to allow the ship to function as a network node had yet to be installed. Major ‘command spaces’ – the spaces in the ship for the major information processing and the spaces for various mission commanders – the admiral in command of the battle force, his intelligence center, the center for integration and information management for different warfare problems such as ‘Air Warfare,’ ‘Anti-Submarine Warfare’ and ‘Electronic Warfare’ – were little more than empty rooms when we took control of the ship. Several spaces literally were bare steel – they had not even been painted and there was nothing in the room. We had 18 months to prepare this ship, as well as manage the IT work on all the other ships in the battle group, to include the complete systems upgrade of two of the Aegis cruisers and all the installations on a brand new Aegis destroyer.
At the same time we faced the same problem others had faced: the systems are all installed by a wide range of contractors and sub-contractors. But, in fact, neither the Admiral nor the captains of any ships actually controlled the contracts. Often they were (and are) not even controlled by anyone within easy reach. Rather, they were controlled by large and remote organizations whose directors are in Washington and who had completely different sets of priorities then we did.
The Result: we did everything that we were required to do, our systems stayed up, and our operational availability figures were unprecedented. Our down times were – across the board – fractions of the normal down times. One example, our most complex system, was a large format SATCOM system used for moving images that, because of the environment (seawater, jet exhaust, extreme heat, complexity (and normal lack of adequate training)) routinely had UP rates of 20% - the best achieved rate prior to our effort was 28% availability over a 6 to 7 month period. We had 3 hours of un-planned non-availability during our entire 6+ month deployment.
How did we do it?
A Clear Goal
We had clearly identified missions and goals, performance and capability standards that we needed to reach or exceed. We needed to achieve those standards two months prior to deploying, and be able to sustain those standards for eight to nine months. It would begin in 16 months from the day I took over, with a month long exercise to test every ship, every person and every system in the group in all of their assigned roles. We knew what we needed to accomplish and we focused on that, placing readiness at that point as the very center of our effort. The key to achieving that focus was planning.
Planning
I began planning immediately. We started with a simple chart on the wall, which actually ended with the ships all returning to the United States 25 months later, was stretched across two walls of the office, and the missions and tasks we knew we would to carry out over that time period. We then worked backwards from there, identifying every possible milestone or hurdle that we would need to overcome to ensure that we were successful. These hurdles were overwhelmingly not IT or even loosely ‘information’ centric. They were simply events we knew (or anticipated) the Group would face over the next 25 months.
Several major iterations of this planning timeline were generated before we were even sure we had all the important items identified. Then we sat down and attempted to identify what systems needed to be in place and operational to support successfully addressing each of those events. Often, particularly in the beginning, this required a good deal of supposition (or simply guessing), but we soon became fairly adept at identifying not only what systems were needed when, but also, and often more importantly, we became adept at identifying which systems were either not needed or might be useful even if not completely installed.
Key to this was regular and frequent meetings with the captains of all the ships and their key officers as well as key personnel who would be involved with each installation. We learned what was in the realm of the possible and what was inflexible for other reasons.
This was significant because it gave us ‘room’ to start adjusting installation and calibration schedules. Onboard a ship (and actually in any building) there are always limits to how much can be done at the same time. Often, and for similar reasons, the companies that are installing pieces of gear will want access to the same places at the same time. By identifying not only what we really needed and when, but also what we really didn’t need yet (though it might at first seem like we would) we were able to control the pace of installations in a way that suited our needs, not the needs of someone who had originally scheduled the installations (usually with little other information).
One example: shipyard schedules are very tightly controlled because there are only a few shipyards available and, if it is the case of an aircraft carrier, very few that can actually fit an aircraft carrier. One of the requirements for the carrier was a large SATCOM antenna. For a number of reasons the antenna could not be placed on the carrier’s superstructure, the small ‘apartment-building’ like structure that sits on the flight deck. So, we needed to build a sponson, a 40 foot long, roughly 40 ton piece of steel that jutted out from the hull, that the SATCOM would sit on. The SATCOM would not be delivered for many months, but the available period in the shipyard would only be available for a short period of time. We identified that period early and adjusted the contractor to send one man down to install some basic gear and pull certain cables while the sponson was being installed, fully six months prior to install of the system. When it came time to install the system it was a significantly more straightforward process than it would have been if we had not first ‘prepared’ the sponson. The result was that the physical installation time was cut in half, and the system grooming time was tripled, as was the training period for our personnel. This was the kind of event that our calendar allowed us to identify early.
Once we had a clear sense of the ‘real’ schedule, that is, what we absolutely needed and when, I presented it to the Admiral (analogously, the Chairman – President – CEO) and received his endorsement.
I was fortunate in that several of my key people (and I) had extensive experience in long range and strategic planning. If you have never engaged in strategic planning, bring in someone for a week of training, it will pay big dividends. I also benefited from the fact that as the Intelligence Officer (Chief Information Officer) I was also a user of the systems I was installing and maintaining. This provided me with a valuable perspective that others before me had not enjoyed (though, at the time ‘enjoyed’ was not a word that was used a great deal).
Communication
Once the Admiral was comfortable, we had to inform everyone else. We began by informing the captains and key personnel on each ship exactly what the schedule was, and as important, we made sure they understood why. Even though they had participated in identifying these ‘reasons why’ a matter of several weeks earlier, this helped them to not only more clearly understand, it also made sure that they had visibility into what each other ship was doing and why. Everyone was informed, everyone was part of the same team, there were no ‘secrets’ and everyone knew that no one was getting a ‘deal’ as well as no one was getting the dirty end of the stick.
When that was done, we informed the information management folks on all the ships, the people who would actually operate and maintain the systems once they were installed. We made it clear to them that we would all be living and breathing based on this schedule. We used as much time as was needed to make sure that these people not only fully understood the schedule, they understood what was the basis for the schedule, the why, what exactly the ships and aircraft and Marines in the group would be doing, why certain systems would be needed, and why certain systems needed to be very precise and what that precision really meant to the user. This understanding on their part was perhaps as important as any other single thing in ensuring the systems remained operating and within proper standards.
Training
Our training program was far and away the most aggressive systems training effort anyone had ever seen. We used every means in the book – extra money, squeezing every dime we could out contractors, using all of our training money, taking advantage of people who simply ‘offered,’ to get more training for our people. Perhaps it is possible to be over-trained, but I‘ve never run across it. Whereas other carriers let their people get trained in other specialties, we focused every additional training hour and training dollar on the specific systems – hardware and software – that we would be working with.
While we were fair, we were very demanding in training and in maintenance. We made it clear from the beginning that we were setting ‘ridiculously high’ standards of availability and that we would train and exercise the people involved until they could achieve those standards. Once people understood the focus of our effort, and they understood the plan, and they understood the importance of the plan, they responded superbly to the training and the exercises.
My Deputy
One key factor in my success was one man: Pico Torres. Pico was a charming and brilliant technician, but his real skill was in leading people. While I was trying to think ‘big thoughts,’ pushing forward the plan and meeting with the captains, the admiral and various contractors, Pico was working with the ‘troops.’ His ability to communicate with the average 22 year old technician, to help him understand why something ‘needed’ to be done, and why it needed to be done right, the first time, his ability to orchestrate and lead training, to conduct extemporaneous drills to test their skills, his energy level in meeting with one young sailor at a time, for hours on end, to help the sailor learn a fine point about this or that piece of hardware or software was noting short of remarkable.
I made a point of making sure everyone knew that Pico had my absolute backing: I might be 500 miles away and out of reach of any phone. But, if Pico said it was so, I would back it up – and I ALWAYS did. If you wanted to get it reversed, only the Admiral could reverse it. And that never happened.
Finding someone as talented as Pico Torres is perhaps the one issue that isn’t readily ‘teachable.’ I was fortunate to have such an individual working with me. You will need to find the most talented person you have and then you must provide some grooming and support. Once you are comfortable with their level of skill, you must provide him or her with both the freedom and the top cover to carry out the detailed plans on a daily basis.
Communication – Part 2
Once the plan was in place and being executed, I devoted considerable effort to keeping the boss, the various ships’ captains and other senior officers in the group, and the more than 1000 technicians, contractors and workers who were installing or would maintain and operate these systems, informed as to our progress as well as informed of any changes, changes both in the specifics of the plan as well as changes in what we as an organization of 14 ships and 13,000 people were doing. This served to keep the meaning of the plan in focus.
This communication consisted of both regular updates at weekly departmental meetings as well as ‘leading by mingling’ on the part of both Pico and me, as well as several other people who worked directly for me. (These often took the part of me stopping by for coffee and chatting with folks at the coffee machine). These informal sessions also served to keep us informed of the detailed problems that they were having in executing discrete elements of the plan. Both were critical.
Lessons learned
Leadership
Taking charge and providing clear goals was the sine qua non of success. So, the most important lesson is simply:
1) Know why you are doing it.
Knowing why we wanted a piece of gear (hardware, software) – the desired capability – was at the center of our success. The reason, the capability or the mission was important, not the piece of gear. Once everyone understood the real goal, and then the role ‘X’ played in achieving the goal, we were over the largest single hurdle. Once you know the goal (the vision), don’t let anyone pull you off that. Our goal was successful combat operations. Most definitively the goal for us was NOT smooth IT operations; your company sells something: hardware, software, turbines, soybeans or penny candy – the IT supports it. Once people understand that “simple” fact it provides remarkable clarity, and sets the foundation for everything else.
2) Communicate
Make sure you communicate enough. First, do the people who work for you understand what it is that you are trying to do and why? Second, does your boss know? Third, do your peers – the other VPs or Directors or however they are titled? And, are you listening to them: your people, your boss, your peers? As a general rule, no one communicates enough. Ted Williams used to say ‘Practice, practice, practice, practice, practice. And when you think you’re done, practice some more.” You can put the word ‘communicate’ in the place of practice and the statement is equally accurate.
Complex organizations require more hands on leadership – so get out of your office: walk around, listen, get a cup of coffee, talk to your people, listen. Remember that you – in a very real sense – work for them: your job is to make sure they have all that they need to do their jobs, the training the gear, etc. So, visit each site, have a cup of coffee and listen.
Set high standards: superb results aren’t achieved with mediocre standards. When I took over as the head of combat systems one of the things I found from talking to other people who had occupied these jobs in the past was a sense of frustration and some fatalism – ‘its always like this – particularly with a new aircraft carrier.’ Simply put, don’t accept that. Make sure everyone understands that you simply aren’t going to accept that. Set the standards high and inform ‘the troops.’ (Your standards will be part of the planning process, but making it work is a leadership issue). People will amaze you with how much they can achieve – so be fair and friendly and keep communicating, but set HIGH standards.
One final thought on standards: make sure your IT people know not only whom they support, but also what those other departments are doing – specifically. If you (the IT department) succeed, in your terms, but it isn’t what the other departments really needed or it wasn’t in time, you haven’t succeeded. This is a team sport, and everyone needs to win.
3) Train your people
It’s almost impossible to image people who received too much training. Load up on the training – it will pay you back.
3A) Training includes exercises.
- Document your exercises – have someone who keeps detailed notes, it is essential to understand what went well and what did not
- Debrief and lessons learned – when the exercise is over, review the notes, make sense of the results and then share it with your people. This is not punishment, it’s learning – you need to treat it that way and you need to demonstrate that fact with your people. One key step in making that point clear is to make sure that any negative comments about your performance in the exercise gets as much ‘air time’ as anyone else’s.
- Make sure you include recommendations for improvement in the debrief – the exercise is to help improve not to criticize.
3B) Train the rest of the organization
Just because you trained your people to maintain and operate the IT gear, does the rest of the organization know how to best use it? Make sure there is a training plan for them, and make sure you coordinate the training schedule with their regular work schedule.
4) Find at least one good (great) deputy
You may not be as fortuitous as I was in having a truly superb leader like Pico Torres working with me, but there are lots of sharp folks out there, and with the right support from you (pushing and pulling, suggestions and help) you can make people rise to the occasion, reach their real potential, and become good leaders.
Once you have several capable deputies, delegate everything that isn’t critical, pushing it completely out of your purview – find someone you can trust and have him give you status reports once or twice per week, but otherwise let him run his own show.
5) Bring your boss in early when you need to. You will have internal roadblocks – another department head is being obdurate and is going to upset the entire timeline and affect not only his own but other timelines in the org., explain, negotiate, try to work with him, but wait so long that that his resistance overturns your timeline. Before that happens you need to both inform the other department head and the boss. It’s not about ego, it’s about reaching the overall organizational goal, and if your planning is right, then something needs to be done: so, negotiate, communicate, etc., but don’t fail to act.
Organization
1) Have a Plan
Once you know the goal, you need a plan to get there. There are many ways to build plans – pick one and build a plan. But always remember – the goal is important – the plan is simply a tool to get you to the goal. You aren’t installing the IT for the heck of it – what are we trying to do and when must we do? Simple flow charts, PERT charts, whatever process you want to use is fine: but have a process, one that is visible and known to all --- and post it. That being said, I found that there were key elements unique to successful IT plans:
A) Keep perspective - Don’t measure with a micrometer if you are cutting with a chain saw - (the sponson) – if you are engaged in heavy construction, don’t start pulling wire, calibrating systems, and loading software
B) Don’t forget the simple stuff: power, cooling, the building, water, people using the stuff and people maintaining the stuff (Can you access everything? Do you have enough toilets?)
C) Take nothing for granted (never assume) – your plan needs detail in IT, particularly when you have a bunch of contractors to sort through. You may not know all the details, but your deputies need to.
D) Have a TCD – a technology cut-off date - beyond which you will not add new or improved pieces of gear. The reason for this is simple: people have to figure out what they have, figure out how to use it, and figure out how to integrate it with other systems. If there is no TCD – or something like it – you set yourself up for some smart guy to come in and ‘sell you’ some really neat (and they are) piece of gear 10 days before the big thing (in our case steaming into a combat zone, but it could just as easily be a takeover or merger or a new product release and when everyone should be focused on that, they are instead struggling with a new piece of gear. So, know when to say ‘no.’
Many organizations used to have TCDs and got rid of them in the recent past because they feel they will lose too much if they don’t have the latest thing. If you can figure out how to manage that, fair enough. But, in any large organization, if you are trying to integrate information from different systems, you need to have some sort of deliberate process and that will result in a stair-step function. You can make the steps small and frequent, but they must be deliberate. The TCD may not even officially exist, but if there is any complexity to the infrastructure, you will need to have deliberation and control.
E) That being said, you also need an upgrade plan. Any large IT infrastructure improvement will last considerably longer then a single generation of any number of different IT systems. Accordingly, a good plan will both provide for substantial room for growth so that elements of your infrastructure will be adequate for many years, while other elements can be more rapidly replaced without causing disruption to the entire system. Key to this is a training plan that ensures the right people in the organization are receiving the right training so that the new box doesn’t arrive and your people aren’t ready to operate and maintain it.
F) Double it – at least – and add some slop in your plan: no matter what anyone says, someone will come up with a need for more: more bandwidth, more storage, more processing power, more displays, more something. And they will convince somebody – this or that Admiral (or President or CEO) that you can’t survive without it. You need to be ready to do battle with the boss: (Boss, can we install it AFTER the merger? (Acquisition, release of the New Widget, etc.)). You also need to have a plan in place in case he says no. It is a wise thing to brief the boss early on the concept of a TCD because it will make him a harder sell when his old friend suggests you need an ‘upgrade.’ Nevertheless, you will lose some of these fights, so include that in your planning.
2) Take notes
This may sound a bit simplistic, but, I witnessed this too many times with others who didn’t do this: you need to have records of where you started and periodic snap-shots of where you have been. Keep track of your decisions and changes to the plan. People have tried this with no effort to track where they started and it has resulted in some un-pleasantries. There is an argument that the systems are too dynamic to engage in configuration management BUT—I submit that not only for simple issues of maintenance and training, but also for the vital issue of Continuity of Operations, there needs to be at least a baseline snapshot – periodically - on every major component of your IT infrastructure and current settings, etc. (This is particularly true of you office is someplace that has hurricanes, tornadoes, or earthquakes.)
So, have a master file with your:
A) Configuration
B) Your data – blinding flash of the obvious – but it needs to be said
C) Have a paper back up of key items – you’ll know what they are – have a master file, in a fire-proof, water proof cabinet.
3) ‘Better’ is not necessarily so
Remember your focus. What is the goal? The goal is important, the IT, just like the plan, is a tool to achieve the goal. Don’t confuse the two. Design for what you need, not for what is possible.
4) Step Back
At least once every month go look at the whole thing
Spend a couple of hours looking at the master schedule for the organization as a whole and your IT plan – does it all still make sense? Spend enough time one afternoon a month – with your key folks – to pull the plan apart and make sure this is all still a solid and workable approach. When possible schedule this for a Friday afternoon, buy a couple of cheese pizzas and a couple of six-packs, and walk through the plan. Get everyone relaxed and let them throw darts. If you can easily defend against the darts – good enough. If not, set it aside and move on, but go back and pull that issue apart later – either later in the session, or on Monday (and you’ll spend the weekend thinking about it).
Remember that your real goal is to lead and motivate your people: they will do the hard work and achieve those things you and your organization need.
Own the Margins
Winston Churchill once observed, as he watched the staffs planning for several different invasions, that ‘the sum of the margins is usually no.’ This was also true in our planning. Regular contact with all the various contractors and Navy Systems Command people always resulted in each of them padding all of their figures in order to make sure they each had enough time, space, personnel and material to adequately install and calibrate their systems. Unfortunately, if they were all added together, it would have taken three years to prepare the aircraft carrier for it’s deployment, and require three times as much space as we actually had, never mind the entire group of ships.
From this comes this one simple rule: Own the Margins. You need to develop enough of a feel for the details of the installation that you know what is possible, what is impossible, and what is simply ridiculous. And when you reach ‘ridiculous,’ you must simply put your foot down and tell how things will proceed. (Again, this is where the master plan is so useful, as it allows much greater clarity then the poor contractor who is trying to install his single piece of gear).
In Conclusion
In the end, you need to be clear on what you want to achieve, plan for it, communicate it, and train to it. You can’t do it all yourself, the people you are leading will do the real work, your job is to lead them, to make sure they have the tools to execute the plan, and to do what is necessary to get obstacles out of their way. If you do that, you will be amazed at what they achieve.
This article is the result of a suggestion from a friend who thought others might learn from some of the issues I have dealt with.
There was only one minor problem: none of the systems were installed.
The task at hand was to install all the information technology gear necessary to handle all the requirements of a modern, nuclear powered aircraft as well as all the ships accompanying her. What are those specific requirements?
First, the aircraft carrier is a small town of 5000, with (as this was still in the early days of networks) 1400 LAN drops (for reasons of both security and issues of RFI WANs are more problematic on a ship then on dry land, and they are also a problem when there are classified networks in some of the same rooms as the internet systems). This was the first aircraft carrier fitted with this many PCs, almost an order of magnitude more PCs than previous aircraft carriers. Similar LANs were being installed on most of our other ships, as well, also for the first time.
Second, the ship is an active airfield with roughly 70 to 80 aircraft operating from it. Additionally, those aircraft require their own classified networks to support planning, maintenance and operations. That too was part of the IT installation.
Third, the aircraft carrier was the flagship of a 14-ship group – known in the Navy as a battle group or strike group, and known during any specific operation as a task group.
The 14 ships included two Aegis cruisers and two Aegis destroyers, similar to the ship that recently shot down the NRO satellite. The Aegis ships were all receiving extensive software upgrades to both their radars and their weapons control systems, upgrades that were substantially more complex then any that had been attempted in a number of years. There were also three amphibious assault ships, which included 2500 Marines and another 25 aircraft.
Fourth, the ship/network needed to be able to handle unclassified information, secret information, top secret information, special intelligence information as well as information that could be shared with allied navies – requiring the ability to accurately handle information from multiple classifications through the same communications systems, while assuring both the correct movement of information between systems when allowed, but protecting that information when not allowed.
Fifth, the ship had to be able to function as both the command ship for a multi-carrier group – known as a ‘battle force’ – that might include as many as four aircraft carriers, 300 aircraft, as many as 60 ships and as many as 60,000 personnel, as well as its own Network Operations Center for the extended networks of the battle force – another ‘first ever.’ As it turned out, we did act as the command or flag ship, as well as network control, for a ‘task force’ that consisted of three aircraft carriers, 25 ships, two submarines, more than 200 aircraft and roughly 30,000 people.
All told, these installations, to include, hardware, software, necessary modifications to the ships to support the upgrades, plus training and maintenance for the total period amounted to something in the neighborhood of $100 million.
When we began this process the carrier was brand new. (Brand new is a bit difficult to define; an aircraft carrier takes more than 5 years to build and after it is ‘finished’ and commissioned, it will be another year and a half of drills before the major fitting out and training of the ship begins). At that start point the systems installed on the carrier are limited to those systems necessary to conduct basic flight operations around the ship – the flight control radars for landing aircraft, one air search radar, several basic navigation radars and the radios for navigation and for flight control.
All of the complex, advanced Satellite Communications systems, advanced air defense radars, and sophisticated data links had yet to be installed, the network internal to ship had not been installed, the network handling system to allow the ship to function as a network node had yet to be installed. Major ‘command spaces’ – the spaces in the ship for the major information processing and the spaces for various mission commanders – the admiral in command of the battle force, his intelligence center, the center for integration and information management for different warfare problems such as ‘Air Warfare,’ ‘Anti-Submarine Warfare’ and ‘Electronic Warfare’ – were little more than empty rooms when we took control of the ship. Several spaces literally were bare steel – they had not even been painted and there was nothing in the room. We had 18 months to prepare this ship, as well as manage the IT work on all the other ships in the battle group, to include the complete systems upgrade of two of the Aegis cruisers and all the installations on a brand new Aegis destroyer.
At the same time we faced the same problem others had faced: the systems are all installed by a wide range of contractors and sub-contractors. But, in fact, neither the Admiral nor the captains of any ships actually controlled the contracts. Often they were (and are) not even controlled by anyone within easy reach. Rather, they were controlled by large and remote organizations whose directors are in Washington and who had completely different sets of priorities then we did.
The Result: we did everything that we were required to do, our systems stayed up, and our operational availability figures were unprecedented. Our down times were – across the board – fractions of the normal down times. One example, our most complex system, was a large format SATCOM system used for moving images that, because of the environment (seawater, jet exhaust, extreme heat, complexity (and normal lack of adequate training)) routinely had UP rates of 20% - the best achieved rate prior to our effort was 28% availability over a 6 to 7 month period. We had 3 hours of un-planned non-availability during our entire 6+ month deployment.
How did we do it?
A Clear Goal
We had clearly identified missions and goals, performance and capability standards that we needed to reach or exceed. We needed to achieve those standards two months prior to deploying, and be able to sustain those standards for eight to nine months. It would begin in 16 months from the day I took over, with a month long exercise to test every ship, every person and every system in the group in all of their assigned roles. We knew what we needed to accomplish and we focused on that, placing readiness at that point as the very center of our effort. The key to achieving that focus was planning.
Planning
I began planning immediately. We started with a simple chart on the wall, which actually ended with the ships all returning to the United States 25 months later, was stretched across two walls of the office, and the missions and tasks we knew we would to carry out over that time period. We then worked backwards from there, identifying every possible milestone or hurdle that we would need to overcome to ensure that we were successful. These hurdles were overwhelmingly not IT or even loosely ‘information’ centric. They were simply events we knew (or anticipated) the Group would face over the next 25 months.
Several major iterations of this planning timeline were generated before we were even sure we had all the important items identified. Then we sat down and attempted to identify what systems needed to be in place and operational to support successfully addressing each of those events. Often, particularly in the beginning, this required a good deal of supposition (or simply guessing), but we soon became fairly adept at identifying not only what systems were needed when, but also, and often more importantly, we became adept at identifying which systems were either not needed or might be useful even if not completely installed.
Key to this was regular and frequent meetings with the captains of all the ships and their key officers as well as key personnel who would be involved with each installation. We learned what was in the realm of the possible and what was inflexible for other reasons.
This was significant because it gave us ‘room’ to start adjusting installation and calibration schedules. Onboard a ship (and actually in any building) there are always limits to how much can be done at the same time. Often, and for similar reasons, the companies that are installing pieces of gear will want access to the same places at the same time. By identifying not only what we really needed and when, but also what we really didn’t need yet (though it might at first seem like we would) we were able to control the pace of installations in a way that suited our needs, not the needs of someone who had originally scheduled the installations (usually with little other information).
One example: shipyard schedules are very tightly controlled because there are only a few shipyards available and, if it is the case of an aircraft carrier, very few that can actually fit an aircraft carrier. One of the requirements for the carrier was a large SATCOM antenna. For a number of reasons the antenna could not be placed on the carrier’s superstructure, the small ‘apartment-building’ like structure that sits on the flight deck. So, we needed to build a sponson, a 40 foot long, roughly 40 ton piece of steel that jutted out from the hull, that the SATCOM would sit on. The SATCOM would not be delivered for many months, but the available period in the shipyard would only be available for a short period of time. We identified that period early and adjusted the contractor to send one man down to install some basic gear and pull certain cables while the sponson was being installed, fully six months prior to install of the system. When it came time to install the system it was a significantly more straightforward process than it would have been if we had not first ‘prepared’ the sponson. The result was that the physical installation time was cut in half, and the system grooming time was tripled, as was the training period for our personnel. This was the kind of event that our calendar allowed us to identify early.
Once we had a clear sense of the ‘real’ schedule, that is, what we absolutely needed and when, I presented it to the Admiral (analogously, the Chairman – President – CEO) and received his endorsement.
I was fortunate in that several of my key people (and I) had extensive experience in long range and strategic planning. If you have never engaged in strategic planning, bring in someone for a week of training, it will pay big dividends. I also benefited from the fact that as the Intelligence Officer (Chief Information Officer) I was also a user of the systems I was installing and maintaining. This provided me with a valuable perspective that others before me had not enjoyed (though, at the time ‘enjoyed’ was not a word that was used a great deal).
Communication
Once the Admiral was comfortable, we had to inform everyone else. We began by informing the captains and key personnel on each ship exactly what the schedule was, and as important, we made sure they understood why. Even though they had participated in identifying these ‘reasons why’ a matter of several weeks earlier, this helped them to not only more clearly understand, it also made sure that they had visibility into what each other ship was doing and why. Everyone was informed, everyone was part of the same team, there were no ‘secrets’ and everyone knew that no one was getting a ‘deal’ as well as no one was getting the dirty end of the stick.
When that was done, we informed the information management folks on all the ships, the people who would actually operate and maintain the systems once they were installed. We made it clear to them that we would all be living and breathing based on this schedule. We used as much time as was needed to make sure that these people not only fully understood the schedule, they understood what was the basis for the schedule, the why, what exactly the ships and aircraft and Marines in the group would be doing, why certain systems would be needed, and why certain systems needed to be very precise and what that precision really meant to the user. This understanding on their part was perhaps as important as any other single thing in ensuring the systems remained operating and within proper standards.
Training
Our training program was far and away the most aggressive systems training effort anyone had ever seen. We used every means in the book – extra money, squeezing every dime we could out contractors, using all of our training money, taking advantage of people who simply ‘offered,’ to get more training for our people. Perhaps it is possible to be over-trained, but I‘ve never run across it. Whereas other carriers let their people get trained in other specialties, we focused every additional training hour and training dollar on the specific systems – hardware and software – that we would be working with.
While we were fair, we were very demanding in training and in maintenance. We made it clear from the beginning that we were setting ‘ridiculously high’ standards of availability and that we would train and exercise the people involved until they could achieve those standards. Once people understood the focus of our effort, and they understood the plan, and they understood the importance of the plan, they responded superbly to the training and the exercises.
My Deputy
One key factor in my success was one man: Pico Torres. Pico was a charming and brilliant technician, but his real skill was in leading people. While I was trying to think ‘big thoughts,’ pushing forward the plan and meeting with the captains, the admiral and various contractors, Pico was working with the ‘troops.’ His ability to communicate with the average 22 year old technician, to help him understand why something ‘needed’ to be done, and why it needed to be done right, the first time, his ability to orchestrate and lead training, to conduct extemporaneous drills to test their skills, his energy level in meeting with one young sailor at a time, for hours on end, to help the sailor learn a fine point about this or that piece of hardware or software was noting short of remarkable.
I made a point of making sure everyone knew that Pico had my absolute backing: I might be 500 miles away and out of reach of any phone. But, if Pico said it was so, I would back it up – and I ALWAYS did. If you wanted to get it reversed, only the Admiral could reverse it. And that never happened.
Finding someone as talented as Pico Torres is perhaps the one issue that isn’t readily ‘teachable.’ I was fortunate to have such an individual working with me. You will need to find the most talented person you have and then you must provide some grooming and support. Once you are comfortable with their level of skill, you must provide him or her with both the freedom and the top cover to carry out the detailed plans on a daily basis.
Communication – Part 2
Once the plan was in place and being executed, I devoted considerable effort to keeping the boss, the various ships’ captains and other senior officers in the group, and the more than 1000 technicians, contractors and workers who were installing or would maintain and operate these systems, informed as to our progress as well as informed of any changes, changes both in the specifics of the plan as well as changes in what we as an organization of 14 ships and 13,000 people were doing. This served to keep the meaning of the plan in focus.
This communication consisted of both regular updates at weekly departmental meetings as well as ‘leading by mingling’ on the part of both Pico and me, as well as several other people who worked directly for me. (These often took the part of me stopping by for coffee and chatting with folks at the coffee machine). These informal sessions also served to keep us informed of the detailed problems that they were having in executing discrete elements of the plan. Both were critical.
Lessons learned
Leadership
Taking charge and providing clear goals was the sine qua non of success. So, the most important lesson is simply:
1) Know why you are doing it.
Knowing why we wanted a piece of gear (hardware, software) – the desired capability – was at the center of our success. The reason, the capability or the mission was important, not the piece of gear. Once everyone understood the real goal, and then the role ‘X’ played in achieving the goal, we were over the largest single hurdle. Once you know the goal (the vision), don’t let anyone pull you off that. Our goal was successful combat operations. Most definitively the goal for us was NOT smooth IT operations; your company sells something: hardware, software, turbines, soybeans or penny candy – the IT supports it. Once people understand that “simple” fact it provides remarkable clarity, and sets the foundation for everything else.
2) Communicate
Make sure you communicate enough. First, do the people who work for you understand what it is that you are trying to do and why? Second, does your boss know? Third, do your peers – the other VPs or Directors or however they are titled? And, are you listening to them: your people, your boss, your peers? As a general rule, no one communicates enough. Ted Williams used to say ‘Practice, practice, practice, practice, practice. And when you think you’re done, practice some more.” You can put the word ‘communicate’ in the place of practice and the statement is equally accurate.
Complex organizations require more hands on leadership – so get out of your office: walk around, listen, get a cup of coffee, talk to your people, listen. Remember that you – in a very real sense – work for them: your job is to make sure they have all that they need to do their jobs, the training the gear, etc. So, visit each site, have a cup of coffee and listen.
Set high standards: superb results aren’t achieved with mediocre standards. When I took over as the head of combat systems one of the things I found from talking to other people who had occupied these jobs in the past was a sense of frustration and some fatalism – ‘its always like this – particularly with a new aircraft carrier.’ Simply put, don’t accept that. Make sure everyone understands that you simply aren’t going to accept that. Set the standards high and inform ‘the troops.’ (Your standards will be part of the planning process, but making it work is a leadership issue). People will amaze you with how much they can achieve – so be fair and friendly and keep communicating, but set HIGH standards.
One final thought on standards: make sure your IT people know not only whom they support, but also what those other departments are doing – specifically. If you (the IT department) succeed, in your terms, but it isn’t what the other departments really needed or it wasn’t in time, you haven’t succeeded. This is a team sport, and everyone needs to win.
3) Train your people
It’s almost impossible to image people who received too much training. Load up on the training – it will pay you back.
3A) Training includes exercises.
- Document your exercises – have someone who keeps detailed notes, it is essential to understand what went well and what did not
- Debrief and lessons learned – when the exercise is over, review the notes, make sense of the results and then share it with your people. This is not punishment, it’s learning – you need to treat it that way and you need to demonstrate that fact with your people. One key step in making that point clear is to make sure that any negative comments about your performance in the exercise gets as much ‘air time’ as anyone else’s.
- Make sure you include recommendations for improvement in the debrief – the exercise is to help improve not to criticize.
3B) Train the rest of the organization
Just because you trained your people to maintain and operate the IT gear, does the rest of the organization know how to best use it? Make sure there is a training plan for them, and make sure you coordinate the training schedule with their regular work schedule.
4) Find at least one good (great) deputy
You may not be as fortuitous as I was in having a truly superb leader like Pico Torres working with me, but there are lots of sharp folks out there, and with the right support from you (pushing and pulling, suggestions and help) you can make people rise to the occasion, reach their real potential, and become good leaders.
Once you have several capable deputies, delegate everything that isn’t critical, pushing it completely out of your purview – find someone you can trust and have him give you status reports once or twice per week, but otherwise let him run his own show.
5) Bring your boss in early when you need to. You will have internal roadblocks – another department head is being obdurate and is going to upset the entire timeline and affect not only his own but other timelines in the org., explain, negotiate, try to work with him, but wait so long that that his resistance overturns your timeline. Before that happens you need to both inform the other department head and the boss. It’s not about ego, it’s about reaching the overall organizational goal, and if your planning is right, then something needs to be done: so, negotiate, communicate, etc., but don’t fail to act.
Organization
1) Have a Plan
Once you know the goal, you need a plan to get there. There are many ways to build plans – pick one and build a plan. But always remember – the goal is important – the plan is simply a tool to get you to the goal. You aren’t installing the IT for the heck of it – what are we trying to do and when must we do? Simple flow charts, PERT charts, whatever process you want to use is fine: but have a process, one that is visible and known to all --- and post it. That being said, I found that there were key elements unique to successful IT plans:
A) Keep perspective - Don’t measure with a micrometer if you are cutting with a chain saw - (the sponson) – if you are engaged in heavy construction, don’t start pulling wire, calibrating systems, and loading software
B) Don’t forget the simple stuff: power, cooling, the building, water, people using the stuff and people maintaining the stuff (Can you access everything? Do you have enough toilets?)
C) Take nothing for granted (never assume) – your plan needs detail in IT, particularly when you have a bunch of contractors to sort through. You may not know all the details, but your deputies need to.
D) Have a TCD – a technology cut-off date - beyond which you will not add new or improved pieces of gear. The reason for this is simple: people have to figure out what they have, figure out how to use it, and figure out how to integrate it with other systems. If there is no TCD – or something like it – you set yourself up for some smart guy to come in and ‘sell you’ some really neat (and they are) piece of gear 10 days before the big thing (in our case steaming into a combat zone, but it could just as easily be a takeover or merger or a new product release and when everyone should be focused on that, they are instead struggling with a new piece of gear. So, know when to say ‘no.’
Many organizations used to have TCDs and got rid of them in the recent past because they feel they will lose too much if they don’t have the latest thing. If you can figure out how to manage that, fair enough. But, in any large organization, if you are trying to integrate information from different systems, you need to have some sort of deliberate process and that will result in a stair-step function. You can make the steps small and frequent, but they must be deliberate. The TCD may not even officially exist, but if there is any complexity to the infrastructure, you will need to have deliberation and control.
E) That being said, you also need an upgrade plan. Any large IT infrastructure improvement will last considerably longer then a single generation of any number of different IT systems. Accordingly, a good plan will both provide for substantial room for growth so that elements of your infrastructure will be adequate for many years, while other elements can be more rapidly replaced without causing disruption to the entire system. Key to this is a training plan that ensures the right people in the organization are receiving the right training so that the new box doesn’t arrive and your people aren’t ready to operate and maintain it.
F) Double it – at least – and add some slop in your plan: no matter what anyone says, someone will come up with a need for more: more bandwidth, more storage, more processing power, more displays, more something. And they will convince somebody – this or that Admiral (or President or CEO) that you can’t survive without it. You need to be ready to do battle with the boss: (Boss, can we install it AFTER the merger? (Acquisition, release of the New Widget, etc.)). You also need to have a plan in place in case he says no. It is a wise thing to brief the boss early on the concept of a TCD because it will make him a harder sell when his old friend suggests you need an ‘upgrade.’ Nevertheless, you will lose some of these fights, so include that in your planning.
2) Take notes
This may sound a bit simplistic, but, I witnessed this too many times with others who didn’t do this: you need to have records of where you started and periodic snap-shots of where you have been. Keep track of your decisions and changes to the plan. People have tried this with no effort to track where they started and it has resulted in some un-pleasantries. There is an argument that the systems are too dynamic to engage in configuration management BUT—I submit that not only for simple issues of maintenance and training, but also for the vital issue of Continuity of Operations, there needs to be at least a baseline snapshot – periodically - on every major component of your IT infrastructure and current settings, etc. (This is particularly true of you office is someplace that has hurricanes, tornadoes, or earthquakes.)
So, have a master file with your:
A) Configuration
B) Your data – blinding flash of the obvious – but it needs to be said
C) Have a paper back up of key items – you’ll know what they are – have a master file, in a fire-proof, water proof cabinet.
3) ‘Better’ is not necessarily so
Remember your focus. What is the goal? The goal is important, the IT, just like the plan, is a tool to achieve the goal. Don’t confuse the two. Design for what you need, not for what is possible.
4) Step Back
At least once every month go look at the whole thing
Spend a couple of hours looking at the master schedule for the organization as a whole and your IT plan – does it all still make sense? Spend enough time one afternoon a month – with your key folks – to pull the plan apart and make sure this is all still a solid and workable approach. When possible schedule this for a Friday afternoon, buy a couple of cheese pizzas and a couple of six-packs, and walk through the plan. Get everyone relaxed and let them throw darts. If you can easily defend against the darts – good enough. If not, set it aside and move on, but go back and pull that issue apart later – either later in the session, or on Monday (and you’ll spend the weekend thinking about it).
Remember that your real goal is to lead and motivate your people: they will do the hard work and achieve those things you and your organization need.
Own the Margins
Winston Churchill once observed, as he watched the staffs planning for several different invasions, that ‘the sum of the margins is usually no.’ This was also true in our planning. Regular contact with all the various contractors and Navy Systems Command people always resulted in each of them padding all of their figures in order to make sure they each had enough time, space, personnel and material to adequately install and calibrate their systems. Unfortunately, if they were all added together, it would have taken three years to prepare the aircraft carrier for it’s deployment, and require three times as much space as we actually had, never mind the entire group of ships.
From this comes this one simple rule: Own the Margins. You need to develop enough of a feel for the details of the installation that you know what is possible, what is impossible, and what is simply ridiculous. And when you reach ‘ridiculous,’ you must simply put your foot down and tell how things will proceed. (Again, this is where the master plan is so useful, as it allows much greater clarity then the poor contractor who is trying to install his single piece of gear).
In Conclusion
In the end, you need to be clear on what you want to achieve, plan for it, communicate it, and train to it. You can’t do it all yourself, the people you are leading will do the real work, your job is to lead them, to make sure they have the tools to execute the plan, and to do what is necessary to get obstacles out of their way. If you do that, you will be amazed at what they achieve.
This article is the result of a suggestion from a friend who thought others might learn from some of the issues I have dealt with.
posted by Pete O'Brien @ 2:42 PM
0 Comments
