Shawn Presson's Response to Christopher Koch's Article in CIO

I asked Shawn Presson, of ITS Services, what he thought of Koch's article in CIO and he kindly sent me this reponse:

Mr. Koch’s article quotes Jay Douglass as saying "You can be a Level 5 organization that produces…garbage." My first response is to question how an organization that produces garbage can sustain true Level 5 operations, or even attain them in the first place. It’s expensive, it is hard, and I marvel at organizations that don’t eventually want to know if the effort is supporting the bottom line, not just the top line. But then, we’ve all heard, "garbage in, garbage out." A flat requirement to report a Maturity Level without understanding is a garbage requirement that has resulted in garbage results. Unfortunately, the credibility of a very useful model is being compromised as a result.

There are many war stories that highlight this state of affairs. One of my colleagues was tasked to perform an appraisal on a company for Maturity Level 4. As part of the normal upfront work, he asked what indicators made this company feel ready for a Level 4 appraisal. The response: "Because we’re profitable!" Further conversation made it clear that the company representative did not even understand Maturity Level 2 principles. In reality, there are many Level 1 shops that are obscenely profitable. The problem is that the capability of these shops can come crashing down as soon as a few key people take another job or fall ill. Mere profitability is not an indicator of maturity.

Another story: at a process improvement conference, I met a man who worked in an offshore shop that had been appraised at Maturity Level 4. This organization had well over 300 projects, and each project had something called "tailoring guidelines." When I asked how thick these guidelines were, this man indicated a size reminiscent of the Manhattan White Pages. I asked if those "guidelines" were the result of adapting the organization’s process definitions to the project. He replied that no, those adopted process descriptions were yet another set of documents. I then asked whether the guidelines were copied from the organization’s set of rules for adapting the standard processes. He said no, they were written uniquely for each project. I then asked how much variability this introduced from one project to the next, and how could processes be quantifiably managed across the organization. I saw the realization flicker in his eyes: they couldn’t be, and the organization was not truly operating at Level 4. I didn’t have the heart to tell him that the organization probably was not Level 3, nor did I ask who the lead appraiser was. Understand that his managers weren’t necessarily "bad" people, they simply were being forced to do something they didn’t understand, and someone subsequently had told them they had succeeded.

So what is a CIO to do? Aside from Mr. Koch’s due-diligence questions (more on those later), first quit paying so much attention to organizations touting their maturity level. Invoke Presson’s (immodestly named) Inverse Font Size Law, which states that the probability of an organization having a certain maturity level is inversely proportional to the size of the banner hung on the building. Realize that "the level number and a dime will get you a cup of coffee."

Secondly, demand to see performance indicators. I enjoy working with organizations that say things like, "Here are our performance metrics. What? Our CMM maturity? Oh, yeah, we’re also Level x against the CMMI." A mature shop should be full of people bragging about their performance. They may get somewhat irritated when you keep harping on the number, because they know it is a single data point.

Thirdly, learn the model. Learn what it means. Learn to spot the ridiculous contradictions that companies try to get away with - and often do. For example, Mr. Koch’s article says, "If the company does not have an excellent training program for all its project managers and developers…the assessment means little." That is an understatement. A reasonable training program is a Maturity Level 3 requirement, and if this is not in place then Level 3 and any higher level is not possible. In other words, the assessment Mr. Koch refers to probably not only "means little," it probably is a deliberate sham. A sponsor should be able to spot that and many other indicators.

Fourthly, do ask questions, do perform the due diligence. Mr. Koch wrote, "Only if CIOs ask tough questions will they be able to distinguish between the companies that are exaggerating their CMM claims and those that are focused on real improvement." I’d like to expand on that statement. Only when CIOs use the CMMs to suggest acquisition risk, and sponsor risk-focused appraisals, will they determine real improvement within their suppliers. Only when CIOs learn the tradeoffs and benefits of the maturity levels, and demand maturity only within the context of the need will they promote real performance among their suppliers. In short, until CIOs mature their acquisition behavior, the models will fail to deliver their potential benefits.

Question #3 on Mr. Koch’s list was, "How long ago was this done?" Ask not only how long ago the appraisal was done, but also find out the rate of growth and turnover in that company. Organizational maturity can be a fragile thing due to growth, mergers, reorganizations, and other upheavals. An appraisal performed six weeks ago could have weakened value under certain circumstances, much less one two years ago.

Question #9 gives the example of a financial services organization wanting to see that at least one of the appraised projects dealt with financial processes. If CIOs take control of this appraisal, they could demand that the scope of an appraisal only include project involved in financial processes. The scope is a negotiable thing, and acquirers should understand how defining the "organization" impacts the appraisal outcomes.

Question #7 asks, "Was the appraiser from inside or outside the organization?" This is a critical question. Some corollaries would be, "Who paid the appraiser? Were the other team members also from outside the organization?" Enough said.

#11 on Mr. Koch’s list is, "How many project managers who were assessed at CMMI Level 5 will be on your project team?" Theoretically, anyone working within the organizational scope at Level 5 should be capable of leading a project about as well as anyone else. Tom DeMarco and Timothy Lister’s Code War research showed that various aspects of the organization were critical enablers or hindrances to developer performance, and that ranges of capability between top and bottom performers in companies were not that variable. Mean performance would vary dramatically from one company to the next, however, due to the organizational characteristics. This doesn't mean that people don't make a difference. If a company runs a Level 5 organization as a subset of an enterprise, then moving people into that organization can represent critical risk if this is not carefully managed (see the next question.)

Question #12 asks "How does the company train new people to be CMM Level 5?" This question touches on orientation, training, mentoring, and other methods used to ensure personnel know how to do their job. As mentioned before, there is an entire process area at Maturity Level 3 focused on this, and all the process areas from Level 2 through Level 5 share a requirement that personnel be trained in performing the functions described within each respective process area. This question alone could merit a focused assessment.

The bottom line is that CIOs and other acquisition entities suffer from a situation of their own collective creation. At risk are the credibility of CIOs basing acquisitions upon CMM ratings, and the credibility of the CMM models themselves. As Mr. Koch writes, there is indeed no substitute for due diligence. CIOs must understand what the CMMs are for, know their contents and understand the history of the legacy appraisal methods. They should use current models and appraisal methods to mitigate risks, not to generate a falsely comforting, single-number indicator.