Ken Dymond
Process Inc US
Abstract
Software process assessments (SPAs), in the style evolved by the Software Engineering Institute of Pittsburgh from 1987 to now, have been used world-wide to start process improvement in software organizations. Successful assessments depend, in the opinion of the author, on the statement quoted in the title above -- assessments are the voice of the working level in software companies. This article explains how assessment provides that voice and the context for which assessment was designed. The article will also describe some ways in which assessment accidents have been mistaken for essentials.
I will begin this article by explaining how the quote in the title came about. This statement was made by an engineer at a one-day briefing on assessment and process improvement I was giving with a colleague. The scene was a meeting room in a large company in a European city. The audience consisted of about 45 people from the company, mostly high-level software managers. The front rows were all occupied by managers. As it happened and as I found out, there were some engineers from the working level sitting in the back row.
My colleague and I from the US had spent the day explaining what software process assessment is all about, what happens during the assessment and why, and the effects assessment is intended to have on a software organization. I was becoming a little frustrated as the day wore on because I felt I was not communicating the message about assessments. The attitude of the managers, mostly everybody in the room, was that "We will do this assessment because the top manager [who was not present] ordered it done, and we will be successful at it. Never mind the buy-in and these other things you are talking about [and that I will outline for you in this paper]."
This attitude, not expressed in these very words but evident all the same, meant that my colleague and I were not successful in conveying to this organization what assessment was for. Finally, just before we finished about 5 PM, I invited the audience to state any concerns they had for doing an assessment. A person from the back row raised his hand and said "Maybe this time the voice of the engineer will be heard." Quite courageous, I thought, given that most of the other persons in the audience were this man's superiors and were of the official "party line" view of assessment that I had been hearing all day. That engineer's bold statement was my reward for a day of toil, for his words meant that the message of assessment was understood by at least one person in the room, and that person was a representative of the group assessment was meant to directly help.
The background of that engineer's statement was one that assessment teams often hear. Software practitioners -- engineers -- often describe the background this way: "We have had plenty of quality initiatives around. Management starts them all the time. But nothing ever changes. The agenda is always management's and nobody ever asks the people who do the work what they think should be changed." In the US one says, "Management hoists the quality flag up the flagpole, everybody salutes, and then we go back to business as usual."
Now I want to explain the message of assessment, why assessments, in my view, continue to be used successfully worldwide, as well as some of the misunderstandings that have arisen about assessment.
Assessment was designed for and should be practiced in a context of continual process improvement. The context can be pictured as in Figure 1.
The Process Improvement Cycle: IAEIL
(Initiate, Agree, Establish, Implement, Leverage)
Figure 1
Assessment is one of the best ways to perform the second step, "Agree on Process Issues". But no part of the process improvement cycle can operate unless it is acted upon by people in the organization. The work has to be done by and with people, and so we must add them to the cycle.
(included in the original copy in the ESI-ISCN'95 Proceedings)
Figure 2
The senior manager initiates the improvement effort and supports it with resources (note the dollar sign), including a company's most valuable asset, its people. Assessment is the next and vital step that generates agreement from management and work force on the process issues that are hampering their work the most. If that step is omitted or performed in an appraisal method that doesn't produce the assessment consensus, the consequences can be failure for the improvement effort to follow (if one does follow). Even if an action plan is written without a preceding assessment to focus the issues, it is likely to contain the agenda of management, not the workers. And if the action plan does not remove the most important bottlenecks of the workers, the changes merely add to the workload instead of rationalizing and streamlining it.
After the assessment, which produces the consensus on process problems, as many people as possible at all levels should participate in the "Establish" step. This wide participation continues the example of the collaborative activity demonstrated by the assessment in deciding what process problems the action plan will address. A process action team should be the coordinator among working groups focusing on process issues in projects, senior and middle management (which supports and oversees the action plan), and the rest of the organization (which expects to hear good news about improvement and to participate when their turn comes).
Figure 2 shows, in the "Implement" step a working group on a particular phase of a project. The working group, of perhaps 2-5 people from the project and the action team, are helping project workers to introduce an improved process step (the triangle replacing the smaller rectangle indicating a process step) in the project phase. Each working group is following a Deming cycle of Plan-Do-Check-Act to implement a process change, with the "Plan" step coming from the organization's overall action plan arising from the assessment. The result of Working Groups implementing the action plan should be an incremental improvement in the organization's capability to deliver software on time, on budget, and with increased quality.
It is easy to miss the point and the outcome of assessment by emphasizing one of its accidental elements as if it were the essence. (I once heard a consultant speak at a conference on assessment and indignantly accuse the SEI of concealing the fact that the SPA was nothing but a means to start an improvement effort.) The remaining sections of this paper try to distinguish the essential features of assessment -- those activities and the manner of conducting them -- that make assessment what it is, from the accidents -- those activities that may be present in any given assessment but need not be.
What is assessment? Software process assessment is a formal and collaborative protocol by which management in a software organization can ask the working level technical people about the process bottlenecks they face in their work.
Why must this simple Process Question be asked with a formal protocol? Why can't management just walk around and ask the "troops"? Why can't management just send out a survey and get the answers?
In most organizations, if you, the manager, walk around and ask the people who work for you about their process problems, they will at the least edit their answers according to what they think you want to hear. So you won't be hearing about their process problems but the ones you have been talking about or hinting at yourself. This distance between manager and worker is a fact of life in many hierarchical organizations. It is not a bad or good fact, merely a natural effect when the boss, who has power over you and can determine your job life, asks you about work performance
Plenty of managers have sent out surveys, usually delegated to
a subordinate or to the Human Resources staff or the "quality"
group. The usual result is enthusiastic answers to the first round
of the survey then disillusionment by the troops when nothing
much happens ("We run the quality flag up the flagpole....").
And how many surveys about process problems generate the commitment
to divert resources to solving the problems?
Another reason why it is not a simple matter to question subordinates about their work problems has to do with the culture of engineering. Engineers and software workers in general, tend to be a "can-do" bunch of people. They think that with enough resources, and if management will let them alone, they can do wondrous things. Usually they are right. They can do wondrous things, like meet an impossible delivery schedule, but at a high cost in overwork, disrupted home lives because of long hours, and burn out. Also, the high cost means that heroic efforts can't be repeated too often. But management keeps expecting heroic efforts and the technical staff keeps delivering them at the horrendous price.
Still another reason to use a formal protocol to ask software workers about their process problems is that all kinds of answers are given, and some cooperative method is needed to organize the responses. Conducting the assessment in a collaborative way to obtain the views of project leaders and technical workers is one way of organizing the process issues. Another organizing factor of assessments is the Capability Maturity Model (CMM) for software, the generic process standard for systematically improving the software process.
Assessment has been characterized as a social ritual. When we try to describe an assessment solely in terms of what happens, we are apt to miss its social nature. In this section I will describe briefly the assessment activities, then in the next two sections point out what is essential about the social ritual involved.
Figure 3 is a version of the basic diagram the SEI and assessment teams had used from about 1987 to 1994 to describe assessment practice. The SPA generally takes 5 days and consists of the activities in the diagram. (I exclude the separate phases of preparation, training, and any ensuing final report.) These activities can be grouped into major phases: formal opening (opening meeting); gathering information (questionnaire analysis, project leader interviews, and functional area discussions); preparing and testing findings (this happens in two sub-phases: (1) preliminary findings and feedback to project leaders and (2) final findings draft and develop briefing through the dry runs); presentation of findings (formal conclusion); and preparation for producing a final report and action plan (draft recommendations, assessment debrief, and next steps).
A team of 5-9 people, trained for the specific assessment, conducts the activities of the 5 days. Most assessment team members are from the organization assessed, but on a first assessment one or two outsiders who are authorized by the SEI as lead assessors may be on the team as guides.
Activities in the Typical 5-day SEI-style Software Process Assessment (SPA)
(Interactions between assessment team and interviewees are outlined with solid borders)
Figure 3
The activities diagram doesn't convey the meaning or experience of assessment. We say in training courses that hearing about assessment is like going to lectures on swimming. The student really doesn't grasp the essence of that activity until he or she enters the water. Assessment is learned by doing.
The essential assessment outcomes are the agreement on the most important process bottlenecks and the commitment to improve them. In other words, assessment produces, almost always, buy-in throughout the software organization for a team effort at process improvement. On whose part does this agreement, commitment, and buy-in occur? On the part of senior management and the software work force. Thus, the result of using this formal technique for management to ask, and the work force to answer, The Process Question, produces much more than a list of findings at the final presentation.
If, as I said above, assessment is a formal technique for management to ask workers about their process problems, one of the crucial activities is the opening meeting. The senior manager must be present and must state publicly the reasons why the assessment is being undertaken and why he or she is asking for the support and enthusiastic participation of everyone. Likewise, the senior manager must attend the management findings presentation on Friday. This formal closing fulfills a commitment the senior manager makes in the opening meeting to hear the process issues of the work force -- that is, to listen to the answers generated by this formal technique for asking The Process Question.
Both the formal opening and closing meetings are "big tent" activities, where everyone involved in software -- the whole site, if possible -- are gathered to witness the commitment of the organization including senior management to do something about the process issues the work force has highlighted in the assessment. The opening and closing meetings are the formal occasion for the senior manager to join the improvement team.
The work force drives the assessment activities and thereby join the team effort during the phases of information-gathering and of preparing and testing findings.
In the project leader discussions, the people responsible for getting software produced on time have the chance to describe their process problems to the assessment team. These discussions are highly structured: one project leader at a time (from a total of 4 to 5), responds to 20 or so open-ended questions about software engineering practices. These questions are based on the process areas in the Capability Maturity Model.
In the Functional Area Representative discussions, the people who actually do the work of software -- designers, coders, testers, quality assurance staff, configuration control staff, documentation specialists -- have the chance to air their process problems to the team. These "FAR" discussions, as they have been called, are for the most part without structure except for what the FARs bring to it. The assessment team listens for about 1 and 1/2 hours and intervenes only to equalize the amount of speaking time between FARs who are more talkative and those who are shy.
The phase of preparing draft findings is done by the assessment team alone, but the practitioners and project leaders -- the interviewees -- test the findings for the team. There are separate feedback sessions for project leaders -- one project leader at a time with the team -- of "preliminary findings." The preliminary findings are rough statements of the process issues the team has summarized from the individual interviews with 4 or 5 project leaders and from the 4 (usually) group discussions with about 8 practitioners each, amounting to a total of about 35 people -- for most sites a fair cross-section of viewpoints on the software process.
The team next drafts the findings briefing, no longer preliminary but in the format to be delivered at the Management Findings Presentation. The team takes into account the feedback of the project leaders on the preliminary findings in drafting the briefing. The team also now applies the Capability Maturity Model (CMM) as a context for the findings. Thus the process issues that concern the interviewees the most are expressed in terms of the CMM whenever possible. This is because process practices at a certain maturity level must be in regular use as a foundation for installing higher level practices, according to the CMM. In any case, the interviewees' process issues, whether expressed in CMM terms or not, become assessment findings -- the voice of the engineer must be heard. Assessment findings are thus also a means of interpreting or customizing the generic CMM to an organization's particular culture, business market, and application domain.
The feedback sessions with project leaders are the assessment team's first test of whether the process issues gathered from the interviewees are the right ones. The team presents the draft briefing 3 times in rehearsals -- called "dry runs" in the American idiom -- in advance of the management briefing. The first dry run is by the assessment team leader to the rest of the team (merely a practice session for the leader to become comfortable with the rhythm and wording of the briefing). There are two more rehearsals for the interviewees. One rehearsal is for all the project leaders together; the other dry run is for all the FARs together.
At both of the latter rehearsals, the team leader presents the draft briefing while the rest of the team writes down comments, reactions, and suggestions from the audience of project leaders or FARs. These rehearsals allow the interviewees to see whether the team has captured their issues and to offer corrections where it has not.
After collecting the comments from the dry run audiences, the team revises the content of the findings as indicated by suggestions from the interviewees and prepares for the management presentation the next day.
To produce the agreement on process issues and the buy-in to do something about them throughout the whole software organization, the assessment must include certain activities which must be conducted in a collaborative way. I will point out from my experience what elements are essential and what makes them collaborative -- a team effort. I will describe in this section the social ritual in these activities that make the assessment different from any other way of appraising an organization's software process.
First of all, the opening and closing meetings are crucial because they bring together, in an unusual and startling format, as many people as possible who are involved in the organization's software process. Clearly something big is happening: a large part -- perhaps all -- of the work force is present. The top manager is there, usually with subordinate levels of management who do not wish to miss what the boss might say. At the opening meeting the team leader explains briefly the assessment process. Many workers will be skeptical ("We run the quality flag up the flagpole...."). To allay their skepticism, the senior manager stands up and says publicly that he or she supports the assessment as well as the ensuing improvement effort. And -- very importantly -- the senior manager asks the interviewees to be candid with the team and promises that the findings and process issues will not be attributed to any person in the organization. No one will be blamed for freely expressing their view of process problems. The senior manager's presence and spoken words start the thinking among the work force that perhaps this time the quality effort will be different.
Next, the face-to-face interviews with the team take place. The interviews are evidence to the people involved that their opinions are being sought and that those opinions, in a non-attributive fashion, will be made known to management. The importance of their opinions is demonstrated by the team writing down the interviews' statements. The promise of non-attribution -- confidentiality of the source of any information the team obtains from interviewees -- is a promise at this point in the assessment, not a demonstrated fact. But there is evidence that the team will keep its promise.
An effort has been made so that there are no reporting relationships between team members and interviewees or among the interviewees. This means there are no managers of any interviewees on the team. And the practitioners have been selected so they are all peers in terms of the organization's hierarchy. Sometimes it happens that a manager is on the team, usually because a small organization just does not have enough people to fill all the team and interviewee roles. When this occurs, the manager-team member steps out of the room before an interview starts. The team leader then asks the interviewees in that group if any one of them would feel uncomfortable or be less candid if the manager-team member were present during the discussion. (Often the discussion may concern an area managed by the absent team member.) If even one person does object, the manager-team member is excluded from that interview. The rest of the team may not share their notes from that interview with the manager-team member (unless the notes are "scrubbed" so that no source of information is revealed). This absolute concern by the team for confidentiality is a sine qua non of the assessment method and evidence to the interviewees that they will not be penalized for giving their true opinions about process problems in their work -- even if the processes causing their problems are favorite ones of their boss!
In the FAR discussions, assessment teams often see a transformation among the interviewees. FAR discussions tend to be lively. Imagine a group of about 8 engineers or technical people, fond of their work and full of suggestions about how to do their work better. They are given free rein for an hour and a half to express their opinions about work problems and how to solve them. None of their managers are present. There is a team of 5 to 9 people who are going to listen to them carefully and take down their opinions. In most organizations nothing like this has ever happened before.
Because no managers are present and because the assessment team has no agenda except a minimal one for this session, the interviewees conduct the discussion. It is entirely theirs. Usually someone starts by talking about a particular problem they are having, say, with not enough time for testing. Someone else may state a similar problem. For a while, the discussion may follow this thread. Then someone may say, "We had that same problem on our project and this is what we did about it. ...." Another person may say: "I didn't know about that. Can we talk about that later? I'd like to find out more about your solution."
This last exchange marks the transition I just referred to: from airing problems to sharing solutions. The most subtle part of the transformation is from skepticism about the assessment to generating the buy-in for a team effort at changing the work process. In many companies the moment marks the first time people from different projects are working together to improve the organization's capability. The transition is an exhilarating one for members of the assessment team. Experienced assessors watch for it to happen during the FAR discussions and consider it part of the "magic" of assessment.
These free-wheeling FAR discussions have been carefully conducted to produce that "magic" -- in other words, the magic is part of the method. The FARs have been selected to include opinion leaders among the work force, people who are respected by their peers and sought out for advice. In case the discussion falters, the assessment team is trained to prompt gently. Just before the end of the session, the team leader asks the "Big Question" which all the interviewees answer one-by-one: "If you could change one thing about your work [and team leaders traditionally add: "other than your boss or your boss's salary"], what would you change?" The team leader then asks each practitioner about strengths of the organization, perhaps with the words: "If everything about the way you work were going to change tomorrow but you could keep one thing the same, what would that be?"
In this way, the team gets a sampling of opinion on the most crucial process problems as well as on process strengths that other parts of the organization might adopt in the follow-on improvement effort. Process strengths and problems, if corroborated in the dry runs, become ingredients in the findings briefing. This is another way that assessment tries to make sure that the voice of the engineer is heard.
The rehearsals of the draft findings are conducted to make sure the team is conveying to management the voice of the whole software organization. Project leaders, interviewed before singly but now in a group, have a chance to approve process findings and to suggest changes before upper management sees them. Neither their managers nor their subordinates -- many of whom were among the FARs -- are present in the dry run for project leaders. Similarly in the findings dry run for the FARs, none of their managers are present.
In these two dry runs, the team usually discovers that the assessment method has worked because practitioners typically suggest few changes to wording, but the changes they do suggest are important ones for nuance and context. Though rehearsing the words for the team to use the next day is vital, there is another essential outcome of the dry runs that is just as important: the practitioners come to "own" the findings. By joining in the dry runs after giving of their time and their opinions in the previous face-to-face meetings of the week, the interviewees now become virtual members of the assessment team. The dry runs are the next-to-the-last step in forging a living example of how the whole software organization can become a process improvement team.
Consider the "social psychology" of the assessed organization at this point, the evening before the culminating presentation to management. On this Thursday night, all the interviewees know substantially what the briefing will contain, except for last-minute changes the team might make as a result of suggestions during the dry runs. The interviewees, as virtual team members, are asked not to discuss the briefing with anyone else until after the final briefing the next day. The most important person who does not yet know the findings is the senior manager. That person has deliberately put himself or herself in this psychologically exposed position in order to give room to the organization to answer the question about process problems. To get that answer freely and without "adulteration" by management's views, the senior manager has sponsored the assessment and has thereby become a member of the improvement team. The senior manager also thereby expresses his or her trust in the validity of the work force's viewpoints. And the workers -- project leaders and FARs - understand by these actions that management is empowering them to improve their work life, productivity, and product quality by going through the assessment.
Finally, on assessment day 5, the last essential piece occurs at the final findings presentation. Here the answer to management's question -- "What process problems are you facing in your work?" -- is presented by the team leader. This short briefing, about 45 minutes, is the high point of the assessment. And the psychological and social dynamics have been set in place by the assessment method.
Of all the people in the room, the 35 or so interviewees and the 5 to 9 assessment team members already know the content of the findings. The senior manager doesn't know what the assessment results are, but many of the people who work under his or her direction have helped to craft the findings. Other members of the organization have come to hear the findings; they have had to make allowance in their work schedule all week for their colleagues who were being interviewed or going through a dry run and are usually curious by this time to hear the result. At least some part of everyone's attention is on the senior manager to judge his or her reaction while the team leader presents the findings, about 7 in number, one-by-one.
And what is the result of the intense activities of assessment week? What is the outcome of the assessment and why was it worth it?
The result the team expects -- and almost always gets -- is for heads in the audience to nod agreement with the findings. The nods are signs that the team's findings are the ones the audience wanted to convey to management and to the rest of the organization. The findings are the main problems the work force faces every day, has to solve with process "patches", and the ones that hamper their creativity and productivity, lower morale and lessen the competitive software capability of the organization.
An even better sign -- and this happens occasionally -- is for someone in the audience, usually a manager who has not been interviewed, to challenge one of the findings. If you are a first time assessment team leader, you may be a little shocked by this challenge and apprehensive about responding. What do you do? You wait a moment before answering. The chances are likely that someone in the audience -- an interviewee perhaps -- will respond for you and say "Yes, this finding is true!" (Remember the engineer's statement quoted in the title of this article.) And if not? You should recall for the audience what you and the team have done in the past week: you have followed the assessment method, designed to produce consensus on process issues; you have rehearsed these findings with knowledgeable people who gave your team this information in the first place; you know that these findings represent the working level issues. At this moment no one in the organization has a better understanding of the process issues than the team.
Finally the briefing comes to an end. The team has prompted the
senior manager beforehand to stand up and say a few words at this
point. Usually managers do not need this prompting because they
recognize full well the improvement impetus that can come from
the unity of opinion in the room. The typical statement of a
senior manager at this point goes something like this: "The
findings are no surprise, people have known about them a long
time. [No wonder, since there have probably been undercurrents
about these problems going on for a long time, but this is the
first time that the whole organization can recognize them as problems.]
The next step is for us to capitalize on the assessment, form
a process improvement team, and build an action plan together.
With the help and input of everyone, we'll write that plan
and implement it and change those processes so we can make a difference
in our capability to deliver software better than our competitors
[or for the business goal appropriate to the organization's
market]. Finally, I commit to implementing the action plan and
expect and want everyone to participate appropriately."
This short findings briefing and the reaction of the audience
and the senior manager may seem a small outcome for a large investment.
What has been produced to justify the investment? Consider what
the organization has achieved:
1) Agreement on the main process issues in the whole organization produced in a systematic way.
2) Buy-in for starting the improvement effort:
- from the work force who were asked their issues by management in a compelling way, face-to face by their peers, not in a mere survey;
- from the senior manager who demonstrated his or her commitment by
-- funding the assessment;
-- participating at the opening and closing meetings;
-- publicly promising support for an improvement program.
3) A worked example -- the assessment -- of how the whole organization can collaborate in an improvement effort.
For my part, as assessment team leader, I appreciate the courage and human understanding of the senior manager who undertakes to put himself or herself in the psychologically vulnerable position resulting from the assessment -- even though momentary. Senior managers are not supposed to show any weakness. But most senior managers instinctively understand the opportunity that assessment, in the style I have outlined above, gives them to focus the whole organization on process improvement and to form a team composed of management and technical workers to carry out improvement.
Assessment is a radically different event than most organizations are used to and marks a cultural change. This culture change is very much a part of the method, because genuine process improvement is a definite break with past ways of operating for most software organizations who have been concentrating --in response to market pressure and rightly so -- on delivering the product to the customer.
Of course, it is possible to just go through the motions of an assessment and not really produce the three outcomes above. Sometimes this happens, usually because there is not a shared understanding between management and work force as to the purpose of the assessment, or because one of the accidents discussed in the next section is considered as the focus of the assessment. But much more often the assessment "magic" happens at the final briefing.
It is important to look at the non-essential elements in assessment activities, especially at this time when other "appraisal" methods are being developed. Many organizations are developing or practicing various appraisal methods: the SEI itself in the CMM-Based Appraisal for Internal Process Improvement or CBA-IPI (initially perhaps with not enough public input from experienced team leaders, judging by remarks at the first meeting of the CBA-IPI Users' Group); ISO with SPICE; ISO with the revised 9001; and governments with their various kinds of evaluations of suppliers.
Another reason to examine what is accidental as opposed to essential in assessments is that there continues to be discussions in the literature and at conferences about this or that failing of the assessment method. Many of these discussions miss the point by focusing on an accident of the assessment method. And many of these discussions have sparked other appraisals, which while perhaps effective for their own purposes are less likely to produce the assessment "magic". I will discuss three viewpoints that in my opinion mistake accidents for essentials and try to describe why these viewpoints stray from the essential.
3.5.1. Assessment is the maturity questionnaire.
In the early days of the SEI assessment method, from 1987 to 1992,
many people thought that the most public aspect of assessment,
the questionnaire on process maturity, was the entire method.
This view mistook the part for the whole. This view is understandable
since those of us conducting assessments at the time hardly paused
to describe adequately the essence of what we were experiencing.
Also, it took assessment practitioners time to understand the
nature of the effects assessment produced and to sort out essence
from the accidents.
For most of those years, the locus classicus of both assessment
and evaluation (the software capability evaluation or SCE, a US
Department of Defense software process audit) was the technical
report SEI/CMU-87-TR-23 "A Method for Assessing the Software
Engineering Capability of Contractors" by Watts Humphrey
and William Sweet. This short report described well the purpose
of assessment to help contractors prepare process improvement
plans to meet DoD requirements for software capability of contractors.
A large part of the report contained a list of 101 questions
used by both assessment and evaluation teams, that is, in both
collaborative self-appraisals and in adversarial, external audits.
The fact that the U.S. government, a very large customer market,
was going to use SCEs based on the questionnaire to evaluate software
suppliers meant that US industry focused on the questionnaire
as the essence of both assessment and evaluation.
In assessment practice, the answers to the questions were used by the assessment team to get an initial estimate of the software maturity level of the site. This estimate helped the team to draft the 20 or so open-ended questions to ask of each project leader in the brief time (about an hour) of an interview. Depending on the maturity level of the questionnaire responses from 4 or 5 project leaders, the team would know whether to concentrate on process maturity issues from level 2, 3, 4 or 5 in interviews with project leaders. Thus the questionnaire became a tool to manage the time and content of the project leader interviews.
After the project leader interviews and FAR discussions were completed, the team also reviewed the initial questionnaire responses to see if the estimate of maturity level was confirmed by the information-gathering done face-to-face. The information from live discussions has always been considered by most experienced assessors as more reliable than responses to the questionnaire. The questionnaire is like a ladder the team uses to climb to a certain plateau of information and then discards as it continues the assessment. Those of us who trained assessment teams at the SEI made this point during the training classes, but it has never been emphasized to my knowledge in the published literature. Other assessment methods that rely heavily on a questionnaire are not likely to achieve the buy-in from practitioners that asking them face-to-face in a collaborative way produces. Assessments can and have been done without a questionnaire, but an appraisal without collaborative interviews is not an assessment
3.5.2. The main purpose of assessment is to gather information.
Audits are very good at finding out detailed information on software practices at a site. Audits are performed by outsiders who examine documents and who may interview practitioners. Because audits are performed by outsiders to ensure objectivity, audit teams almost never include practitioners of the process under view. Assessment teams must include peers of the interviewees (provided the free flow of opinions is not hampered by a reporting relationship between a team member and an interviewee, as mentioned above in Section 3.4).
There are three reasons why assessment teams include practitioners of the site's software process: first, assessment is a living example that the site people can perform a process improvement activity in the midst of the schedule pressures inherent in the software industry; second, peers feel more comfortable talking to peers about their process problems rather than to outsiders or to managers; and third, some of the team members will more than likely serve on a process action team (in the "Establish" step of the improvement cycle), and the assessment interviews introduce them to a cross-section of the site's software work, a viewpoint which no one at the site may have had before. Thus, assessments do gather information but for specific future improvement actions at the site.
3.5.3. Assessment results should be quantitative.
It is likely that quantitative process findings are valuable for a site that has already gone through the improvement cycle once and has a basis of comparison. But in the usual case of an organization starting its first improvement effort, there are few baseline process metrics collected and thus little or no data to report. It is also apparent that a large database of quantitative process information is needed to make comparisons of software capability across companies and internationally. Questionnaire responses and the maturity levels have been collected by the SEI for studies of trends in assessment results. While comparison data helps to get a macro view of software capabilities on an industry, on a national or international scale, the purpose of assessment is to start an improvement effort on the micro scale of one site. For that purpose, industry comparisons may be interesting but are largely irrelevant to the systematic actions an organization may take for improving its software capability.
It is sometimes argued that managers need quantitative information about the return on investment to process improvement before they will undertake something like an assessment. My experience has been that in some cases such data may help but even data will not convince a reluctant manager ("Yes, those numbers are impressive, but our organization is different.") There seems to be no "silver bullet" or magic remedy that removes a senior manager's skepticism. Some managers take a long time to finally agree after repeated approaches by their subordinates or by hearing the idea at conferences. Others seem to grasp quickly the value of process improvement and see assessment as its starting point and as an opportunity to establish a relationship of regular two-way communication with the work force. Of course, managers have the ultimate responsibility for their organizations and with their knowledge of current business conditions may rightly judge that the added stress assessment brings is inopportune at the time.
My last argument on this point is that assessment is less about science, which broad data gathering helps, than about an organization's internal transfer of technology. Process improvement is different from scientific research. If you wait to start a process improvement effort until all the evidence of its value is in -- as seems to be urged by the research view of software engineering -- your company may have lost advantage to competitors who have the head start that assessment provides.
My intention in this article was to show the nature of assessment: that it is a method for a senior manager to ask his or her organization for a candid view of their process problems and to forge an organization-wide, team impetus for improving that process. This dual purpose captures the double-, even triple-, acting nature of the collaborative style of the SEI assessment. Though apparently only one activity may be going on at a time, there is a great deal of parallel processing in assessment. For example, the FAR discussions not only gather information about process problems facing practitioners but also generate some solutions from colleagues in the discussion and start the idea that improvement is possible. The FAR discussions also embody an example of projects cooperating across organization boundaries to reuse their best process and technology practices.
Assessment SEI-style, is a powerful means for senior managers
to turn the attention of a whole organization away from daily
routine for a time and onto long-term process improvement. For
most organizations who have not started a systematic improvement
effort, an assessment is a routine-breaking way of beginning.
The unique nature of assessment and the results it produces should
be understood by managers and practitioners contemplating its
use and by national and international committees developing alternate
appraisal methods.
ISCN Information Request
I.S.C.N. International Software Consulting Network
Tel: +353 1 286 1583, Fax: +353 1 286 5078
email: office@iscn.ie