ami: a Taylorable Framework for Software Process Improvement
C. Debou, N. Fuchs, M. Haux,
Alcatel Austria AG
Scheydgasse 41
A-1210 Vienna
At the first ISCN seminar, ami (application of metrics in industry) was introduced as a new paradigm for software process improvement. It was one of the first attempts to reengineer successful technologies e.g. the G/Q/M [BaR88] into a general framework. Measurement was stressed as being an essential component of software process improvement for understanding, monitoring the changes and evaluating the return on investment.
This paper describes how major software process improvement approaches like CMM, Bootstrap or even ISO 9001 fit into the ami framework. The objective is to demonstrate on the one hand the ability of ami activities to be taylored to those specific methods and on the other hand the necessity of having an overall strategy (cycle) for software process improvement e.g. ami. Furthermore, newly defined software process improvement cycles (SEI IDEAL, SPICE) are described and also compared to ami.
When the software process issue was first raised, the emphasis had been on how to assess your process [HuS87]. Then W. Humphrey wrote his famous book "Managing The Software Process" [Hum89] which served as basis for the development of the Capability Maturity Model [CMM93a, CMM93b]. The CMM and other derived methods (Bootstrap, Trillium) are a real step forward for performing software process improvement, describing software engineering best practices in an evolutionary way but are still not sufficient. The overall vision (all necessary detailed steps towards an appropriate software process improvement programme) was missing. We now finally reach that phase ("re-engineering phase”). In software re-engineering, all components/modules are studied to derive the design. For SPI, components like assessment procedure, CMM, ... were existing but not integrated within an improvement cycle. ami (application of metrics in Industry) was one of the first attempts to combine together successful methods in one improvement cycle.
The ami approach has been extensively promoted during the past years in major conferences and journals [DLP92, DPF92, DFS93, RoW94]. At the first ISCN conference who took place in Dublin in May 1994, an in depth presentation of ami as a framework for software process improvement [Deb94] was performed and illustrated by case studies from the industry. In October 1994, during the first ami user group in Basel [AMI94], one of the main concerns was how ami relates to other methods like SEI CMM, Bootstrap [MHK94], SPICE, ISO 9001. Is ami working totally separately from major initiatives? Users always worry about the compatibility and adaptability aspects of the methodology.
The purpose of this paper is twofold:
After a brief introduction of ami as an improvement framework, the previously listed methods will be mapped into the ami cycle. The same framework will be applied for each method: A brief introduction and a description of the 4 generic ami activities (assess, analyze, metricate, improve) instantiated with one of the methods.
ami (application of metrics in industry) was a two-year project which started in December 1990 under sponsorship of DG XIII of the Commission of the European Communities through the ESPRIT programme promoting the use of measurement in software development. The goal of the project was to develop a practical approach and to validate it on a variety of projects all over Europe [PFS92, DLS93]. The approach is described in the ami handbook [AMI92]. The ami paradigm (assess, analyze, metricate, improve) is similar to the Shewart cycle (plan, do, check, act) for process improvement. ami has taken this cycle, based on common sense principles, and developed it for software measurement mainly in the context of software process improvement. The ami method is an stepwise, iterative, incremental, goal-oriented procedure coupling together a model-based process assessment technique with a quantitative approach to software development issues from the viewpoint of the process, product and resources.
The twelve steps are grouped by three in activities which are highlighted next (see schematic description in figure 1) .
The first step deals with the assessment of the software development environment for defining primary goals for metrication. Weaknesses and critical parts of the software development process are first pointed out. From those findings and business objectives as defined by the management, primary goals are defined (step 2). A hierarchy of top level goals is being considered depending on the maturity of the development process. Before setting up "change'"goals (e.g. to improve productivity while maintaining quality), one should envisaged "understanding" goals e.g. support of project management with process and product metrics. The assumption behind improvement goals is that the process is well defined. Then a validation of goals against the assessment conclusions, the timescale and the budget is performed to avoid too ambitious goals are set up (step 3).
The aim of the second activity is to build a so-called goal-tree which is the translation of the primary goal into sub-goals and metrics (step 4). Primary goals are broken down into more manageable sub-goals until directly quantifiable goals are reached. The process is based on the Goal/Question/Metric paradigm [BaR88]. After having verified the consistency of the tree (step 5), metrics are derived from these bottom goals with the help of questions (step 6). The results are a documented goal tree and the associated set of metrics.
The metricate activity encompasses three steps:
The exploitation of measures has to be performed in reference with the goals defined in the analyse activity. The improvement activity starts with an appropriate presentation of the measurement data (step 10). Graphics (histograms, pie charts, scatterplot, ...) should present the data so that both trend and outliers can be detected. Then, by relating data to goals (steps 11 and 12), it has to be determined whether the goals are fulfilled, how quickly they are fulfilled or why they did fail. Further corrective and/or improving actions are based on this determination.
The first loop is achieved. Now, it is worth quantifying the first benefits of the measurement programme. The iterative aspects of the approach permits not only -by a reassessment- the refinement of goals and consequently the metrics set, but also the improvement of the metrication process.
Figure 1. ami improvement cycle
The comprehensive Capability Maturity Model, CMM, with its current version released in 1993 [CMM93a,b], is extensively used in industry for
A primary business goal of the top management is to control and eventually reduce software development costs. It has been shown by experience that one way to achieve this is to use the SEI Software Process Assessment (SPA) method for investigating strengths and weaknesses and creating change momentum and then use the SEI CMM as a roadmap for improvement actions.
SEI SPA is a method for assessing the organization's software development process. It is an intensive 5-day investigation lead by a team of 6 to 8 individuals. During the week this assessment team gets input from 30 to 40 of the organization's employees, and out of this input establishes the organization's maturity level and formulates about 8 findings, i.e. main problem areas which the organization is facing. The consequences of these problems in the organization are considered as well.
The already mentioned CMM, see table 1, contains a number of key process areas (KPA) for level 2 to 5. Each key process area has its specific goals and a set of key practices for reaching these goals. The key practices are categorized in commitment, abilities, activities, measurement & analysis, and verification - the so called common features.
The final report of the SPA contains the already mentioned findings and consequences as well as suggestions concerning what to do to improve the process. The findings and suggestions are often mapped to, but are not limited to, the key process areas of the CMM. On the basis of the final report concrete planning of improvement actions is done, resulting in an action plan with detailed descriptions of improvement tasks with responsibilities, effort estimates, etc.
Table 1. Capability Maturity Model Overview
One major principle of the CMM is that the sequence in which changes are performed is important. Thus the action plan will concentrate on the key process areas one level higher than the organization's own maturity level, e.g. a level one organization may have one action domain concerning requirements management (RM).
The top management must now allocate enough resources, i.e. people for working groups, to implement the action plan contents within 12-18 months. Main focal point for the software process improvements is the newly established SEPG, software engineering process group. Its tasks will be to coordinate, control and promote the process improvement program [FoR90].
The primary management goal is at this stage to successfully implement the action plan, while keeping track of costs. The organization was assessed to be level 1- thus the top goal is G1-Reach CMM level 2.
As 2nd level goals, the findings from the final report has benn chosen. The corresponding consequences of the deficiencies might provide ideas on what metrics may be used (see figure 2 where we show the subtree in full depth). For example one major consequence of the lacks in requirements management was high rework effort caused by non-controlled requirements changes and weak requirements traceability.
3rd level goals are the domains (high level definition of improvement actions) of the action plan. Thus the ami goals tree was used to show the mapping between findings and these domains, see figure 1.
In the 4th level there are two paths, one for each of the management's main concerns. The left one covers implementation/ institutionalization of the incomplete key practices. For the domains of the action plan not representing key process areas of the CMM a similar structure is possible. The right path goal is to monitor the costs and benefits of the new process element, in this case requirements management.
The metrics for implementation/ institutionalization measure to what extent the key practices of a certain common feature of requirements management has been covered, separated in definition and usage, as appropriate.
The measures of costs and benefits are on one hand the cost of requirements management activities and on the other hand, relating to the already discussed consequence with high rework, rework effort related to lacks in requirements management.
Figure 2. Goal tree
As shown in figure 2, some metrics that we will collect for the action domain requirements management are
The necessary information is gathered through forms or, in the case of CFF, by the SEPG in periodic meetings with the working groups. For capturing information on CFF, the Interim Profile method of SEI might be useful [WNH94].
The data will be presented by the SEPG periodically in reviews with working groups and management.
Management will like to see that the primary goal, reaching CMM level 2, is accomplished in time. Using the radar chart (Kiviat) in 3, the SEPG can make progress within each action domain towards this goal visible. For management information, an additional radar chart with compound information, i.e. for each action domain one degree (like NS=Not Satisfied, PS=Partially Satisfied, FS=Fully Satisfied), could be useful. This would be similar to the SEI Interim Profile.
Figure 3. Monitoring the implementation/institutionalization (mapping to goal G1.n.1.1)
Also, the cost increase will be monitored and compared with the reduction in rework effort (remember - high rework effort was recognized to be a major consequence of lacks in the RM process in this organization). Any anomalies in the relation between these cost deltas can be easily recognized and causes analyzed to make corrections of the improvement program in time.
These measured benefits of the improved requirement management process could also be used for return on investment calculation. Of course, to do this we need to measure the effort put into definition of new procedures, training, etc.
After 12 to 18 months a full SPA-style re-assessment will be made, to validate the improvements made and set pace to continue improvements towards level 3.
As ami, the Bootstrap project [HMK94] was performed within the frame of the ESPRIT programme. Its goal was to develop a method for software-process assessment, quantitative measurement and improvement. Bootstrap enhanced and refined the SEI's process assessment method and adapted it to the needs of the European software industry i.e. including ISO 9000-3 (guidelines for software quality assurance) attributes and the ESA's PSS05 software engineering standards.
The Bootstrap method covers the assessment activity. A maturity level is calculated for each attribute of the process quality model by answering a questionnaire. This model takes into account three main dimensions in the process: organization, technology and methodology issues, methodology being broken down in life-cycle-independent functions, life cycle functions and process related functions. Bootstrap provides a detailed profiling technique as well as a standard format of an action plan. But there is no goal-oriented approach to map the maturity profile onto improvement activities and goals. The idea would be to analyse the profile and to derive main goals (mainly based on the improvement of weak quality atrributes). Then a goal tree is derived which nodes contains goals and necessary actions (Action plan). Metrics are associated to each goal in order to measure whether the goals are achieved.
The proposed approach is illustrated with a real case study [HMK94] derived from a Bootstrap article in IEEE Software of july 1994. The assessed organization is developing a control system to visualise industrial processes. The assessed project is currently in the maintenance phase, and the project is continued with development of a larger system in which the assessed system is integrated. An improvement project was started and runs in parallel with the development introducing software engineering practices based on the findings of the Bootstrap assessment. The bootstrap profile showed big weaknesses in project management and design.
Design: 12 Person days were needed to correct an error during maintenance because there was no graphical or textual representation of the module structure, the links among modules and the product interfaces
Project management: No detailed workflow charts and cost and effort based on sound historical data were available despite the fact that a project-management tool has been purchased. But since no training had been performed, it is never been used properly.
Consequently, the top level goals and related actions are:
G1: To reduce effort for maintenance
A1: Introducing a case tool based on structured design
G2: To produce reliable estimates
A2: To implement a project management methodology based on the existing tool.
From the analysis of the top level goals and related actions, a framework for action plan and measurement is built (see figure 4).
The goal tree merges main improvement actions and related goals. The leafs define the necessary metrics to ensure the achievement of the goals.
With regards to G1, a redesign of software will be performed. The benefits will be evaluated on the next maintenance activities. With regards to G2, the existing project management methodology shall be applied and supported with the definition of a historical and tracking database for estimation purposes.
For supporting and evaluating the benefits of the redesign, the following metrics were collected:
The data analysis revealed the following facts:
G1.1.1
The redesign took only 4 person-months. The redesign was not really important but the effect of design on the maintenance seemed to be important.
After re-design of parts of the system and the use of professional design techniques in a further project, a reduced meaningful effort per maintenance activity (about 4 person days) was achieved, which was about one third of the previous effort. Moreover the redesign effort was amortized in about 6 months of maintenance activities. The quality of the product significantly increases.
G1.2.1
The implementation time was reduced by half with the same amount of effort. As a matter of fact, the graphical and textual design facilitated the decomposition of the product into different units that could be developed in parallel.
G2.1
The main benefits were the clear breakdown of a project into workpackages together with a more reliable resource estimation. In the requirements phase resources were estimated within 30% and after analysis and design within 10%.
Repeating the Bootstrap assessment will check whether the weak process quality attributes detected at the first reassessment have been improved.
Figure 4. Goal trees
In the last years ISO 9000 series became more and more THE quality standard. Never before a standard has got such a broad acceptance all over the world.
As ISO 9000 is independent from the industry segment, IT industry has some problems with the application. 1991 ISO 9000-3 Guidelines for the application of ISO 9001 to the development, supply and maintenance of software attacked this problem but was only a guideline. TickIT is an initiative to take ISO 9000-3 as a baseline and uses it for certification purposes. This is not a standard but a UK based certification scheme towards the ISO 9001 application in IT.
To demonstrate a quality management system according to ISO 9000-3 (or ISO 9001) internal audits have to be performed regularly. The nonconformities are handled through so-called corrective actions. Corrective actions aim at either improving a specific process or improving the way a process is followed.
In the following we show how ami can be apply to use this mechanism of internal audits for starting a process improvement cycle. The tracking of corrective actions is required by ISO 9000-3, nevertheless there are no guidelines how to do it. We will see how ami can be used for tracking and visualisation of the improvements.
As described in section 2 the first phase of the ami method deals with the assessment of the software development environment. In our example the assessment required by ami can be done through an internal audit. ISO 9000-3 requires a checklist. On hand of this checklist as well the quality system as written on paper as well as how far daily life differs from the written procedures is assessed. The weaknesses are described in corrective actions describing the nonconformities. For the application of ami the next step is to define goals as a baseline for the analysis phase. In our example TickIT certification is used as top-level goal.
In the analysis phase the results of the internal audit are analysed in order to define improvement actions. As ami requires the result of this phase is a goal-tree. How our example is decomposed can be seen in figure 5. The goals are broken down until quantifiable goals are reached. In our case all the bottom goals are either metrics concerning the definition or concerning the deployment.
The Measurement plan for these very simple metrics has to be defined here. In our example we have two different kinds of metrics: level of definition and percentage of institutionalisation.
For the definition metrics (DF definition factor) we use three categories
For the institutionalisation metrics (IF) we use five categories:
0% little effective usage
25% applied to about one-quarter of the potential
50% applied to about half the potential
75% applied to about three-quarters of the potential
100% applied to full potential in all relevant areas and activities.
Figure 5. ISO 9001 / TickIT goal tree
Through follow-up audits an assessment of the actual status of the progress can be made. The following diagram can be used for monitoring. After closing all corrective actions the next higher level goal is reached.
Tab. 2: Status of Quality System
The ISO Software process assessment initiative has grown out of an initial UK MOD effort call ImproveIT. ISO/JTC1/SC7/WG10 develops an international standard for software process assessment.
The objective of SPICE (Software Process Improvement and Capability Determination) is to provide a common approach and framework for assessment and improvement. Initial field trials of SPICE have started in January 1995.
We will concentrate on the so-called Process improvement guide [PIG94] which describes steps to perform within a complete improvement cycle (based on the SPICE assessment technology). ami has advocated and successfully applied the measurement techniques in a process improvement context. [DeS93, DLS93, DKR94]. SPICE makes use of the ami concepts to design its improvement cycle. The figure 6 show a mapping of the PIG improvement steps onto the ami activities.
In parallel, a mapping is made with IDEAL, a framework for software process improvement as recently being defined by the SEI [RaP94]. SPICE has generalized IDEAL.
A process improvement programme is triggered by the organization's needs, for instance, to fill in the market expectations or from a customer request in terms of maturity level. Initiating process improvement consists in defining an overall plan like any types of project, including responsibilities, resources, timeframe (Initiating phase of IDEAL).
How to conduct a SPICE assessment is described in the Process Assessment Guide. [PAG94]. The output mainly consists in conformance measurements which describe to what extend an organization's software process conforms to a specification of industry base practices contained in the Baseline Practices guide [BPG94] as well as additionnal comments (example of good practice, suggestions, etc.) consider as useful for the analysis. Conformance measurement is undirectly connected with the organization's needs and may be therefore complemented with other metrics, the effectiveness measures (resulting from the previous cycle - see next analyse activity). It corresponds to the diagnosing phase of IDEAL.
Figure 6. SPICE process improvement cycle (included in original copy in the proceedings)
The analysis (Establishing phase of IDEAL) is split into three steps:
An action plan is derived describing in detail process improvement actions to meet the process goals and quantified targets.
The process measurement framework is illustrated figure 7 and compare with ami's measurement structure.
A detailed action plan is developed and implemented (Acting phase of IDEAL)
When the action plan is completed, the organization monitor whether the current measurement of process effectiveness confirm the achievement of improvement targets and goals. Risks and cost and benefits should be also evaluated.
As soon as the improvement is confirmed, the improvement gains need to be sustained, i.e. to monitor institutionalisation of the improved process and to give encouragement when necessary. Then, the performance of the organization's software process should be continuously monitored using the effectiveness and conformance measures previously defined. New assessment can take place when a long term goal is about to be achieved or when the organization needs changes (e.g. higher capability). This is called the leveraging phase in SEI IDEAL
Figure 7: SPICE process measurement framework (included in the original copy in the proceedings).
An overview on existing software process assessment and improvement approaches and how they fit in the ami framework was provided. They all converge to the same improvement cycle as initiated by ami and taking its roots in Deming's work [Dem86]. They are all based on a few agreed basic principles such as continuous and iterative process, business needs driven, essential role of measurement to support process improvement.
The added value of ami relates to the effective implementation of measurement whatever the context is, i.e. software process improvement program, ISO 9001 certification etc.. Even though measurement is being identified as a crucial management tool, the same mistakes are still made, lack of goals, no vertical commitments (at all levels of management) within the organization and consequently misunderstandings between practictioners and management on what the objectives are. Those points may have caused that two third of all software process improvement initiative in the US have failed (this gossip is being confirmed at the last 1995 SEPG conference in Boston by B. Curtis and M. Paulk).
What is new about ami? The ami user group has been officially started beginning of 1995. An ami method update workshop took place in May 1995. A version 2 of the ami Handbook should be available by the end of this year, including more guidelines towards applying ami in different context i.e. SPI, procurement, ... Large companies known as being at the leading edge in their field are applying ami as part of their software process improvement programme e.g. HP, Nokia, Schlumberger. A mailing list has been also set up1.
Finally, to answer the question raised in the introduction, ami partners are involved in major initiatives such as SPICE and are collaborating with the European Software Institute. Hence, the ami method and the related tools will evolve in accordance with those standards.
Acknowledgements: The authors would like to thank all partners involved in the ami project who permit this work to be carried out. Special thanks to A. Combelles from Objectif Technology and one of the contributors to the PIG and R. Messnarz from ISCN as well as the reviewers of the European Software Institute for their valuable comments.
[AMI92] Esprit II 5494 ami: ami Handbook, Published version, March 1992.
[AMI94] Proceedings of the first ami user group meeting, Basel, October 1994.
[BaR88] Basili, V., Rombach, H.: The TAME Project: Towards Improvement-Oriented Software Environments, IEEE Transactions on Software Engineering, Vol 14, Number 6, June 1988.
[BPG94] SPICE consortium: The Base Practices Guide, Draft
[CMM93a] Paulk M C, Curtis B, Chrissis M B: Capability Maturity Model for Software, version 1.1, CMU/SEI-93-TR-24, February 1993.
[CMM93b] Paulk M C, Weber C V, Garcia S M, Chrissis M : Key practices of the Capability Maturity Model, version 1.1, CMU/SEI-93-TR-25, February 1993.
[Deb94] Debou C: ami a new paradigm for software process improvement, In: Proceedings of the first ISCN Seminar, Dublin, May1994
[DeS93] Debou C, Stainer S.: Improving the maintenance process: a quantitative approach, In: Proceedings of the 6th international conference on software engineering and its application, Paris, Nov 1993.
[DFS93] C.Debou, N. Fuchs, H. Saria, Selling believable technology, IEEE Software, Nov 1993.
[DKR94] Debou C, Kuntzmann-Combelles, Rowe A.: A quantitative approach to software process, In: Proceedings of the 2nd international symposium on software metrics, Oct 1994, London.
[DLP92] Debou C, Lipták J, Pescoller L: Managing Software Process by Applying ami. In: Proceedings of the MSP-92 IFAC - annual review in Automatic programming, volume 16, Graz, May 1992.
[DLS93] Debou C, Lipták J, Shippers H.: Decision making for software process improvement: a quantitative approach, In: Proceedings of the 2nd internation conference on "achieving quality in software" ACQUIS 93, Venice (Italy), pp 363-377, Oct 1993.
Also In: The Journal of Systems and Software, 1994; 26:43-52, Elsevier Science Inc.
[DPF92] Debou, C., Pescoller, L. and Fuchs, N.: Software Measurements on telecom systems - Success stories ? : Proceedings of the 3rd European conference on software quality, Madrid, November 1992.
[Dem86] Demin W.E., Out of the Crisis, MIT Press
[FoR90] Fowler, P. and Rifkin, S., Software Engineering Process Group Guide, CMU/SEI-90-TR-24, September.
[HMK94] Haasse V, Messnarz R, Koch G., Kugler H, Decrinis P: Bootstrap: Fine-Tuning Process Assessment, In: IEEE Software pp25-35, July 1994
[Hum89] Humphrey, W.: Managing the Software Process, Addison-Wesley, Reading, Mass., 1989.
[HuS87] Humphrey W.S., Sweet W.L.: A Method for Assessing the Software Engineering Capability of Contractors, Software Engineering Institute, Sept 1987
[MKH94] Messnarz R, H-J. Kugler, Haase V.: BOOTSTRAP and ISO 9000: A Quantitative Approach to Objective Quality Management, In: Proceedings of the first ISCN Seminar, Dublin, May1994
[PAG94] SPICE Consortium, The Process Assessment Guide, Draft
[PeR94] Perterson B, Radice R.: IDEAL: an integrared approach to software process improvement (SPI), SEI Symposium, Pittsburgh, August 1994.
[PFS92] I. Perez, P. Ferrer, A. Fernandez: Application of Metrics in Industry : Proceedings of the 3rd European conference on software quality, Madrid, November 1992.
[PIG94] SPICE consortium: The process improvement guide, issue 0.05, Draft, October 1994
[RoW94] Rowe A., Whitty R.: ami: a Quantitative Approach to Software Management, In: Software Quality Journal 2, 291-296, 1994
[WNH94] Whitney, R., Nawrocky, E., Hayes, W., Siegel, J. (1994) Interim Profile: Development and Trial of a Method to Rapidly Measure Software Engineering maturity Status, CMU/SEI-94-TR-4, March.
ISCN Information Request
I.S.C.N. International Software Consulting Network
Tel: +353 1 286 1583, Fax: +353 1 286 5078
email: office@iscn.ie