Miklós Biró, Éva Feuer, Tibor Remzsô, Piroska Turchányi
Computer and Automation Institute
Hungarian Academy of Sciences
Budapest, Lágymányosi u. 11. H-1111 Hungary
Tel +36 1 269 8270
Fax +36 1 269 8269
e-mail: miklos.biro@sztaki.hu
This paper analyses critical business decision problems supported by software product and process assessment which can potentially benefit from a systematic application of multiple criteria decision making theory and techniques. An additional purpose of the paper is to provide a new approach to the topic with a didactic introduction appealing to a business oriented audience as well. The first issue is product quality where standards support purchasing type decisions. The decision problem of an external customer seeking reliable partners is supported by ISO 9000 certification which is performed according to a well defined list of decision criteria. This list is helpful for the decision but does not tell how to proceed in case of failure on one or several of the criteria. The BOOTSTRAP software process assessment methodology provides an action plan which supports the management in the detailed choice of the most appropriate steps and decisions depending on the actual standing of the organization.
Multiple Criteria Decision Making (MCDM) is a classical research field whose objective is the improvement of decision maker performance when there are several alternatives and a variety of criteria for evaluating and comparing them. The subject of this paper is to show how Software Product Evaluation and Software Process Improvement methods can benefit from the systematic application of MCDM theory and techniques.
The fundamental notions of the MCDM field are introduced in the following section with a focus on software quality. What kinds of decisions are supported by software product quality evaluations based on more or less defined systems of decision criteria? This question is analyzed in the third section. Are there critical decisions which are not supported by software product quality evaluations? Two different types of such critical decisions are given in the fourth and fifth sections. The last section contains a summary and directions for further research.
MCDM research starts back in the 18th century with Daniel Bernoulli [Bernoulli, 1738]. He argued that a person's utility of wealth is not a linear function of the money measure of his wealth. In our century, it was first von Neumann and Morgenstern [von Neumann, Morgenstern, 1944] who gave a scientific articulation to MCDM. They referred to "several conflicting maximum problems". Since then, due in part to the general successes of the operations research/management science field, the literature of MCDM has exhibited a huge growth.
With a mind trained in the traps of the software development process, we can discover however, that many approaches are strongly biased by the solution methods they offer themselves. An analogous problem is encountered within the software life-cycle, when user requirements specification is not expressly separated from software requirements specification.
There are of course approaches which discipline themselves to focus first on the analysis of the real nature of decision problems. These include [Tversky, 1969] and [Zeleny, 1982]. In this paper, we introduce the MCDM approach following [Zeleny, 1982] but focusing on the concepts relevant to Software Product and Process Assessments.
Multiple criteria decisions involve alternatives which are usually evaluated on the basis of a hierarchical system of criteria. There is a large variety of evaluation process types. The alternatives for example are most of the time to be ranked, but there are cases where only a single alternative has to be accepted.
The starting point of the introduction to MCDM must be the clarification of the fundamental concepts which are attributes, objectives, goals, criteria, strategies and alternatives. The subject of the following sections is essentially the discussion of the high relevance of these notions to software quality and process assessment. These terms are typically used however not defined in the otherwise meticulous standards.
Attributes are the descriptors of reality. They can be identified and measured essentially independently from the decision maker's or user's requirements. Objectives consist of individual attributes or sets of attributes together with their directions of preference. They are clearly identifiable with the decision maker's or user's requirements. Two or more objectives can form higher level objectives resulting in a hierarchy which helps overcoming the cognitive psychological barriers in case of more than 72 simultaneous objectives [Biró, Maros, 1992]. Goals mean target levels of achievement defined in terms of either attributes or objectives. They are determined by the decision maker's or user's requirements. Criteria is a general term for the attributes, objectives, or goals which are judged relevant in a given decision situation by an individual or a group.
A quality standard can be considered as the result of a group consensus on a system of relevant quality criteria.
Strategies are the means of changing the current situation. Alternatives are strategies which are candidates to be employed for changing the current situation according to a current system of citeria. Alternatives are usually expected to be mutually exclusive. Nevertheless, if the decision situation allows for the simultaneous selection of several alternatives, then it is enough to state that there is at least one pair of alternatives which are mutually exclusive, otherwise there is no real decision to be made.
The evaluation of the quality of a software product is one of the basic issues in information technology. Many books [Gilb, 1988] [Gillies, 1992] [Boehm, 1981] and papers deal with this subject, suggesting MCDM approaches to a smaller or larger extent. A system of decision criteria is summarized in the ISO/IEC 9126 standard. The first level of the criterion hierarchy contains the following elements: functionality, reliability, usability, maintainability, portability, and efficiency. This standard should be considered as the result of a group consensus on a system of relevant criteria. Nevertheless, there is no consensus for the moment about subcharacteristics for example, which are only provided as an illustrative annex to the standard. In MCDM terms, the ISO/IEC 9126 standard defines high level objectives only without specifying the corresponding lower level measurable attributes themselves, thus currently, it fails to satisfy its fundamental purpose of assisting consistent software product quality evaluation. An approach to filling this gap is provided by the SCOPE guide for software product quality evaluation [Boegh, 1994].
The decisions that ISO/IEC 9126 intends to support are the following:
Alternatives product1 Result: status related to the
ISO/IEC 9126 standard
product2
.
productn
Criteria functionality Evaluation: scaling, weights
reliability
usability
maintainability
portability,
efficiency
The fact that a company was able to eliminate defects from some of its products does not mean that those same defects will not reoccur in future software products.
A fundamental business issue is by consequent the customer's decision problem:
ISO-9000 certification is intended to support the above decision by focusing on the process rather than on the product. The standard generally applied to the software development process is ISO 9001 which is complemented by the ISO 9000-3 notes for guidance on the application of ISO 9001 to software development.
ISO 9001 lists 20 quality system requirements which can be considered as a set of high level criteria for the above decision problem.
These criteria have more or less detailed subcriteria spelled out in the standard's subparagraphs on multiple levels. Thus, we are again faced with a hierarchical system of criteria. Here the lowest level attributes are mostly measurable in binary terms, that is only the existence of required policies and practices has to be checked.
ISO 9000-3, formulated in the form of guidelines for the application of ISO 9001 to the development, supply and maintenance of software, is essentially a restructuring and reformulating of the system of criteria from top to bottom but finally covering all of the ISO 9001 requirements.
ISO 9001 ISO 9000-3
Alternatives Certification (Yes or No) Certification (Yes or No)
Criteria Tree structure Same as ISO 9001 in a
20 elements (high level different structure.
criteria)
Multiple subparagraphs (lower
level criteria)
Certification provides little support for improvement. The system of requirements is rather large and their simultaneous satisfaction may mean an inhibitive burden to the company. In addition, conformance must be maintained after registration as well, which can be best achieved by implementing continuous improvement. By consequent, the natural question before or after certification is how a company can improve its capability for reliably producing high quality products. This is the supplier's decision problem:
While the customer's decision problem discussed in the preceding section involved only two alternatives (yes or no), this decision problem admits a large number of alternative courses of action the choice among which is supported by the BOOTSTRAP method for software process assessment and improvement.
BOOTSTRAP was a European ESPRIT project (5441) which was carried out and finished in February 1993 by a consortium of European software companies and universities. The aim of this project was to develop a method for software process assessment and improvement adapted to the European software industry. BOOTSTRAP designed a very detailed process quality attribute hierarchy and enhanced the questionnaire based on the Software Engineering Institute's (SEI) Capability Maturity Model (CMM) [Paulk, Curtis, Chrissis, 1991,1993] by taking into account the ISO 9000-3 guidelines for software quality and the ESA (European Space Agency) PSS-05 software engineering standards. In addition, the questions are answered on a four choice percentage scale instead of the yes-no choices of the SEI method.
Contrary to the yes-no conclusions of ISO 9000, CMM provided an ordinal scale for measuring process maturity, which made it possible to move further than Software Capability Evaluation (SCE), to the possibility of prioritizing improvement efforts.
In CMM, the maturity levels are decomposed into key process areas which are targeted by specific yes-no questions during capability evaluation. From the decision making perspective, the key process areas are high level criteria which lead to the decision to be made about the maturity level of the evaluated software producing unit as a whole. The decision alternatives in this case are the maturity levels themselves.
SCE
Alternatives Five maturity levels Result: maturity level status
(integer)
Criteria Key process areas Evaluation: binary
As pointed out in [Haase, Messnarz, Koch, Kugler, Degrinis, 1994], a CMM key process area is only valid within one maturity level, thus, it cannot be used for mapping separate attributes onto a maturity level scale. BOOTSTRAP, on the other hand, results in a profile containing the maturity-levels of the individual attributes.
BOOTSTRAP for process assessment
Alternatives Profiles with five Result: refined maturity level
possible maturity levels status (the overall status
for each attribute between two consecutive
maturity levels, and the
maturity levels of process
attributes)
Criteria Detailed hierarchy of Evaluation: refined scaling and
process-quality attributes evaluation algorithm
When, instead of evaluation, the decision problem is improvement related, a strategy must be worked out. The key process areas or BOOTSTRAP attributes become decision alternatives as the potential components of the improvement action plan: which key process areas or which attribute(s) should be improved in order to achieve the goal, for example maturity level 4. In this case, the number of alternatives is considerable.
BOOTSTRAP for process improvement
Alternatives Process-quality attributes Result: Action plan
to include into the action
plan
Criteria Maturity levels of Evaluation: Trained BOOTSTRAP
process-quality attributes assessors
Having based the work on the ISO 9000-3 standard, BOOTSTRAP can determine about 85% of the ISO attributes as well whether they are satisfied or not [Biró, Feuer, Haase, Koch, Kugler, Messnarz, Remzsô, 1993], [Haase, Messnarz, Koch, Kugler, Degrinis, 1994]. Therefore BOOTSTRAP can also be used as a decision support tool for undertaking ISO certification.
BOOTSTRAP for ISO certification
Alternatives satisfied or not Result: satisfied or not
Criteria 85% of ISO 9000-3 Evaluation: mapping
attributes algorithm
We analyzed critical software related business decision problems. Different types of criteria and alternatives were recognized in each case. Some of the criteria are standardized, like ISO/IEC 9126 and ISO 9001. Methods rank from simple binary evaluation to multiple level weighted average. The table below summarizes the decision criteria and alternatives in case of various approaches.
Summary table Alternatives Criteria
ISO 9000 yes or no 20 elements
SCE Global maturity level Key process areas
SEI Improvement Key process areas to Global maturity level
improve
BOOTSTRAP Profiles with five Detailed hierarchy of
assessment possible maturity process-quality attributes
levels for each
attribute
BOOTSTRAP Improvement action Profiles with five possible
improvement plan maturity levels for each
attribute
In the nineties, a new trend started in the field of MCDM. Instead of providing an exact but strict evaluation procedure, flexible learning tools based on visual interactive technology have been developed, for example [Angehrn, 1991], [Biró, Csáki, Vermes, 1991] and [Csáki, 1995]. The large variety of MCDM problems justifies the development of humanized multiple criteria decision systems where the decision makers are allowed to learn about the decision problem itself while experimenting with both its formulation and solution. These kinds of systems merely put different tools and instruments at the decision makers' disposal. The user is allowed to define the set of alternatives, the set of criteria, the evaluation method. He can trace the effect of any change in one problem component to any other problem component. Consequently, with the help of such a flexible system for the case of process improvement, the decision maker could determine its preferred improvement action plan in a most creative and challenging way.
BOOTSTRAP can be considered as a special MCDM method for comparing and evaluating alternatives on the basis of hierarchically structured criteria. It would be highly attractive to provide a flexible working environment in which the direct or indirect effect of any changes to problem components could be automatically monitored. Such a feature would certainly increase the interest and commitment of the participants in the assessment and improvement process.
[Bernoulli, 1738] Bernoulli,D. Specimen Theoriae Novae de Mensura Sortis. Comment.Acad.Sci.Imper.Petropolitanae, 5(1738)175-192. English translation by L.Sommer, Exposition of a New Theory on the Measurement of Risk. Econometrica, 22(1954)23-36.
[Biró, Csáki, Vermes, 1991] Biró,M.; Csáki,P.; Vermes,M. WINGDSS Group Decision Support System under MS-Windows. In: Proceedings of the Second Conference on Artificial Intelligence (ed. by I.Fekete and P.Koch). (John von Neumann Society for Computer Sciences, Budapest, Hungary, 1991) pp.263-274.
[Biró, Maros, 1992] Biró,M.; Maros,I. The Use of Deep Knowledge from the Perspectives of Cooperative Problem Solving, Systems Modeling, and Cognitive Psychology. In: Shifting Paradigms in Software Engineering (ed. R. Mittermeir). (Springer-Verlag, Wien, New York, 1992) pp.56-67.
[Biró, Feuer, Haase, Koch, Kugler, Messnarz, Remzsô, 1993] Biró,M.; Feuer,É.; Haase,V.; Koch,G.R.; Kugler,H.J.; Messnarz,R.; Remzsô,T. BOOTSTRAP and ISCN a current look at the European Software Quality Network. In: The Challenge of Networking: Connecting Equipment, Humans, Institutions (ed. by D. Sima, G. Haring). (R.Oldenbourg, Wien, München, 1993) pp.97-106.
[Boegh, 1994] Boegh, J.: SCOPE: A guide for Software Product Quality Evaluation, Proceedings of the ISCN '94 Conference on Practical Improvement of Software Processes and Products, (Dublin, Ireland, 1994).
[Boehm, 1981] Boehm, B.: Software Engineering Economics. Prentice Hall, Englewood Cliffs, NJ 1981
[Csáki, 1995] Csáki, P., Csiszár, L., Fölsz, F., Keller, K., Mészáros, Cs., Rapcsák, T., Turchányi, P.: A flexible framework for group decision support, Annals of Operations Research, Special Volume on APMOD, accepted
[Gilb, 1988] Gilb, T.: Principles of Software Engineering Management, Addision-Wesley 1988.
[Gillies, 1992] Gillies, A.C.: Software Quality Theory and Management, Chapman & Hall, 1992
[Haase, Messnarz, Koch, Kugler, Degrinis, 1994] Haase,V.; Messnarz,R.; Koch,G.; Kugler,H.J.; Degrinis,P. Bootstrap: Fine-Tuning Process Assessment. IEEE Software (July 1994) pp.25-35.
[Paulk, Curtis, Chrissis, 1991] Paulk, M.C., Curtis, C. and Chrissis, M.B.: Capability Maturity Model for Software, Software Engineeering Institute, Carnegie Mellon University, Pittsburgh, August 1991.
[Paulk, Curtis, Chrissis, 1993] Paulk, M.C., Curtis, C., Chrissis, M.B.and Weber,C.V.: Capability Maturity Model, Version 1.1, IEEE Software, July 1993, pp.18-27.
[Tversky, 1969] Tversky,A. Intransivity of Preferences. Psychological Review, 76(1969)31-48.
[von Neumann, Morgenstern, 1944] von Neumann,J. and Morgenstern,O. Theory of Games and Economic Behavior. Princeton University Press, Princeton, NJ. 1944.
[Zeleny, 1982] Zeleny,M. Multiple Criteria Decision Making. McGraw-Hill Book Company, 1982.
ISCN Information Request
I.S.C.N. International Software Consulting Network
Tel: +353 1 286 1583, Fax: +353 1 286 5078
email: office@iscn.ie