read.pudn.comread.pudn.com/downloads454/doc/1911479/iso_iec 9126-3.pdf · 2011-11-11 · iso/iec tr...
TRANSCRIPT
Reference numberISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003
TECHNICAL REPORT
ISO/IECTR
9126-3
First edition2003-07-01
Software engineering — Product quality — Part 3: Internal metrics
Génie du logiciel — Qualité des produits —
Partie 3: Métrologie interne
ISO/IEC TR 9126-3:2003(E)
PDF disclaimer This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but shall not be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat accepts no liability in this area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In the unlikely event that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO/IEC 2003 All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO's member body in the country of the requester.
ISO copyright office Case postale 56 • CH-1211 Geneva 20 Tel. + 41 22 749 01 11 Fax + 41 22 749 09 47 E-mail [email protected] Web www.iso.org
Published in Switzerland
ii © ISO/IEC 2003 – All rights reserved
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved iii
Contents Page
Foreword............................................................................................................................................................ vi
Introduction ...................................................................................................................................................... vii
1 Scope........................................................................................................................................................... 1
2 Conformance .............................................................................................................................................. 2
3 Normative references................................................................................................................................. 2
4 Terms and definitions ................................................................................................................................ 2
5 Abbreviated terms...................................................................................................................................... 2
6 Use of software quality metrics ................................................................................................................ 3
7 How to read and use the metrics tables .................................................................................................. 4
8 Metrics tables.............................................................................................................................................. 4 8.1 Functionality metrics............................................................................................................................... 4 8.1.1 Suitability metrics ........................................................................................................................ 5 8.1.2 Accuracy metrics ......................................................................................................................... 5 8.1.3 Interoperability metrics................................................................................................................ 5 8.1.4 Security metrics............................................................................................................................ 5 8.1.5 Functionality compliance metrics .............................................................................................. 5 8.2 Reliability metrics .................................................................................................................................. 12 8.2.1 Maturity metrics.......................................................................................................................... 12 8.2.2 Fault tolerance metrics .............................................................................................................. 12 8.2.3 Recoverability metrics ............................................................................................................... 12 8.2.4 Reliability compliance metrics.................................................................................................. 12 8.3 Usability Metrics .................................................................................................................................... 17 8.3.1 Understandability metrics ......................................................................................................... 17 8.3.2 Learnability metrics ................................................................................................................... 17 8.3.3 Operability metrics..................................................................................................................... 17 8.3.4 Attractiveness metrics............................................................................................................... 17 8.3.5 Usability compliance metrics.................................................................................................... 17 8.4 Efficiency metrics .................................................................................................................................. 24 8.4.1 Time behaviour metrics ............................................................................................................. 24 8.4.2 Resource utilization metrics ..................................................................................................... 24 8.4.3 Efficiency compliance metrics.................................................................................................. 24 8.5 Maintainability metrics .......................................................................................................................... 28 8.5.1 Analysability metrics ................................................................................................................. 28 8.5.2 Changeability metrics ................................................................................................................ 28 8.5.3 Stability metrics.......................................................................................................................... 28 8.5.4 Testability metrics...................................................................................................................... 28 8.5.5 Maintainability compliance metrics.......................................................................................... 28 8.6 Portability metrics.................................................................................................................................. 34 8.6.1 Adaptability metrics ................................................................................................................... 34 8.6.2 Installability metrics................................................................................................................... 34
ISO/IEC TR 9126-3:2003(E)
iv © ISO/IEC 2003 – All rights reserved
8.6.3 Co-existence metrics..................................................................................................................34 8.6.4 Replaceability metrics................................................................................................................34 8.6.5 Portability compliance metrics..................................................................................................34
Annex A (informative) Considerations When Using Metrics........................................................................41 A.1 Interpretation of measures....................................................................................................................41 A.1.1 Potential differences between test and operational contexts of use .....................................41 A.1.2 Issues affecting validity of results .............................................................................................42 A.1.3 Balance of measurement resources ..........................................................................................42 A.1.4 Correctness of specification.......................................................................................................42 A.2 Validation of Metrics ..............................................................................................................................42 A.2.1 Desirable Properties for Metrics ...............................................................................................42 A.2.2 Demonstrating the Validity of Metrics ......................................................................................43 A.3 Use of metrics for estimation (judgement) and prediction (forecast) ..............................................44 A.3.1 Quality characteristics prediction by current data...................................................................44 A.3.2 Current quality characteristics estimation on current facts ...................................................44 A.4 Detecting deviations and anomalies in quality problem prone components ..................................45 A.5 Displaying measurement results..........................................................................................................45
Annex B (informative) Use of Quality in Use, External & Internal Metrics (Framework Example) ...........46 B.1 Introduction ............................................................................................................................................46 B.2 Overview of Development and Quality Process .................................................................................46 B.3 Quality Approach Steps ........................................................................................................................47 B.3.1 General.........................................................................................................................................47 B.3.2 Step #1 Quality requirements identification ............................................................................47 B.3.3 Step #2 Specification of the evaluation....................................................................................48 B.3.4 Step #3 Design of the evaluation ..............................................................................................50 B.3.5 Step #4 Execution of the evaluation .........................................................................................50 B.3.6 Step #5 Feedback to the organization ......................................................................................50
Annex C (informative) Detailed explanation of metric scale types and measurement types ...................51 C.1 Metric Scale Types.................................................................................................................................51 C.2 Measurement Types...............................................................................................................................52 C.2.1 Size Measure Type......................................................................................................................52 C.2.2 Time measure type .....................................................................................................................55 C.2.2.0 General.........................................................................................................................................55 C.2.3 Count measure type ...................................................................................................................56
Annex D (informative) Term(s).........................................................................................................................58 D.1 Definitions...............................................................................................................................................58 D.1.1 Quality..........................................................................................................................................58 D.1.2 Software and user.......................................................................................................................58 D.1.3 Measurement...............................................................................................................................59
Annex E (informative) Pure Internal Metrics ..................................................................................................60 E.1 Pure Internal Metrics..............................................................................................................................60
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved v
Table 8.1.1 Suitability metrics........................................................................................................................... 6 Table 8.1.2 Accuracy metrics............................................................................................................................ 8 Table 8.1.3 Interoperability metrics.................................................................................................................. 9 Table 8.1.4 Security metrics............................................................................................................................ 10 Table 8.1.5 Functionality compliance metrics............................................................................................... 11 Table 8.2.1 Maturity metrics ............................................................................................................................ 13 Table 8.2.2 Fault tolerance metrics ................................................................................................................ 14 Table 8.2.3 Recoverability metrics ................................................................................................................. 15 Table 8.2.4 Reliability compliance metrics .................................................................................................... 16 Table 8.3.1 Understandability metrics ........................................................................................................... 18 Table 8.3.2 Learnability metrics...................................................................................................................... 19 Table 8.3.3 Operability metrics .................................................................................................................... 20 Table 8.3.4 Attractiveness metrics ................................................................................................................. 22 Table 8.3.5 Usability compliance metrics ...................................................................................................... 23 Table 8.4.1 Time behaviour metrics ............................................................................................................ 25 Table 8.4.2 Resource utilisation metrics ................................................................................................... 26 Table 8.4.3 Efficiency compliance metrics .................................................................................................... 27 Table 8.5.1 Analysability metrics.................................................................................................................... 29 Table 8.5.2 Changeability metrics .................................................................................................................. 30 Table 8.5.3 Stability metrics ............................................................................................................................ 31 Table 8.5.4 Testability metrics ........................................................................................................................ 32 Table 8.5.5 Maintainability compliance metrics............................................................................................ 33 Table 8.6.1 Adaptability metrics ..................................................................................................................... 35 Table 8.6.2 Installability metrics ..................................................................................................................... 37 Table 8.6.3 Co-existence metrics ................................................................................................................... 38 Table 8.6.4 Replaceability metrics.................................................................................................................. 39 Table 8.6.5 Portability compliance metrics ................................................................................................... 40 Table B.1 Quality Measurement Model ......................................................................................................... 46 Table B.2 User Needs Characteristics & Weights ........................................................................................ 47 Table B.3 Quality measurement tables ......................................................................................................... 48 Table B.4 Measurement plan........................................................................................................................... 50
ISO/IEC TR 9126-3:2003(E)
vi © ISO/IEC 2003 – All rights reserved
Foreword ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission) form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC participate in the development of International Standards through technical committees established by the respective organization to deal with particular fields of technical activity. ISO and IEC technical committees collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the work. In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting. Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote.
In exceptional circumstances, the joint technical committee may propose the publication of a Technical Report of one of the following types:
— type 1, when the required support cannot be obtained for the publication of an International Standard, despite repeated efforts;
— type 2, when the subject is still under technical development or where for any other reason there is the future but not immediate possibility of an agreement on an International Standard;
— type 3, when the joint technical committee has collected data of a different kind from that which is normally published as an International Standard (“state of the art”, for example).
Technical Reports of types 1 and 2 are subject to review within three years of publication, to decide whether they can be transformed into International Standards. Technical Reports of type 3 do not necessarily have to be reviewed until the data they provide are considered to be no longer valid or useful.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC TR 9126-3:2003, which is a Technical Report of type 2, was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology, Subcommittee SC 7, Software and system engineering.
This document is being issued in the Technical Report (type 2) series of publications (according to the Procedures for the technical work of ISO/IEC JTC 1) as a “prospective standard for provisional application” in the field of external metrics for quantitatively measuring external software because there is an urgent need for guidance on how standards in this field should be used to meet an identified need.
This document is not to be regarded as an “International Standard”. It is proposed for provisional application so that information and experience of its use in practice may be gathered. Comments on the content of this document should be sent to the ISO Central Secretariat.
A review of this Technical Report (type 2) will be carried out not later than three years after its publication with the options of: extension for another three years; conversion into an International Standard; or withdrawal.
ISO/IEC 9126 consists of the following parts, under the general title Software engineering — Product quality :
Part 1: Quality model
Part 2: External metrics
Part 3: Internal metrics
Part 4: Quality in use metrics
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved vii
Introduction This Technical Report provides internal metrics for measuring attributes of six external quality characteristics defined in ISO/IEC 9126-1. The metrics listed in this Technical Report are not intended to be an exhaustive set. Developers, evaluators, quality managers and acquirers may select metrics from this Technical Report for defining requirements, evaluating software products, measuring quality aspects and other purposes. They may also modify the metrics or use metrics which are not included here. This Technical Report is applicable to any kind of software product, although each of the metrics is not always applicable to every kind of software product.
ISO/IEC 9126-1 defines terms for the software quality characteristics and how these characteristics are decomposed into subcharacteristics. ISO/IEC 9126-1, however, does not describe how any of these subcharacteristics could be measured. ISO/IEC TR 9126-2 defines external metrics, ISO/IEC TR 9126-3 defines internal metrics and ISO/IEC 9126-4 defines quality in use metrics, for measurement of the characteristics or the subcharacteristics. Internal metrics measure the software itself, external metrics measure the behaviour of the computer-based system that includes the software, and quality in use metrics measure the effects of using the software in a specific context of use.
This Technical Report is intended to be used together with ISO/IEC 9126-1. It is strongly recommended to read ISO/IEC 14598-1 and ISO/IEC 9126-1, prior to using this Technical Report, particularly if the reader is not familiar with the use of software metrics for product specification and evaluation.
Clauses 1 to 7 and Annexes A to D are common to ISO/IEC TR 9126-2, ISO/IEC TR 9126-3, and ISO/IEC 9126-4. Annex E is for ISO/IEC TR 9126-3 use.
TECHNICAL REPORT ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 1
Software engineering — Product quality —
Part 3: Internal metrics
1 Scope
This Technical Report defines internal metrics for quantitatively measuring external software quality in terms of characteristics and subcharacteristics defined in ISO/IEC 9126-1, and is intended to be used together with ISO/IEC 9126-1.
This Technical Report contains:
I. an explanation of how to apply software quality metrics
II. a basic set of metrics for each subcharacteristic
III. an example of how to apply metrics during the software product life cycle
This Technical Report does not assign ranges of values of these metrics to rated levels or to grades of compliance, because these values are defined for each software product or a part of the software product, by its nature, depending on such factors as category of the software, integrity level and users' needs. Some attributes may have a desirable range of values, which does not depend on specific user needs but depends on generic factors; for example, human cognitive factors.
This Technical Report can be applied to any kind of software for any application. Users of this Technical Report can select or modify and apply metrics and measures from this Technical Report or may define application-specific metrics for their individual application domain. For example, the specific measurement of quality characteristics such as safety or security may be found in International Standards or Technical Reports provided by IEC 65 and ISO/IEC JTC 1/SC 27.
Intended users of this Technical Report include:
— Acquirer (an individual or organization that acquires or procures a system, software product or software service from a supplier);
— Evaluator (an individual or organization that performs an evaluation. An evaluator may, for example, be a testing laboratory, the quality department of a software development organization, a government organization or a user);
— Developer (an individual or organization that performs development activities, including requirements analysis, design, and testing through acceptance during the software life cycle process);
— Maintainer (an individual or organization that performs maintenance activities);
— Supplier (an individual or organization that enters into a contract with the acquirer for the supply of a system, software product or software service under the terms of the contract) when validating software quality at qualification test;
— User (an individual or organization that uses the software product to perform a specific function) when evaluating quality of software product at acceptance test;
— Quality manager (an individual or organization that performs a systematic examination of the software product or software services) when evaluating software quality as part of quality assurance and quality control.
ISO/IEC TR 9126-3:2003(E)
2 © ISO/IEC 2003 – All rights reserved
2 Conformance
There are no conformance requirements in this Technical Report.
NOTE General conformance requirements for metrics are in ISO/IEC 9126-1 Quality model.
3 Normative references
ISO/IEC 9126-1:2001, Software engineering — Product quality — Part 1: Quality model
ISO/IEC TR 9126-21), Software engineering — Product quality — Part 2: External metrics
ISO/IEC 9126-41), Software engineering — Product quality — Part 4: Quality in use metrics
ISO/IEC 14598-1:1999, Information technology — Software product evaluation — Part 1: General overview
ISO/IEC 14598-2:2000, Software engineering — Product evaluation — Part 2: Planning and management
ISO/IEC 14598-3:2000, Software engineering — Product evaluation — Part 3: Process for developers
ISO/IEC 14598-4:1999, Software engineering — Product evaluation — Part 4: Process for acquirers
ISO/IEC 14598-5:1998, Information technology — Software product evaluation — Part 5: Process for evaluators
ISO/IEC 14598-6:2001, Software engineering — Product evaluation — Part 6: Documentation of evaluation modules
ISO/IEC 12207:1995, Information technology — Software life cycle processes
ISO/IEC 14143-1:1998, Information technology — Software measurement — Functional size measurement — Part 1: Definition of concepts
ISO 2382-20:1990, Information technology — Vocabulary — Part 20: System development
ISO 9241-10:1996, Ergonomic requirements for office work with visual display terminals (VDTs) — Part 10: Dialogue principles
4 Terms and definitions
For the purposes of this document, the terms and definitions given in ISO/IEC 14598-1:1999 and ISO/IEC 9126-1:2001 apply. They are also listed in Annex D.
5 Abbreviated terms
The following abbreviations are used in this Technical Report:
SQA — Software Quality Assurance (Group)
SLCP — Software Life Cycle Processes
1) To be published.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 3
6 Use of software quality metrics
These Technical Reports (ISO/IEC TR 9126-2 External metrics, ISO/IEC TR 9126-3 Internal metrics and ISO/IEC 9126-4 Quality in use metrics) provide a suggested set of software quality metrics (external, internal and quality in use metrics) to be used with the ISO/IEC 9126-1 Quality model. The user of these Technical Reports may modify the metrics defined, and/or may also use metrics not listed. When using a modified or a new metric not identified in these Technical Reports, the user should specify how the metrics relate to the ISO/IEC 9126-1 quality model or any other substitute quality model that is being used.
The user of these Technical Reports should select the quality characteristics and subcharacteristics to be evaluated, from ISO/IEC 9126-1; identify the appropriate direct and indirect measures, identify the relevant metrics and then interpret the measurement result in an objective manner. The user of these Technical Reports also may select product quality evaluation processes during the software life cycle from the ISO/IEC 14598 series of standards. These give methods for measurement, assessment and evaluation of software product quality. They are intended for use by developers, acquirers and independent evaluators, particularly those responsible for software product evaluation (see Figure 1).
software product effect of softwareproduct
quality in usemetrics
quality inuse
internalquality
internal metrics external metrics
externalquality
contexts ofusedepends on
influences influences
depends on
Figure 1 – Relationship between types of metrics
The internal metrics may be applied to a non-executable software product during its development stages (such as request for proposal, requirements definition, design specification or source code). Internal metrics provide the users with the ability to measure the quality of the intermediate deliverables and thereby predict the quality of the final product. This allows the user to identify quality issues and initiate corrective action as early as possible in the development life cycle.
The external metrics may be used to measure the quality of the software product by measuring the behaviour of the system of which it is a part. The external metrics can only be used during the testing stages of the life cycle process and during any operational stages. The measurement is performed when executing the software product in the system environment in which it is intended to operate.
The quality in use metrics measure whether a product meets the needs of specified users to achieve specified goals with effectiveness, productivity, safety and satisfaction in a specified context of use. This can be only achieved in a realistic system environment.
User quality needs can be specified as quality requirements by quality in use metrics, by external metrics, and sometimes by internal metrics. These requirements specified by metrics should be used as criteria when a product is evaluated.
It is recommended to use internal metrics having a relationship as strong as possible with the target external metrics so that they can be used to predict the values of external metrics. However, it is often difficult to design a rigorous theoretical model that provides a strong relationship between internal metrics and external metrics. Therefore, a hypothetical model that may contain ambiguity may be designed and the extent of the relationship may be modelled statistically during the use of metrics.
Recommendations and requirements related to validity and reliability are given in ISO/IEC 9126-1, Clause A.4. Additional detailed considerations when using metrics are given in Annex A of this Technical Report.
ISO/IEC TR 9126-3:2003(E)
4 © ISO/IEC 2003 – All rights reserved
7 How to read and use the metrics tables
The metrics listed in Clause 8 are categorized by the characteristics and subcharacteristics in ISO/IEC 9126-1. The following information is given for each metric in the table:
a) Metric name: Corresponding metrics in the internal metrics table and external metrics table have similar names.
b) Purpose of the metric: This is expressed as the question to be answered by the application of the metric.
c) Method of application: Provides an outline of the application.
d) Measurement, formula and data element computations: Provides the measurement formula and explains the meanings of the used data elements.
NOTE In some situations more than one formula is proposed for a metric.
e) Interpretation of measured value: Provides the range and preferred values.
f) Metric scale type: Type of scale used by the metric. Scale types used are; Nominal scale, Ordinal scale, Interval scale, Ratio scale and Absolute scale.
NOTE A more detailed explanation is given in Annex C.
g) Measure type: Types used are; Size type (e.g. Function size, Source size), Time type (e.g. Elapsed time, User time), Count type (e.g. Number of changes, Number of failures).
NOTE A more detailed explanation is given in Annex C.
h) Input to measurement: Source of data used in the measurement.
i) ISO/IEC 12207 SLCP Reference: Identifies software life cycle process(es) where the metric is applicable.
j) Target audience: Identifies the user(s) of the measurement results.
8 Metrics tables
The metrics listed in this clause are not intended to be an exhaustive set and may not have been validated. They are listed by software quality characteristics and subcharacteristics, in the order introduced in ISO/IEC 9126-1.
Metrics, which may be applicable, are not limited to these listed here. Additional specific metrics for particular purposes are provided in other related documents, such as functional size measurement or precise time efficiency measurement.
NOTE 1 It is recommended to refer a specific metric or measurement form from specific standards, technical reports or guidelines. Functional size measurement is defined in ISO/IEC 14143. An example of precise time efficiency measurement can be referred from ISO/IEC 14756.
Metrics should be validated before application in a specific environment (see Annex A).
NOTE 2 This list of metrics is not finalized, and may be revised in future versions of this Technical Report. Readers of this Technical Report are invited to provide feedback.
8.1 Functionality metrics
Internal functionality metrics are used for predicting if the software product in question will satisfy prescribed functional requirements and implied user needs.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 5
8.1.1 Suitability metrics
Internal suitability metrics indicate a set of attributes for assessing explicitly functions to prescribed tasks, and for determining their adequacy for performing the tasks.
8.1.2 Accuracy metrics
Internal accuracy metrics indicate a set of attributes for assessing the capability of the software product to achieve correct or agreeable results.
8.1.3 Interoperability metrics
Internal Interoperability metrics indicate a set of attributes for assessing the capability of the software product’s interaction with designated systems.
8.1.4 Security metrics
Internal security metrics indicate a set of attributes for assessing the capability of the software product to avoid illegal access to the system and/or data.
8.1.5 Functionality compliance metrics
Internal compliance metrics indicate a set of attributes for assessing the capability of the software product to comply to such items as standards, conventions or regulations of the user organization in relation to functionality.
ISO/IEC TR 9126-3:2003(E)
6 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
1.1
Suita
bilit
y m
etric
s In
tern
al s
uita
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Sour
ces
of
inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SL
CP
Ref
eren
ce
Targ
et
audi
ence
Func
tiona
l ad
equa
cy
How
ade
quat
e ar
e th
e ch
ecke
d fu
nctio
ns?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
func
tions
that
ar
e su
itabl
e fo
r per
form
ing
the
spec
ified
task
s, th
en
mea
sure
the
ratio
of i
t to
func
tions
impl
emen
ted.
Th
e fo
llowi
ng m
ay b
e m
easu
red;
-a
ll or p
arts
of d
esig
n sp
ecific
atio
ns
-com
plet
ed m
odul
es/p
arts
of
sof
twar
e pr
oduc
ts
X=1-
A/B
A=
Num
ber o
f fun
ctio
ns in
whi
ch p
robl
ems
are
dete
cted
in e
valu
atio
n
B= N
umbe
r of f
unct
ions
che
cked
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
mor
e ad
equa
te.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n So
urce
cod
e R
evie
w re
port
6.5
Valid
atio
n 6.
6
Join
t rev
iew
Req
uire
rs
Dev
elop
ers
Func
tiona
l im
plem
enta
tion
com
plet
enes
s
How
com
plet
e is
the
func
tiona
l im
plem
enta
tion?
Cou
nt th
e nu
mbe
r of
miss
ing
func
tions
det
ecte
d in
eva
luat
ion
and
com
pare
wi
th th
e nu
mbe
r of f
unct
ion
desc
ribed
in th
e re
quire
men
t spe
cifica
tions
.
X=1-
A/B
A=N
umbe
r of m
issin
g fu
nctio
ns d
etec
ted
in
eval
uatio
n.
B=N
umbe
r of f
unct
ions
des
crib
ed in
re
quire
men
t spe
cifica
tions
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
mor
e co
mpl
ete.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n So
urce
cod
e R
evie
w re
port
6.5
Valid
atio
n 6.
6 J
oint
revie
w
Req
uire
rs
Dev
elop
ers
FOO
TNO
TE
Inpu
t to
the
mea
sure
men
t pro
cess
is th
e up
date
d re
quire
men
t spe
cifica
tions
. Any
cha
nges
iden
tified
dur
ing
life c
ycle
mus
t be
appl
ied
to th
e re
quire
men
t spe
cifica
tions
bef
ore
usin
g in
mea
sure
men
t pr
oces
s.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 7
Tabl
e 8.
1.1
(con
tinue
d)
Inte
rnal
sui
tabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Sour
ces
of
inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SL
CP
Ref
eren
ce
Targ
et
audi
ence
Func
tiona
l im
plem
enta
tion
cove
rage
How
cor
rect
is th
e fu
nctio
nal
impl
emen
tatio
n?
Cou
nt th
e nu
mbe
r of
inco
rrect
ly im
plem
ente
d or
m
issin
g fu
nctio
ns a
nd
com
pare
with
the
num
ber o
f fu
nctio
ns d
escr
ibed
in th
e re
quire
men
t spe
cifica
tions
.
X=1-
A/B
A=
Num
ber o
f inc
orre
ctly
impl
emen
ted
or
miss
ing
func
tions
det
ecte
d B=
Num
ber o
f fun
ctio
ns d
escr
ibed
in
requ
irem
ent s
pecif
icatio
ns
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
mor
e co
rrect
.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n So
urce
cod
e R
evie
w re
port
6.5
Va
lidat
ion
6.6
Join
t rev
iew
Req
uire
rs
Dev
elop
ers
FOO
TNO
TES
1 R
evie
w b
y fu
nctio
nal it
em.
2 In
put t
o th
e m
easu
rem
ent p
roce
ss is
the
upda
ted
requ
irem
ent s
pecif
icatio
ns. A
ny c
hang
es id
entif
ied
durin
g life
cyc
le m
ust b
e ap
plie
d to
the
requ
irem
ent s
pecif
icatio
ns b
efor
e us
ing
in m
easu
rem
ent
proc
ess.
Fu
nctio
nal
spec
ifica
tion
stab
ility
(v
olat
ility
)
How
sta
ble
is th
e fu
nctio
nal s
pecif
icatio
n du
ring
the
deve
lopm
ent l
ife
cycle
?
Cou
nt th
e nu
mbe
r of
func
tions
cha
nged
(add
ed,
mod
ified,
or d
elet
ed) d
urin
g de
velo
pmen
t life
cyc
le
phas
e, th
en c
ompa
re w
ith
the
num
ber o
f fun
ctio
ns
desc
ribed
in th
e re
quire
men
t spe
cifica
tions
.
X=1-
A/B
A=N
umbe
r of f
unct
ions
cha
nged
dur
ing
deve
lopm
ent l
ife c
ycle
pha
ses
B=N
umbe
r of
func
tions
des
crib
ed in
re
quire
men
t spe
cifica
tions
0 <=
X <
= 1
The
close
r to
1 th
e m
ore
stab
le.
abso
lute
A=C
ount
B=
Cou
nt
X=C
ount
/C
ount
Req
uire
men
t sp
ecific
atio
ns
Rev
iew
repo
rt
6.5
Va
lidat
ion
6.3
Qua
lity
Assu
ranc
e 5.
3 Q
ualifi
catio
n te
stin
g 6.
8 Pr
oble
m
Res
olut
ion
5.4
Ope
ratio
n
Dev
elop
ers
Mai
ntai
ners
ISO/IEC TR 9126-3:2003(E)
8 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
1.2
Accu
racy
met
rics
Inte
rnal
acc
urac
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Com
puta
tiona
l Ac
cura
cy
How
com
plet
ely
have
th
e ac
cura
cy
requ
irem
ents
bee
n im
plem
ente
d?
Cou
nt th
e nu
mbe
r of
func
tions
that
hav
e im
plem
ente
d th
e ac
cura
cy
requ
irem
ents
and
com
pare
wi
th th
e nu
mbe
r of
func
tions
with
spe
cific
accu
racy
requ
irem
ents
.
X=A/
B
A= N
umbe
r of f
unct
ions
in w
hich
spe
cific
accu
racy
requ
irem
ents
had
bee
n im
plem
ente
d, a
s co
nfirm
ed in
eva
luat
ion
B= N
umbe
r of f
unct
ions
for w
hich
spe
cific
accu
racy
requ
irem
ents
nee
d to
be
impl
emen
ted
0 <=
X <
= 1.
Th
e clo
ser t
o 1,
the
mor
e co
mpl
ete.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
uire
men
t sp
ecific
atio
n D
esig
n So
urce
cod
e R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
Prec
isio
n H
ow c
ompl
ete
was
the
impl
emen
tatio
n of
sp
ecific
leve
ls of
pr
ecisi
on fo
r the
dat
a ite
ms?
Cou
nt th
e nu
mbe
r of d
ata
item
s th
at m
eet t
he
requ
irem
ents
of s
pecif
ic le
vels
of p
recis
ion
and
com
pare
to th
e to
tal
num
ber o
f dat
a ite
ms
with
sp
ecific
leve
l of p
recis
ion
requ
irem
ents
.
X=A/
B
A= N
umbe
r of d
ata
item
s im
plem
ente
d wi
th
spec
ific le
vels
of p
recis
ion,
con
firm
ed in
ev
alua
tion
B= N
umbe
r of d
ata
item
s th
at re
quire
spe
cific
leve
ls of
pre
cisio
n
0 <=
X <
= 1.
Th
e clo
ser t
o 1,
the
mor
e co
mpl
ete.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
uire
men
t sp
ecific
atio
n D
esig
n So
urce
cod
e R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 9
Tabl
e 8.
1.3
Inte
rope
rabi
lity
met
rics
Inte
rnal
inte
rope
rabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Dat
a ex
chan
geab
ility
(Dat
a fo
rmat
ba
sed)
How
cor
rect
ly ha
ve
the
inte
rface
dat
a fo
rmat
s be
en
impl
emen
ted?
Cou
nt th
e nu
mbe
r of
inte
rface
dat
a fo
rmat
s th
at
have
bee
n im
plem
ente
d co
rrect
ly as
in th
e sp
ecific
atio
ns a
nd c
ompa
re
to th
e nu
mbe
r of d
ata
form
ats
to b
e ex
chan
ged
as in
the
spec
ificat
ions
.
X=A/
B
A=N
umbe
r of i
nter
face
dat
a fo
rmat
s th
at h
ave
been
impl
emen
ted
corre
ctly
as in
the
spec
ificat
ions
B=
Num
ber o
f dat
a fo
rmat
s to
be
exch
ange
d as
in th
e sp
ecific
atio
ns
0 <=
X <
= 1.
T
he c
lose
r to
1, th
e m
ore
corre
ct.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n So
urce
cod
e R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s R
equi
rers
Inte
rface
co
nsis
tenc
y (p
roto
col)
How
cor
rect
ly ha
ve
the
inte
rface
pro
toco
ls be
en im
plem
ente
d?
Cou
nt th
e nu
mbe
r of
inte
rface
pro
toco
ls th
at
were
impl
emen
ted
corre
ctly
as in
the
spec
ificat
ions
and
co
mpa
re w
ith th
e nu
mbe
r of
inte
rface
pro
toco
ls to
be
impl
emen
ted
as in
the
spec
ificat
ions
.
X=A/
B
A=N
umbe
r of i
nter
face
pro
toco
ls im
plem
entin
g co
nsist
ent f
orm
at a
s in
the
spec
ificat
ion
conf
irmed
in re
view
B=
Num
ber o
f int
erfa
ce p
roto
cols
to b
e im
plem
ente
d as
in th
e sp
ecific
atio
ns
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
mor
e co
nsist
ent.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n So
urce
cod
e R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s R
equi
rers
ISO/IEC TR 9126-3:2003(E)
10 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
1.4
Secu
rity
met
rics
Inte
rnal
sec
urity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Acce
ss
audi
tabi
lity
How
aud
itabl
e is
acce
ss lo
gin?
C
ount
the
num
ber o
f ac
cess
type
s th
at a
re b
eing
lo
gged
cor
rect
ly as
in th
e sp
ecific
atio
ns a
nd c
ompa
re
with
the
num
ber o
f acc
ess
type
s th
at a
re re
quire
d to
be
logg
ed in
the
spec
ificat
ions
.
X=A/
B
A= N
umbe
r of a
cces
s ty
pes
that
are
bei
ng
logg
ed a
s in
the
spec
ificat
ions
B=
Num
ber o
f acc
ess
type
s re
quire
d to
be
logg
ed in
the
spec
ificat
ions
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
mor
e au
dita
ble.
Abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
uire
men
t sp
ecific
atio
n D
esig
n So
urce
cod
e R
evie
w re
port
6.5
Valid
atio
n 6.
6 Jo
int r
evie
w
Req
uire
rs
Dev
elop
ers
Acce
ss
cont
rolla
bilit
y H
ow c
ontro
llabl
e is
acce
ss to
the
syst
em?
Cou
nt th
e nu
mbe
r of
acce
ss c
ontro
llabi
lity
requ
irem
ents
impl
emen
ted
corre
ctly
as in
the
spec
ificat
ions
and
com
pare
wi
th th
e nu
mbe
r of a
cces
s co
ntro
llabi
lity re
quire
men
ts
in th
e sp
ecific
atio
ns.
X=A/
B A=
Num
ber o
f acc
ess
cont
rolla
bility
re
quire
men
ts im
plem
ente
d co
rrect
ly as
in th
e sp
ecific
atio
ns
B= N
umbe
r of a
cces
s co
ntro
llabi
lity
requ
irem
ents
in th
e sp
ecific
atio
ns
0 <=
X <
= 1
T
he c
lose
r to
1, th
e m
ore
cont
rolla
ble.
Abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
uire
men
t sp
ecific
atio
n D
esig
n So
urce
cod
e R
evie
w re
port
6.5
Va
lidat
ion
6.6
Join
t rev
iew
Req
uire
rs
Dev
elop
ers
Dat
a co
rrup
tion
prev
entio
n
How
com
plet
e is
the
impl
emen
tatio
n of
dat
a co
rrupt
ion
prev
entio
n? C
ount
the
num
ber o
f im
plem
ente
d in
stan
ces
of
data
cor
rupt
ion
prev
entio
n as
spe
cifie
d an
d co
mpa
re
with
the
num
ber o
f in
stan
ces
of o
pera
tions
/ ac
cess
spe
cifie
d in
re
quire
men
ts a
s ca
pabl
e of
co
rrupt
ing/
des
troyin
g da
ta.
X=A/
B
A= N
umbe
r of im
plem
ente
d in
stan
ces
of d
ata
corru
ptio
n pr
even
tion
as s
pecif
ied
conf
irmed
in
revie
w B=
Num
ber o
f ins
tanc
es o
f ope
ratio
n/ac
cess
id
entif
ied
in re
quire
men
ts a
s ca
pabl
e of
co
rrupt
ing/
dest
royin
g da
ta
0 <=
X <
= 1
T
he c
lose
r to
1, th
e m
ore
com
plet
e.
Abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
uire
men
t sp
ecific
atio
n D
esig
n So
urce
cod
e R
evie
w re
port
6.5
Va
lidat
ion
6.6
Jo
int r
evie
w
Dev
elop
ers
FOO
TNO
TE
Con
sider
sec
urity
leve
ls w
hen
usin
g th
is m
etric
.
Data
enc
rypt
ion
How
com
plet
e is
the
impl
emen
tatio
n of
dat
a en
cryp
tion?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
inst
ance
s of
en
cryp
tabl
e/de
cryp
tabl
e da
ta it
ems
as s
pecif
ied
and
com
pare
with
the
num
ber o
f in
stan
ces
of d
ata
item
s re
quiri
ng d
ata
encr
yptio
n/de
cryp
tion
facil
ity a
s in
spe
cifica
tions
.
X=A/
B
A=N
umbe
r of im
plem
ente
d in
stan
ces
of
encr
ypta
ble/
decr
ypta
ble
data
item
s as
sp
ecifie
d co
nfirm
ed in
revie
w
B= N
umbe
r of d
ata
item
s re
quirin
g da
ta
encr
yptio
n/de
cryp
tion
facil
ity a
s in
sp
ecific
atio
ns
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
mor
e co
mpl
ete.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
uire
men
t sp
ecific
atio
n D
esig
n So
urce
cod
e R
evie
w re
port
6.5
Valid
atio
n D
evel
oper
s
FOO
TNO
TE
Dat
a en
cryp
tion:
e.g
., da
ta in
ope
n da
taba
se, d
ata
in p
ublic
com
mun
icatio
n fa
cility
.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 11
Tabl
e 8.
1.5
Func
tiona
lity
com
plia
nce
met
rics
In
tern
al fu
nctio
nalit
y co
mpl
ianc
e m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Func
tiona
l co
mpl
ianc
e H
ow c
ompl
iant
is th
e fu
nctio
nality
of t
he
prod
uct t
o ap
plica
ble
regu
latio
ns, s
tand
ards
an
d co
nven
tions
?
Cou
nt th
e nu
mbe
r of i
tem
s re
quiri
ng c
ompl
ianc
e th
at
have
bee
n m
et a
nd
com
pare
with
the
num
ber o
f ite
ms
requ
iring
com
plia
nce
as in
the
spec
ificat
ion.
X=A/
B
A= N
umbe
r of c
orre
ctly
impl
emen
ted
item
s re
late
d to
func
tiona
lity c
ompl
ianc
e co
nfirm
ed
in e
valu
atio
n B=
Tot
al n
umbe
r of c
ompl
ianc
e ite
ms
0 <=
X <
= 1.
Th
e clo
ser t
o 1,
the
mor
e co
mpl
iant
.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Spec
ificat
ion
of c
ompl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
or
regu
latio
ns.
Des
ign
Sour
ce c
ode
Rev
iew
repo
rt
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
Inte
rsys
tem
st
anda
rd
com
plia
nce
How
com
plia
nt a
re th
e in
terfa
ces
to
appl
icabl
e re
gula
tions
, st
anda
rds
and
conv
entio
ns?
Cou
nt th
e nu
mbe
r of
inte
rface
s th
at m
eet
requ
ired
com
plia
nce
and
com
pare
with
the
num
ber o
f in
terfa
ces
requ
iring
co
mpl
ianc
e as
in th
e sp
ecific
atio
ns.
X=A/
B
A= N
umbe
r of c
orre
ctly
impl
emen
ted
inte
rface
s as
spe
cifie
d, c
onfir
med
in re
view
B= T
otal
num
ber o
f int
erfa
ces
requ
iring
com
plia
nce
0 <=
X <
= 1.
Th
e clo
ser t
o 1,
the
mor
e co
mpl
iant
.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n So
urce
cod
e R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s R
equi
rers
FOO
TNO
TE
All s
pecif
ied
attri
bute
s of
a s
tand
ard
mus
t be
chec
ked.
ISO/IEC TR 9126-3:2003(E)
12 © ISO/IEC 2003 – All rights reserved
8.2 Reliability metrics
Internal reliability metrics are used for predicting if the software product in question will satisfy prescribed reliability needs, during the development of the software product.
8.2.1 Maturity metrics
Internal maturity metrics indicate a set of attributes for assessing the maturity of the software.
8.2.2 Fault tolerance metrics
Internal fault tolerance metrics indicate a set of attributes for assessing the software products capability in maintaining a desired performance level in case of operational faults or infringement of its specified interface.
8.2.3 Recoverability metrics
Internal recoverability metrics indicate a set of attributes for assessing the software product’s capability to re-establish an adequate level of performance and recover the data directly affected in case of a failure.
8.2.4 Reliability compliance metrics
Internal compliance metrics relating to reliability indicate a set of attributes for assessing the capability of the software product to comply to such items as standards, conventions or regulations of the user organization in relation to reliability.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 13
Tabl
e 8.
2.1
Mat
urity
met
rics
Inte
rnal
mat
urity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Faul
t det
ectio
n
How
man
y fa
ults
wer
e de
tect
ed in
revie
wed
prod
uct?
Cou
nt th
e nu
mbe
r of
dete
cted
faul
ts in
revie
w an
d co
mpa
re it
to th
e nu
mbe
r of e
stim
ated
faul
ts
to b
e de
tect
ed in
this
phas
e.
X=A/
B
A=Ab
solu
te n
umbe
r of f
aults
det
ecte
d in
re
view
B=
Num
ber o
f est
imat
ed fa
ults
to b
e de
tect
ed
in re
view
(usin
g pa
st h
istor
y or
refe
renc
e m
odel
)
0 <=
X
A hi
gh v
alue
fo
r X im
plie
s go
od p
rodu
ct
qual
ity, w
hile
A=
0 do
es n
ot
nece
ssar
ily
impl
y fa
ult f
ree
stat
us o
f the
re
viewe
d ite
m.
Abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Valu
e A
com
es fr
om
revie
w re
port
Valu
e B
com
es fr
om
the
orga
niza
tion
data
base
.
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
FOO
TNO
TES
1 Th
is m
etric
sho
uld
only
be u
sed
for p
redi
ctio
n du
ring
deve
lopm
ent.
2 It
is ne
cess
ary
to c
onve
rt th
is va
lue(
X) to
the
<0,1
> in
terv
al if
mak
ing
sum
mar
izatio
n of
cha
ract
erist
ics.
Faul
t rem
oval
H
ow m
any
faul
ts h
ave
been
cor
rect
ed?
Wha
t is
the
prop
ortio
n of
faul
ts re
mov
ed?
Cou
nt th
e nu
mbe
r of f
aults
re
mov
ed d
urin
g de
sign/
codi
ng a
nd c
ompa
re
it to
the
num
ber o
f fau
lts
dete
cted
in re
view
durin
g de
sign/
codi
ng.
X=A
A=
Num
ber o
f cor
rect
ed fa
ults
in d
esig
n/co
ding
Y=
A/B
A=N
umbe
r of c
orre
cted
faul
ts d
esig
n/co
ding
B=
Num
ber o
f fau
lts d
etec
ted
in re
view
0 <=
X
A hi
gh v
alue
of
X im
plie
s, th
at
less
faul
ts
rem
ain.
0
<= Y
<=
1 Th
e clo
ser t
o 1,
the
bette
r. (m
ore
faul
ts
rem
oved
)
ratio
ab
solu
te
X=co
unt
A=co
unt
Y=co
unt/
coun
t B=
coun
t
Valu
e A
com
es fr
om
faul
t rem
oval
re
port.
Va
lue
B co
mes
from
re
view
repo
rt.
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
FOO
TNO
TE
It is
nece
ssar
y to
con
vert
this
valu
e (X
) to
the
<0,1
> in
terv
al if
mak
ing
sum
mar
izatio
n of
cha
ract
erist
ics.
Test
ade
quac
y H
ow m
uch
of th
e re
quire
d te
st c
ases
ar
e co
vere
d by
the
test
pla
n?
Cou
nt th
e nu
mbe
r of
test
ca
ses
plan
ned
and
com
pare
it to
the
num
ber o
f te
st c
ases
requ
ired
to
obta
in a
dequ
ate
test
co
vera
ge.
X=A/
B
A=N
umbe
r of t
est c
ases
des
igne
d in
test
pla
n an
d co
nfirm
ed in
revie
w B=
Num
ber o
f tes
t cas
es re
quire
d
0 <=
X
Whe
re X
is
grea
ter t
he
bette
r ad
equa
cy
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Valu
e A
com
es fr
om
test
pla
n Va
lue
B co
mes
from
re
quire
men
ts
QA
Prob
lem
re
solu
tion
Verif
icatio
n
Dev
elop
ers
Mai
ntai
ners
ISO/IEC TR 9126-3:2003(E)
14 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
2.2
Faul
t tol
eran
ce m
etric
s In
tern
al fa
ult t
oler
ance
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Failu
re
avoi
danc
e H
ow m
any
faul
t pa
ttern
s we
re b
roug
ht
unde
r con
trol t
o av
oid
critic
al a
nd s
erio
us
failu
res?
Cou
nt th
e nu
mbe
r of
avoi
ded
faul
t pat
tern
s an
d co
mpa
re it
to th
e nu
mbe
r of
faul
t pat
tern
s to
be
cons
ider
ed.
X=A/
B
A=N
umbe
r of f
ault
patte
rns
havin
g av
oida
nce
in d
esig
n/co
de
B=N
umbe
r of f
ault
patte
rns
to b
e co
nsid
ered
CO
MM
ENT(
S) F
ault
patte
rn e
xam
ples
out
of
rang
e da
ta d
eadl
ock.
CO
MM
ENT(
S) F
ault
tree
anal
ysis
tech
niqu
e m
ay b
e us
ed to
det
ect f
ault
patte
rns.
0 <=
X
Whe
re X
is
grea
ter t
he
bette
r fai
lure
av
oida
nce.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Valu
e A
com
es fr
om
revie
w re
port
Valu
e B
com
es fr
om
requ
irem
ent
spec
ificat
ion
docu
men
t.
Verif
icatio
n Va
lidat
ion
Join
t rev
iew
Prob
lem
re
solu
tion
Dev
elop
ers
Req
uire
rs
Mai
ntai
ners
Inco
rrec
t op
erat
ion
avoi
danc
e
How
man
y fu
nctio
ns
are
impl
emen
ted
with
in
corre
ct o
pera
tions
av
oida
nce
capa
bility
?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
func
tions
to
avoi
d cr
itical
and
ser
ious
fa
ilure
s ca
used
by
inco
rrect
op
erat
ions
and
com
pare
it to
the
num
ber o
f inc
orre
ct
oper
atio
n pa
ttern
s to
be
cons
ider
ed.
COM
MEN
T(S)
Also
dat
a da
mag
e in
add
ition
to
syst
em fa
ilure
.
X=A/
B
A=N
umbe
r of f
unct
ions
impl
emen
ted
to a
void
in
corre
ct o
pera
tion
patte
rns
B=N
umbe
r of i
ncor
rect
ope
ratio
n p
atte
rns
to
be c
onsid
ered
CO
MM
ENT(
S) In
corre
ct o
pera
tion
patte
rns
Inco
rrect
dat
a ty
pes
as p
aram
eter
s In
corre
ct s
eque
nce
of d
ata
inpu
t In
corre
ct s
eque
nce
of o
pera
tion.
CO
MM
ENT(
S) F
ault
tree
anal
ysis
tech
niqu
e m
ay b
e us
ed to
det
ect i
ncor
rect
ope
ratio
n pa
ttern
s.
0 <=
X
Whe
re X
is
grea
ter t
he
bette
rinc
orre
ct
oper
atio
n av
oida
nce.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Valu
e A
com
es fr
om
revie
w re
port
Valu
e B
com
es fr
om
requ
irem
ent
spec
ificat
ion
docu
men
t.
Verif
icatio
n Va
lidat
ion
Join
t rev
iew
Prob
lem
re
solu
tion
Dev
elop
ers
Req
uire
rs
Mai
ntai
ners
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 15
Tabl
e 8.
2.3
Rec
over
abili
ty m
etric
s In
tern
al re
cove
rabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Res
tora
bilit
y H
ow c
apab
le is
the
prod
uct i
n re
stor
ing
itsel
f afte
r abn
orm
al
even
t or a
t req
uest
?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
rest
orat
ion
requ
irem
ents
and
com
pare
it
to th
e nu
mbe
r of
rest
orat
ion
requ
irem
ents
in
the
spec
ificat
ions
. R
esto
ratio
n re
quire
men
t ex
ampl
es: d
atab
ase
chec
kpoi
nt, t
rans
actio
n ch
eckp
oint
, red
o fu
nctio
n,
undo
func
tion.
X=A/
B
A=N
umbe
r of im
plem
ente
d re
stor
atio
n re
quire
men
ts c
onfir
med
in re
view
B=
Num
ber o
f res
tora
tion
requ
irem
ents
in th
e sp
ecific
atio
ns
0 <=
X <
= 1
Whe
re X
is
grea
ter,
the
bette
r re
stor
abilit
y
Abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
A co
mes
from
re
view
docu
men
t B
com
es fr
om
requ
irem
ents
or
des
ign
docu
men
t
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s M
aint
aine
rs
Res
tora
tion
Effe
ctiv
enes
s H
ow e
ffect
ive is
the
rest
orat
ion
capa
bility
? C
ount
the
num
ber o
f im
plem
ente
d re
stor
atio
n re
quire
men
ts m
etin
g ta
rget
re
stor
atio
n tim
e (b
y ca
lcula
tions
or s
imul
atio
ns)
and
com
pare
it to
the
num
ber o
f res
tora
tion
requ
irem
ents
with
spe
cifie
d ta
rget
tim
e.
X=A/
B
A=N
umbe
r of im
plem
ente
d re
stor
atio
n re
quire
men
ts m
eetin
g ta
rget
rest
ore
time
B=
Num
ber o
f res
tora
tion
requ
irem
ents
with
sp
ecifie
d ta
rget
times
0 <=
X <
= 1
Whe
re X
is
grea
ter,
the
bette
r ef
fect
ivene
ss
Abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
A co
mes
from
re
view
docu
men
t B
com
es fr
om
requ
irem
ents
or
des
ign
docu
men
t
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s M
aint
aine
rs
ISO/IEC TR 9126-3:2003(E)
16 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
2.4
Rel
iabi
lity
com
plia
nce
met
rics
In
tern
al re
liabi
lity
com
plia
nce
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Rel
iabi
lity
co
mpl
ianc
e H
ow c
ompl
iant
is th
e re
liabi
lity o
f the
pr
oduc
t to
appl
icabl
e re
gula
tions
, sta
ndar
ds
and
conv
entio
ns?
Cou
nt th
e nu
mbe
r of i
tem
s re
quiri
ng c
ompl
ianc
e th
at
have
bee
n m
et a
nd
com
pare
with
the
num
ber o
f ite
ms
requ
iring
com
plia
nce
as in
the
spec
ificat
ion.
X=A/
B
A= N
umbe
r of c
orre
ctly
impl
emen
ted
item
s re
late
d to
relia
bility
com
plia
nce
conf
irmed
in
eval
uatio
n B=
Tot
al n
umbe
r of c
ompl
ianc
e ite
ms
0 <=
X <
= 1.
Th
e clo
ser t
o 1,
the
mor
e co
mpl
iant
.
Abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Spec
ificat
ion
of c
ompl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
or
regu
latio
ns.
Des
ign
Sour
ce c
ode
Rev
iew
repo
rt
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 17
8.3 Usability Metrics
Internal usability metrics are used for predicting the extent to which the software in question can be understood, learned, operated, attractive and compliant with usability regulations and guidelines.
NOTE It should be possible for the measures taken to be used to establish acceptance criteria or to make comparisons between products. This means that the measures should be counting items of known value. Results should report the mean value and the standard error of the mean
8.3.1 Understandability metrics
Users should be able to select a software product which is suitable for their intended use. Internal understandability metrics assess whether new users can understand:
• whether the software is suitable
• how it can be used for particular tasks.
8.3.2 Learnability metrics
Internal learnability metrics assess how long users take to learn how to use particular functions, and the effectiveness of help systems and documentation.
Learnability is strongly related to understandability, and understandability measurements can be indicators of the learnability potential of the software.
8.3.3 Operability metrics
Internal operability metrics assess whether users can operate and control the software. Operability metrics can be categorized by the dialogue principles in ISO 9241-10:
• suitability of the software for the task
• self-descriptiveness of the software
• controllability of the software
• conformity of the software with user expectations
• error tolerance of the software
• suitability of the software for individualization
The choice of functions to test will be influenced by the expected frequency of use of functions, the criticality of the functions, and any anticipated usability problems.
8.3.4 Attractiveness metrics
Internal attractiveness metrics assess the appearance of the software, and will be influenced by factors such as screen design and colour. This is particularly important for consumer products.
8.3.5 Usability compliance metrics
Internal compliance metrics assess adherence to standards, conventions, style guides or regulations relating to usability.
ISO/IEC TR 9126-3:2003(E)
18 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
3.1
Und
erst
anda
bilit
y m
etric
s In
tern
al u
nder
stan
dabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Com
plet
enes
s of
de
scrip
tion
Wha
t pro
porti
on o
f fu
nctio
ns (o
r typ
es o
f fu
nctio
n) a
re
desc
ribed
in th
e pr
oduc
t des
crip
tion?
Cou
nt th
e nu
mbe
r of
func
tions
whi
ch a
re
adeq
uate
ly de
scrib
ed a
nd
com
pare
with
the
tota
l nu
mbe
r of f
unct
ions
in th
e pr
oduc
t.
X= A
/B
A= N
umbe
r of f
unct
ions
(or t
ypes
of
func
tions
) des
crib
ed in
the
prod
uct
desc
riptio
n B=
Tot
al n
umbe
r of f
unct
ions
(or t
ypes
of
func
tions
)
0<=X
<=1
The
close
r to
1 th
e m
ore
com
plet
e.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
FOO
TNO
TES
1 Th
is in
dica
tes
whe
ther
pot
entia
l use
rs w
ill un
ders
tand
the
capa
bility
of t
he p
rodu
ct a
fter r
eadi
ng th
e pr
oduc
t des
crip
tion.
2
See
also
ISO
/IEC
912
7 C
onsu
mer
sof
twar
e pa
ckag
e.
Dem
onst
ratio
n ca
pabl
ity
Wha
t pro
porti
on o
f fu
nctio
ns re
quiri
ng
dem
onst
ratio
n ha
ve
dem
onst
ratio
n ca
pabi
lity?
Cou
nt th
e nu
mbe
r of
func
tions
that
are
ad
equa
tely
dem
onst
rabl
e an
d co
mpa
re w
ith th
e to
tal
num
ber o
f fun
ctio
ns
requ
iring
dem
onst
ratio
n ca
pabi
lity.
X=A/
B
A= N
umbe
r of f
unct
ions
dem
onst
rate
d an
d co
nfirm
ed in
revie
w B=
Tot
al n
umbe
r of f
unct
ions
requ
iring
de
mon
stra
tion
capa
bility
0<=X
<=1
The
close
r to
1 th
e m
ore
capa
ble.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
FOO
TNO
TE
Dem
onst
ratio
ns s
tep
thro
ugh
the
proc
ess
show
ing
how
the
prod
uct is
use
d. T
his
inclu
des
“wiza
rds”
. Ev
iden
t fun
ctio
nsW
hat p
ropo
rtion
of
the
prod
uct f
unct
ions
ar
e ev
iden
t to
the
user
?
Cou
nt th
e nu
mbe
r of
func
tions
that
are
evid
ent
to th
e us
er a
nd c
ompa
re
with
the
tota
l num
ber o
f fu
nctio
ns.
X= A
/B
A= N
umbe
r of f
unct
ions
(or t
ypes
of
func
tions
) evid
ent t
o th
e us
er
B= T
otal
num
ber o
f fun
ctio
ns (o
r typ
es o
f fu
nctio
ns)
0<=X
<=1
The
close
r to
1 th
e be
tter.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
FOO
TNO
TE
This
indi
cate
s w
heth
er u
sers
will
be a
ble
to lo
cate
func
tions
by
expl
orin
g th
e in
terfa
ce (e
.g. b
y in
spec
ting
the
men
us).
Func
tion
unde
rsta
ndab
ility
Wha
t pro
porti
on o
f th
e pr
oduc
t fun
ctio
ns
will t
he u
ser b
e ab
le
to u
nder
stan
d co
rrect
ly?
Cou
nt th
e nu
mbe
r of
user
inte
rface
func
tions
w
here
pur
pose
s is
un
ders
tood
by
the
user
an
d co
mpa
re w
ith th
e nu
mbe
r of u
ser i
nter
face
fu
nctio
ns.
X= A
/B
A= N
umbe
r of u
ser i
nter
face
func
tions
w
hose
pur
pose
is u
nder
stoo
d by
the
user
B= N
umbe
r of u
ser i
nter
face
func
tions
0 <=
X <
= 1
The
clos
er to
1,
the
bette
r.
abso
lute
X=co
unt/
coun
t
A=co
unt
B=co
unt
Req
spe
c
Des
ign
Rev
iew
re
port
Verif
icat
ion
Join
t rev
iew
Req
uire
rs
Dev
elop
ers
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 19
Tabl
e 8.
3.2
Lear
nabi
lity
met
rics
Inte
rnal
lear
nabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Com
plet
enes
s of
use
r do
cum
enta
tion
and/
or h
elp
faci
lity
Wha
t pro
porti
on o
f fu
nctio
ns a
re d
escr
ibed
in
the
user
do
cum
enta
tion
and/
or
help
facil
ity?
Cou
nt th
e nu
mbe
r of
func
tions
impl
emen
ted
with
he
lp fa
cility
and
/or
docu
men
tatio
n an
d co
mpa
re
with
the
tota
l num
ber o
f fu
nctio
ns in
pro
duct
.
X= A
/B
A= N
umbe
r of f
unct
ions
des
crib
ed
B= T
otal
of n
umbe
r of f
unct
ions
pro
vided
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e m
ore
com
plet
e.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
FOO
TNO
TE
Thre
e m
etric
s ar
e po
ssib
le: c
ompl
eten
ess
of th
e do
cum
enta
tion,
com
plet
enes
s of
the
help
facil
ity o
r com
plet
enes
s of
the
help
and
doc
umen
tatio
n us
ed in
com
bina
tion.
ISO/IEC TR 9126-3:2003(E)
20 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
3.3
Ope
rabi
lity
met
rics
Inte
rnal
Ope
rabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Inpu
t val
idity
ch
ecki
ng
Wha
t pro
porti
on o
f in
put i
tem
s pr
ovid
e ch
eck
for v
alid
dat
a?
Cou
nt th
e nu
mbe
r of i
nput
ite
ms,
whi
ch c
heck
for v
alid
da
ta a
nd c
ompa
re w
ith th
e nu
mbe
r of i
nput
item
s,
which
cou
ld c
heck
for v
alid
da
ta.
X=A/
B A=
Num
ber o
f inp
ut it
ems
which
che
ck fo
r va
lid d
ata
B=N
umbe
r of i
nput
item
s wh
ich c
ould
che
ck
for v
alid
dat
a
0 <=
X <
= 1
The
close
r to
1, th
e be
tter.
Abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s R
equi
rers
Use
r ope
ratio
n ca
ncel
labi
lity
Wha
t pro
porti
on o
f fu
nctio
ns c
an b
e ca
ncel
led
prio
r to
com
plet
ion?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
func
tions
, wh
ich c
an b
e ca
ncel
led
by
the
user
prio
r to
com
plet
ion
and
com
pare
it wi
th th
e nu
mbe
r of f
unct
ions
re
quiri
ng th
e pr
ecan
cella
tion
capa
bility
.
X=A/
B A=
Num
ber o
f impl
emen
ted
func
tions
whi
ch
can
be c
ance
lled
by th
e us
er
B= N
umbe
r of f
unct
ions
requ
iring
the
prec
ance
llatio
n ca
pabi
lity
0 <=
X <
= 1
The
close
r to
1, th
e be
tter
canc
ella
bility
.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s R
equi
rers
Use
r ope
ratio
n U
ndoa
bilit
y W
hat p
ropo
rtion
of
func
tions
can
be
undo
ne?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
func
tions
, wh
ich c
an b
e un
done
by
the
user
afte
r com
plet
ion
and
com
pare
it wi
th th
e nu
mbe
r of f
unct
ions
.
X=A/
B A=
Num
ber o
f impl
emen
ted
func
tions
whi
ch
can
be u
ndon
e by
the
user
B=
Num
ber o
f fun
ctio
ns
0 <=
X <
= 1
The
close
r to
1, th
e be
tter
undo
abilit
y.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s R
equi
rers
FOO
TNO
TE
Eith
er s
ingl
e un
doab
ility
or m
ultip
le u
ndoa
bility
afte
r sev
eral
sub
sequ
ent a
ctio
ns c
an b
e as
sess
ed.
Cus
tom
isab
ility
W
hat p
ropo
rtion
of
func
tions
can
be
cust
omise
d du
ring
oper
atio
n?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
func
tions
, wh
ich c
an b
e cu
stom
ized
by th
e us
er d
urin
g op
erat
ion
and
com
pare
it
with
the
num
ber o
f fu
nctio
ns re
quiri
ng th
e cu
stom
izatio
n ca
pabi
lity.
X=A/
B A=
Num
ber o
f fun
ctio
ns w
hich
can
be
cust
omise
d du
ring
oper
atio
n
B=N
umbe
r of f
unct
ions
requ
iring
the
cust
omiza
tion
capa
bility
0 <=
X <
= 1
The
close
r to
1,th
e be
tter
cust
omisa
bility
.abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s R
equi
rers
Phys
ical
ac
cess
ibili
ty
Wha
t pro
porti
on o
f fu
nctio
ns c
an b
e cu
stom
ised
for a
cces
s by
use
rs w
ith p
hysic
al
hand
icaps
?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
func
tions
, wh
ich c
an b
e cu
stom
ised
and
com
pare
it wi
th th
e nu
mbe
r of f
unct
ions
.
X=A/
B A=
Num
ber o
f fun
ctio
ns w
hich
can
be
cust
omise
d
B=N
umbe
r of f
unct
ions
0 <=
X <
= 1
The
close
r to
1, th
e be
tter
phys
ical
acce
ssib
ility.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s R
equi
rers
FOO
TNO
TE
Exam
ples
of p
hysic
al a
cces
sibilit
y ar
e in
abilit
y to
use
a m
ouse
and
blin
dnes
s.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 21
Tabl
e 8.
3.3
(con
tinue
d)
Inte
rnal
Ope
rabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Ope
ratio
n st
atus
m
onito
ring
capa
bilit
y
Wha
t pro
porti
on o
f fu
nctio
ns h
ave
oper
atio
ns s
tatu
s m
onito
ring
capa
bility
?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
func
tions
, wh
ich s
tatu
s ca
n be
m
onito
red
and
com
pare
it wi
th th
e nu
mbe
r of f
unct
ions
re
quiri
ng th
e m
onito
ring
capa
bility
.
X=A/
B A=
Num
ber o
f fun
ctio
ns h
avin
g st
atus
mon
itorin
g ca
pabi
lity
B=N
umbe
r of f
unct
ions
that
are
requ
ired
to
have
mon
itorin
g ca
pabi
lity
0 <=
X <
= 1
The
close
r to
1,
the
bette
r m
onito
ring
capa
bility
.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
wD
evel
oper
s R
equi
rers
FOO
TNO
TE
Stat
us in
clude
s pr
ogre
ss m
onito
ring.
O
pera
tiona
l co
nsis
tenc
y W
hat p
ropo
rtion
of
oper
atio
ns b
ehav
e th
e sa
me
way
to s
imila
r op
erat
ions
in o
ther
pa
rts o
f the
sys
tem
?
Cou
nt th
e nu
mbe
r of
inst
ance
s of
ope
ratio
ns w
ith
inco
nsist
ent b
ehav
iour
and
co
mpa
re it
with
the
tota
l nu
mbe
r of o
pera
tions
X=1
- A/B
A=
Num
ber o
f ins
tanc
es o
f ope
ratio
ns w
ith
inco
nsist
ent b
ehav
iour
B=
Tota
l num
ber o
f ope
ratio
ns
0 <=
X <
= 1
The
close
r to
1,
the
mor
e co
nsist
ent.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
wD
evel
oper
s R
equi
rers
Mes
sage
Cla
rity
Wha
t pro
porti
on o
f m
essa
ges
are
self-
expl
anat
ory?
Cou
nt th
e nu
mbe
rs o
f im
plem
ente
d m
essa
ges
with
cle
ar e
xpla
natio
ns
and
com
pare
it w
ith th
e to
tal n
umbe
r of m
essa
ges
impl
emen
ted.
X=A/
B
A=N
umbe
r of i
mpl
emen
ted
mes
sage
s w
ith
clea
r exp
lana
tions
B=N
umbe
r of m
essa
ges
impl
emen
ted
0 <=
X <
= 1
The
clos
er to
1,
the
mor
e cl
ear.
abso
lute
X=co
unt/
coun
t
A=co
unt
B=co
unt
Req
spe
c
Des
ign
Rev
iew
re
port
Verif
icat
ion
Join
t re
view
Dev
elop
ers
Req
uire
rs
FOO
TNO
TE
Cle
ar e
rror m
essa
ges
expl
ain
to th
e us
er w
hat a
ctio
n to
take
to re
cove
r fro
m th
e er
ror.
Inte
rfac
e el
emen
t cla
rity
Wha
t pro
porti
on o
f in
terfa
ce e
lem
ents
ar
e se
lf-ex
plan
ator
y?
Cou
nt th
e nu
mbe
r of
inte
rface
ele
men
ts w
hich
ar
e se
lf ex
plan
ator
y an
d co
mpa
re it
with
the
tota
l nu
mbe
r of i
nter
face
el
emen
ts
X=A/
B
A=N
umbe
r of i
nter
face
ele
men
ts w
hich
are
se
lf-ex
plan
ator
y
B=To
tal n
umbe
r of i
nter
face
ele
men
ts
0 <=
X <
= 1
The
clos
er to
1,
the
mor
e cl
ear.
abso
lute
X=co
unt/
coun
t
A=co
unt
B=co
unt
Req
spe
c
Des
ign
Rev
iew
re
port
Verif
icat
ion
Join
t re
view
Dev
elop
ers
Req
uire
rs
FOO
TNO
TE
Elem
ents
are
sel
f exp
lana
tory
whe
n th
ey u
se p
lain
text
or p
rovid
e “h
over
-hel
p” o
r “to
ol ti
ps”.
Ope
ratio
nal
erro
r re
cove
rabi
lity
Wha
t pro
porti
on o
f fu
nctio
ns c
an to
lera
te
user
erro
r?
Cou
nt th
e nu
mbe
r of
func
tions
impl
emen
ted
with
use
r erro
r tol
eran
ce
and
com
pare
it to
the
tota
l nu
mbe
r of f
unct
ions
re
quiri
ng th
e to
lera
nce
capa
bilit
y
X=A/
B
A=N
umbe
r of f
unct
ions
impl
emen
ted
with
us
er e
rror t
oler
ance
B=To
tal n
umbe
r of f
unct
ions
requ
iring
the
tole
ranc
e ca
pabi
lity
0 <=
X <
= 1
The
clos
er to
1,
the
mor
e re
cove
rabl
e.
abso
lute
X=co
unt/
coun
t
A=co
unt
B=co
unt
Req
spe
c
Des
ign
Rev
iew
re
port
Verif
icat
ion
Join
t re
view
Dev
elop
ers
Req
uire
rs
ISO/IEC TR 9126-3:2003(E)
22 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
3.4
Attr
activ
enes
s m
etric
s In
tern
al a
ttrac
tiven
ess
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Attr
activ
e in
tera
ctio
n H
ow a
ttrac
tive
is th
e in
terfa
ce to
the
user
? Q
uest
ionn
aire
to u
sers
Q
uest
ionn
aire
to a
sses
s th
e at
tract
ivene
ss o
f th
e in
terfa
ce to
use
rs, t
akin
g ac
coun
t of
attri
bute
s su
ch a
s co
lour
and
gra
phica
l des
ign.
C
OM
MEN
T(S)
Issu
es th
at p
oten
tially
co
ntrib
ute
to a
ttrac
tiven
ess
incl
ude:
Al
ignm
ent o
f ite
ms
(Ver
tical
and
H
oriz
onta
l), G
roup
ing,
Use
of c
olou
rs,
Appr
opria
te a
nd re
ason
able
siz
ed
grap
hics
, Use
of w
hite
spac
e/se
para
tors
/ bo
rder
s, A
nim
atio
n, T
ypog
raph
y, a
nd 3
D
inte
rface
.
Asse
ssm
ent
class
ificat
ion.
O
rdin
al
X= C
ount
(C
ount
is a
sc
ore)
Req
spe
c D
esig
n
Rev
iew
repo
rt
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
FOO
TNO
TE
This
coul
d be
bas
ed o
n sc
reen
ske
tche
s or
moc
k-up
s.
Use
r Int
erfa
ce
appe
aran
ce
cust
omis
abili
ty
Wha
t pro
porti
on o
f us
er in
terfa
ce
elem
ents
can
be
cust
omise
d in
ap
pear
ance
?
Insp
ectio
n (b
y ex
pert)
X=
A/B
A=N
umbe
r of t
ypes
of i
nter
face
ele
men
ts th
at
can
be c
usto
mise
d B=
Tota
l num
ber o
f typ
es o
f int
erfa
ce e
lem
ents
0 <=
X <
= 1
The
close
r to
1, th
e be
tter.
Abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 23
Tabl
e 8.
3.5
Usa
bilit
y co
mpl
ianc
e m
etric
s
Inte
rnal
usa
bilit
y co
mpl
ianc
e m
etric
s M
etric
nam
e Pu
rpos
e M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Usa
bilit
y co
mpl
ianc
e
How
com
plia
nt is
the
prod
uct t
o ap
plica
ble
regu
latio
ns, s
tand
ards
an
d co
nven
tions
for
usab
ility?
Cou
nt th
e nu
mbe
r of i
tem
s re
quiri
ng c
ompl
ianc
e th
at
have
bee
n m
et a
nd
com
pare
with
the
num
ber o
f ite
ms
requ
iring
com
plia
nce
as in
the
spec
ificat
ion.
X=A/
B
A= N
umbe
r of c
orre
ctly
impl
emen
ted
item
s re
late
d to
usa
bility
com
plia
nce
conf
irmed
in
eval
uatio
n B=
Tot
al n
umbe
r of c
ompl
ianc
e ite
ms
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
mor
e co
mpl
iant
.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Spec
ificat
ion
of c
ompl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
or
regu
latio
ns.
Des
ign
Sour
ce c
ode
Rev
iew
repo
rt
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
ISO/IEC TR 9126-3:2003(E)
24 © ISO/IEC 2003 – All rights reserved
8.4 Efficiency metrics
Internal efficiency metrics are used for predicting the efficiency of behaviour of the software product during testing or operating. To measure efficiency, the stated conditions should be defined, i.e., the hardware configuration and the software configuration of a reference environment (which has to be defined in the software specifications) should be defined. When citing measured time behaviour values the reference environment should be referred.
8.4.1 Time behaviour metrics
Internal time behaviour metrics indicate a set of attributes for predicting the time behaviour of the computer system including the software product during testing or operating.
8.4.2 Resource utilization metrics
Internal resource utilization metrics indicate a set of attributes for predicting the utilization of hardware resources by the computer system including the software product during testing or operating.
8.4.3 Efficiency compliance metrics
Internal compliance metrics relating to efficiency indicate a set of attributes for assessing the capability of the software product to comply to such items as standards, conventions or regulations of the user organization in relation to efficiency.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 25
Tabl
e 8.
4.1
Tim
e be
havi
our m
etric
s In
tern
al ti
me
beha
viou
r met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Res
pons
e tim
eW
hat i
s th
e es
timat
ed
time
to c
ompl
ete
a sp
ecifie
d ta
sk?
Eval
uate
the
effic
ienc
y of
th
e op
erat
ing
syst
em a
nd
the
appl
icatio
n sy
stem
cal
ls.
Estim
ate
the
resp
onse
time
base
d on
this.
Th
e fo
llowi
ng m
ay b
e m
easu
red,
-a
ll or p
arts
of d
esig
n sp
ecific
atio
ns
-test
com
plet
e tra
nsac
tion
path
-te
st c
ompl
ete
mod
ules
/par
ts o
f sof
twar
e pr
oduc
t -c
ompl
ete
softw
are
prod
uct
durin
g te
st p
hase
.
X=tim
e (c
alcu
late
d or
sim
ulat
ed)
The
shor
ter
the
bette
r. ra
tio
X=tim
e Kn
own
oper
atin
g sy
stem
. Es
timat
ed
time
in s
yste
m
calls
.
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s R
equi
rers
Thro
ughp
ut
time
Wha
t is
the
estim
ated
nu
mbe
r of t
asks
that
ca
n be
per
form
ed o
ver
a un
it of
time?
Eval
uate
the
effic
ienc
y of
ha
ndlin
g re
sour
ces
in th
e sy
stem
. Mak
e a
fact
or
base
d up
on th
e ap
plica
tion
calls
to th
e sy
stem
in
hand
ling
the
reso
urce
s.
X=N
o of
task
s pe
r uni
t of t
ime
The
grea
ter
the
bette
r. ra
tio
X=co
unt
Know
n op
erat
ing
syst
em.
Estim
ated
tim
e in
sys
tem
ca
lls.
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s R
equi
rers
Turn
arou
nd
time
Wha
t is
the
estim
ated
tim
e to
com
plet
e a
grou
p of
rela
ted
task
s as
a jo
b lo
t?
Eval
uate
the
effic
ienc
y of
th
e op
erat
ing
syst
em a
nd
the
appl
icatio
n sy
stem
cal
ls.
Estim
ate
the
resp
onse
time
to c
ompl
ete
a gr
oup
of
rela
ted
task
s ba
sed
on th
is.Th
e fo
llowi
ng m
ay b
e m
easu
red,
-a
ll or p
arts
of d
esig
n sp
ecific
atio
ns
-test
com
plet
e tra
nsac
tion
path
-te
st c
ompl
ete
mod
ules
/par
ts o
f sof
twar
e pr
oduc
t -c
ompl
ete
softw
are
prod
uct
durin
g te
st p
hase
.
X=tim
e (c
alcu
late
d or
sim
ulat
ed)
The
shor
ter
the
bette
r. ra
tio
X=tim
e Kn
own
oper
atin
g sy
stem
. Es
timat
ed
time
in s
yste
m
calls
.
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s R
equi
rers
ISO/IEC TR 9126-3:2003(E)
26 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
4.2
Res
ourc
e ut
ilisa
tion
met
rics
Inte
rnal
reso
urce
util
isat
ion
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
I/O U
tiliz
atio
n W
hat i
s th
e es
timat
ed
I/O u
tiliza
tion
to
com
plet
e a
spec
ified
task
?
Estim
ate
the
I/O u
tiliza
tion
requ
irem
ent f
or th
e ap
plica
tion.
X=nu
mbe
r of b
uffe
rs(c
alcu
late
d or
sim
ulat
ed)
The
shor
ter
the
bette
r. ra
tio
X=siz
e So
urce
cod
e Ve
rifica
tion
Dev
elop
ers
I/O U
tiliz
atio
n M
essa
ge
Den
sity
Wha
t is
the
dens
ity o
f m
essa
ges
rela
ting
to
I/O u
tiliza
tion
in th
e lin
es o
f cod
e re
spon
sible
in m
akin
g sy
stem
cal
ls?
Cou
nt th
e nu
mbe
r of e
rrors
pe
rtain
ing
to I/
O fa
ilure
and
wa
rnin
gs a
nd c
ompa
re it
to
the
estim
ated
num
ber o
f lin
es o
f cod
e re
spon
sible
in
syst
em c
alls.
X=A/
B A=
num
ber o
f I/O
rela
ted
erro
r mes
sage
s.
B=nu
mbe
r of l
ines
of c
ode
dire
ctly
rela
ted
to
syst
em c
alls
The
grea
ter
the
bette
r. Ab
solu
teX=
coun
t/ co
unt
A=co
unt
B=co
unt
Sour
ce c
ode
Verif
icatio
n D
evel
oper
s
Mem
ory
ut
iliza
tion
Wha
t is
the
estim
ated
m
emor
y siz
e th
at th
e pr
oduc
t will
occu
py to
co
mpl
ete
a sp
ecifie
d ta
sk?
Estim
ate
the
mem
ory
requ
irem
ent.
X=siz
e in
byt
es (c
alcu
late
d or
sim
ulat
ed)
The
less
er th
e be
tter.
ratio
X=
size
Estim
ated
size
of
mem
ory
utiliz
atio
n.
Verif
icatio
n D
evel
oper
s
Mem
ory
ut
iliza
tion
mes
sage
de
nsity
Wha
t is
the
dens
ity o
f m
essa
ges
rela
ting
to
mem
ory
utiliz
atio
n in
th
e lin
es o
f cod
e re
spon
sible
in m
akin
g sy
stem
cal
ls?
Cou
nt th
e nu
mbe
r of e
rror
mes
sage
s pe
rtain
ing
to
mem
ory
failu
re a
nd
warn
ings
and
com
pare
it to
th
e es
timat
ed n
umbe
r of
lines
of c
ode
resp
onsib
le in
sy
stem
cal
ls.
X=A/
B A=
Num
ber o
f mem
ory
rela
ted
erro
r m
essa
ges.
B=N
umbe
r of l
ines
of c
ode
dire
ctly
rela
ted
to s
yste
m c
alls
The
grea
ter
the
bette
r. ra
tio
X=co
unt/
coun
t A=
coun
t B=
coun
t
Sour
ce c
ode
Verif
icatio
n D
evel
oper
s
Tran
smis
sion
U
tiliz
atio
n W
hat i
s th
e es
timat
ed
amou
nt o
f tra
nsm
issio
n re
sour
ces
utiliz
atio
n?
Estim
ate
the
Tran
smiss
ion
reso
urce
utili
zatio
n re
quire
men
ts b
y es
timat
ing
the
trans
miss
ion
volu
mes
.
X=bi
ts/tim
e (c
alcu
late
d or
sim
ulat
ed)
The
less
er th
e be
tter.
ratio
X=
time
Know
n op
erat
ing
syst
em.
Estim
ated
tim
e in
sys
tem
ca
lls.
Verif
icatio
n D
evel
oper
s
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 27
Tabl
e 8.
4.3
Effic
ienc
y co
mpl
ianc
e m
etric
s
Inte
rnal
Effi
cien
cy c
ompl
ianc
e m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Effic
ienc
y C
ompl
ianc
e H
ow c
ompl
iant
is th
e ef
ficie
ncy
of th
e pr
oduc
t to
appl
icabl
e re
gula
tions
, sta
ndar
ds
and
conv
entio
ns?
Cou
nt th
e nu
mbe
r of i
tem
s re
quiri
ng c
ompl
ianc
e th
at
have
bee
n m
et a
nd
com
pare
with
the
num
ber o
f ite
ms
requ
iring
com
plia
nce
as in
the
spec
ificat
ion.
X=A/
B
A= N
umbe
r of c
orre
ctly
impl
emen
ted
item
s re
late
d to
effic
ienc
y co
mpl
ianc
e co
nfirm
ed in
ev
alua
tion
B= T
otal
num
ber o
f com
plia
nce
item
s
0 <=
X <
= 1
The
close
r to
1, th
e m
ore
com
plia
nt.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Spec
ificat
ion
of c
ompl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
or
regu
latio
ns.
Des
ign
Sour
ce c
ode
Rev
iew
repo
rt
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
ISO/IEC TR 9126-3:2003(E)
28 © ISO/IEC 2003 – All rights reserved
8.5 Maintainability metrics
Internal maintainability metrics are used for predicting the level of effort required for modifying the software product.
8.5.1 Analysability metrics
Internal analysability metrics indicate a set of attributes for predicting the maintainer’s or user’s spent effort or spent resources in trying to diagnose for deficiencies or causes of failure, or for identification of parts to be modified in the software product.
8.5.2 Changeability metrics
Internal changeability metrics indicate a set of attributes for predicting the maintainer’s or user’s spent effort when trying to implement a specified modification in the software product.
8.5.3 Stability metrics
Internal stability metrics indicate a set of attributes for predicting how stable the software product would be after any modification.
8.5.4 Testability metrics
Internal testability metrics indicate a set of attributes for predicting the amount of designed and implemented autonomous test aid functions present in the software product.
8.5.5 Maintainability compliance metrics
Internal compliance metrics relating to maintainability indicate a set of attributes for assessing the capability of the software product to comply to such items as standards, conventions or regulations of the user organization in relation to software maintainability.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 29
Tabl
e 8.
5.1
Anal
ysab
ility
met
rics
Inte
rnal
ana
lysa
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Activ
ity
reco
rdin
g H
ow th
orou
gh is
the
reco
rdin
g of
the
syst
em s
tatu
s?
Cou
nt th
e nu
mbe
r of i
tem
s lo
gged
in th
e ac
tivity
log
as
spec
ified
and
com
pare
it to
th
e nu
mbe
r of i
tem
s re
quire
d to
be
logg
ed.
X=A/
B
A=N
umbe
r of im
plem
ente
d da
ta lo
gin
item
s as
sp
ecifie
d c
onfir
med
in re
view
B=
Num
ber o
f dat
a ite
ms
to b
e lo
gged
def
ined
in
the
spec
ificat
ions
0 <=
X <
= 1
The
close
r to
1, m
ore
data
pr
ovid
ed to
re
cord
sys
tem
st
atus
. CO
MM
ENT(
S)It
is ne
cess
ary
to c
onve
rt th
is va
lue
to th
e <0
,1>
inte
rval
if
mak
ing
sum
mar
izatio
n of
ch
arac
teris
tics.
Abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Valu
e A
com
es fr
om
revie
w re
port.
Va
lue
B co
mes
from
re
quire
men
t sp
ecific
atio
ns.
Verif
icatio
n Jo
int r
evie
w M
aint
aine
rs
Use
rs
Rea
dine
ss o
f di
agno
stic
fu
nctio
n
How
thor
ough
is th
e pr
ovisi
on o
f the
di
agno
stic
func
tions
?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
diag
nost
ic fu
nctio
ns a
s sp
ecifie
d an
d co
mpa
re it
to th
e nu
mbe
r of
diag
nost
ic fu
nctio
ns
requ
ired
in s
pecif
icatio
ns.
COM
MEN
T(S)
Thi
s m
etric
is
also
use
d to
mea
sure
fa
ilure
ana
lysis
capa
bility
an
d ca
usal
ana
lysis
capa
bility
.
X=A/
B
A=N
umbe
r of im
plem
ente
d di
agno
stic
func
tions
as
spec
ified
conf
irmed
in re
view
B=
Num
ber o
f dia
gnos
tic fu
nctio
ns re
quire
d
0 <=
X
The
close
r to
1, th
e be
tter
impl
emen
tatio
n of
dia
gnos
tic
func
tions
. CO
MM
ENT(
S)It
is ne
cess
ary
to c
onve
rt th
is va
lue
to th
e <0
,1>
inte
rval
if
mak
ing
sum
mar
izatio
n of
ch
arac
teris
tics.
Abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Valu
e A
com
es fr
om
revie
w re
port.
Va
lue
B co
mes
from
re
quire
men
t sp
ecific
atio
ns.
Verif
icatio
n Jo
int r
evie
w M
aint
aine
rs
Use
rs
ISO/IEC TR 9126-3:2003(E)
30 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
5.2
Cha
ngea
bilit
y m
etric
s In
tern
al c
hang
eabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Cha
nge
reco
rdab
ility
Ar
e ch
ange
s to
sp
ecific
atio
ns a
nd
prog
ram
mod
ules
re
cord
ed a
dequ
atel
y in
the
code
with
co
mm
ent li
nes?
Rec
ord
ratio
of m
odul
e ch
ange
info
rmat
ion.
X=
A/B
A=
Num
ber o
f cha
nges
in fu
nctio
ns/m
odul
es
havin
g ch
ange
com
men
ts c
onfir
med
in re
view
B=To
tal n
umbe
r of f
unct
ions
/mod
ules
ch
ange
d fro
m o
rigin
al c
ode
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
mor
e re
cord
able
. Th
e ch
ange
co
ntro
l 0
indi
cate
s po
or
chan
ge c
ontro
l or
little
ch
ange
s, h
igh
stab
ility.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Con
figur
atio
n co
ntro
l sys
tem
Vers
ion
logs
Sp
ecific
atio
ns
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s M
aint
aine
rs
Req
uire
rs
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 31
Tabl
e 8.
5.3
Stab
ility
met
rics
Inte
rnal
sta
bilit
y m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Cha
nge
impa
ct W
hat i
s th
e fre
quen
cy
of a
dver
se im
pact
s af
ter m
odific
atio
n?
Cou
nt th
e nu
mbe
r of
dete
cted
adv
erse
impa
cts
afte
r mod
ificat
ion
and
com
pare
it to
the
num
ber o
f m
odific
atio
ns p
erfo
rmed
.
X=1-
A/B
A=N
umbe
r of d
etec
ted
adve
rse
impa
cts
afte
r m
odific
atio
ns
B=N
umbe
r of m
odific
atio
ns m
ade
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
bette
r.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
A co
mes
from
re
view
repo
rt B
com
es fr
om
revie
w re
port
Join
t rev
iew
Verif
icatio
n D
evel
oper
s M
aint
aine
rs
Req
uire
rs
Mod
ifica
tion
impa
ct
loca
lizat
ion
How
larg
e is
the
impa
ct o
f the
m
odific
atio
n on
the
softw
are
prod
uct?
Cou
nt th
e nu
mbe
r of
affe
cted
var
iabl
es fr
om a
m
odific
atio
n an
d co
mpa
re it
to th
e to
tal n
umbe
r of
varia
bles
in th
e pr
oduc
t. CO
MM
ENT(
S) Im
pact
ed
varia
ble
is a)
All v
aria
bles
in th
e in
stru
ctio
n w
hich
was
ch
ange
d.
b) V
aria
ble
whi
ch is
in th
e sa
me
inst
ruct
ion
with
the
varia
ble
defin
ed b
y a)
.
X=A/
B A=
Num
ber o
f affe
cted
var
iabl
e da
ta b
y m
odific
atio
n, c
onfir
med
in re
view
B=To
tal n
umbe
r of v
aria
bles
0 <=
X <
= 1
Th
e clo
ser t
o 0,
the
less
er
impa
ct o
f m
odific
atio
n.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
A co
mes
from
re
view
repo
rt B
com
es fr
om
revie
w re
port
Join
t rev
iew
Verif
icatio
n D
evel
oper
s M
aint
aine
rs
Req
uire
rs
ISO/IEC TR 9126-3:2003(E)
32 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
5.4
Test
abili
ty m
etric
s In
tern
al te
stab
ility
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Com
plet
enes
s of
bui
lt-in
test
fu
nctio
n
How
com
plet
e is
the
built-
in te
st c
apab
ility?
C
ount
the
num
ber o
f im
plem
ente
d bu
ilt-in
test
fu
nctio
ns a
s sp
ecifie
d an
d co
mpa
re it
to th
e nu
mbe
r of
built-
in te
st fu
nctio
ns in
the
requ
irem
ents
.
X=A/
B
A=N
umbe
r of im
plem
ente
d bu
ilt-in
test
fu
nctio
n as
spe
cifie
d co
nfirm
ed in
revie
w
B=N
umbe
r of b
uilt-
in te
st fu
nctio
n re
quire
d
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
mor
e co
mpl
ete.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
A co
mes
from
re
view
docu
men
t B
com
es fr
om
requ
irem
ents
or
des
ign
docu
men
t
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s M
aint
aine
rs
Req
uire
rs
Auto
nom
y of
te
stab
ility
H
ow in
depe
nden
tly
can
the
softw
are
be
test
ed?
Cou
nt th
e nu
mbe
r of
depe
nden
cies
on o
ther
sy
stem
s fo
r tes
ting
that
ha
ve b
een
simul
ated
with
st
ubs
and
com
pare
it w
ith
the
tota
l num
ber o
f tes
t de
pend
encie
s on
oth
er
syst
ems.
X=A/
B
A=N
umbe
r of d
epen
denc
ies
on o
ther
sys
tem
s fo
r tes
ting
that
hav
e be
en s
imul
ated
with
stu
bs
B= T
otal
num
ber o
f tes
t dep
ende
ncie
s on
ot
her s
yste
ms
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
bette
r.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
A co
mes
from
re
view
docu
men
t B
com
es fr
om
requ
irem
ents
or
des
ign
docu
men
t
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s M
aint
aine
rs
Req
uire
rs
Test
pro
gres
s ob
serv
abili
ty
How
com
plet
e ar
e th
e bu
ilt in
test
resu
lt di
spla
ys d
urin
g te
stin
g?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
chec
kpoi
nts
as s
pecif
ied
and
com
pare
it
to th
e nu
mbe
r sp
ecifie
d ch
eckp
oint
s re
quire
d by
de
sign.
X=A/
B
A=N
umbe
r of im
plem
ente
d ch
eckp
oint
s as
sp
ecifie
d co
nfirm
ed in
revie
w
B=N
umbe
r of d
esig
ned
chec
kpoi
nts
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
bette
r.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
A co
mes
from
re
view
docu
men
t B
com
es fr
om
desig
n do
cum
ent
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s M
aint
aine
rs
Req
uire
rs
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 33
Tabl
e 8.
5.5
Mai
ntai
nabi
lity
com
plia
nce
met
rics
In
tern
al m
aint
aina
bilit
y co
mpl
ianc
e m
etric
s M
etric
nam
e Pu
rpos
e of
the
met
rics
Met
hod
of a
pplic
atio
n M
easu
rem
ent,
form
ula
and
da
ta e
lem
ent c
ompu
tatio
ns
Inte
rpre
tatio
n of
mea
sure
d va
lue
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Mai
ntai
nabi
lity
com
plia
nce
How
com
plia
nt is
the
mai
ntai
nabi
lity o
f the
pr
oduc
t to
appl
icabl
e re
gula
tions
, sta
ndar
ds
and
conv
entio
ns?
Cou
nt th
e nu
mbe
r of i
tem
s re
quiri
ng c
ompl
ianc
e th
at
have
bee
n m
et a
nd
com
pare
with
the
num
ber o
f ite
ms
requ
iring
com
plia
nce
as in
the
spec
ificat
ion.
X=A/
B
A= N
umbe
r of c
orre
ctly
impl
emen
ted
item
s re
late
d to
mai
ntai
nabi
lity c
ompl
ianc
e co
nfirm
ed in
eva
luat
ion
B= T
otal
num
ber o
f com
plia
nce
item
s
0 <=
X <
= 1
Th
e clo
ser t
o 1,
the
mor
e co
mpl
iant
.
abso
lute
X=co
unt/
coun
t A=
coun
t B=
coun
t
Spec
ificat
ion
of c
ompl
ianc
e an
d re
late
d st
anda
rds,
co
nven
tions
or
regu
latio
ns.
Des
ign
Sour
ce c
ode
Rev
iew
repo
rt
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
ISO/IEC TR 9126-3:2003(E)
34 © ISO/IEC 2003 – All rights reserved
8.6 Portability metrics
Internal portability metrics are used for predicting the effect the software product may have on the behaviour of the implementor or system during the porting activity.
8.6.1 Adaptability metrics
Internal adaptability metrics indicate a set of attributes for predicting the impact the software product may have on the effort of the user who is trying to adapt the software product to different specified environments.
8.6.2 Installability metrics
Internal installability metrics indicate a set of attributes for predicting the impact the software product may have on the effort of the user who is trying to install the software in a user specified environment.
8.6.3 Co-existence metrics
Internal co-existence metrics indicate a set of attributes for predicting the impact the software product may have on other software products sharing the same operational hardware resources.
8.6.4 Replaceability metrics
Internal replaceability metrics indicate a set of attributes for predicting the impact the software product may have on the effort of the user who is trying to use the software in place of other specified software in a specified environment and context of use.
8.6.5 Portability compliance metrics
Internal compliance metrics relating to portability indicate a set of attributes for assessing the capability of the software product to comply to such items as standards, conventions or regulations of the user organization in relation to portability.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 35
Tabl
e 8.
6.1
Adap
tabi
lity
met
rics
Inte
rnal
ada
ptab
ility
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Adap
tabi
lity
of
data
str
uctu
res
How
ada
ptab
le is
the
prod
uct t
o th
e da
ta
stru
ctur
e ch
ange
s?
Cou
nt th
e nu
mbe
r of d
ata
stru
ctur
es, w
hich
are
op
erab
le a
nd h
as n
o lim
itatio
n af
ter a
dapt
atio
n an
d co
mpa
re it
to th
e to
tal
num
ber o
f dat
a st
ruct
ures
re
quiri
ng a
dapt
atio
n ca
pabi
lity.
X=A/
B A=
Num
ber o
f dat
a st
ruct
ures
whi
ch a
re
oper
able
and
has
no
limita
tion
afte
r ada
ptat
ion,
co
nfirm
ed in
revie
w B=
Tota
l num
ber o
f dat
a st
ruct
ures
requ
iring
ad
apta
tion
capa
bility
0 <=
X <
= 1
The
clo
ser t
o 1,
the
bette
r.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s M
aint
aine
rs
Req
uire
rs
Har
dwar
e en
viro
nmen
tal
adap
tabi
lity
(ada
ptab
ility
to
hard
war
e de
vice
s an
d ne
twor
k fa
cilit
ies)
How
ada
ptab
le is
the
prod
uct t
o th
e H/
W
rela
ted
envir
onm
enta
l ch
ange
?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
func
tions
whi
ch
are
capa
ble
of a
chie
ving
requ
ired
resu
lts in
spe
cifie
d m
ultip
le H
/W e
nviro
nmen
ts
as s
pecif
ied
and
com
pare
it
to th
e nu
mbe
r of f
unct
ions
wi
th H
/W e
nviro
nmen
t ad
apta
tion
capa
bility
re
quire
men
ts.
X=A/
B
A=N
umbe
r of im
plem
ente
d fu
nctio
ns w
hich
are
ca
pabl
e of
ach
ievin
g re
quire
d re
sults
in
spec
ified
mul
tiple
H/W
env
ironm
ent a
s sp
ecifie
d, c
onfir
med
in re
view
B=
Tota
l num
ber o
f fun
ctio
ns w
ith H
/W
envir
onm
ent a
dapt
atio
n ca
pabi
lity re
quire
men
ts 0
<= X
<=
1
The
close
r to
1,
the
bette
r.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s M
aint
aine
rs
Req
uire
rs
Org
anis
atio
nal
envi
ronm
ent
adap
tabi
lity
(Org
anis
atio
n ad
apta
bilit
y to
in
frast
ruct
ure
of
orga
nisa
tion)
How
ada
ptab
le is
the
prod
uct t
o or
gani
zatio
nal c
hang
e? C
ount
the
num
ber o
f im
plem
ente
d fu
nctio
ns w
hich
ar
e ca
pabl
e of
ach
ievin
g re
quire
d re
sults
in s
pecif
ied
mul
tiple
org
aniza
tiona
l and
bu
sines
s en
viron
men
ts a
s sp
ecifie
d an
d co
mpa
re it
to
the
num
ber o
f fun
ctio
ns w
ith
orga
niza
tiona
l env
ironm
ent
adap
tatio
n ca
pabi
lity
requ
irem
ents
.
X=A/
B
A=nu
mbe
r of im
plem
ente
d fu
nctio
ns w
hich
are
ca
pabl
e of
ach
ievin
g re
quire
d re
sults
in
spec
ified
mul
tiple
org
aniza
tiona
l and
bus
ines
s en
viron
men
t as
spec
ified,
con
firm
ed in
revie
w
B=To
tal n
umbe
r of f
unct
ions
with
org
aniza
tiona
l en
viron
men
t ada
ptat
ion
capa
bility
requ
irem
ents
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e be
tter.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s M
aint
aine
rs
Req
uire
rs
Port
ing
user
fr
iend
lines
s H
ow e
ffortl
ess
is it t
o pe
rform
por
ting
oper
atio
ns o
n th
e pr
oduc
t?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
func
tions
whi
ch
are
capa
ble
of s
uppo
rting
ea
se-o
f-ada
ptat
ion
by u
ser
as s
pecif
ied
and
com
pare
it
to th
e nu
mbe
r of f
unct
ions
wi
th e
asy-
to-a
dapt
cap
abilit
y re
quire
men
ts.
X=A/
B
A=N
umbe
r of f
unct
ions
sup
porti
ng e
ase-
of-
adap
tatio
n by
use
r as
spec
ified,
con
firm
ed in
re
view
B=To
tal n
umbe
r of f
unct
ions
with
eas
e-to
-ada
pt
capa
bility
requ
irem
ents
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e m
ore
frien
dly.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s M
aint
aine
rs
Req
uire
rs
ISO/IEC TR 9126-3:2003(E)
36 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
6.1
(con
tinue
d)
Inte
rnal
ada
ptab
ility
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Syst
em
softw
are
envi
ronm
enta
l ad
apta
bilit
y (a
dapt
abilit
y to
O
S, n
etw
ork
softw
are
and
co-
oper
ated
ap
plic
atio
n so
ftwar
e)
How
ada
ptab
le is
the
prod
uct t
o sy
stem
so
ftwar
e re
late
d en
viron
men
tal
chan
ges?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
func
tions
whi
ch
are
capa
ble
of a
chie
ving
requ
ired
resu
lts in
spe
cifie
d m
ultip
le s
yste
m s
oftw
are
envir
onm
ents
as
spec
ified
and
com
pare
it to
the
num
ber o
f fun
ctio
ns w
ith
syst
em s
oftw
are
envir
onm
ent a
dapt
atio
n ca
pabi
lity re
quire
men
ts.
X=A/
B
A=N
umbe
r of im
plem
ente
d fu
nctio
ns w
hich
are
ca
pabl
e of
ach
ievin
g re
quire
d re
sults
in
spec
ified
mul
tiple
sys
tem
sof
twar
e en
viron
men
t as
spe
cifie
d, c
onfir
med
in re
view
B=
Tota
l num
ber o
f fun
ctio
ns w
ith s
yste
m
softw
are
envir
onm
ent a
dapt
atio
n ca
pabi
lity
requ
irem
ents
B=T
otal
num
ber o
f fun
ctio
ns w
ith
syst
em s
oftw
are
envir
onm
ent a
dapt
atio
n ca
pabi
lity re
quire
men
ts
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e be
tter.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Req
spe
c D
esig
n R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w D
evel
oper
s M
aint
aine
rs
Req
uire
rs
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 37
Tabl
e 8.
6.2
Inst
alla
bilit
y m
etric
s In
tern
al in
stal
labi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Ease
of S
etup
R
e-tr
y
How
eas
y is
it to
repe
at
setu
p op
erat
ion?
C
ount
the
num
ber o
f im
plem
ente
d se
tup
retry
op
erat
ions
and
com
pare
it to
th
e nu
mbe
r of s
etup
retry
op
erat
ions
requ
ired.
X=A/
B A=
Num
ber o
f impl
emen
ted
retry
ope
ratio
ns fo
r se
tup,
con
firm
ed in
revie
w B=
Tota
l num
ber o
f set
up o
pera
tions
requ
ired
0 <=
X <
= 1
The
close
r to
1,
the
easie
r.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Rev
iew
repo
rt 6.
5
Valid
atio
n D
evel
oper
s
Inst
alla
tion
effo
rt
Wha
t lev
el o
f effo
rt is
requ
ired
for
inst
alla
tion?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
inst
alla
tion
auto
mat
ed s
teps
and
co
mpa
re it
to th
e nu
mbe
r of
pres
crib
ed in
stal
latio
n st
eps.
X=A/
B
A=N
umbe
r of a
utom
ated
inst
alla
tion
step
s co
nfirm
ed in
revie
w
B=N
umbe
r of i
nsta
llatio
n st
eps
requ
ired
COM
MEN
T(S)
Pre
scrib
ed: e
.g.,
num
ber o
f w
indo
ws/
com
man
ds/m
anua
l ope
ratio
n to
reac
h ta
rget
ope
ratio
n.
0 <=
X <
= 1
The
close
r to
1,
the
bette
r.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Rev
iew
repo
rt 6.
5
Valid
atio
n D
evel
oper
s
Inst
alla
tion
flexi
bilit
y H
ow fl
exib
le a
nd
cust
omiza
ble
is th
e in
stal
latio
n ca
pabi
lity?
Cou
nt th
e nu
mbe
r of
impl
emen
ted
cust
omiza
ble
inst
alla
tion
oper
atio
ns a
s sp
ecifie
d an
d co
mpa
re it
to
the
num
ber o
f ins
talla
tion
oper
atio
ns w
ith
cust
omiza
tion
capa
bility
re
quire
men
ts.
X=A/
B
A=N
umbe
r of im
plem
ente
d cu
stom
izabl
e in
stal
latio
n op
erat
ion
as s
pecif
ied
conf
irmed
in
revie
w
B=N
umbe
r of c
usto
miza
ble
inst
alla
tion
oper
atio
n re
quire
d CO
MM
ENT(
S) C
usto
miza
ble:
e.g
., ne
stin
g de
pth,
num
ber o
f pan
els.
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e m
ore
flexib
le.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Req
uire
men
t sp
ecific
atio
n R
evie
w re
port
6.5
Va
lidat
ion
Dev
elop
ers
ISO/IEC TR 9126-3:2003(E)
38 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
6.3
Co-
exis
tenc
e m
etric
s In
tern
al c
o-ex
iste
nce
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Avai
labl
e co
-ex
iste
nce
How
flex
ible
is th
e pr
oduc
t in
shar
ing
its
envir
onm
ent w
ith o
ther
pr
oduc
ts w
ithou
t ad
vers
e im
pact
s on
ot
her p
rodu
cts?
Cou
nt th
e nu
mbe
r of e
ntitie
s wi
th w
hich
pro
duct
can
co-
exist
as
spec
ified
and
com
pare
it to
the
num
ber o
f en
titie
s in
pro
duct
ion
envir
onm
ent t
hat r
equi
re c
o-ex
isten
ce.
X=A/
B A=
Num
ber o
f ent
ities
with
whi
ch p
rodu
ct c
an
co-e
xist a
s sp
ecifie
d B=
Num
ber o
f ent
ities
in p
rodu
ctio
n en
viron
men
t tha
t req
uire
co-
exist
ence
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e be
tter.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Req
uire
men
ts
spec
ificat
ion
Rev
iew
repo
rt Te
st re
port
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s M
aint
aine
rs
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 39
Tabl
e 8.
6.4
Rep
lace
abili
ty m
etric
s In
tern
al re
plac
eabi
lity
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Con
tinue
d us
e of
dat
a W
hat i
s th
e am
ount
of
orig
inal
dat
a th
at
rem
ain
unch
ange
d af
ter r
epla
cem
ent w
ith
this
prod
uct?
Cou
nt th
e nu
mbe
r of d
ata
item
s, th
at c
ontin
ue to
be
used
afte
r rep
lace
men
t as
sp
ecifie
d, a
nd c
ompa
re it
to
the
tota
l num
ber o
f dat
a ite
ms
requ
ired
to b
e us
ed
from
the
old
data
afte
r so
ftwar
e re
plac
emen
t.
X=A/
B
A=N
umbe
r of s
oftw
are
data
item
s th
at c
ontin
ue
to b
e us
ed a
s sp
ecifie
d af
ter r
epla
cem
ent,
conf
irmed
in e
valu
atio
n B=
Num
ber o
f old
dat
a ite
ms
requ
ired
to b
e us
ed
from
old
sof
twar
e
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e be
tter.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Des
ign
Sour
ce c
ode
Rev
iew
repo
rt Te
st re
port
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s M
aint
aine
rs
Func
tion
incl
usiv
enes
s W
hat’s
the
amou
nt o
f fu
nctio
ns th
at re
mai
n un
chan
ged?
Cou
nt th
e nu
mbe
r of
func
tions
cov
ered
by
new
softw
are
that
pro
duce
s sim
ilar r
esul
ts a
nd c
ompa
re
it to
the
num
ber o
f fun
ctio
n in
th
e ol
d so
ftwar
e.
X=A/
B
A=N
umbe
r of f
unct
ions
cov
ered
by
new
softw
are
that
pro
duce
s sim
ilar r
esul
ts,
conf
irmed
in re
view
B=
Num
ber o
f fun
ctio
ns in
old
sof
twar
e
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e be
tter.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Des
ign
Sour
ce c
ode
Rev
iew
repo
rt Te
st re
port
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s M
aint
aine
rs
ISO/IEC TR 9126-3:2003(E)
40 © ISO/IEC 2003 – All rights reserved
Tabl
e 8.
6.5
Port
abili
ty c
ompl
ianc
e m
etric
s
Inte
rnal
por
tabi
lity
com
plia
nce
met
rics
Met
ric n
ame
Purp
ose
of th
e m
etric
s M
etho
d of
app
licat
ion
Mea
sure
men
t, fo
rmul
a an
d
data
ele
men
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
scal
e ty
pe
Mea
sure
ty
pe
Inpu
t to
mea
sure
-m
ent
ISO
/IEC
12
207
SLC
P R
efer
ence
Targ
et
audi
ence
Port
abili
ty
com
plia
nce
How
com
plia
nt is
the
porta
bility
of t
he
prod
uct t
o ap
plica
ble
regu
latio
ns, s
tand
ards
an
d co
nven
tions
?
Cou
nt th
e nu
mbe
r of i
tem
s re
quiri
ng c
ompl
ianc
e th
at
have
bee
n m
et a
nd c
ompa
re
with
the
num
ber o
f ite
ms
requ
iring
com
plia
nce
as in
th
e sp
ecific
atio
n.
X=A/
B
A= N
umbe
r of c
orre
ctly
impl
emen
ted
item
s re
late
d to
por
tabi
lity c
ompl
ianc
e co
nfirm
ed in
ev
alua
tion
B= T
otal
num
ber o
f com
plia
nce
item
s
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e m
ore
com
plet
e.
abso
lute
X=
coun
t/ co
unt
A=co
unt
B=co
unt
Spec
ificat
ion
of
com
plia
nce
and
rela
ted
stan
dard
s,
conv
entio
ns o
r re
gula
tions
. D
esig
n So
urce
cod
e R
evie
w re
port
Verif
icatio
n Jo
int r
evie
w R
equi
rers
D
evel
oper
s
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 41
Annex A (informative)
Considerations When Using Metrics
A.1 Interpretation of measures
A.1.1 Potential differences between test and operational contexts of use
When planning the use of metrics or interpreting measures it is important to have a clear understanding of the intended context of use of the software, and any potential differences between the test and operational contexts of use. For example, the “time required to learn operation” measure is often different between skilled operators and unskilled operators in similar software systems. Examples of potential differences are given below.
a) Differences between testing environment and the operational environment
Are there any significant differences between the testing environment and the operational execution in user environment?
The following are examples:
• testing with higher / comparable / lower performance of CPU of operational computer;
• testing with higher / comparable / lower performance of operational network and communication;
• testing with higher / comparable / lower performance of operational operating system;
• testing with higher / comparable / lower performance of operational user interface.
b) Differences between testing execution and actual operational execution
Are there any significant differences between the testing execution and operational execution in user environment?
The following are examples:
• coverage of functionality in test environment;
• test case sampling ratio;
• automated testing of real time transactions;
• stress loads;
• 24 hour 7 days a week (non stop) operation;
• appropriateness of data for testing of exceptions and errors;
• periodical processing;
• resource utilisation;
• levels of interruption;
• production pressures;
• distractions.
ISO/IEC TR 9126-3:2003(E)
42 © ISO/IEC 2003 – All rights reserved
c) User profile under observation
Are there any significant differences between test user profiles and operational user profiles?
The following are examples:
• mix of type of users;
• user skill levels;
• specialist users or average users;
• limited user group or public users.
A.1.2 Issues affecting validity of results
The following issues may affect the validity of the data that is collected.
(a) procedures for collecting evaluation results:
• automatically with tools or facilities / manually collected / questionnaires or interviews;
(b) source of evaluation results
• developers' self reports / reviewers’ report / evaluator’s report;
(c) results data validation
• developers' self check / inspection by independent evaluators.
A.1.3 Balance of measurement resources
Is the balance of measures used at each stage appropriate for the evaluation purpose?
It is important to balance the effort used to apply an appropriate range of metrics for internal, external and quality in use measures.
A.1.4 Correctness of specification
Are there significant differences between the software specification and the real operational needs?
Measurements taken during software product evaluation at different stages are compared against product specifications. Therefore, it is very important to ensure by verification and validation that the product specifications used for evaluation reflect the actual and real needs in operation.
A.2 Validation of Metrics
A.2.1 Desirable Properties for Metrics
To obtain valid results from a quality evaluation, the metrics should have the properties stated below. If a metric does not have these properties, the metric description should explain the associated constraint on its validity and, as far as possible, how that situation can be handled.
a) Reliability (of metric): Reliability is associated with random error. A metric is free of random error if random variations do not affect the results of the metric.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 43
b) Repeatability (of metric): repeated use of the metric for the same product using the same evaluation specification (including the same environment), type of users, and environment by the same evaluators, should produce the same results within appropriate tolerances. The appropriate tolerances should include such things as fatigue, and learning effect.
c) Reproducibility (of metric): use of the metric for the same product using the same evaluation specification (including the same environment), type of users, and environment by different evaluators, should produce the same results within appropriate tolerances.
NOTE 1 It is recommended to use statistical analysis to measure the variability of the results.
d) Availability (of metric): The metric should clearly indicate the conditions (e.g. presence of specific attributes) which constrain its usage.
e) Indicativeness (of metric): Capability of the metric to identify parts or items of the software which should be improved, given the measured results compared to the expected ones.
NOTE 2 The selected or proposed metric should provide documented evidence of the availability of the metric for use, unlike those requiring project inspection only.
f) Correctness (of measure): The metric should have the following properties:
1) Objectivity (of measure): the metric results and its data input should be factual: i.e., not influenced by the feelings or the opinions of the evaluator, test users, etc. (except for satisfaction or attractiveness metrics where user feelings and opinions are being measured).
2) Impartiality (of measure): the measurement should not be biased towards any particular result.
3) Sufficient precision (of measure): Precision is determined by the design of the metric, and particularly by the choice of the material definition used as the basis for the metric. The metric user will describe the precision and the sensitivity of the metric.
g) Meaningfulness (of measure): the measurement should produce meaningful results about the software behaviour or quality characteristics.
The metric should also be cost effective: that is, more costly metrics should provide higher value results.
A.2.2 Demonstrating the Validity of Metrics
The users of metrics should identify the methods for demonstrating the validity of metrics, as shown below:
(a) Correlation
The variation in the quality characteristics values (the measures of principal metrics in operational use) explained by the variation in the metric values, is given by the square of the linear coefficient.
An evaluator can predict quality characteristics without measuring them directly by using correlated metrics.
(b) Tracking
If a metric M is directly related to a quality characteristics value Q (the measures of principal metrics in operational use), for a given product or process, then a change value Q(T1) to Q(T2), would be accompanied by a change metric value from M(T1) to M(T2), in the same direction (for example, if Q increases, M increases).
An evaluator can detect movement of quality characteristics along a time period without measuring directly by using those metrics which have tracking ability.
ISO/IEC TR 9126-3:2003(E)
44 © ISO/IEC 2003 – All rights reserved
(c) Consistency
If quality characteristics values (the measures of principal metrics in operational use) Q1, Q2,..., Qn, corresponding to products or processes 1, 2,..., n, have the relationship Q1 > Q2 > ...> Qn, then the corresponding metric values would have the relationship M1 > M2 > ...> Mn.
An evaluator can notice exceptional and error prone components of software by using those metrics which have consistency ability.
(d) Predictability
If a metric is used at time T1 to predict a quality characteristic value Q (the measures of principal metrics in operational use) at T2, prediction error, which is {(predicted Q(T2) - actual Q(T2) ) / actual Q(T2)}, would be within allowed prediction error range.
An evaluator can predict the movement of quality characteristics in the future by using these metrics, which measure predictability.
(e) Discriminative
A metric would be able to discriminate between high and low quality software.
An evaluator can categorize software components and rate quality characteristics values by using those metrics which have discriminative ability.
A.3 Use of metrics for estimation (judgement) and prediction (forecast)
Estimation and prediction of the quality characteristics of the software product at the earlier stages are two of the most rewarding uses of metrics.
A.3.1 Quality characteristics prediction by current data
(a) Prediction by regression analysis
When predicting the future value (measure) of the same characteristic (attribute) by using the current value (data) of it (the attribute), a regression analysis is useful based on a set of data that is observed in a sufficient period of time.
For example, the value of MTBF (Mean Time Between Failures) that is obtained during the testing stage (activities) can be used to estimate the MTBF in operation stage.
(b) Prediction by correlation analysis
When predicting the future value (measure) of a characteristic (attribute) by using the current measured values of a different attribute, a correlation analysis is useful using a validated function which shows the correlation.
For example, the complexity of modules during coding stage may be used to predict time or effort required for program modification and test during maintenance process.
A.3.2 Current quality characteristics estimation on current facts
(a) Estimation by correlation analysis
When estimating the current values of an attribute which are directly unmeasurable, or if there is any other measure that has strong correlation with the target measure, a correlation analysis is useful.
For example, because the number of remaining faults in a software product is not measurable, it may be estimated by using the number and trend of detected faults.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 45
Those metrics which are used for predicting the attributes that are not directly measurable should be estimated as explained below:
• Using models for predicting the attribute;
• Using formula for predicting the attribute;
• Using basis of experience for predicting the attribute;
• Using justification for predicting the attribute.
Those metrics which are used for predicting the attributes that are not directly measurable may be validated as explained below:
• Identify measures of attributes which are to be predicted;
• Identify the metrics which will be used for prediction;
• Perform a statistical analysis based validation;
• Document the results;
• Repeat the above periodically.
A.4 Detecting deviations and anomalies in quality problem prone components
The following quality control tools may be used to analyse deviations and anomalies in software product components:
(a) process charts (functional modules of software)
(b) Pareto analysis and diagrams
(c) histograms and scatter diagrams
(d) run diagrams, correlation diagrams and stratification
(e) Ishikawa (Fishbone) diagrams
(f) statistical process control (functional modules of software)
(g) check sheets
The above tools can be used to identify quality issues from data obtained by applying the metrics.
A.5 Displaying measurement results
(a) Displaying quality characteristics evaluation results
The following graphical presentations are useful to display quality evaluation results for each of the quality characteristic and subcharacteristic.
Radar chart; Bar chart numbered histogram, multi-variates chart, Importance Performance Matrix, etc.
(b) Displaying measures
There are useful graphical presentations such as Pareto chart, trend charts, histograms, correlation charts, etc.
ISO/IEC TR 9126-3:2003(E)
46 © ISO/IEC 2003 – All rights reserved
Annex B (informative)
Use of Quality in Use, External & Internal Metrics (Framework Example)
B.1 Introduction This framework example is a high level description of how the ISO/IEC 9126 Quality model and related metrics may be used during the software development and implementation to achieve a quality product that meets user’s specified requirements. The concepts shown in this example may be implemented in various forms of customization to suit the individual, organization or project. The example uses the key life cycle processes from ISO/IEC 12207 as a reference to the traditional software development life cycle and quality evaluation process steps from ISO/IEC 14598-3 as a reference to the traditional Software Product Quality evaluation process. The concepts can be mapped onto other models of software life cycles if the user so wishes as long as the underlying concepts are understood.
B.2 Overview of Development and Quality Process Table B.1 depicts an example model that links the Software Development life cycle process activities (activity 1 to activity 8) to their key deliverables and the relevant reference models for measuring quality of the deliverables (i.e., Quality in Use, External Quality, or Internal Quality).
Row 1 describes the software development life cycle process activities. (This may be customized to suit individual needs). Row 2 describes whether an actual measure or a prediction is possible for the category of measures (i.e., Quality in Use, External Quality, or Internal Quality). Row 3 describes the key deliverable that may be measured for Quality and Row 4 describes the metrics that may be applied on each deliverable at each process activity.
Table B.1 Quality Measurement Model
Activity 1 Activity 2 Activity 3 Activity 4 Activity 5 Activity 6 Activity 7 Activity 8 Phase Requirement analysis
(Software and systems)
Architectural design (Software and systems)
Software detailed design
Software coding and testing
Software integration and software qualification testing
System integration and system qualification testing
Software installation
Software acceptance support
9126 series model reference
Required user quality, Required internal quality, Required external quality
Predicted quality in use, Predicted external quality, Measured internal quality
Predicted quality in use, Predicted external quality, Measured internal quality
Predicted quality in use, Measured external quality, Predicted external quality, Measured internal quality
Predicted quality in use, Measured external quality, Predicted external quality, Measured internal quality
Predicted quality in use, Measured external quality, Measured internal quality
Predicted quality in use, Measured external quality, Measured internal quality
Measured quality in use, Measured external quality, Measured internal quality
Key deliverables of activity
User quality requirements (specified), External quality requirements (specified), Internal quality requirements (specified)
Architecture design of Software / system
Software detailed design
Software code, Test results
Software product, Test results
Integrated system, Test results
Installed system
Delivered software product
Metrics used to measure
Internal metrics (External metrics may be applied to validate specifications)
Internal metrics
Internal metrics
Internal metrics External metrics
Internal metrics External metrics
Internal metrics External metrics
Internal metrics External metrics
Quality in use metrics Internal metrics External metrics
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 47
B.3 Quality Approach Steps
B.3.1 General
Evaluation of the Quality during the development cycle is divided into the following steps. Step 1 has to be completed during the Requirement Analysis activity. Steps 2 to 5 have to be repeated during each process activity defined above.
B.3.2 Step #1 Quality requirements identification
For each of the Quality characteristics and subcharacteristics defined in the Quality model determine the User Needs weights using the two examples in Table B.2 for each category of the measurement. (Quality in Use, External and Internal Quality). Assigning relative weights will allow the evaluators to focus their efforts on the most important subcharacteristics.
Table B.2 User Needs Characteristics & Weights (a)
Quality in Use
CHARACTERISTIC WEIGHT (High/Medium/Low)
Effectiveness H Productivity H Safety L
Satisfaction M (b)
External & Internal Quality
CHARACTERISTIC SUBCHARACTERISTIC WEIGHT (High/Medium/Low)
Suitability H Accuracy H Interoperability L Security L
Functionality
Compliance M Maturity (hardware/software/data) L
Fault tolerance L Recoverability (data, process, technology) H
Reliability
Compliance H Understandability M Learnability L Operability H Attractiveness M
Usability
Compliance H Time behaviour H Resource utilization H
Efficiency
Compliance H Analyzability H Changeability M Stability L Testability M
Maintainability
Compliance H Adaptability H Installability L Co-existence H Replaceability M
Portability
Compliance H
NOTE Weights can be expressed in the High/Medium/Low manner or using the ordinal type scale in the range 1-9 (e.g.: 1-3 = low, 4-6 = medium, 7-9 = high).
ISO/IEC TR 9126-3:2003(E)
48 © ISO/IEC 2003 – All rights reserved
B.3.3 Step #2 Specification of the evaluation
This step is applied during every development process activity.
For each of the Quality subcharacteristics defined in the Quality model identify the metrics to be applied and the required levels to achieve the User Needs set in Step 1 and record as shown in the example in Table B.3.
Basic input and directions for the content formulation can be obtained from the example in Table B.1 that explains what can be measured at this stage of the development cycle.
NOTE It is possible, that some of the rows of the tables would be empty during the specific activities of the development cycle, because it would not be possible to measure all of the subcharacteristics early in the development process.
Table B.3 Quality measurement tables (a)
Quality in Use Measurement Category
CHARACTERISTIC METRICS REQUIRED LEVEL
ASSESSMENT ACTUAL RESULT
Effectiveness Productivity Safety
Satisfaction (b)
External Quality Measurement Category
CHARACTERISTIC SUBCHARACTERISTIC METRICS REQUIRED LEVEL
ASSESSMENT ACTUAL RESULT
Suitability Accuracy Interoperability Security
Functionality
Compliance Maturity (hardware/software/data)
Fault tolerance Recoverability (data, process, technology)
Reliability
Compliance Understandability Learnability Operability Attractiveness
Usability
Compliance Time behaviour Resource utilisation
Efficiency
Compliance Analyzability Changeability Stability Testability
Maintainability
Compliance
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 49
Adaptability Instability Co-existence Replaceability
Portability
Compliance
(c)
Internal Quality Measurement Category
CHARACTERISTIC SUBCHARACTERISTIC METRICS REQUIRED LEVEL
ASSESSMENT ACTUAL RESULT
Suitability Accuracy Interoperability Security
Functionality
Compliance Maturity (hardware/software/data)
Fault tolerance Recoverability (data, process, technology)
Reliability
Compliance Understandability Learnability Operability Attractiveness
Usability
Compliance Time behaviour Resource utilisation
Efficiency
Compliance Analyzability Changeability Stability Testability
Maintainability
Compliance Adaptability Instability Co-existence Replaceability
Portability
Compliance
ISO/IEC TR 9126-3:2003(E)
50 © ISO/IEC 2003 – All rights reserved
B.3.4 Step #3 Design of the evaluation
This step is applied during every development process activity.
Develop a measurement plan (similar to example in Table B.4) containing the deliverables that are used as input to the measurement process and the metrics to be applied.
Table B.4 Measurement plan
SUBCHARACTERISTIC DELIVERABLES TO BE EVALUATED
INTERNAL METRICS TO BE
APPLIED
EXTERNAL METRICS TO BE
APPLIED
QUALITY IN USE METRICS TO BE
APPLIED
1. Suitability 1. 2. 3.
1. 2. 3.
1. 2. 3.
(Not Applicable)
2. Satisfaction 1. 2. 3.
(Not Applicable) (Not Applicable) 1. 2. 3.
3. 4. 5. 6.
B.3.5 Step #4 Execution of the evaluation
This step is applied during every development process activity.
Execute the evaluation plan and complete the column as shown in the examples in Table B.3. ISO/IEC 14598 series of standards should be used as a guidance for planning and executing the measurement process.
B.3.6 Step #5 Feedback to the organization
This step is applied during every development process activity.
Once all measurements have been completed map the results into Table B.1 and document conclusions in the form of a report. Also identify specific areas where quality improvements are required for the product to meet the user needs.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 51
Annex C (informative)
Detailed explanation of metric scale types and measurement types
C.1 Metric Scale Types
One of the following measurement metric scale types should be identified for each measure, when a user of metrics has the result of a measurement and uses the measure for calculation or comparison. The average, ratio or difference values may have no meaning for some measures. Metric scale types are: Nominal scale, Ordinal scale, Intervals scale, Ratio scale, and Absolute scale. A scale should always be defined as M'=F(M), where F is the admissible function. Also the description of each measurement scale type contains a description of the admissible function (if M is a metric then M'=F(M) is also a metric).
(a) Nominal Scale
M'=F(M) where F is any one-to-one mapping.
This includes classification, for example, software fault types (data, control, other). An average has a meaning only if it is calculated with frequency of the same type. A ratio has a meaning only when it is calculated with frequency of each mapped type. Therefore, the ratio and average may be used to represent a difference in frequency of only the same type between early and later cases or two similar cases. Otherwise, they may be used to mutually compare the frequency of each other type respectively.
Examples: Town transport line identification number, Compiler error message identification number
Meaningful statements are Numbers of different categories only.
(b) Ordinal Scale
M'=F(M) where F is any monotonic increasing mapping that is, M(x)>=M(y) implies M'(x)>=M'(y).
This includes ordering, for example, software failure by severity (negligible, marginal, critical, catastrophic). An average has a meaning only if it is calculated with frequency of the same mapped order. A ratio has a meaning only when it is calculated with the frequency of each mapped order. Therefore, the ratio and the average may be used to represent a difference in frequency of only the same order between early and later cases or two similar cases. Otherwise, they may be used to compare mutually the frequency of each order.
Examples: School exam.result (excellent, good, acceptable, not acceptable),
Meaningful statements: Each will depend on its position in the order, for example the median.
(c) Interval Scale
M'=aM+b (a>0)
This includes ordered rating scales where the difference between two measures has an empirical meaning. However the ratio of two measures in an interval scale may not have the same empirical meaning.
Examples: Temperature (Celsius, Fahrenheit, Kelvin), difference between the actual computation time and the time predicted
Meaningful statements: An arithmetic average and anything that depends on an order
ISO/IEC TR 9126-3:2003(E)
52 © ISO/IEC 2003 – All rights reserved
(d) Ratio Scale
M'=aM (a>0)
This includes ordered rating scales, where the difference between two measures and also the proportion of two measures have the same empirical meaning. An average and a ratio have meaning respectively and they give actual meaning to the values.
Examples: Length, Weight, Time, Size, Count
Meaningful statements: Geometrical mean, Percentage
(e) Absolute Scale
M'=M they can be measured only in one way.
Any statement relating to measures is meaningful. For example the result of dividing one ratio scale type measure by another ratio scale type measure where the unit of measurement is the same is absolute. An absolute scale type measurement is in fact one without any unit.
Example: Number of lines of code with comments divided by the total lines of code
Meaningful statements: Everything
C.2 Measurement Types
C.2.0 General In order to design a procedure for collecting data, interpreting fair meanings, and normalizing measures for comparison, a user of metrics should identify and take account of the measure type of measurement employed by a metric.
C.2.1 Size Measure Type
C.2.1.0 General
A measure of this type represents a particular size of software according to what it claims to measure within its definition.
NOTE Software may have many representations of size (like any entity can be measured in more than one dimension - mass, volume, surface area etc.).
Normalizing other measures with a size measure can give comparable values in terms of units of size. The size measures described below can be used for software quality measurement.
C.2.1.1 Functional Size Type
Functional size is an example of one type of size (one dimension) that software may have. Any one instance of software may have more than one functional size depending on, for example:
(a) the purpose for measuring the software size (It influences the scope of the software included in the measurement);
(b) the particular functional sizing method used (It will change the units and scale).
The definition of the concepts and process for applying a functional size measurement method (FSM Method) is provided by the standard ISO/IEC 14143-1.
In order to use functional size for normalization it is necessary to ensure that the same functional sizing method is used and that the different software being compared have been measured for the same purpose and consequently have a comparable scope.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 53
Although the following often claim that they represent functional sizes, it is not guaranteed they are equivalent to the functional size obtained from applying a FSM Method compliant with ISO/IEC 14143-1. However, they are widely used in software development:
1. number of spread sheets;
2. number of screens;
3. number of files or data sets which are processed;
4. number of itemized functional requirements described in user requirements specifications.
C.2.1.2 Program size type
In this clause, the term ‘programming’ represents the expressions that when executed result in actions, and the term ‘language’ represents the type of expression used.
1. Source program size
The programming language should be explained and it should be provided how the non executable statements, such as comment lines, are treated. The following measures are commonly used.
Non-comment source statements (NCSS) include executable statements and data declaration statements with logical source statements.
NOTE 1 New program size
A developer may use newly developed program size to represent development and maintenance work product size.
NOTE 2 Changed program size
A developer may use changed program size to represent size of software containing modified components.
NOTE 3 Computed program size
Example of computed program size formula is new lines of code + 0.2 x lines of code in modified components (NASA Goddard).
It may be necessary to distinguish a type of statements of source code into more detail as follows:
i. Statement Type
Logical Source Statement (LSS). The LSS measures the number of software instructions. The statements are irrespective of their relationship to lines and independent of the physical format in which they appear.
Physical Source Statement (PSS). The PSS measures the number of software source lines of code.
ii. Statement attribute
Executable statements;
Data declaration statements;
Compiler directive statements;
Comment source statements.
iii. Origin
Modified source statements;
Added source statements;
ISO/IEC TR 9126-3:2003(E)
54 © ISO/IEC 2003 – All rights reserved
Removed source statements;
♦ Newly Developed source statements: (= added source statements + modified source statements);
♦ Reused source statements: (= original - modified - removed source statements);
2. Program word count size The measurement may be computed in the following manner using the Halstead's measure:
Program vocabulary = n1+n2; Observed program length = N1+N2, where:
• n1: Is the number of distinct operator words which are prepared and reserved by the program language in a program source code;
• n2: Is the number of distinct operand words which are defined by the programmer in a program source code;
• N1: Is the number of occurrences of distinct operators in a program source code;
• N2: Is the number of occurrences of distinct operands in a program source code.
3. Number of modules The measurement is counting the number of independently executable objects such as modules of a program.
C.2.1.3 Utilized resource measure type
This type identifies resources utilized by the operation of the software being evaluated. Examples are:
(a) Amount of memory, for example, amount of disk or memory occupied temporally or permanently during the software execution;
(b) I/O load, for example, amount of traffic of communication data (meaningful for backup tools on a network);
(c) CPU load, for example, percentage of occupied CPU instruction sets per second (This measure type is meaningful for measuring CPU utilization and efficiency of process distribution in multi-thread software running on concurrent/parallel systems);
(d) Files and data records, for example, length in bytes of files or records;
(e) Documents, for example, number of document pages.
It may be important to take note of peak (maximal), minimum and average values, as well as periods of time and number of observations done.
C.2.1.4 Specified operating procedure step type
This type identifies static steps of procedures which are specified in a human-interface design specification or a user manual.
The measured value may differ depending on what kinds of description are used for measurement, such as a diagram or a text representing user operating procedures.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 55
C.2.2 Time measure type
C.2.2.0 General
The user of metrics of time measure type should record time periods, how many sites were examined and how many users took part in the measurements.
There are many ways in which time can be measured as a unit, as the following examples show.
(a) Real time unit
This is a physical time: i.e. second, minute, or hour. This unit is usually used for describing task processing time of real time software.
(b) Computer machinery time unit
This is computer processor's clock time: i.e. second, minute, or hour of CPU time.
(c) Official scheduled time unit
This includes working hours, calendar days, months or years.
(d) Component time unit
When there are multiple sites, component time identifies individual site and it is an accumulation of individual time of each site. This unit is usually used for describing component reliability, for example, component failure rate.
(e) System time unit
When there are multiple sites, system time does not identify individual sites but identifies all the sites running, as a whole in one system. This unit is usually used for describing system reliability, for example, system failure rate.
C.2.2.1 System operation time type
System operation time type provides a basis for measuring software availability. This is mainly used for reliability evaluation. It should be identified whether the software is under discontinuous operation or continuous operation. If the software operates discontinuously, it should be assured that the time measurement is done on the periods the software is active (this is obviously extended to continuous operation).
(a) Elapsed time
When the use of software is constant, for example in systems operating for the same length of time each week.
(b) Machine powered-on time
For real time, embedded or operating system software that is in full use the whole time the system is operational.
(c) Normalized machine time
As in “machine powered-on time”, but pooling data from several machines of different “powered-on-time” and applying a correction factor.
C.2.2.2 Execution time type
Execution time type is the time which is needed to execute software to complete a specified task. The distribution of several attempts should be analysed and mean, deviation or maximal values should be
ISO/IEC TR 9126-3:2003(E)
56 © ISO/IEC 2003 – All rights reserved
computed. The execution under the specific conditions, particularly overloaded condition, should be examined. Execution time type is mainly used for efficiency evaluation.
C.2.2.3 User time type
User time type is measured upon time periods spent by individual users on completing tasks by using operations of the software. Some examples are:
(a) Session time
Measured between start and end of a session. Useful, as example, for drawing behaviour of users of a home banking system. For an interactive program where idling time is of no interest or where interactive usability problems only are to be studied.
(b) Task time
Time spent by an individual user to accomplish a task by using operations of the software on each attempt. The start and end points of the measurement should be well defined.
(c) User time
Time spent by an individual user using the software from time started at a point in time. (Approximately, it is how many hours or days user uses the software from beginning).
C.2.2.4 Effort type
Effort type is the productive time associated with a specific project task.
(a) Individual effort
This is the productive time which is needed for the individual person who is a developer, maintainer, or operator to work to complete a specified task. Individual effort assumes only a certain number of productive hours per day.
(b) Task effort
Task effort is an accumulated value of all the individual project personnel: developer, maintainer, operator, user or others who worked to complete a specified task.
C.2.2.5 Time interval of events type
This measure type is the time interval between one event and the next one during an observation period. The frequency of an observation time period may be used in place of this measure. This is typically used for describing the time between failures occurring successively.
C.2.3 Count measure type
If attributes of documents of the software product are counted, they are static count types. If events or human actions are counted, they are kinetic count types.
C.2.3.1 Number of detected fault type
The measurement counts the detected faults during reviewing, testing, correcting, operating or maintaining. Severity levels may be used to categorize them to take into account the impact of the fault.
C.2.3.2 Program structural complexity number type
The measurement counts the program structural complexity. Examples are the number of distinct paths or the McCabe's cyclomatic number.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 57
C.2.3.3 Number of detected inconsistency type
This measure counts the detected inconsistent items which are prepared for the investigation.
(a) Number of failed conforming items
Examples:
• Conformance to specified items of requirements specifications;
• Conformance to rule, regulation, or standard;
• Conformance to protocols, data formats, media formats, character codes.
(b) Number of failed instances of user expectation
The measurement is to count satisfied/unsatisfied list items, which describe gaps between user's reasonable expectation and software product performance.
The measurement uses questionnaires to be answered by testers, customers, operators, or end users on what deficiencies were discovered.
The following are examples:
• Function available or not;
• Function effectively operable or not;
• Function operable to user's specific intended use or not;
• Function is expected, needed or not needed.
C.2.3.4 Number of changes type
This type identifies software configuration items which are detected to have been changed. An example is the number of changed lines of source code.
C.2.3.5 Number of detected failures type
The measurement counts the detected number of failures during product development, testing, operating or maintenance. Severity levels may be used to categorize them to take into account the impact of the failure.
C.2.3.6 Number of attempts (trial) type
This measure counts the number of attempts at correcting the defect or fault. For example, during reviews, testing, and maintenance.
C.2.3.7 Stroke of human operating procedure type
This measure counts the number of strokes of user human action as kinetic steps of a procedure when a user is interactively operating the software. This measure quantifies the ergonomic usability as well as the effort to use. Therefore, this is used in usability measurement. Examples are number of strokes to perform a task, number of eye movements, etc.
C.2.3.8 Score type
This type identifies the score or the result of an arithmetic calculation. Score may include counting or calculation of weights checked on/off on checklists. Examples: Score of checklist; score of questionnaire; Delphi method; etc.
ISO/IEC TR 9126-3:2003(E)
58 © ISO/IEC 2003 – All rights reserved
Annex D (informative)
Term(s)
D.1 Definitions
Definitions are from ISO/IEC 14598-1 and ISO/IEC 9126-1 unless otherwise indicated.
D.1.1 Quality
External quality: The extent to which a product satisfies stated and implied needs when used under specified conditions.
Internal quality: The totality of attributes of a product that determine its ability to satisfy stated and implied needs when used under specified conditions.
NOTE 1 The term “attribute” is used (rather than the term “characteristic” used in 3.1.3) as the term “characteristic” is used in a more specific sense in ISO/IEC 9126 series.
Quality: The totality of characteristics of an entity that bear on its ability to satisfy stated and implied needs.
NOTE 2 In a contractual environment, or in a regulated environment, such as the nuclear safety field, needs are specified, whereas in other environments, implied needs should be identified and defined.
Quality in use: The capability of the software product to enable specified users to achieve specified goals with effectiveness, productivity, safety and satisfaction in specified contexts of use.
NOTE 3 Quality in use is the user’s view of the quality of an environment containing software, and is measured from the results of using the software in the environment, rather than properties of the software itself.
NOTE 4 The definition of quality in use in ISO/IEC 14598-1 does not currently include the new characteristic of “safety”.
Quality model: The set of characteristics and the relationships between them, which provide the basis for specifying quality requirements and evaluating quality.
D.1.2 Software and user
Software: All or part of the programs, procedures, rules, and associated documentation of an information processing system. (ISO/IEC 2382-1:1993)
NOTE 1 Software is an intellectual creation that is independent of the medium on which it is recorded.
Software product: The set of computer programs, procedures, and possibly associated documentation and data designated for delivery to a user. [ISO/IEC 12207]
NOTE 2 Products include intermediate products, and products intended for users such as developers and maintainers.
User: An individual that uses the software product to perform a specific function.
NOTE 3 Users may include operators, recipients of the results of the software, or developers or maintainers of software.
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 59
D.1.3 Measurement
Attribute: A measurable physical or abstract property of an entity.
Direct measure: A measure of an attribute that does not depend upon a measure of any other attribute.
External measure: An indirect measure of a product derived from measures of the behaviour of the system of which it is a part.
NOTE 1 The system includes any associated hardware, software (either custom software or off-the-shelf software) and users.
NOTE 2 The number of faults found during testing is an external measure of the number of faults in the program because the number of faults are counted during the operation of a computer system running the program to identify the faults in the code.
NOTE 3 External measures can be used to evaluate quality attributes closer to the ultimate objectives of the design.
Indicator: A measure that can be used to estimate or predict another measure.
NOTE 4 The measure may be of the same or a different characteristic.
NOTE 5 Indicators may be used both to estimate software quality attributes and to estimate attributes of the production process. They are indirect measures of the attributes.
Indirect measure: A measure of an attribute that is derived from measures of one or more other attributes.
NOTE 6 An external measure of an attribute of a computing system (such as the response time to user input) is an indirect measure of attributes of the software as the measure will be influenced by attributes of the computing environment as well as attributes of the software.
Internal measure: A measure derived from the product itself, either direct or indirect; it is not derived from measures of the behaviour of the system of which it is a part.
NOTE 7 Lines of code, complexity, the number of faults found in a walk through and the Fog Index are all internal measures made on the product itself.
Measure (noun): The number or category assigned to an attribute of an entity by making a measurement.
Measure (verb): Make a measurement.
Measurement: The process of assigning a number or category to an entity to describe an attribute of that entity.
NOTE 8 “Category” is used to denote qualitative measures of attributes. For example, some important attributes of software products, e.g. the language of a source program (ADA, C, COBOL, etc.) are qualitative.
Metric: A measurement scale and the method used for measurement.
NOTE 9 Metrics can be internal or external.
Metrics includes methods for categorizing qualitative data.
ISO/IEC TR 9126-3:2003(E)
60 © ISO/IEC 2003 – All rights reserved
Annex E (informative)
Pure Internal Metrics
E.1 Pure Internal Metrics
Pure Internal metrics are used to measure certain attributes of the software design and code of the software product that will influence the same or all of the overall software characteristics and subcharacteristics
ISO/IEC TR 9126-3:2003(E)
© ISO/IEC 2003 – All rights reserved 61
Tabl
e E.
1.1
Pur
e In
tern
al M
etric
s
Met
ric N
ame
Purp
ose
of th
e m
etric
s M
etho
d of
ap
plic
atio
n M
easu
rem
ent,
form
ula
and
data
el
emen
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
Scal
e ty
peM
easu
re ty
pe
Inpu
t to
mea
sure
men
t IS
O/IE
C
1220
7 re
fere
nce
Targ
et
Audi
ence
s
Coh
eren
ce
Tr
acea
bilit
y To
mea
sure
ef
fect
ivene
ss o
f do
cum
enta
tion
and
desig
n st
ruct
ure
and
code
of s
oftw
are
prod
uct
in m
appi
ng fu
nctio
ns
from
requ
irem
ents
to
impl
emen
tatio
n.
X=
A/B
A=N
umbe
r of t
race
able
item
s co
nfirm
ed in
revie
w B=
Num
ber o
f ite
ms
chec
ked
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e be
tter.
Abso
lute
X=
coun
t/cou
nt
A=co
unt
B=co
unt
Cyc
lom
atic
nu
mbe
r To
mea
sure
the
leve
l of
com
plex
ity o
f the
so
ftwar
e de
sign
and
codi
ng s
truct
ure.
e-
n+2p
e:
# o
f sid
es
n: #
of e
dges
p:
# o
f adj
acen
t com
pone
nts
Info
rmat
ion
Flow
C
ompl
exity
To
mea
sure
com
plex
ity
of d
esig
n co
ntro
l st
ruct
ure.
(re
fer t
o IE
EE 9
82.1
)
IF
C(In
form
atio
n Fl
ow C
ompl
exity
) =(
fani
n x
fano
ut)2
Self-
desc
riptiv
enes
s
Mod
ular
ity
To m
easu
re th
e ea
sine
ss to
upd
ate
and
gene
raliz
e th
e fu
nctio
nal k
now
ledg
e ba
se o
n pr
ogra
m
func
tion/
data
, se
quen
ce o
f exe
cutio
n,
and
hier
arch
y of
con
trol
flow
.
X1
=A1/
B1
Whe
re A
1=th
e nu
mbe
r of m
odul
es
that
are
func
tiona
lly a
ssoc
iate
d w
ith e
ach
othe
r, an
d B1
=the
nu
mbe
r of m
odul
es
X2=A
2/B2
W
here
A2=
the
num
ber o
f mod
ules
th
at a
re a
ssoc
iate
d w
ith e
ach
othe
r in
data
stru
ctur
e, a
nd
B2=t
he n
umbe
r of m
odul
es
Self-
cont
aine
dnes
s
Prog
ram
siz
e To
mea
sure
the
prog
ram
sc
ale.
(N1+
N2)
log2
(n1+
n2)
N
1: o
pera
tor o
ccur
renc
es
N2:
ope
rand
occ
urre
nces
n1
: tot
al #
of o
pera
tors
n2
: tot
al #
of o
pera
nds
ISO/IEC TR 9126-3:2003(E)
62 © ISO/IEC 2003 – All rights reserved
Tabl
e E.
1.1
(con
tinue
d)
Met
ric N
ame
Purp
ose
of th
e m
etric
s M
etho
d of
ap
plic
atio
n M
easu
rem
ent,
form
ula
and
data
el
emen
t com
puta
tions
In
terp
reta
tion
of m
easu
red
valu
e
Met
ric
Scal
e ty
peM
easu
re ty
pe
Inpu
t to
mea
sure
men
t IS
O/IE
C
1220
7 re
fere
nce
Targ
et
Audi
ence
s
Con
ditio
nal
stat
emen
t To
mea
sure
the
com
plex
ity le
vel o
f cod
ed
mod
ules
.
X=
A
A= N
umbe
r of c
ondi
tiona
l st
atem
ents
0 <=
X
X=
size
A=siz
e
Uni
fied
data
re
fere
nce
To m
easu
re th
e da
ta
unific
atio
n.
X=
A/B
A=
Num
ber o
f dat
a re
fere
nces
with
un
ified
nam
e co
nfirm
ed in
revie
w
B=To
tal n
umbe
r of d
ata
refe
renc
es
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e be
tter.
Abso
lute
X=
coun
t/cou
nt
A=co
unt B
=cou
nt
Adeq
uacy
of
varia
ble
nam
es
To m
easu
re th
e va
riabl
e na
mes
ade
quac
y.
X=
A/B
A=
Num
ber o
f var
iabl
es w
ith
adeq
uate
nam
es c
onfir
med
in re
view
B=
Tota
l num
ber o
f var
iabl
es
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e be
tter.
Abso
lute
X=
coun
t/cou
nt
A=co
unt B
=cou
nt
Dat
a-co
uple
d m
odul
e ra
tio
To m
easu
re th
e da
ta-
coup
led
mod
ule
ratio
.
X=A/
B
A=N
umbe
r of d
ata-
coup
led
mod
ules
co
nfirm
ed in
revie
w B=
Tota
l num
ber
of a
ll mod
ules
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e be
tter.
Abso
lute
X=
coun
t/cou
nt
A=co
unt B
=cou
nt
Prog
ram
st
atem
ents
To
mea
sure
the
prog
ram
so
urce
sta
tem
ent.
X=
A
A=To
tal n
umbe
r of p
rogr
am
stat
emen
ts
0 <=
X
X=
size
A=siz
e
Aver
age
mod
ule
size
To
mea
sure
the
aver
age
mod
ule
size.
X=A/
B
A=To
tal li
nes
of s
ourc
e st
atem
ents
in
all m
odul
es B
=Tot
al n
umbe
r of a
ll m
odul
es
0 <=
X
Abso
lute
X=
size
A=siz
e
Func
tion-
coup
led
mod
ule
ratio
To m
easu
re th
e fu
nctio
n-co
uple
d m
odul
e ra
tio.
X=
A/B
A=
Num
ber o
f fun
ctio
n-co
uple
d m
odul
es c
onfir
med
in re
view
B=To
tal n
umbe
r of a
ll mod
ules
0 <=
X <
= 1
Th
e clo
ser t
o 1,
th
e be
tter.
Abso
lute
X=
coun
t/cou
nt
A=co
unt B
=cou
nt
ISO/IEC TR 9126-3:2003(E)
ICS 35.080 Price based on 62 pages
© ISO/IEC 2003 – All rights reserved