Title: test case outcomes: semantics
Jacques,
I agree with the addition of a fourth
"final state" for a Test Case (exception) and your definitions of each. I
think we can add this as an insert, while still keeping the "concrete" meaning
of
terms (in terms of actual Test Case execution) at
the end of that section. In fact, I would suggest having both the
"semantic" meaning of these final states, along with the "concrete" (for lack
of
a better word) meaning.
Mike
----- Original Message -----
From:
Jacques
Durand
To: ebXML IIC - main list (E-mail)
(E-mail)
Sent: Tuesday, August 17, 2004 2:33
PM
Subject: [ebxml-iic] test case outcomes:
semantics
Miekand all:
here is a proposed insert on the meaning of test
case outcomes:
For review
(note: there may be a blurred distinction between som cases
of "Undetermined" and "system exception")
Jacques
Pass:
-----
Meaning:
The material under test (e.g. messages from the
implementation under test) is satisfying
the test requirement associated with the test case.
Fail:
-----
Meaning:
The material under test (e.g. messages from the
implementation under test)
does not
satisfy the test requirement associated with the test case.
Action:
The expected remedial actions include fixing the implementation under
test,
as this is a failure to
conform to the specification as being verified by this test suite.
Undetermined:
-------------
Meaning:
The test case execution could not conclude by either
a Fail or a Pass,
because of the
irrelevance of the material under test, or because the right context for this
test case could not
be created.
These reasons could be:
(a)- the
material under test is simply not relevant to the test requirement being
tested, and a preliminary test assertion (acting as a pre-requisite or
pre-condition) filtered this material out. (e.g. test case on conformance to
"simple SOAP", will not be relevant to messages generated as SWA, and the test
case may not be able to "force" the implementation under test to generate the
former instgead of the latter.)
(b)- The environment of the test case other
than the testbed itself (e.g. expected events from other parties assisting the
test case, expected resources external to the testbed such as an error log
site) cannot be obtained, or failed during execution, so that the outcome
would not be meaningful.
Action:
The expected remedial actions include, for (a) possibly replay the test
case (in case there are some chances the proper material can be obtained), and
for (b) fixing the test case environment, possibly the test case itself, so
that the test case execution can proceed.
System Exception:
----------------
Meaning:
The test case execution failed due to the inability
of the testbed to complete its execution
because of problems related to the testbed itself. E.g., a connection
between testbed components failed.
Or, material being manipulated (e.g. message data) is too large to be
handled by this version of the tsetbed.
Action:
The expected remedial actions includes either upgrading / fixing the
testbed, or modifying the testcases so that the material being handled can be
processed by the testbed.
