Title: call tomorrow Wed 8th, 3pm PT
Mike:
inline +
attached comments on Section 7.1.
Jacques
-----Original Message-----
From: Michael Kass
[mailto:]
Sent: Tuesday, September 07, 2004
3:59 PM
To: Jacques Durand; ebXML IIC - main list (E-mail)
(E-mail)
Subject: Re: [ebxml-iic] call tomorrow Wed 8th, 3pm
PT
Jacques,
I
modified section 7.1 based on comments from last conference call (changed
structure/design of logic flow HTML table. Plus I modified the "pass", "fail",
"undetermined" description at the end.
Please see attached document (turn on "tracking" to see
changes).
It seems that the last sticking point is the "default"
behavior of a Test Driver when
encountering a "failed" TestAssertion.
Please see my comments below:
[Durand]
It is
annoying that the interpretation of an assertion failure (what to do if fails)
, is context-dependent (presence of a step behind or not...) Shouldn't we say
that an assertion failure just aborts the thread, in the absence of any other
explicit exit statement?
[MIKE] - Why can't there be a "default" behavior for a
failed TestAssertion. The
fundamental meaning of TestAssertion is just that.. testing an assertion. Generally, if such an assertion is not
met, this means the implementation failed. However, we have created a more
flexible meaning..saying that one can now "continue" from a failed
TestAssertion..or conceivably even "pass" after failing a TestAsssertion
operation. I believe that the
meaning has been "watered down" to the point where it is now nothing more than
a logic flow operator. I
believe that the default behavior should be "fail" for a "failed"
TestAssertion. One can still
"override" it using the <Return> instruction ( or <Abort>...
although I think that you suggested that we not use <Abort>). I am in favor of trying to keep close
to the meaning of <TestAssertion> as we originally defined it.
[Jacques
Durand] I think we have this default meaning now (Section 7.1) , that
a "false" TA will by default exit the test case on failure. I think
this is OK for now. I suggest we may introduce "return" but not "abort" which
requires new semantics that is a bit late to introduce for this release. We
need more time to think of it.
[Durand]
This thread may be or-joined later and
the entire test case may still succeed thanks to other concurrent threads that
have successfully completed.
[MIKE] - I'm OK with that. I just do not think that
<Abort> should be the default Test Driver behavior when a TestAssertion
fails. In our 216 ebMS Test
Cases, that would result in 216 "undetermined" Test Case results.
[Jacques
Durand] agree.
Can't we
say that an aborted test case wil be by default interpreted as
"Undetermined"?
[MIKE] - An "aborted" Test Case should result in an "undetermined"
result. But I do not believe that
a "false" TestAssertion result
should not (by default) <Abort> a Test Case.
Below are the rules added to the end of section 7.1
for your review
[Jacques Durand] I commented on these in the
attachment. They look OK to me overall - only I would handle separately the
first "continue" state (would not call it a state, but as default "action" on
the boolean value of a TA) (see my attached
comemnts)
A "default" Test Case state of
"continue" exists after a
TestAssertion operation returns a boolean result of "true" , or a
"Return" instruction is encountered by the Test Driver, indicating that the
current Thread MUST be terminated, but that logic flow will continue within
the "calling" Thread.
A final Test Case state of
"pass" occurs when the
Test Driver encounters an explicit "exit/pass" instruction from within a
TestAssertion, OR logical testing execution proceeds from beginning to end
without:
a) An explicit "exit/fail" or
"exit/undetermined"
b) A system "exception"
condition
c) A failed Join that precludes further
testing execution beyond the Join
A final Test Case state of
"fail" is given to a TestCase where :
a) A TestAssertion boolean operation
returns a result of "false" (default behavior) (i.e. it is assumed that
a TestAssertion is a meaningful measurement of conformance/interoperability
that MUST pass.. unless explcitly overriden by the test writer in such cases
where a TestAssertion verifies a "precondition" to further testing, in which
case the test writer may wish to "exit" with a state of
"undetermined". Additionally, the test writer may with to
"continue" if the failed result of the TestAssertion is used to alter the flow
of Test Case execution.
b) The
Test Driver encounters an
explicit "exit/fail" instruction within a TestAssertion
operation.
A final Test Case state of
"undetermined" occurs if:
1) An explicit "exit/undetermined"
instruction occurs from a TestAssertion
operation
2) Testing logic
flow cannot proceed beyond a Join due to failed andJoin or orJoin
condition
Comments?
Mike
----- Original Message -----
From:
Jacques
Durand
To: ebXML IIC - main list (E-mail)
(E-mail)
Sent: Tuesday, September 07, 2004 5:27
PM
Subject: [ebxml-iic] call tomorrow Wed
8th, 3pm PT
We'll have exceptionally a meeting tomorrow Wednesday at
3pm PT.
Primary objective is to close on Test Framework spec
remaining points being discussed.
I'll mostly
give a quick update on #2, #3 in agenda below.
We'll likely postpone #4 to Monday, unless Patrick is on the
call.
Jacques
Time: Wed 8th, 3pm-4pm PT
Host: Fujitsu
Toll only : 1-512-225-3050
Participant code: 385218#
tentative Agenda :
1. Test Framework 1.1 technical:
- use cases , BPSS relevance.
- Threads: abort semantics, return
feature.
- Monica
comments.
- levels of
conformance, if any?
- any
"binding" to consider? (optional, but normative)
2. ETSI follow-up:
- CD status for the test plan
- feedback on test suites: few bugs detected,
updates needed.
3. Request from ebXML Asia for a n
Interop test suite without:
-
sync messaging (bus responses only?)
- empty messages.
4. Others:
- follow-up/update on Patrick Hogan / Linux
plans.
section7.1-JD1_MK_JD2_MK2-JD3.doc
