Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC- Proposal
Hi Martin,
As I said, I think I would be satisfied with one implementation that
passes the test suite. Then again, of the two announced
implementations I'm aware of, I don't know how close the non-Tuscany
one is to passing the test suite. If it passes already, then the
distinction between 0, 1, and 2 is mostly academic, isn't it?
-Eric.
On 2/25/11 6:25 AM, Martin Chapman wrote:
Eric,
See my answer to Jim. We can argue why we built
test suites, either to validate the spec or to test
conformance of implementations (they are both right IMHO)
but the real question is do we want two implementations to
pass the TC Test suite before we can progress to an OASIS
Standard vote.
Martin.
From: Eric Johnson
[mailto:]
Sent: 24 February 2011 21:40
To: Jim Marino
Cc: OASIS Assembly
Subject: Re: [sca-assembly] Concrete Exit
Criteria for the SCA Assembly TC - Proposal
Hi Jim, Jeff,
On 2/24/11 10:40 AM, Jim Ma! rino
wrote:
Hi Jeff,
Comments inline.
Jim
On Feb 23, 2011, at 3:05 PM, Jeff
Mischkinsky wrote:
On Feb 23, 2011, at 11:34 AM, Jim Marino wrote:
Hi Martin,
I guess I may now be even more
confused, probably because of my ignorance of OASIS
rules. Why would the exit criteria for the TC
require an implementation to adhere to something
that is not no! rmative,
specifically running and passing the TC-supplied
test suite? Also, who determines whether a runtime
passes or not?
I am having a hard time pinning down your confusion.
I guess my confusion was not clear
:-)
I understand that the the TC-supplied
test suite is not normative and that an entity
responsible for an SCA runtime can legitimately claim
conformance without running the TC-supplied test suite.
It is up to the court of public opinion to validate
conformance claims.
My question was why the exit criteria
would require two runtimes to also implement/run
non-normative TC artifacts? That seems inconsistent
with! how
other non-normative aspects of the specification are
handled, such as optional sections, which are not
required. Are there other OASIS TCs that have adopted
exit criteria which require implementations to pass
conformance tests?
<eej>
This point, at least, I understand. As I understand it, the
test suite is normative material, although not, as you note,
a criteria for conforming.
If I understood him correctly, during the bindings call this
morning, Mike Edwards indicated he disagrees with me, but I
do think that a key part of the test suite is to validate
the normative statements of the spec. If you can't write a
test for a particular normative statement, or a particular
clause of a normative statement, then perhaps the normative
statement shouldn't be normative. Or maybe you write the
test, and realize while thinking through all the cases that
the normative statement leaves possibilities out. Or maybe
writing t! he tests
highlights an area where there's too much implementation
leeway, and more normative statements might be in order. The
best way to find any of that out is to write a test suite.
As I recall, all of these situations arose during the
development of the test suites.
Now that we have the tests, do we need the them to run
against two implementations? Personally, I'd be satisfied by
one.
</eej>
There is a test suite that has been
developed by the TC. When you take your implementation
and run it through the test suite, you get an
"answer". That's the answer you are looking for.
The exit criteria require that there be 2 independent
impls that run the test suite, and get answer that
says "yes" (or all green ;-).
Did you intend to say "the proposed
exit criteria"? As it currently stands, the TC charter
says nothing about mandating an implementation run the
test suite. Assuming exit criteria were adopted that
mandated the test suite be passed by two independent
runtimes, how would that be done? In other words, who
decides and verifies the tests were "all green"? Does
the entity responsible for a runtime need to make the
test suite and integration code (or binaries) required
to plug the runtime into the test suite publicly
available? Or is it enough for the entity to claim its
runtime passes?
<eej>
I concur with Jim here. If an implementation professes to
implement the spec, but then fails the test suite, that's a
fairly obvious point that the marketplace can sort out.
Vendors might declare "we pass 90+%" for example. Leave it to
the end users to figure out whether that last X perc! ent
matters.
</eej>
jeff
Jim
On Feb 23, 2011, at 10:54 AM,
Martin Chapman wrote:
Jim,
I think you are confusing the
normative-ness of the test suite with TC exit
criteria.
You are correct that to claim
conformance you don’t have to pass the TC’s test
suite, but that is different from the TC saying
that in order to progress the spec to Committee
Specification/OASIS Standard we want two
implementations passing the TC’s test suite.
Martin.
From: Jim Marino [mailto:]
Sent: 22 February 2011 20:08
To: OASIS Assembly
Subject: Re: [sca-assembly]
Concrete Exit Criteria for t! he SCA Assembly TC -
Proposal
Hi,
The first bullet makes sense to
me. Concerning the second, my understanding is the
TC Test Suite is not normative and the only thing
required for conformance is for the entity
responsible for a particular runtime to declare
that it adheres to the normative portions of the
Assembly Specification. In other words, it should
be valid for a runtime to claim conformance that
supplies its own test suite, which may or may not
be based on the TC one. This essentially goes back
to using the "court of public opinion" to judge
conformance claims as opposed to something akin to
a TCK.
Jim
On Feb 18, 2011, at 6:19 AM,
Mike Edwards wrote:
Folks,
I propose the following as the
Exit Criteria to be adopted by the SCA Ass! embly
TC:
"The Concrete Exit Criteria for
the SCA Assembly V1.1 specification are that:
o there shall be 2 independent
SCA runtimes that are compliant with all normative
portions of the
specification as described in!
Section 12.2 of the SCA Assembly V1.1
specification
o the 2 independent runtimes
shall pass the Test Suite for SCA Assembly as
described in the document
'TestCases for the SCA Assembly
Model Version 1.1 Specification' "
I believe that this is
sufficient to cover:
- the question of conformance
to the spec
- the issues of portability and
interoperability
since the test suite is a
pretty thorough examination of conformance to the
spec (there are testcases for
all normative statements with
testable & observable consequences) and the
test suite is also a test of portability,
since the test suite has an
extensive set of ar! tifacts
that are ported to each runtime and
interoperability is
checked through the use of Web
services being used between the SCA runtime and a
non-SCA (JAXWS) client.
Yours, Mike
Dr Mike Edwards
Mail Point 146, Hursley Park
<Mail Attachment.gif>
STSM
Winchester, Hants SO21 2JN
SCA & Services Standards
United Kingdom
Co-Chair OASIS SCA Assembly TC
IBM Software Group
Phone:
+44-1962 818014
Mobile:
+44-7802-467431 (274097)
e-mail:
Unless stat! ed otherwise
above:
IBM United Kingdom Limited -
Registered in England and Wales with number
741598.
Registered office: PO Box 41,
North Harbour, Portsmouth, Hampshire PO6 3AU
--
Jeff Mischkinsky
Sr. Director, Oracle Fusion Middleware +1(650)506-1975
and Web Services
Standards 500
Oracle Parkway, M/S 2OP9
Oracle Redwood
Shores, CA 94065