Re: [ebxml-iic-msg] MS Conformance checklist: comments

To all,

 

    Sorry it took this long to respond.  
I was digesting the mail sent, and in particular studying the situation
we have regarding the need for a "testing API" if we are to follow
the approach Jacques outlines in his paper:  "ebXML MS Conformance
Testing: An Approach".   This paper clearly infers that
an API exists between the Test Driver and Candidate MSH ( an MSH adapter
) for "Sending Pattern" and "Receiving Pattern" . Two-way pattern" does
not require control on the Candidate side, and appears to be the only choice
we have without an MSH adapter on the candidate implementation.

    I have been talking to network people here regarding
testing efforts of protocols in the past, with the main question being..
what do we lose if we do a "black box" approach to testing ( via
a "service" as opposed to an API approach to testing? ).

    The answer is, of course that you lose a good amount
of control over the process by not having access to the candidate's inner
workings, and will have to rely on returned message responses, or message
traces in the header to determine the inner workings and conformance of
a candidate implementation. Lack of a standardized API for ebXML MS certainly
complicates matters, and asking implementors at this point to agree on
an API, or for that matter a testing API I believe would be difficult.

    Perhaps a "black box" testing effort, done
via an external service will get the conformance effort going, and as testing
evolves, a standardized API will be developed.

     The "Two-way pattern" Jacques describes in
his paper would appear to be the scenario that we could implement without
a testing API.

      I reply to Jacques and Matt's comments
below.

Jacques Durand wrote:
 

I suspect Michael Kass has not subscribed to this elist,

as he is unusually mute on all these mails (in fact,
I am almost sure of that,

as it is my privilege to get a notification of who is
subscribing :)

Michael: you have to subscribe (see OASIS site), or else
we assume you are

OK with all the work items we deflect on you!

Matt: just a few more comments below.

Jacques

-----Original Message-----

From: Matthew MacKenzie [mailto:]

Sent: Saturday, March 30, 2002 7:37 PM

To: Jacques Durand

Cc: 

Subject: Re: [ebxml-iic-msg] MS Conformance checklist:
comments

 

On Friday, March 29, 2002, at 05:01  PM, Jacques
Durand wrote:

> Matt, Michael:

>

> Looks like the "test requirement" material is almost
finalized.

> You seem also to have detailed ideas on how best execute
the test items.

>

> Would it be possible, by end of next week, to send
to the group

> (in preparation to April review):

>

> (a)- a "readable" list of proposed test items for MS
conformance

> (I guess pretty close to what XMLG has sent out)

[Matt] I worked on this for a couple hours today, so an
initial list based on

the items donated by XMLG will be sent to the list by
the end of the

week.  Its a big data entry job, basically translating
the XMLG list

into the more useful Requirements XML format.  The
good news is that

Michael has been developing a stylesheet along with the
schema, which

does a good job of making a readable list of the requirements.

> (b)- the subset of these items for "core features" (Level
1), cast into

> your XML format.

[Matt]See above.  The XMLG list contains things beyond
the level 1 scope, but

I am classifying them all as level one so that I can
get something out

faster.  What I will do is create a level two when
I am done and move

the appropriate items into it later on.

[Mike] I am looking at the stylized XML that Matt produced. 
It looks good right now from a readability standpoint.  Matt's perl
script did a good job of pulling out the requirements.  As Matt had
mentioned, the use of the <Condition> <Assertion> logic is not used
in his document, but could easily be modified to accommodate it.. Just
a lot of text editing.  But it is a great starting document.
 

> (c)- some overall design of the testing procedure(s)
and scenarios that

> would be used to

> execute these test items?

[Matt]Michael?

[Mike]  If it is agreed that a "service" ( non API ) approach
is to be taken toward conformance testing, then I can go forward with this
concept and sketch out a testing procedure and scenario document.
 

>

> Matt: just some comments on your comments:

>

> >I think a "Conformance service" is probably a better
way to request

> that

> >the MSH create messages and that sort of thing to
be tested.  It seems

> >impractical to me to have custom code for each vendor
in the

> conformance

> >test suite.

>

> If vendors accept to implement this conformance service,

> that will indeed be the shortest path to conformance
testing

> (and also to interoperability test later). So it seems
like something

> to explore further. (but we need to publish this design
ASAP in Impl.

> Guidelines, if we push for it, so that implementors
are aware of it.)

> I assume then that vendors who do not want/plan to
implement this

> "conformance service" will still have the option to
"drive" their MSH

> through

> its API, for test scenarios - their choice?

>

[Matt]Ok, I will give this some thought and send a note
to the chap doing the

impl guidelines.

[Mike]  It seems worth exploring, at least to get the ball
rolling toward establishing either a testing api, or ( since they're building
it ) a standardized API for MSH.  At this stage in the game,
I think that starting a conformance service based on Jacques "2-way
pattern", utilizing a service makes the most sense.  If an API evolves,
then "sending" and "receiving" scenarios could be added to the test
framework.
 

[Jacques] probably we should call it more genrally a "test
service", as it would also

be very helpful for interoperability tests?

(but again, as we must accommodate anyone who implements
the standard and only the standard,

we should also make sure that they'll have a plan B to
support conformance test procedures

via a driver and MSH API (even if they have to write
code for an API adapter).

That could also be described in the Impl Guidelines.)

> >I'm not suggesting we require vendors to adopt a certain
log format,

> >what I am referring to here is more of a tracing format. 
In the

> absence

> >of a reference implementation, we need to do this,
or employ network

> >sniffing.  Maybe I have my head in the mud, please
set me straight if I

> >am missing an easier way to capture a complete exchange/transaction.

>

> What I had in mind is: the candidate MSH could send
its messages to an

> URL that

> would just happen to be our testing end-point listening
to this HTTP

> port

> (not even an MSH - just a servlet able to dig in the
MIME envelope and

> analyze themessage parts).

> Again that may not be sufficient - we need to look
at all your test

> cases.

> So we may all have heads in the mud until that's done
:)

[matt] I am going to deflect this to Michael, as he probably
has a better idea

of how the tests will be implemented.
    [Mike] Out network person suggested the same approach
as suggested by

Jacques.. setting up a "service" that serves as an endpoint for messaging
and

parses the header trace... seems reasonable to me

 

 
 

>

> >Either way, it doesn't really matter.  Extracting
from an actual CPA is

> >a breeze.

>

> I would favor that, as again there might not be agreement
on a common

> config file format

> across implementations (some MSH could actually directly
use/fetch CPA

> docs, if I recall).

> (also, CPA data will actually be more dynamic than
config files: new

> CPAs may

> show up any time - and expire - during the liefetime
of an MSH

> instance, based on new

> collaborations)

[matt] Maybe something simple that uses XPath would be
in order.  That way,

changes to CPP/A are easy to handle.

 
[I heavily support XPath... even in the testing requirements document if
it can

be done in a reasonable way]

 
 

[jacques] sounds good, as we only need a small subset
of CPA. (that way, we

could also use just a subset of a full CPA when configuring
a candidate MSH for testing.)

 I suspect XPath could also be useful to access
and test elements of a captured message,

when checking conformance rules one after the other.

[matt] Regards, and Happy Easter.

--

Matthew MacKenzie

XML Global R&D

PGP Key available upon request.