RE: [ebxml-iic-interop] interop update

Title: interop update

Eric:

 

My 
comments - (Steve and Hatem may add theirs especially if they 
disagree!)

  
-----Original Message-----
From: Eric VanLydegraf 
  [mailto:]
Sent: Thursday, September 26, 2002 2:45 
  PM
To: 'Jacques Durand'
Cc: 
  ''
Subject: RE: 
  [ebxml-iic-interop] interop update

  
Well 
  thanks jacques,

  
    in that case here are my comments on the 0.40 
  document

  
 

  
Introduction 1 states conformance testing (should be interoperability 
  correct ?)

  
 

  
<Jacques> 
  right

  
 

  
 

  
It 
  seems the document needs to have an interoperability definition and motivation 
  statement.

  
The 
  term is used everywhere but only defined in Appendix A (which BTW follows 
  Appendix B ???) and properly differentiated from 
  conformance.

  
 

  

  
<Jacques> correct. I ahve not seen a 
  "formal-enough" definition so far, so I tried to craft one some time 
  ago.

  
Here it is. Let me know if you know of any 
  other. 
  
we can use it if you guys 
  agree.
I think the notion of "contract" is essential 
  - allows you to test things

  
that are out of scope of the spec, like 
  interop with "large payloads", or interop with options from spec that 
  are

  
narrowed down (according to the 
  "contract"):

  
Interoperability 
  Testing: 
  Process of verifying that two implementations of the same specification, 
  or

  
 that 
  an implementation and its operational environment, can interoperate according 
  to the requirements 

  
of 
  an assumed agreement or contract. This contract does not belong necessarily to 
  the specification, 

  
but 
  its terms and elements should be defined in it with enough detail, so that 
  such a contract, combined 

  
with 
  the specification, will be sufficient to determine precisely the expected 
  behavior of an implementation, 

  
and 
  to test it.

  
We may need to talk a bit about the expected 
  methodology (matrix testing, vs. hub-and-spoke testting 
  against

  
some "reference" 
  implementation)

  
 

  
 

  
 

  
I would specifically add a Pre Requesite heading section stipulating 
  the conformance to have passed prior to running 
  interop.

  
<Jacques> absolutely.

  
 

  
I 
  was also thinking over the Test Service & Test Driver framework and 
  wondering about

  
having the Test Service be initiated with ebXML messages itself and 
  having notifications and events signalled via messages.

  
This 
  has the advantage of keeping the Test Driver and Test Service separate and 
  using ebXML messagging as the logging and tracing.

  
It 
  also means we can configured the Test Service entirely through a message - 
  with the test cases as payload. The conversation_ID with the Test 
  Driver

  
determines the Test Case being referenced. With timeouts the Test 
  Driver can determine test failure. So the Test Driver is both initiatior and 
  logger of the test run, but the actuall testing occurs between candidate 
  MSH's.

  
 
<Jacques> tricky issue. My main concern 
  about initiating test cases via messages sent by the Test Driver 
  (like

  
we do in conformance testing - but for 
  conformance that is not a problem), is that 
  this introduces a third end-point 
  

  
on the communication. In a deployed 
  environment, where network access may be subject to all kinds of 
  restrictions,

  
this may be a problem (and we need be 
  able to test interop at any time, on a periodical 
  basis.)

  
 You may not be able to send easily 
  messages to your own MSH. In addition, it may 
  craete

  
"noise" in the log, send back messages 
  or errors, you need to create an additional conversation, there may be 
  security problems, 

  
you don't want your MSH to interact with 
  your driver using same CPA as the ones (several different) used for interop 
  test cases, 

  
so you'd need to switch 
  to 
  a special CPA to be installed, 
  etc. 

  
All that 
  may 
  get messy and restrictive, even if the architecture looks attractive because 
  just a building block game.

  
I think in the f-2-f we came-up with the 
  idea that :

  
we would have an integration  Test 
  Driver / Test Service, i.e. a bundled version of the test driver would be able 
  to directly

  
invoke the Initiator service action 
  (from inside same process), without going through the wire/MSH to do 
  so.

  
That seems much less problem-prone, and 
  leaves the communication and MSH layer untouched - no side effects, 
  no

  
intrusion, no reconfiguration, etc. 
  The 
  test driver will have a mode where instead of crafting messages to 
  send

  
a payload to the local Test Service 
  (Initiator action), it will just directly invoke the Initiator action 
  

  
(via the same internal interface that is 
  used by the MSH) and directly pass the payload to 
  it.

  
For the notification, same thing: 
  actions will notify message material (header data + payload) to test driver 
  directly through 

  
its 
  interface.

  
 

  
 

  
it 
  seems we need a template, as the tests will be very similar and only certain 
  tests will be excluded rather than different.

  
It 
  seems more appropriate to list all tests and have a PreCondition table stating 
  for what protocols it applies to.

  
That 
  way all testers step through each Test Case and test if needed otherwise skip 
  to next one.

  
Or 
  alternatively write up the Test Cases in one section and make a table 
  with checkmarks on which protocols apply to the Test Case.

  
 

  
e.g.

  
Test 
  Case 1.1 No payload basic exchange 

  
Protocols [HTTP,SMTP]

  
...

  
Test 
  Case 1.12 Synchronous Unsigned Acknowledgement  
  exchange

  

  
Protocols [HTTP]

  
 

  
<Jacques> I guess it is mostly a matter 
  of specification style - I 'd still like to clearly and 
  unambiguously

  
identify each Interop Profile (i.e. their 
  corresponding test suite) as "optionless".

  
I.e. each test suite is totally deterministic 
  (interop "contract" is fully defined within, e.g. with the precise protocol 
  you

  
will use, e.g. HTTP), and ther eis only one 
  way to execute it.

  
So this still allows for using a parameterized 
  "profile template" if you will - but we would clearly distinguish two 
  instances of

  
it (e.g.in our case, one for HTTP, one for 
  SMTP), and consider them as two different interop profiles 
  .

  
Would that be consistent with what you are 
  proposing?

  
e.g.. :

  
TemplateTestSuiteXYZ ( parameters: 
  protocolname,  protocoltype )

  
{

  

  
Test Case 1.1 No payload basic exchange 
precondition(nil); 
 ... 

  
Test Case 1.12 Synchronous Unsigned 
  Acknowledgement  
  exchange 

  
precondition(protocoltype="synchronous");

  
}

  
 

  
Interop Profile A:

  
TemplateTestSuiteXYZ ( HTTP, 
  synchronous)

  
 

  

  
Interop Profile B:

  
TemplateTestSuiteXYZ ( SMTP, 
  asynchronous)

  
 

  
 

  
Cheers,

  
 

  
jacques

  
 

  
I 
  think with some more discussion and thought about the test framework - namely 
  the test driver/test service we could more thoroughly

  
go 
  through the test steps.

  
    
-----Original Message-----
From: Jacques Durand 
    [mailto:]
Sent: Tuesday, September 24, 2002 
    11:14 AM
To: 'Eric VanLydegraf'
Cc: 
    ''
Subject: RE: 
    [ebxml-iic-interop] interop update

    
Yes, latest draft is 0.4 you got by mail. Not posted yet on our 
    site.

    
We 
    need to review in particular Section 3:

    
- 
    the detail of the test steps for each test case.

    
- 
    the notion of having two "basic interop profiles", concretized by a test 
    suite for HTTP, 

    
and another test suite for SMTP. (no "option" inside a 
    profile.)

    
- 
    the content of these basic interop profiles (as drafted, they roughly match 
    what remains

    
from the DGI tests, after removing (a) conformance tests, (b) 
    optional tests, (c) specific tests like large payloads.)

    
 

    
Cheers,

    
 

    
Jacques

    
      
-----Original Message-----
From: Eric VanLydegraf 
      [mailto:]
Sent: Monday, September 23, 2002 6:06 
      PM
To: 'Jacques Durand'; 
      ''
Subject: RE: 
      [ebxml-iic-interop] interop update

      
Just to confirm 0.40 is the latest interop document 
      ?

      
        
-----Original Message-----
From: Jacques Durand 
        [mailto:]
Sent: Monday, September 09, 
        2002 5:45 PM
To: 
        ''
Subject: 
        [ebxml-iic-interop] interop update

        
Steve, Hatem, Eric: 

        
Here is an incremental update of the Interop 
        suite document, for your review 
based 
        on improvement we discussed in past conf calls as well as F-2-F. 
        

        
- Section 1.2 (concept of operations) is 
        expanded. 

        
- References to third-party tests (here, DGI) 
        has been removed, as this is a general, abstract test suite, 
        
that will be used as a reference by third 
        parties. Comparisons with UCC/DGI test suite will be done 
        separately, 
as we expect for other 
        existing test suites (ECOM). (We'll need to do such "comparative 
        evaluations" 
if only as a basis for 
        discussion when negotiating with these entities, so that they understand 
        how "compliant" 
they are with our 
        tests, and also so that they can give feedback on what may be missing in 
        ours. 

        
- list of Actions of  Test Service 
        (Section 2.2) is slightly updated (configurator, also the message 
        sending behavior 
of all these actions 
        is the same whether the Test Service is associated or not with a Test 
        Driver.) 

        
- still need to refine the way a Test Driver 
        will drive the test cases by directly invoking the Test Service 
        "Initiator" action.

        
- only one interop profile test suite 
        described here for HTTP (Section 3), based on Steve's list of test 
        cases 
 as we went through in 
        F-2-F. 
It is roughly equivalent to: { 
        DGI, minus conformance tests, minus optional tests, minus HTTP/S 
        } 
Detail of test cases (test steps, 
        CPA...) still needs to borrow from Mike's material. 
Still to discuss: idea that there is an interop 
        profile for HTTP, another vfor SMTP. (see Section 3.3) 

        
Regards, 

        
Jacques 

        
<<ebxml_ms_interoperability_testsuite_0.4.doc>>