RE: [ebxml-iic-interop] interop update

Title: interop update

 

  
-----Original Message-----
From: Jacques Durand 
  [mailto:]
Sent: Thursday, September 26, 2002 
  4:37 PM
To: 'Eric VanLydegraf'
Cc: 
  ''
Subject: RE: 
  [ebxml-iic-interop] interop update

  
Eric:

  
 

  
My 
  comments - (Steve and Hatem may add theirs especially if they 
  disagree!)

  
    
-----Original Message-----
From: Eric VanLydegraf 
    [mailto:]
Sent: Thursday, September 26, 2002 2:45 
    PM
To: 'Jacques Durand'
Cc: 
    ''
Subject: RE: 
    [ebxml-iic-interop] interop update

    
Well thanks jacques,

    
    in that case here are my comments on the 0.40 
    document

    
 

    
Introduction 1 states conformance testing (should be interoperability 
    correct ?)

    
 

    
<Jacques> 
    right

    
 

    
 

    
It 
    seems the document needs to have an interoperability definition and 
    motivation statement.

    
The term is used everywhere but only defined in Appendix A (which BTW 
    follows Appendix B ???) and properly differentiated from 
    conformance.

    
 

    

    
<Jacques> correct. I ahve not seen a 
    "formal-enough" definition so far, so I tried to craft one some time 
    ago.

    
Here it is. Let me know if you know of any 
    other. 
    
we can use it if you guys 
    agree.
I think the notion of "contract" is 
    essential - allows you to test things

    
that are out of scope of the spec, like 
    interop with "large payloads", or interop with options from spec that 
    are

    
narrowed down (according to the 
    "contract"):

    
Interoperability 
    Testing: 
    Process of verifying that two [Eric VanLydegraf] (or 
    more) implementations of the same specification, 
    or

    
 that 
    an implementation and its operational environment, can interoperate 
    according to the requirements 

    
of 
    an assumed agreement or contract. This contract does not belong necessarily 
    to the specification, 

    
but 
    its terms and elements should be defined in it with enough detail, so that 
    such a contract, combined 

    
with 
    the specification, will be sufficient to determine precisely the expected 
    behavior of an implementation, 

    
and 
    to test it.

    
We may need to talk a bit about the expected 
    methodology (matrix testing, vs. hub-and-spoke testting 
    against

    
some "reference" 
    implementation)

    
 

    
 

    
 

    
I would specifically add a Pre Requesite heading 
    section stipulating the conformance to have passed prior to running 
    interop.

    
<Jacques> absolutely.

    
 

    
I 
    was also thinking over the Test Service & Test Driver framework and 
    wondering about

    
having the Test Service be initiated with ebXML messages itself and 
    having notifications and events signalled via messages.

    
This has the advantage of keeping the Test Driver and Test Service 
    separate and using ebXML messagging as the logging and 
    tracing.

    
It 
    also means we can configured the Test Service entirely through a message - 
    with the test cases as payload. The conversation_ID with the Test 
    Driver

    
determines the Test Case being referenced. With 
    timeouts the Test Driver can determine test failure. So the Test Driver is 
    both initiatior and logger of the test run, but the actuall testing occurs 
    between candidate MSH's.

    

<Jacques> tricky issue. My main concern 
    about initiating test cases via messages sent by the Test Driver 
    (like

    
we do in 
    conformance testing - but for conformance that is not a problem), is that 
    this introduces a third end-point 
    

    
on the 
    communication. In a deployed environment, where network access may be 
    subject to all kinds of 
    restrictions,

    
this may be a 
    problem (and we need be able to test interop at any time, on a periodical 
    basis.)

    
 You may not 
    be able to send easily messages to your own MSH. In addition, it may 
    craete

    
"noise" in the 
    log, send back messages or errors, you need to create an additional 
    conversation, there may be security problems, 
    

    
you don't want 
    your MSH to interact with your driver using same CPA as the ones (several 
    different) used for interop test cases, 
    

    
so you'd need to 
    switch to a special CPA to be installed, 
    etc. 

    
All that 
    may get messy and restrictive, even if the 
    architecture looks attractive because just a building block 
    game.

    
I think in the 
    f-2-f we came-up with the idea that 
    :

    
we would have an 
    integration  Test Driver / Test Service, i.e. a bundled version of the 
    test driver would be able to 
    directly

    
invoke the 
    Initiator service action (from inside same process), without going through 
    the wire/MSH to do so.

    
That seems much 
    less problem-prone, and leaves the communication and MSH layer untouched - 
    no side effects, no

    
intrusion, no 
    reconfiguration, etc. The test driver will have a mode where 
    instead of crafting messages to 
send

    
a payload to the 
    local Test Service (Initiator action), it will just directly invoke the 
    Initiator action 

    
(via the same 
    internal interface that is used by the MSH) 
    and directly pass the payload to 
    it.

    
For the 
    notification, same thing: actions will notify message material (header data 
    + payload) to test driver directly through 
    

    
its interface.
[Eric VanLydegraf] See Monica's 
    email about describing both but from the concept level we only need to 
    specify the role/requirement of the trigger function of the Test 
    Driver

    
 

    
 

    
it 
    seems we need a template, as the tests will be very similar and only certain 
    tests will be excluded rather than different.

    
It 
    seems more appropriate to list all tests and have a PreCondition table 
    stating for what protocols it applies to.

    
That way all testers step through each Test Case and test if needed 
    otherwise skip to next one.

    
Or 
    alternatively write up the Test Cases in one section and make a table 
    with checkmarks on which protocols apply to the Test 
    Case.

    
 

    
e.g.

    
Test Case 1.1 No payload basic exchange 

    
Protocols [HTTP,SMTP]

    
...

    
Test Case 1.12 Synchronous Unsigned Acknowledgement  
    exchange

    

    
Protocols [HTTP]

    
 

    
<Jacques> I guess it is mostly a 
    matter of specification style - I 'd still like to clearly and 
    unambiguously

    
identify each Interop Profile (i.e. their 
    corresponding test suite) as "optionless".

    
I.e. each test suite is totally 
    deterministic (interop "contract" is fully defined within, e.g. with the 
    precise protocol you

    
will use, e.g. HTTP), and ther eis only one 
    way to execute it.

    
So this still allows for using a 
    parameterized "profile template" if you will - but we would clearly 
    distinguish two instances of

    
it (e.g.in our case, one for HTTP, one for 
    SMTP), and consider them as two different interop profiles 
    .

    
Would that be consistent with what you are 
    proposing?

    
e.g.. :

    
TemplateTestSuiteXYZ ( parameters: 
    protocolname,  protocoltype )

    
{

    

    
Test Case 1.1 No payload basic 
    exchange 
precondition(nil); 
 ... 

    
Test Case 
    1.12 Synchronous Unsigned Acknowledgement  
    exchange 

    
precondition(protocoltype="synchronous");

    
}

    
 

    
Interop Profile 
A:

    
TemplateTestSuiteXYZ ( HTTP, 
    synchronous)

    
 

    

    
Interop Profile 
B:

    
TemplateTestSuiteXYZ ( SMTP, 
    asynchronous)

    
 

    
 

    
Cheers,

    
 

    
jacques

    
 

    
I 
    think with some more discussion and thought about the test framework - 
    namely the test driver/test service we could more 
    thoroughly

    
go through the test steps.
[Eric VanLydegraf] Yes your example conveys 
    the idea, we can bang up on it in the document to find the best 
    format.

    
      
-----Original Message-----
From: Jacques Durand 
      [mailto:]
Sent: Tuesday, September 24, 
      2002 11:14 AM
To: 'Eric VanLydegraf'
Cc: 
      ''
Subject: RE: 
      [ebxml-iic-interop] interop update

      
Yes, latest draft is 0.4 you got by mail. Not posted yet on our 
      site.

      
We need to review in particular Section 3:

      
- the detail of the test steps for each test 
      case.

      
- the notion of having two "basic interop profiles", concretized by 
      a test suite for HTTP, 

      
and another test suite for SMTP. (no "option" inside a 
      profile.)

      
- the content of these basic interop profiles (as drafted, they 
      roughly match what remains

      
from the DGI tests, after removing (a) conformance tests, (b) 
      optional tests, (c) specific tests like large 
      payloads.)

      
 

      
Cheers,

      
 

      
Jacques

      
        
-----Original Message-----
From: Eric VanLydegraf 
        [mailto:]
Sent: Monday, September 23, 2002 
        6:06 PM
To: 'Jacques Durand'; 
        ''
Subject: RE: 
        [ebxml-iic-interop] interop update

        
Just to confirm 0.40 is the latest interop document 
        ?

        
          
-----Original Message-----
From: Jacques Durand 
          [mailto:]
Sent: Monday, September 09, 
          2002 5:45 PM
To: 
          ''
Subject: 
          [ebxml-iic-interop] interop update

          
Steve, Hatem, Eric: 

          
Here is an incremental update of the 
          Interop suite document, for your review 
based on improvement we discussed in past conf calls as well as 
          F-2-F. 

          
- Section 1.2 (concept of operations) is 
          expanded. 

          
- References to third-party tests (here, 
          DGI) has been removed, as this is a general, abstract test 
          suite, 
that will be used as a 
          reference by third parties. Comparisons with UCC/DGI test suite will 
          be done separately, 
as we expect 
          for other existing test suites (ECOM). (We'll need to do such 
          "comparative evaluations" 
if only 
          as a basis for discussion when negotiating with these entities, so 
          that they understand how "compliant" 
they are with our tests, and also so that they can give 
          feedback on what may be missing in ours. 

          
- list of Actions of  Test Service 
          (Section 2.2) is slightly updated (configurator, also the message 
          sending behavior 
of all these 
          actions is the same whether the Test Service is associated or not with 
          a Test Driver.) 

          
- still need to refine the way a Test 
          Driver will drive the test cases by directly invoking the Test Service 
          "Initiator" action.

          
- only one interop profile test suite 
          described here for HTTP (Section 3), based on Steve's list of test 
          cases 
 as we went through in 
          F-2-F. 
It is roughly equivalent to: 
          { DGI, minus conformance tests, minus optional tests, minus HTTP/S 
          } 
Detail of test cases (test steps, 
          CPA...) still needs to borrow from Mike's material. 
Still to discuss: idea that there is an interop 
          profile for HTTP, another vfor SMTP. (see Section 3.3) 

          
Regards, 

          
Jacques 

          
<<ebxml_ms_interoperability_testsuite_0.4.doc>>