FW: WS-I: Updated Basic Profile Test Report Summary September 15, 2010

See below a past test report on BP1.2 and BP2.0 interop testing, showing that IBM, Oracle and Microsoft have implemented these profiles.

(Please do not forward that email outside this list, except internally to your own company if you are among the above testers.)

 

This could be used as a basis for statements of use from these member companies.

 

Thanks,

-jacques

 

From: Jacques Durand 
Sent: Monday, October 27, 2014 6:24 PM
To: Jacques Durand
Subject: WS-I: Updated Basic Profile Test Report Summary September 15, 2010

 

 

 

From:  [mailto:] On Behalf Of Monica Martin
Sent: Thursday, September 16, 2010 8:16 AM
To: Ram Jeyaraman; 
Subject: [wsi_wsbasic] Updated Basic Profile Test Report Summary September 15, 2010

 

This report includes the interoperability update from IBM and reflects recent TA changes to validation report.

 

15 September 2010

Test report summary

 

IBM, Oracle and Microsoft successfully achieved interoperability for Basic Profile 1.2 [WGAD link] and 2.0 [WGAD link].  Web service instances and clients of IBM Websphere, Oracle WebLogic, and Microsoft WCF and Windows Web Services API (WWSAPI) participated during the 2-year test period.

 

Using the Interoperability Test Suite, IBM, MS WCF and WWSAPI, and Oracle achieved successful interoperability results across 184 scenarios, [see tables at end of this email] With the new test methodology (herein called the Test Tools Process) and tools, we also verified conformance of the enabled test assertions for the Basic Profile requirements. 

 

Cumulative Interoperability 

Results

BP 1.2

BP 2.0

95%

93.7%

Existing: 96

New: 82.9

Existing: 94.6

New: 82.9

BOM: 91.7

BOM: 91.7

 

In some instances, a client and instance were unable to support or enable specific scenarios (i.e. some instances don’t support non-anonymous responses). The interoperability suite also includes negative scenarios difficult to create (require emitting invalid WSDL or using a special client).  These factors and a small number of documented issues remain.[1]   Overall, interoperability has significantly improved and the robustness of the test materials increased exponentially (for example, more than 170 interoperability issues have been processed).

                                                                                                                 

To meet exit criteria outlined in the Basic Profile WG charter, testing was conducted in 2 phases: interoperability testing, and monitoring and analysis. The Interoperability Test Suite (interoperability message exchange using defined scenarios) [link] and the Testing Tools Package with updates identified here (monitoring, log capture and test analysis, and reporting) [link] were used with new updates to provide:

1.      Evidence of interoperability between web services instances and artifacts.

2.      An indication of conformance to the Profile of web service instances and artifacts analyzed.

 

Interoperability summary results for BP 1.2 and 2.0 are also available [link]. Detailed test reports are available on the BP WG site.

BP 1.2: [folder] See: BP12-TestReports-20100915

BP 2.0: [folder] See: BP20-TestReports-20100915

               Wire-validation summary: [folder] See: BP12-20-WireValidation-Results-20100915

 

All issues in test reporting – test assertions, test participant interoperability and test report bugs or scenario updates - have been processed and re-verified, and/or documented where appropriate. [link] 

 

Enhancements to the BdAD test assertions and supporting test materials (scenarios) were used to improve the fidelity of and the assurance for the test results provided here. The most recent testing uncovered gaps in the test assertions that resulted, for example, in false negatives or left test code unexercised in specific situations.  In addition, several negative scenarios In Basic Profile trigger test assertion failures, and in specific cases, actually provided evidence the profile requirements were exercised.   

 

The overall test results are representative of the actual status of the web services instances under test. Overall, about 30% of the enabled test assertions were improved.[2]

 

This effort was only possible because of the commitment of all Basic Profile WG members including current members – Asynchrony, Fujitsu, IBM, Oracle and Microsoft. 

 

Interoperability Test Results

 

Basic Profile 1.2

Oracle->MS WCF

MS WCF->Oracle

Oracle->IBM

IBM->Oracle

MS WCF->IBM

IBM->MS WCF

MS WCF->MS WWSAPI

MS WWSAPI->MS WCF

IBM->MS WWSAPI

MS WWSAPI->IBM

Percentage Pass

98.9%

100.0%

98.9%

93.3%

93.3%

100.0%

95.6%

92.2%

85.6%

92.2%

Percentage Fail

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

Percentage Not Enabled

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

Not supported or tested 

1.1%

0.0%

1.1%

6.7%

6.7%

0.0%

4.4%

7.8%

14.4%

7.8%

 

Basic Profile 2.0

Oracle->MS WCF

MS WCF->Oracle

Oracle->IBM

IBM->Oracle

MS WCF->IBM

IBM->MS WCF

MS WCF->MS WWSAPI

MS WWSAPI->MS WCF

IBM->MS WWSAPI

MS WWSAPI->IBM

Percentage Pass

98.9%

100.0%

98.9%

91.5%

92.6%

100.0%

90.4%

91.5%

81.9%

91.5%

Percentage Fail

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

0.0%

Percentage Not Enabled

0.0%

0.0%

0.0%

1.1%

0.0%

0.0%

1.1%

0.0%

2.1%

0.0%

Not supported or tested 

1.1%

0.0%

1.1%

7.4%

7.4%

0.0%

8.5%

8.5%

16.0%

8.5%

 

 

PostSOAP results 1.2

(Special client scenario)

To Oracle

To IBM

To MS WCF

To MS WWSAPI

Pass

100.0%

100.0%

100.0%

66.7%

Fail

0.0%

0.0%

0.0%

33.3% [3]

Not enabled

0.0%

0.0%

0.0%

0.0%

Other: Not supported or not tested

0.0%

0.0%

0.0%

0.0%

 

PostSOAP results 2.0

(Special client scenario)

To Oracle

To IBM

To MS WCF

To MS WWSAPI

Pass

100.0%

100.0%

100.0%

66.7%

Fail

0.0%

0.0%

0.0%

33.3% [3]

Not enabled

0.0%

0.0%

0.0%

0.0%

Other: Not supported or not tested

0.0%

0.0%

0.0%

0.0%

[1] Results are based on partners that supported and have executed the tests. In BP testing, core features are supported and interoperable between participating partners.  Basic Profile allows rather than dictates what web services instances could support (anonymous responses only or wsam:Addressing, for example). 

[2] BP 1.2: 35 of 108 (32.4%); BP 2.0: 34 of 104 (32.7%)

[3] Potential tool issue – See Known Issues.