RE: More on message bundling

Comments inline:

-jacques

From: Pim van der Eijk 
[mailto:] 
Sent: Wednesday, July 01, 2009 10:58 
AM
To: Jacques R. Durand; 

Subject: More on message 
bundling

The latest Part 2 draft already has a good start on 
bundling.

Here are some more 
ideas, building on that.

 

A relatively 
simple way of controlling bundling is to view bundles as combinations of a 
single "primary" user message (the driver) and other "secondary" user messages 
that piggy back onto that message. The "primary" 
user message determines the P-mode settings applied by the MSH to the entire bundle.   

 

<JD> So I guess some implicit 
compatibility rules are needed, when bundling from different PModes, 
e.g:

- it is OK to put a message from a PMode 
without RM, into a bundle with primary u.message under 
RM.

- it is OK to sign the entire SOAP header 
according to the PMode of a primary u.message, even if secondary u.messages did 
not require security.

- however, Receipts are only on a 
per-u.message basis and should remain so. So if 
NRR is required for u.m. 1,2,3 but not for u.m. 4,5,6 in the same bundle, then a 
single signature may cover it all (e.g. all payloads are is the SOAP body), but 
there should be only 3 Receipts in response (for 1,2,3) each one of them 
will use however the same digest(s) in its nonRepudiationInformation element 
(digest may cover all u.m. 1,2,3,4,5,6).

 

 

 Assumption: 
in a bundle, the primary user message is the first message unit and the first 
message unit can be assumed to be the primary user 
message.

 

Since user messages are 
submitted separately and usually not at the exact same time, the MSH should 
temporarily store submitted user messages to allow combining them later into 
bundles.  The MSH may already compute the eb3:UserMessage XML header 
including the eb3:PayloadInfo references to message parts. It may also already 
perform some CPU intensive ebMS3/AS4 functions such as compressing payloads and 
perhaps already compute secure hashes, as an optimization to speed up security 
operations in the security module. The MSH cannot yet assemble the ebMS3 SOAP 
header and does not perform any security operations such as encryption, since 
that depends on the Pmode of the primary message that a user message is going to 
be bundled with, nor does it apply any 
reliability headers for the same reason. 

 

<JD> By definition, the message (the bundle) must be complete on 
the sender side before applying any RM and Security. Even if the primary u.m. is 
known from the start, I would assume that so signing is done until the message 
is complete. A signature is likely to cover all the SOAP Body (and contained 
payloads), and several SOAP headers. All u.m. headers will be signed together, 
very likely.

 

Bundling therefore would be controlled controlled by Pmodes.  
Suggested new Pmode parameters:

 

Pmode[].bundling.standalone: values "always", "optional", 
"never"

 

Indicates whether 
the user message is to be sent as a standalone message without piggy-backed user 
messages
 ("always"),  

 

<JD> the name of this parameter is 
confusing (standalone is conflicting with "bundling"). We could 
have:

PMode[].bundling.policy = {always. 
optional, never}

with opposite meanings as 
yours.

 

 

 may be sent separately but may also be 
bundled with other user messages ("optional") or is never sent as a 
standalone message ("never"). An ebMS 3 implementation that strictly conforms to part 1 of the ebMS 3.0 Core 
Specification and does not support any form of bundling will only support the 
option "always" for this parameter. A parameter value "never" means that the 
message needs to be bundled with some other message to be sent at all. It cannot act as a primary user message. 

 

Pmode[].bundling.bundleswith 

 

<JD> we could have two ways so specify bundling 
compatibility:

(1) by 
explicit listing (sane as your "bundlewith" 
above):

PMode[].bundling.compatibility.pmodelist 
(e.g. = "PMode['abc'][1],PMode['def'][2],PMode['ghi'][1]" where  'abc', 
'def', 'ghi' are IDs of PModes)

(2) by correlation rule (same technique used as for 
PMode[1].Reliability.Correlation 
):

PMode[].bundling.compatibility.class 
(e.g. = "eb:UserMessage/eb:CollaborationInfo/eb:Service, 
eb:UserMessage/eb:CollaborationInfo/eb:Action")

would mean that only 
messages units with same Service/Action as the primary unit, can be 
bundled.

NOTE: A more complete 
expression woudl make explicit reference to the "primary" unit using the 
reserved variable 
$primary:

PMode[].bundling.compatibility.class = 
"eb:UserMessage/eb:CollaborationInfo/eb:Service = 
$primary/eb:CollaborationInfo/eb:Service and  eb:UserMessage/eb:CollaborationInfo/eb:Action = 
$primary/eb:CollaborationInfo/eb:Action", 

meaning every unit for which this condition is true, belongs to the same 
bundling compatibility class as the 
primary.)

 

If both (1) and (2) are 
present, the bundling must satisfy (1) AND 
(2).

 

This parameter is to be 
specified in case the "standalone" parameter has a value other than "never". Its 
value is a list of Pmodes.  The semantics of the parameter is that a user 
message may (if standalone is "optional") or must (if standalone is "never") be 
bundled with other user messages associated with the Pmodes identified in the 
list.  The parameter may list other types of Pmodes that describe messages 
that the user message can be bundled with.  In general these Pmodes SHOULD 
be destined to the same target MSH (even in a multihop network) and SHOULD 
satisfy the same or stricter QoS 
requirements, but the responsibility for making sure this is true is with the 
Pmode configuration administrator.  

 

<JD> Not sure there is any 
value in "SHOULD be destined to the same target MSH 
". In a multihop context these PModes may not know the actual destination 
endpoint address, and all we can ask for, is for PMode designers to know 
whether the logical destination of 2 PModes is the same or not. In a borderline 
case, a PMode could be designed leaving blank the To/PartyId, e.g. a PMode 
governing messages sent for a particular Service/Action regardless of the actual 
destination party, which could vary and be served by a different MSH from 
another. The sender would pass the To/PartyId when submitting the message, which 
would determine the routing. So in that case, we can't even bundle any message 
from the same PMode: we need in addition a compatibility rule (added as bundling 
parameter in the above generic Service/Action 
PMode):

PMode[].bundling.compatibility.class = 
"fn:contains('party100;party102;party104;party105;party106', xsd:string( eb:UserMessage/eb:PartyInfo/eb:To/eb:PartyId) and 
fn:contains('party100;party102;party104;party105;party106', xsd:string( $primary/eb:PartyInfo/eb:To/eb:PartyId) 
)

Where it is known that 
all the above party IDs are served by the same 
MSH.

In case there is another 
set of parties that can be bundled together (for a different MSH endpoint), we 
would have an additional instance of PMode[].bundling.compatibility.class 
parameter specified in this 
PMode.

 

 

 Typically, the list value of the 
"bundleswith" parameter for Pmode P1 will also include that same Pmode P1, 
indicating that more than one user message of this type may be sent.  But 
if a particular user message is known to be very large or to be sent rarely, the 
parameter could only list some Pmodes of messages that are known to be small and 
sent frequently. 

 

The MSH should support 
configuration of the maximum duration of the time it waits for more user 
messages to be submitted to be added to bundles, the maximum number of user 
messages it bundles and/or the maximum size of the user message content 
(including business documents and attachments, after AS4 compression) of the 
bundle.

 

Pmode[].bundling.maxdelay
Pmode[].bundling.maxmessages
Pmode[].bundling.maxsize

 

<JD> instead of restricting this to the bundling case: PMode.Message.maxSize: overall, maximum size of any 
message in this PMode (bundled or 
not).

 

 

Note that the MSH 
is not required to wait before the maxdelay interval has expired before sending 
the message. It may send the message before the interval has finished. The maxdelay is a contract with the submitting 
business application which may have some time-to-perform to be respected. 

 

<JD> wondering if we should have the same approach for the 
timing as suggested for the size: independently from bundling, this could be 
seen as a general guideline for any message of this PMode (a QoS contract): what 
is the maximum delay that an MSH should not exceed, from submission time to 
sending time: PMode.Message.maxDelay. 
(not sure Ian will like it better, 
though...)

 

 

If the "bundleswith" 
list includes multiple Pmodes, and more than one candidate waiting "primary" 
user message is available that a user message can be bundled with, the 
allocation is implementation-dependent.

 

If the value of 
the "standalone" parameter is "optional", the decision by the MSH to send the 
message as a standalone message or to bundle it with another user message is 
implementation-dependent. 

 

If the value 
of the "standalone" is "never", and no primary message is available to bundle 
the message with within a "maxdelay" duration, the MSH MUST generate an 
ebms:xxxx BundlingError sender error.

 

<JD> So, if we have: 
PMode[].bundling.policy = {always, optional, never}, then no error should be 
sent back: the "always" (your "never") only means that the MSH must do its best 
effort to bundle. If it cannot bundle, then it just send the message 
alone.

 

Bundling and 
Pull

 

The description 
above assumes that submitted user messages are queued temporarily between ebMS 
processing and between further security and reliability processing.  This 
means that a submitted user message may 
not yet be available for 
pulling, leading to unintuitive 
results. This could be optimized in 
advanced implementations as some kind of on-demand bundling, or 
on-demand MSH completion, triggered by an incoming Pull Request.  
Left to implementations.

 

<JD> Intuitively, as soon as you send a PullRequest, you want to 
get an immediate response, bundling or not... So waiting is not an option for 
sure. The PullRequest could take the initiative of requesting the bundling. I 
see three ways to do this: 

(a) on-the-wire-indication: several PullRequests units could be bundled 
together: as many as the maximum u.m. you may expect bundled in response. 
Thats easy but not very elegant. 

(b) some additional attribute in PullRequest indicate the max bundling 
you can handle. Problem is , schema needs be extended for 
this. 

(c) PMode configured: the PMode says so. The receiver needs to know this. 
Less flexible too.

 

Bundling and 
Sync

 

The above is really about requests and responses for 
both Push and Pull binding, but not for response user messages to Two Way / Sync 
MEP bindings, which must be transmitted on the HTTP back channel without 
delay.  The synchronous responses to 
each of the user messages in the bundle should be sent without waiting 
"maxdelay" for more messages to be bundled with them, to prevent HTTP transport 
timeouts. 

 

<JD> But a Two-way / Sync still depends on the Responding 
application for not causing HTTP timeout, bundling or not. A policy could be 
that when a timeout is near and not all responses have been obtained yet, an 
implementation  has the right to send back a bundle with whatever responses 
it has, along with an (new) error like "timeout constraint" to excuse the 
missing responses.

 

Minor comment:  the 
current bundling chapter writes:

 

"However, when InOrder 
delivery assurance is required for User message units, bundling SHOULD NOT be in 
effect for these messages as there is no trace of the submission order on the 
sending side, and the order of message units under eb:Messaging is not 
significant."

 

I think we could 
assume that message units are to be 
processed in document order.  There is still no guarantee that 
messages are assigned to the same bundle and/or that different bundles would 
arrive in-order though, so the warning against 
bundling is valid.  Perhaps an advanced MSH API could allow bundling or 
ordering to be controlled. Out of scope for 
spec.

 

Bundling and multihop

 

Assumption:  
intermediaries (such as the "transparent" intermediaries) may not bundle or 
unbundle messages. The MSH may not know which Pmodes target the same (logical) 
destination (as this requires knowledge of the routing function). So what can 
and cannot be bundled cannot be predicted based on Pmode parameters other than the proposed "bundleswith".  

 

<JD> But we could still have a default: any two messages from the 
same PMode leg can be bundled together, except when the above compatibility 
parameters are present, in which case they take 
over.

 

 It seems acceptable to assume that in a 
multi-hop context any routing function should be able to look at the first user 
message in a bundle to determine the destination. 
So the "first unit determines .." principle would apply to both bundling and 
routing. 

 

<JD> The "primary" unit sets the tone for bundling and routing. Now 
we may want to control whcih units can be "primary" and which ones cannot. E.g. 
the ICloud is configured with routing functions that only know about some "lead 
PartyId" for each MSH endpoint. There is always the resort of adding the 
routingInput ref parameter in that case. But if a sending gateway always get 
enough messages for this lead partyId, it should always use some of these as 
primary.

 

<JD> we may need 
these additional parameters:

- 
PMode[1].Bundling.Receipts: yes = bundling of Receipts 
requested for messages bundled together.

- PMode[1].Bundling.Errors: yes = bundling 
of Errors requested for messages bundled 
together.

 

Pim