> Sorry, I don't get this.
> Do you have small examples (e.g. one for translate, on for text analysis)
> of the difference?
My understanding is that to work with an ITS processor we would change each span marked by <sm/>/<em/> to a set of <mrk>/</mrk>. At
least this is what I read from your algorithm (not from Fredrik's option).
For example:
<sm id='1' translate='no'/>French <pc id='2'>Canadian<em startRef='1'/> hockey</pc>.
Would be changed to:
<mrk id='1' translate='no'>French </mrk><pc id='2'><mrk id='m1' translate='no'>Canadian</mrk> hockey</pc>.
This would get you the same properties as the original for each character of the content.
But you can't do this for a data category where the content is related in a meaningful way to the attributes of the data category.
For example:
<sm id='1' type='term' ref="http://en.wikipedia.org/wiki/Qu%C3%A9b%C3%A9cois'/>French <pc id='2'>Canadian<em startRef='1'/>
hockey</pc>.
Cannot be split into two instances:
<mrk id='1' type='term' ref="http://en.wikipedia.org/wiki/Qu%C3%A9b%C3%A9cois'>French </mrk><pc id='2'><mrk id='3' type='term'
ref="http://en.wikipedia.org/wiki/Qu%C3%A9b%C3%A9cois'>Canadian</mrk> hockey</pc>.
That is: the term is not "French " or "Canadian" it is "French Canadian".
But maybe I didn't get your algorithm correctly and it doesn't result in multiple <mrk>/</mrk> for a single <sm/>/<em/>.
Looking at Fredrik's note: there are various ways to reduce the <sm/>/<em/> in favor of well-formed <mrk> elements (joining all
segments, prioritizing well-formness of annotation over inline codes, etc.) but I'm not sure it can be an absolute solution: the
markup may originate from the XLIFF tool and care little about well-formness, and one always can have overlapping annotations.
But maybe this is a restriction we can live with.
-ys
-----Original Message-----
From: Felix Sasaki [mailto:]
Sent: Thursday, October 9, 2014 12:20 PM
To: Yves Savourel
Cc: XLIFF Main List; public-i18n-its-ig
Subject: Re: [xliff] ITS scope with sm/em
Am 09.10.2014 um 14:18 schrieb Yves Savourel <>:
> Yes, something like the MT Confidence value is different, but that
> conversion can be described in the mapping itself (If I recall correctly). So an ITS processor has nothing 'special' to do: it
just applies the rules.
>
> I suppose we could have additional pre-processing steps for a case
> like <sm>/<em>. But that means you can't really use a 'pure' ITS processor to look at an XLIFF file because it would not know how
to do the pre-processing.
> But that is probably acceptable, especially if we provide generic ways to do the transformation.
>
> This said, I'm not 100% sure you can transform <sm>/<em> into
> <mrk>/</mrk> for all data categories: it would be ok for things like translate, domain, etc. But info like terminology, Text
Analysis, LQI make sense only when set as a single content.
Sorry, I don't get this. Do you have small examples (e.g. one for translate, on for text analysis) of the difference?
- Felix
>
> -ys
>
>
> -----Original Message-----
> From: [mailto:]
> On Behalf Of Felix Sasaki
> Sent: Thursday, October 9, 2014 5:53 AM
> To: Yves Savourel
> Cc: Dr. David Filip; Estreen, Fredrik; XLIFF Main List;
> public-i18n-its-ig
> Subject: Re: [xliff] ITS scope with sm/em
>
> Hi Yves, al,
>
> understand. Though: aren't there also other parts of the xliff/its
> mapping that require from an ITS1 or 2 processor special handling?
> E.g. mt confidence
> https://www.w3.org/International/its/wiki/XLIFF_2.0_Mapping#MT_Confide
> nce_.28.3D.3D.3D.3D.3D.3D.3D.3D.3D.3DTO_REVIEW.29
> which requires a computation of values. In the other thread you just listed the types of processors:
>
> "- An XLIFF Extractor aware of both ITS and the ITS module for any data coming from the original source document.
> - An XLIFF Modifier aware of the ITS Module for data generated during the life time of the XLIFF document.
> - An XLIFF Merger aware of both the ITS Module and the ITS syntax if any of that data is merged back into the translated
document."
>
> Couldn't we require in the mapping specification that before using a
> general ITS processor uses XLIFF+ITS content, it has to do the preprocessing described in this thread, the one for MT confidence
etc?
>
> Cheers,
>
> Felix
>
> Am 09.10.2014 um 13:33 schrieb Yves Savourel <>:
>
>> Hi all,
>>
>> Thanks for the input Fredrik and Felix.
>>
>> I'm not worried about the XLIFF implementation of those cases: We
>> have had working code for those since a long time (a good use
> case is mrk with translate='yes|no').
>>
>> I was thinking more about the ITS aspect of it.
>>
>>> From an ITS viewpoint something like this: <sm id='1'
>>> itx:domain='travel'/>...<em startRef='1'/> the scope of the domain
>>> is an
> empty content (the content of <sm/>). There is nothing in ITS that allows to use distinct elements to annotate a span.
>>
>> Because, while on the XLIFF side the processing expectation is to
>> treat the content between a given <sm/> and its corresponding
> <em/> as a span, on the ITS side there is no semantic for such construct.
>>
>> Cheers,
>> -ys
>>
>>
>> From: Dr. David Filip [mailto:]
>> Sent: Thursday, October 9, 2014 5:08 AM
>> To: Felix Sasaki
>> Cc: Estreen, Fredrik; Yves Savourel; XLIFF Main List;
>> public-i18n-its-ig
>> Subject: Re: [xliff] ITS scope with sm/em
>>
>> Felix, I like the algorithmic approach that is open to different implementations.
>>
>> After all ITS is a set of abstract categories that should not be restricted to hierarchical structured formats.
>>
>> Now to your proposed algorithm.
>>
>> Unlike native codes, annotations MUST have the opening and closing tag in the same unit.
>> So you will be always creating <mrk> nodes from <sm/> tags if you consider the whole <unit> content, which is the point..
>>
>> Cheers
>> dF
>>
>>
>> Dr. David Filip
>> =======================
>> OASIS XLIFF TC Secretary, Editor, and Liaison Officer LRC | CNGL |
>> CSIS University of Limerick, Ireland
>> telephone: +353-6120-2781
>> cellphone: +353-86-0222-158
>> facsimile: +353-6120-2734
>> http://www.cngl.ie/profile/?i=452
>> mailto:
>>
>> On Thu, Oct 9, 2014 at 3:49 AM, Felix Sasaki <> wrote:
>> I agree with Fredrik. Processing of overlapping hierarchies is a task
>> that cannot be solved in general and discarding
> non-hierarchical structures is a good strategy for XML / HTML content.
>>
>>
>> If people don't want to specify an XSLT conversion we could also define the conversion process in an algorithmic way like this:
>>
>> 0) set current content to whole content to be processed.
>> 1) is there an s tag in current content?
>> Then output text before s tag and do 2)
>> else just output all text in current content.
>> 2) has the s tag an e tag with corresponding id?
>> Then create a mrk node
>> set the content between s and e to new current content
>> do 1)
>> else discard s and go to 1)
>> 3) output rest of text
>>
>> and say: you can implement this as XSLT (example given) or in
>> different programing languages. That would have the benefit to keep
> the door open to future non XML, API focsued XLIFF.
>>
>> - Felix
>>
>> Am 08.10.2014 um 18:41 schrieb Estreen, Fredrik <>:
>>
>>> Hi Yves,
>>>
>>>> Hi all,
>>>>
>>>> Looking at the ITS mapping: In many case we put the ITS information
>>>> on a marker (<mrk> element).
>>>>
>>>> But such element can be represented by <sm/>...<em/> when it's
>>>> overlapping another element.
>>>> In that case the normal ITS scope mechanism can't work because it
>>>> applies to the empty content of <sm/>, not the content between
>>>> <sm/> and the corresponding <em/>.
>>>>
>>>> We can have provision for this in the XLIFF module. But I'm not
>>>> sure it's doable in the ITS rules, especially with inheritance when
>>>> there are nested annotations.
>>>
>>> This is an interesting problem and I doubt it is solvable in a
>>> general way without additional steps. It might be solvable when
> the <sm/> and <em/> is in the same segment, but I doubt it is in the case where they start and end in different segments (ie.
> different sibling trees).
>>>
>>> One potentially workable solution would be to apply an XSLT transform on the XLIFF that merges all segments in each unit.
> Discards any non ITS carrying marker (to reduce risk of overlapping
> markers) and finally normalize the remaining markers to the
> <mrk></mrk> spanning form. Since ITS information will likely be coming
> from and going to an XML source there should not be any overlapping
> markers at that stage as they would be difficult to represent in the source format. It is not guaranteed but we could declare that
ill-formed. ITS global rules could then be evaluated against the transformed version. Admittedly not the most beautiful solution but
I think it could work.
>>>
>>>> I vaguely recall that such topic was discussed at some point in the ITS-WG no?
>>>> Does anyone recall the outcome?
>>>>
>>>> Cheers,
>>>> -ys
>>>
>>> Regards,
>>> Fredrik Estreen
>>>
>>> --------------------------------------------------------------------
>>> - To unsubscribe from this mail list, you must leave the OASIS TC
>>> that generates this mail. Follow this link to all your TCs in OASIS
>>> at:
>>> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.p
>>> h
>>> p
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe from this mail list, you must leave the OASIS TC that
>> generates this mail. Follow this link to all your TCs in OASIS at:
>> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.ph
>> p
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that
> generates this mail. Follow this link to all your TCs in OASIS
> at:
> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
>
>
>
