Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC- Proposal

I have a simple answer:

    

    On 2/28/11 3:17 PM, Anish Karmarkar wrote:
    So
      what I'm wondering is: why would we not have adapting and passing
      the test suite be a requirement to claiming conformance (necessary
      but not sufficient condition) to the main spec? Wouldn't that keep
      implementations and vendor's marketing departments honest? It
      always happens that there are ambiguities and various
      interpretation of a spec despite best efforts. The test suite, is
      a runnable artifact with little doubt as to what one is supposed
      to do. Certainly, the test suite won't cover everything and isn't
      intended to do so. But I think a as close to an iron-clad
      conformance criteria that we can get to, the better off we are wrt
      ensuring portability and interop.
      

    
    

    My take is simple - the assembly spec in particular, is about a
    model for describing services. The test suite, as such, is simply a
    projection of that model coupled to a Java-based execution
    environment. As with all mappings some details are lost, and some
    are gained as part of that projection. Do you know which details
    those are? I certainly don't.

    

    For me to understand the answer to that question, I would want a
    significant chunk of time, and I expect the same for others in TC,
    and what would be the added value? This extra level of conformance
    would add to the "marketplace" a binary flag for
    conformance/non-conformance? But who cares? For the near term, SCA
    end users will likely trend towards medium to large corporations. It
    won't include random developers who are content to kick around Ruby
    or Python code, download open source projects, and cobble them
    together in whatever way works. In contrast to those small time
    developers, those medium and large corporations will come to their
    own conclusions about how important conformance to all of SCA might
    be. If you run the test suite, and get 90% conformance, is that good
    enough? It might be, if the runtime in question is 50% faster than
    all the others. So I think, if you raise the conformance bar to
    include the test suite, it *still* won't be a binary answer.

    

    In addition, conforming to the spec means looking at a very limited
    set of conformance claims. Looking at the test suite, how do I know
    whether a line of code is considered "normative". What if it happens
    to break in my runtime? The authority for deciding whether the test
    is incorrect already comes from the existing spec. What if a
    newer version of Java breaks some aspect of the test suite? What
    does that even mean?

    

    We know that Microsoft asked us to jump through hoops so that they
    could claim conformance with an implementation type not based on
    Java. And we agreed to that. Are you suggesting we go back on that?

    

    Now, to change that, by requiring that they also pass the test suite
    feels like you're trying to overturn a previous TC decision. In
    addition, all previous votes on the test suite have been held with
    the expectation that the the test suite itself was *not* a
    conformance criteria, and thus for my votes at least, I've
    considered it a much lower threshold to clear.

    

    -Eric.