Re: [xml-dev] Two tips for dramatically increasing the speed of validating a large number of XML files
and parallelize execution with GNU parallel or xargs
On Wed, Feb 10, 2021 at 1:06 PM Roger L Costello <[email protected]> wrote:
Hi Folks,
Note: The following only applies if you are using SAXON for schema validation.
I have over 6,000 XML files that I need to validate against an XML Schema. Initially, I used commands in a batch (.bat) file to loop over all the XML files and call SAXON to validate them. I never got through all 6,000 files because it was taking so long; I estimate it would have taken 3 hours or more.
I looked on the SAXON web page and found some tips for dramatically increasing the speed of validating a large number of XML documents.
Tip #1: Compile your XML Schema to a Schema Component Model (SCM). Here's how to generate the SCM file:
java com.saxonica.Validate -export:file.scm -xsd:file.xsd
That results in creating file.scm which is a compiled version of file.xsd
You can use file.scm anywhere you would use file.xsd and it will run much faster.
Tip #2: Specify all your XML files to be validated using the glob syntax, *.xml
Suppose all the XML files to be validated are in the folder named instances. Then use this command to validate all of them:
java com.saxonica.Validate -xsd:file.scm instances\*.xml
Notice that that uses the SCM file not the XSD file.
I used those two tips to validate my 6,000 XML files. How long did it take to validate the entire bunch? Answer: Immediate (i.e., way less than one second). So, the time to validate all 6,000 XML files dropped from 3 hours to less than one second. Pretty cool, aye?
/Roger
_______________________________________________________________________
XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.
[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: [email protected]
subscribe: [email protected]
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php