Virtual Token Descriptor for eXtensible Markup Language (VTD-XML) refers to a collection of cross-platform XML
XML
Extensible Markup Language is a set of rules for encoding documents in machine-readable form. It is defined in the XML 1.0 Specification produced by the W3C, and several other related specifications, all gratis open standards....
processing technologies centered around a non-extractive XML
XML
Extensible Markup Language is a set of rules for encoding documents in machine-readable form. It is defined in the XML 1.0 Specification produced by the W3C, and several other related specifications, all gratis open standards....
, "document-centric" parsing technique called Virtual Token Descriptor (VTD). Depending on the perspective, VTD-XML can be viewed as one of the following:
Binary XML refers to any specification which defines the compact representation of XML in a binary format. While there are several competing formats, none has been widely adopted by a standards organization or accepted as a de facto standard...
XPath is a language for selecting nodes from an XML document. In addition, XPath may be used to compute values from the content of an XML document...
evaluator
VTD-XML is developed by XimpleWare and dual-licensed under GPL and proprietary license. It is originally written in Java, but is now available in C
C (programming language)
C is a general-purpose computer programming language developed between 1969 and 1973 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system....
C++ is a statically typed, free-form, multi-paradigm, compiled, general-purpose programming language. It is regarded as an intermediate-level language, as it comprises a combination of both high-level and low-level language features. It was developed by Bjarne Stroustrup starting in 1979 at Bell...
In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens. A program or function which performs lexical analysis is called a lexical analyzer, lexer or scanner...
represents tokens (the small units of indivisible character values) as discrete string objects. This approach is designated extractive parsing. In contrast, non-extractive tokenization mandates that one keeps the source text intact, and uses offsets and lengths to describe those tokens.
Virtual Token Descriptor
Virtual Token Descriptor (VTD) applies the concept of non-extractive, document-centric parsing to XML processing. A VTD record uses a 64-bit integer to encode the offset, length, token type and nesting depth of a token in an XML document. Because all VTD records are 64-bit in length, they can be stored efficiently and managed as an array.
Location Cache
Location Caches (LC) build on VTD records to provide efficient random access. Organized as tables, with one table per nesting depth level, LCs contain entries modeling an XML document's element hierarchy. An LC entry is a 64-bit integer encoding a pair of 32-bit values. The upper 32 bits identify the VTD record for the corresponding element. The lower 32 bits identify that element's first child in the LC at the next lower nesting level.
Overview
Virtually all the core benefits of VTD-XML are inherent to non-extractive, document-centric parsing which provides these characteristics:
The source XML text is kept intact in memory without decoding.
The internal representation of VTD-XML is inherently persistent
Persistence
Persistence may refer to:* Image persistence, in LCD monitors* Multidrug tolerance, a dormant, persistent state of a bacterial population* Persistence , the characteristic of data that outlives the execution of the program that created it...
.
Obviates object-oriented modeling of the hierarchical representation as it relies entirely on primitive data types (e.g., 64-bit integers) to represent the XML hierarchy, thus reducing object creation cost to nearly zero.
Combining those characteristics permits thinking of XML purely as syntax (bits, bytes, offsets, lengths, fragments, namespace-compensated fragments, and document composition) instead of the serialization
Serialization
In computer science, in the context of data storage and transmission, serialization is the process of converting a data structure or object state into a format that can be stored and "resurrected" later in the same or another computer environment...
/deserialization of objects. This is a powerful way to think about XML/SOA
Service-oriented architecture
In software engineering, a Service-Oriented Architecture is a set of principles and methodologies for designing and developing software in the form of interoperable services. These services are well-defined business functionalities that are built as software components that can be reused for...
applications.
Conformance
VTD-XML conforms strictly to XML 1.0 (Except the DTD part) and XML Namespace 1.0. It essentially conforms to XPath 1.0 spec (with some subtle differences in terms of underlying data model) with extension to XPath 2.0 built-in functions.
As Parser
When used in parsing mode, VTD-XML is a general purpose, extremely high performance XML parser which compares favorably with others:
VTD-XML typically outperforms SAX (with NULL content handler) while still providing full random access and built-in XPath
XPath
XPath is a language for selecting nodes from an XML document. In addition, XPath may be used to compute values from the content of an XML document...
support.
VTD-XML typically consumes 1.3-1.5 times the XML document's size in memory, which is about 1/5 the memory usage of DOM
Applications written in VTD-XML are usually much shorter and cleaner than their DOM or SAX versions.
As Indexer
Because of the inherent persistence of VTD-XML, developers can write the internal representation of a parsed XML document to disk and later reload it to avoid repetitive parsing. To this end, XimpleWare has introduced VTD+XML as a binary packaging format combining VTD, LC and the XML text. It can typically be viewed in one of the following two ways:
A native XML index that completely eliminates the parsing cost and also retains all benefits of XML. It is a file format that is human readable and backward compatible with XML.
Binary XML refers to any specification which defines the compact representation of XML in a binary format. While there are several competing formats, none has been widely adopted by a standards organization or accepted as a de facto standard...
format that uses binary data to enhance the processing of the XML text.
XML Content Modifier
Because VTD-XML keeps the XML text intact without decoding, when an application intends to modify the content of XML it only needs to modify the portions most relevant to the changes. This is in stark contrast with DOM, SAX, or StAx parsing, which incur the cost of parsing and re-serialization no matter how small the changes are.
Since VTDs refer to document elements by their offsets, changes to the length of elements occurring earlier in a document require adjustments to VTDs referring to all later elements. However, those adjustments are integer additions, albeit to many integers in multiple tables, so they are quick.
XML Slicer/Splitter/Assembler
An application based on VTD-XML can also use offsets and lengths to address tokens, or element fragments. This allows XML documents to be manipulated like arrays of bytes.
As a slicer, VTD-XML can "slice" off a token or an element fragment from an XML document, then insert it back into another location in the same document, or into a different document.
As a splitter, VTD-XML can split sub-elements in an XML document and dump each into a separate XML document.
As an assembler, VTD-XML can "cut" chunks out of multiple XML documents and assemble them into a new XML document.
XML Editor/Eraser
Used as an editor/eraser, VTD-XML can directly edit/erase the underlying byte content of the XML text, provided that the token length is wider than the intended new content. An immediate benefit of this approach is that the application can immediately reuse the original VTD and LC. In contrast, when using VTD-XML to incrementally update an XML document, an application needs to reparse the updated document before the application can process it.
An editor can be made smart enough to track the location of each token, permitting new, longer tokens to replace existing, shorter tokens by merely addressing the new token in separate memory outside that used to store the original document. Likewise, when reordering the document, element text does not need to be copied; only the LCs need to be updated. When a complete, contiguous XML document is needed, such as when saving it, the disparate parts can be reassembled into a new, contiguous document.
Other Benefits
VTD-XML also pioneers the non-blocking, stateless XPath evaluation approach.
Weaknesses
VTD-XML also exhibits a few noticeable shortcomings:
As an XML parser, it does not support external entities declared in the DTD.
As a file format, it increases the document size by about 30% to 50%.
The Document Object Model is a cross-platform and language-independent convention for representing and interacting with objects in HTML, XHTML and XML documents. Aspects of the DOM may be addressed and manipulated within the syntax of the programming language in use...
SAX is an event-based sequential access parser API developed by the XML-DEV mailing list for XML documents. SAX provides a mechanism for reading data from an XML document that is an alternative to that provided by the Document Object Model...
.
It is difficult to support certain validation techniques, employed by DTD and XML Schema (e.g., default attributes and elements), that require modifications to the XML instances being parsed.
General-purpose Replacement for DOM or SAX
Because of VTD-XML's performance and memory advantages, it covers a larger portion of XML use cases than either DOM or SAX.
Compared to DOM, VTD-XML processes bigger (3x~5x) XML documents for the same amount of physical memory at about 3 to 10 times the performance.
Compared to SAX, VTD-XML provides random access and XPath support and outperforms SAX by at least 2x.
XPath over Huge XML documents
The extended edition of VTD-XML combining with 64-bit JVM makes possible XPath-based XML processing over huge XML documents (up to 256 GB) in size.
For SOA/WS/XML Security
The combination of VTD-XML's high performance and incremental-update capability makes it essential to achieve the desired level of Quality of Service
Quality of service
The quality of service refers to several related aspects of telephony and computer networks that allow the transport of traffic with special requirements...
In software engineering, a Service-Oriented Architecture is a set of principles and methodologies for designing and developing software in the form of interoperable services. These services are well-defined business functionalities that are built as software components that can be reused for...
An enterprise service bus is a software architecture model used for designing and implementing the interaction and communication between mutually interacting software applications in Service Oriented Architecture...
es, and services aggregation points. All those applications perform the basic "store and forward" operations for which retaining the original XML is critical for minimizing latency. VTD-XML's incremental update capability also contributes significantly to the forwarding performance.
VTD-XML's random-access capability lends itself well to XPath
XPath
XPath is a language for selecting nodes from an XML document. In addition, XPath may be used to compute values from the content of an XML document...
-based XML routing/switching/filtering common in AJAX
Ajax
- Mythology :* Ajax , son of Telamon, ruler of Salamis and a hero in the Trojan War, also known as "Ajax the Great"* Ajax the Lesser, son of Oileus, ruler of Locris and the leader of the Locrian contingent during the Trojan War.- People :...
and SOA deployment.
Intelligent SOA/WS/XML Load-balancing and Offloading
When an XML document travels through several middle-tier SOA components, the first message stop, after finishing the inspection of the XML document, can choose to send the VTD+XML file format to the downstream components to avoid repetitive parsing, thus improving throughput.
By the same token, an intelligent SOA load balancer can choose to generate VTD+XML for incoming/outgoing SOAP messages to offload XML parsing from the application servers that receive those messages.
XML Persistence Data Store
When viewed from the perspective of native XML persistence, VTD-XML can be used as a human-readable, easy to use, general-purpose XML index. XML documents stored this way can be loaded into memory to be queried, updated, or edited without the overhead of parsing/re-serialization.
Schemaless XML Data Binding
VTD-XML's combination of high performance, low memory usage, and efficient XPath evaluation makes possible a new XML data binding
XML data binding
XML data binding refers to a means of representing information in an XML document as an object in computer memory. This allows applications to access the data in the XML from the object rather than using the DOM or SAX to retrieve the data from a direct representation of the XML itself.An XML data...
approach based entirely on XPath. This approach's biggest benefit is it no longer requires XML schema, avoids needless object creation, and takes advantage of XML's inherent loose encoding.
It is worth noting that data binding discussed in the article mentioned above needs to be implemented by the application: VTD-XML itself only offers accessors. In this regard VTD-XML is not a data binding solution itself (unlike JiBX, JAXB, XMLBeans), although it offers extraction functionality for data binding packages, much like other XML parsers (STAX, StAX).
Essential Classes
As of Version 2.6, the Java and C# versions of VTD-XML consist of the following classes:
VTDGen (VTD Generator) is the class that encapsulates the main parsing, index loading and index writing functions.
VTDNav (VTD Navigator) is the class that (1) encapsulates XML, VTD, and hierarchical info, (2) contains various navigation methods,(3) performs various comparisons between VTD records and strings, and (4) converts VTD records to primitive data types.
AutoPilot is a class containing functions that perform node-level iteration and XPath.
XMLModifier is a class that offers incremental update capability, such as delete, insert and update.
The extended VTD-XML consists of the following classes:
VTDGenHuge (Extended VTD Generator) encapsulates the main parsing.
XMLBuffer performs in-memory loading of XML documents.
XMLMemMappedBuffer performs memory mapped loading of XML documents.
VTDNavHuge (Extended VTD Navigator)1) encapsulates XML, Extended VTD, and hierarchical info, (2) contains various navigation methods,(3) performs various comparisons between VTD records and strings, and (4) converts VTD records to primitive data types.
AutoPilotHuge performs node-level iteration and XPath.
Code Sample
/* In this java program, we demonstrate how to use XMLModifier to incrementally
update a simple XML purchase order.
a particular name space. We also are going
to use VTDGen's parseFile to simplify programming.
/
import com.ximpleware.*;
public class Update {
public static void main(String argv[]) throws NavException, ModifyException, IOException{
// open a file and read the content into a byte array
VTDGen vg = new VTDGen;
if (vg.parseFile("oldpo.xml", true)){
VTDNav vn = vg.getNav;
AutoPilot ap = new AutoPilot(vn);
XMLModifier xm = new XMLModifier(vn);
ap.selectXPath("/purchaseOrder/items/item[@partNum='872-AA']");
int i = -1;
while((i=ap.evalXPath)!=-1){
xm.remove;
xm.insertBeforeElement("\n");
}
ap.selectXPath("/purchaseOrder/items/item/USPrice[.<40]/text");
while((i=ap.evalXPath)!=-1){
xm.updateToken(i,"200");
}
xm.output("newpo.xml");
}
}
}
The source of this article is wikipedia, the free encyclopedia. The text of this article is licensed under the GFDL.
