Parsers

This article is part of the Advanced User's Guide. It presents the available parsers that can be used to import various data sources in BaseX databases. Please visit the Serialization article if you want to know how to export data.

XML Parsers

BaseX provides two alternatives for parsing XML:

• By default, Java’s SAXParser is used to parse XML documents.
• The internal, built-in XML parser is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out of the box, and it is faster than the default parser, in particular if small documents are to be parsed. However, the internal parser does not support the full range of DTD features and cannot resolve catalogs.

GUI

Go to Menu Database → New, then choose the Parsing tab and (de)activate Use internal XML parser. The parsing of DTDs can be turned on/off by selecting the checkbox below.

Command Line

To turn the internal XML parser and DTD parsing on/off, modify the INTPARSE and DTD options:

 SET INTPARSE true
 SET DTD true

XQuery

The db:add and db:put functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database DB that are found in the directory 2Bimported:

for $file in file:list("2Bimported")
return db:add('DB', $file, '', map { 'intparse': true() })

HTML Parser

If TagSoup is found in the classpath, HTML input is converted to well-formed XML documents.

If TagSoup is not available on a system, the default XML parser will be used, and the import will only succeed if the input is already well-formed XML (which is usually the case for XHTML documents).

Installation

Downloads

TagSoup is already included in the full BaseX distributions (BaseX.zip, BaseX.exe, etc.). It can also be manually downloaded and embedded on the appropriate platforms.

Maven

An easy way to add TagSoup to your project is to as follows:

1. Visit MVN TagSoup Repository

2. Click on the version you want

3. On the first tab, you can see an XML snippet like this:

<dependency>
  <groupId>org.ccil.cowan.tagsoup</groupId>
  <artifactId>tagsoup</artifactId>
  <version>1.2.1</version>
</dependency>

4. Insert the XML fragment into the <dependencies> element of your project’s pom.xml file.

Debian

With Debian, TagSoup will automatically be detected and included after it has been installed via:

 apt-get install libtagsoup-java

Options

TagSoup offers a variety of options to customize the HTML conversion. For the complete list, please visit the TagSoup website. BaseX supports most options, with a few exceptions:

• encoding: BaseX tries to guess the input encoding, but this can be overwritten by this option.
• files: not supported as the input documents are piped directly to the XML parser.
• method: set to 'xml' as default. If this is set to 'html' ending tags may be missing for instance.
• version: dismissed, as TagSoup always falls back to 'version 1.0', no matter what the input is.
• standalone: deactivated.
• pyx, pyxin: not supported as the XML parser can't handle this kind of input.
• output-encoding: not supported, BaseX already takes care of that.
• reuse, help: not supported.

GUI

Go to Menu Database → New and select "HTML" in the input format combo box. There's an info in the "Parsing" tab about whether TagSoup is available or not. The same applies to the "Resources" tab in the "Database Properties" dialog.

These two dialogs come with an input field 'Parameters' where TagSoup options can be entered. Keys and values are separated by the equal sign, multiple options are separated by commas, e.g.: method=xml,nons=true.

Command Line

Turn on the HTML Parser before parsing documents, set a file filter and optional parsing options:

 SET PARSER html
 SET CREATEFILTER *.html
 SET HTMLPARSER method=xml,nons=true

XQuery

The HTML Module provides XQuery functions for converting HTML to XML. In addition, all functions that accept Parsing Options can be used as well to convert HTML:

fetch:doc("index.html", map {
  'parser': 'html',
  'htmlparser': map { 'method': 'xml', 'nons': true() }
})

JSON Parser

With BaseX, JSON documents can be imported. The JSON parsing options are described in the documentation of the JSON Module.

GUI

Go to Menu Database → New and select "JSON" in the input format combo box. You can set the following options for parsing JSON documents in the "Parsing" tab.

Command Line

Turn on the JSON Parser before parsing documents, and set some optional, parser-specific options and a file filter:

 SET PARSER json
 SET JSONPARSER encoding=utf-8, jsonml=true
 SET CREATEFILTER *.json

XQuery

The JSON Module provides functions for converting JSON to XML.

In addition, all functions that accept Parsing Options can be used as well to convert JSON. The following query converts a JSON string to an XML document, using the conversion format basic:

db:create('json-db', '{ "key": "value" }', 'example.json', map {
  'parser': 'json',
  'jsonparser': map { 'format': 'basic' }
})

CSV Parser

BaseX can be used to import CSV documents. The CSV parsing options are described in the documentation of the CSV Module.

GUI

Go to Menu Database → New and select "CSV" in the input format combo box. You can set the following options for parsing CSV documents in the "Parsing" tab.

Command Line

Enable CSV Parsing before parsing documents, and set optional, parser-specific options and a file filter:

 SET PARSER csv
 SET CSVPARSER encoding=utf-8,header=true,separator=semicolon
 SET CREATEFILTER *.csv

XQuery

The CSV Module provides a function for converting CSV to XML.

In addition, all functions that accept Parsing Options can be used as well to convert CSV. In the following, all CSV files from a directory are added to a database, with the first line of each file interpreted as column header:

for $file in file:list("2Bimported", false(), "*.csv")
return db:add("csv-db", $file, "", map {
  'parser': 'csv',
  'csvparser': map { 'header': true() }
})

Changelog

Version 11.0

• Removed: Text Parser

Version 7.8

• Updated: parser options

Version 7.7.2

• Removed: CSV option "format"

Version 7.3

• Updated: the CSV SEPARATOR option may now be assigned arbitrary single characters

Version 7.2

• Updated: Enhanced support for TagSoup options