Changes

This article is part of the [[Getting StartedAdvanced User's Guide]] Section.It presents the available parsers that can be used to import various data sources in BaseX databases.Please visit the [[Serialization]] article if you want to know how to export data.

BaseX provides two parsers to import alternatives for parsing XML data:

* By default, the Java’s [https://docs.oracle.com/en/java/javase/11/docs/api/java.xml/javax/xml/parsers/SAXParser.html SAXParser] is used to parse XML documents.* The internal, built-in XML parser is used, which is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out-of-the-box, and it is faster than the default parser, in most casesparticular if small documents are to be parsed. In turnHowever, it the internal parser does not support all oddities specified by DTDs, the full range of DTD features and cannot resolve [[Catalog_Resolver|catalogs]].* Java’s [http://download.oracle.com/javase/6/docs/api/javax/xml/parsers/SAXParser.html SAXParser] can also be selected for parsing XML documents. This parser is stricter than the built-in parser, but it refuses to process some large documents.

SET [[Options#INTPARSE{{Option|INTPARSE]] }} true SET [[Options#DTD{{Option|DTD]] }} true

The [[{{Function|Database Module#db:add|db:add]] }} and [[{{Function|Database Module#db:replace|db:replace]] put}} functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database <code>DB</code> that are found in the directory <code>2Bimported</code>:

<pre classlang="brush:'xquery"'>

With If [http://homevrici.ccillojban.org/~cowan/XML/tagsoup/ TagSoup] is found in the [[Startup#Distributions|classpath]], HTML can be imported in BaseX without any problems. TagSoup ensures that only input is converted to well-formed HTML arrives at the XML parser (correct opening and closing tags, etcdocuments.). Hence, if  If TagSoup is not available on a system, there the default XML parser will be a lot of cases where importing HTML failsused, no matter whether you use and the GUI or import will only succeed if the input is already well-formed XML (which is usually the standalone modecase for XHTML documents).

====Downloads====

====Maven====

An easy way to add TagSoup to your own project is to follow this stepsas follows:

1. visit Visit [httphttps://mvnrepository.com/artifact/org.ccil.cowan.tagsoup/tagsoup/ MVN TagSoup Repository]

2. click Click on the version you want

3. On the first tab, you can see on the first tab called Maven a an XML snippet like this :

<pre classlang="brush:xml">

4. copy that in Insert the XML fragment into the <code><dependencies></code> element of your own maven project's project’s <code>pom.xml under the <dependencies/code> tagfile.

====Debian==== With Debian, TagSoup will automatically be automatically detected and included after it has been installed via:

==TagSoup Options==

TagSoup offers a variety of options to customize the HTML conversion. For the complete list, please visit the [http://homevrici.ccillojban.org/~cowan/XMLtagsoup/tagsoup#program TagSoup] website. BaseX supportsmost of these options , with a few exceptions:

* '''encoding''': BaseX tries to guess the input encoding , but this can be overwritten by the user if necessarythis option.* '''files''': not supported as the input documents are piped directly to the XML parser.

===GUI===

Go to Menu ''Database'' → ''New'' and select "HTML" in the input format combo box.There's an info in the "Parsing" tab about whether TagSoup is available or not. Thesame applies to the "Resources" tab in the "Database Properties" dialog.

These two dialogs come with an input field 'Parameters' where TagSoup optionscan be entered. Keys and values are separated by the equal sign, multiple options are separated by commas, e.g.: <code>method=xml,nons=true</code>.

===Command Line===

Turn on the HTML Parser before parsing documents, and set a file filterand optional parsing options:

SET [[Options#{{Option|PARSER}} html SET {{Option|PARSER]] CREATEFILTER}} *.html SET [[Options#HTMLPARSER{{Option|HTMLPARSER]] }} method=xml,nons=true,nocdata=true,nodefaults=true,nobogons=true,nocolons=true,ignorable=true SET [[Options#CREATEFILTER|CREATEFILTER]] *.html

===XQuery===

The [[HTML Module]] provides a function XQuery functions for converting HTML to XML documents.In addition, all functions that accept [[Options#Parsing|Parsing Options]] can be used as well to convert HTML:

Documents can also be converted by specifying the parser and additional optionsas function arguments: <pre classlang="brush:'xquery"'>fetch:xmldoc("index.html", map {

'htmlparser': map { 'htmlmethod': false()'xml', 'nodefaultsnons': true() }

With BaseX can also import , JSON documentscan be imported. The resulting format is JSON parsing options are described in the documentation for of the XQuery [[JSON Module]]:.

Go to Menu ''Database'' → ''New'' and select "JSON" in the input format combo box.You can set the following options for parsing JSON documents in the "Parsing" tab: * '''Encoding''': Choose the appropriate encoding of the JSON file.* '''JsonML''': Activate this option if the incoming file is a JsonML file.

SET [[Options#PARSER{{Option|PARSER]] }} json SET [[Options#JSONPARSER{{Option|JSONPARSER]] }} encoding=utf-8, jsonml=true SET [[Options#CREATEFILTER{{Option|CREATEFILTER]] }} *.json

The [[JSON Module]] provides functions for converting JSON objects to XML documents. In addition, all functions that accept [[Options#Parsing|Parsing Options]] can be used as well to convert JSON. The following query converts a JSON string to an XML document, using the conversion format {{Code|[[JSON Module#Basic|basic]]}}: <pre lang='xquery'>db:create('json-db', '{ "key": "value" }', 'example.json', map { 'parser': 'json', 'jsonparser': map { 'format': 'basic' }})</pre>

BaseX can be used to import CSV documents. Different alternatives how to proceed The CSV parsing options are shown described in the following:documentation of the [[CSV Module]].

You can set the following options for parsing CSV documents in the "Parsing" tab: * '''Encoding''': Choose the appropriate encoding of the CSV file.* '''Separator''': Choose the column separator of the CSV file. Possible: <code>comma</code>, <code>semicolon</code>, <code>tab</code> or <code>space</code> or an arbitrary character.* '''Header''': Activate this option if the incoming CSV files have a header line.

Turn on the Enable CSV Parser Parsing before parsing documents, and set some optional, parser-specific options and a file filter. Unicode code points can be specified as separators; {{Code|32}} is the code point for spaces:

SET [[Options#PARSER{{Option|PARSER]] }} csv SET [[Options#CSVPARSER{{Option|CSVPARSER]] }} encoding=utf-8, linesheader=true, header=false, separator=spacesemicolon SET [[Options#CREATEFILTER{{Option|CREATEFILTER]] }} *.csv

The [[CSV Module]] provides a function for converting CSV to XML documents.

Documents In addition, all functions that accept [[Options#Parsing|Parsing Options]] can also be converted by specifying used as well to convert CSV. In the parser in an XQuery function.The following example query adds , all CSV files that from a directory are located in thedirectory {{Code|2Bimported}} added to the a database {{Code|DB}} and interprets, with the first lines line of each file interpreted as column headersheader:

<pre classlang="brush:'xquery"'>

return db:add("DBcsv-db", $file, "", map {

=Text Parser= Plain text can be imported as well: ==GUI=Changelog=

Go to Menu ''Database'' → ''New'' and select "TEXT" in the input format combobox;Version 11.You can set the following option for parsing text documents in the "Parsing" tab:0* '''Encoding'''Removed: Choose the appropriate encoding of the text file.* '''Lines''': Activate this option to create a <code>&lt;line&gt;...&lt;/line&gt;</code> element for each line of the input text file. ==Command Line== Turn on the CSV Text Parser before parsing documents and set some optional, parser-specific options and a file filter:  SET [[Options#PARSER|PARSER]] text SET [[Options#TEXTPARSER|TEXTPARSER]] lines=yes SET [[Options#CREATEFILTER|CREATEFILTER]] * ==XQuery== Similar to the other formats, the text parser can also be specified via XQuery: <pre class="brush:xquery">for $file in file:list("2Bimported", true(), "*.txt")return db:add("DB", $file, "", map { 'parser': 'text' })</pre> =Changelog=