Changes

This article is part of the [[Getting StartedAdvanced User's Guide]] Section.It presents the available parsers that can be used to import various data sources in BaseX databases.Please visit the [[Serialization]] article if you want to know how to export data.The names of the parser options have been updated with {{Version|7.8}}.

==XML Parsers==

BaseX provides two parsers to import alternatives for parsing XML data:

* By default, the Java’s [https://docs.oracle.com/en/java/javase/11/docs/api/java.xml/javax/xml/parsers/SAXParser.html SAXParser] is used to parse XML documents.* The internal, built-in XML parser is used, which is more fault-tolerant than Java’s XML parser. It supports standard HTML entities out-of-the-box, and it is faster than the default parser, in most casesparticular if small documents are to be parsed. In turnHowever, it the internal parser does not support all oddities specified by DTDs, the full range of DTD features and cannot resolve [[Catalog_Resolver|catalogs]].* Java’s [http://download.oracle.com/javase/6/docs/api/javax/xml/parsers/SAXParser.html SAXParser] can also be selected for parsing XML documents. This parser is stricter than the built-in parser, but it refuses to process some large documents.

===GUI===

===Command Line===

To turn the internal XML parser and DTD parsing on/off, modfify modify the <code>INTPARSE</code> and <code>DTD</code> options:

SET [[Options#INTPARSE{{Option|INTPARSE]] }} true SET [[Options#DTD{{Option|DTD]] }} true

===XQuery===

The [[{{Function|Database Module#db:add|db:add]] }} and [[{{Function|Database Module#db:replace|db:replace]] put}} functions can also be used to add new XML documents to the database. The following example query uses the internal XML parser and adds all files to the database <code>DB</code> that are found in the directory <code>2Bimported</code>:

<pre classlang="brush:'xquery"'>declare option db:intparse "yes";

return db:add('DB', $file, '', map { 'intparse': true() })

==HTML Parser==

With If [http://homevrici.ccillojban.org/~cowan/XML/tagsoup/ TagSoup] is found in the [[Startup#Distributions|classpath]], HTML can be imported in BaseX without any problems. TagSoup ensures that only input is converted to well-formed HTML arrives at the XML parser (correct opening and closing tags, etc.). Hence, if TagSoup is not available on a system, there will be a lot of cases where importing HTML fails, no matter whether you use the GUI or the standalone modedocuments.

==Installation== =Downloads==Downloads===

=====Maven=====

An easy way to add TagSoup to your own project is to follow this stepsas follows:

1. visit Visit [httphttps://mvnrepository.com/artifact/org.ccil.cowan.tagsoup/tagsoup/ MVN TagSoup Repository]

2. click Click on the version you want

3. On the first tab, you can see on the first tab called Maven a an XML snippet like this :

<pre classlang="brush:xml">

4. copy that in Insert the XML fragment into the <code><dependencies></code> element of your own maven project's project’s <code>pom.xml under the <dependencies/code> tagfile.

=====Debian===== With Debian, TagSoup will automatically be automatically detected and included after it has been installed via:

===TagSoup Options===

TagSoup offers a variety of options to customize the HTML conversion. For the complete list, please visit the [http://homevrici.ccillojban.org/~cowan/XMLtagsoup/tagsoup#program TagSoup] website. BaseX supportsmost of these options , with a few exceptions:

* '''encoding''': BaseX tries to guess the input encoding , but this can be overwritten by the user if necessarythis option.* '''files''': not supported as the input documents are piped directly to the XML parser.

====GUI==== Go to Menu ''Database'' → ''New'' and select "HTML" in the input format combo box.There's an info in the "Parsing" tab about whether TagSoup is available or not. Thesame applies to the "Resources" tab in the "Database Properties" dialog.

These two dialogs come with an Go to Menu ''Database'' → ''New'' and select "HTML" in the input field format combo box. There'Parameters' where s an info in the "Parsing" tab about whether TagSoup optionscan be enteredis available or not. The same applies to the "Resources" tab in the "Database Properties" dialog.

These two dialogs come with an input field 'Parameters' where TagSoup options can be entered. Keys and values are separated by the equal sign, multiple options are separated by commas, e.g.: <code>method=xml,nons===Command Line====true</code>.

SET [[Options#PARSER|PARSER]] html SET [[Options#HTMLPARSER|HTMLPARSER]] method=xmlTurn on the HTML Parser before parsing documents,nons=true,ncdata=true,nodefaults=true,nobogons=true,nocolons=true,ignorable=true SET [[Options#CREATEFILTER|CREATEFILTER]] *.htmlset a file filter and optional parsing options:

SET {{Option|PARSER}} html SET {{Option|CREATEFILTER}} *.html SET {{Option|HTMLPARSER}} method=xml,nons===XQuery====true

Documents The [[HTML Module]] provides XQuery functions for converting HTML to XML. In addition, all functions that accept [[Options#Parsing|Parsing Options]] can also be converted by specifying the parser and additional optionsin the query prologused as well to convert HTML:

<pre classlang="brush:'xquery"'>declare option dbfetch:parser doc("index.html";, map {declare option db 'parser':htmlparser "'html=false";',doc 'htmlparser': map { 'method': 'xml', 'nons': true("index.html") }})

==JSON Parser== BaseX can also import JSON documents. The resulting format is described in the documentation for the XQuery [[JSON Module]]:

* Go to Menu ''Database'Encoding'→ '': Choose New'' and select "JSON" in the appropriate encoding of input format combo box. You can set the following options for parsing JSON file.* '''JsonML''': Activate this option if documents in the incoming file is a JsonML file"Parsing" tab.

===Command Line===

SET [[Options#PARSER{{Option|PARSER]] }} json SET [[Options#JSONPARSER{{Option|JSONPARSER]] }} encoding=utf-8, jsonml=true SET [[Options#CREATEFILTER|CREATEFILTER]] *.json ===XQuery=== The [[JSON Module]] provides functions for converting JSON objects to XML documents. ==CSV Parser== BaseX can be used to import CSV documents. Different alternatives how to proceed are shown in the following: ===GUI=== Go to Menu ''Database'' → ''New'' and select "CSV" in the input format combo box.You can set the following options for parsing CSV documents in the "Parsing" tab: * '''Encoding''': Choose the appropriate encoding of the CSV file.* '''Separator''': Choose the column separator of the CSV file. Possible: <code>comma</code>, <code>semicolon</code>, <code>tab</code> or <code>space</code> or an arbitrary character.* '''Header''': Activate this option if the incoming CSV files have a header line. ===Command Line=== Turn on the CSV Parser before parsing documents, and set some optional, parser-specific options and a file filter. Unicode code points can be specified as separators; {{CodeOption|32CREATEFILTER}} is the code point for spaces:  SET [[Options#PARSER|PARSER]] csv SET [[Options#CSVPARSER|CSVPARSER]] encoding=utf-8, lines=true, header=false, separator=space SET [[Options#CREATEFILTER|CREATEFILTER]] *.csvjson

===XQuery===

The [[CSV JSON Module]] provides a function functions for converting CSV JSON to XML documents.

Documents In addition, all functions that accept [[Options#Parsing|Parsing Options]] can also be converted by specifying the parser in the query prologused as well to convert JSON.The following example query adds all CSV files that are located in converts a JSON string to an XML document, using thedirectory conversion format {{Code|2Bimported}} to the database {{Code[[JSON Module#Basic|DBbasic]]}} and interpretsthe first lines as column headers:

<pre classlang="brush:'xquery"'>declare option db:parser "csv";declare option create('json-db:csvparser ', '{ "header=yeskey";for $file in file:list("2Bimportedvalue"}', false()'example.json', "*.csv")map {return db 'parser':add('DBjson', $file 'jsonparser': map { 'format': 'basic' }})

==Text CSV Parser==

Plain text BaseX can be imported as well:used to import CSV documents. The CSV parsing options are described in the documentation of the [[CSV Module]].

===GUI===

Go to Menu ''Database'' → ''New'' and select "TEXTCSV" in the input format comboboxcombo box.You can set the following option options for parsing text CSV documents in the "Parsing" tab:.

Turn on the CSV Parser before parsing documents and set some optional SET {{Option|PARSER}} csv SET {{Option|CSVPARSER}} encoding=utf-8, parser-specific options and a file filter:header=true,separator=semicolon SET {{Option|CREATEFILTER}} *.csv

SET [[Options#PARSER|PARSER]] text SET [[Options#TEXTPARSER|TEXTPARSER]] lines=yes SET [[Options#CREATEFILTER|CREATEFILTER]] *=XQuery==

Similar In addition, all functions that accept [[Options#Parsing|Parsing Options]] can be used as well to convert CSV. In the other formats the text parser can be specified in following, all CSV files from a directory are added to a database, with the prolog first line of anXQuery expressioneach file interpreted as column header:

<pre classlang="brush:'xquery"'>declare option db:parser "text";for $file in file:list("2Bimported", truefalse(), "*.txtcsv")return db:add("csv-db", $file, "", map { 'parser': 'DBcsv', $file 'csvparser': map { 'header': true() }})