Changes

Jump to navigation Jump to search
1,276 bytes added ,  18:39, 1 December 2023
m
Text replacement - "syntaxhighlight" to "pre"
This [[Module Library|XQuery Module]] contains a single function to parse CSV input. [httphttps://en.wikipedia.org/wiki/Comma-separated_values CSV] (comma-separated values) is a popular representation for tabular data, exported e. g. from Excel.
=Conventions=
All functions and errors in this module are assigned to the <code><nowiki>http://basex.org/modules/csv</nowiki></code> namespace, which is statically bound to the {{Code|csv}} prefix.<br/>All errors are assigned to the <code><nowiki>http://basex.org/errors</nowiki></code> namespace, which is statically bound to the {{Code|bxerr}} prefix.
==ConversionFormats==
===XML: Direct, Attributes===
If the {{Code|direct}} or {{Code|attributes}} format is chosen, a CSV string is converted to XML as follows:
* The resulting XML document has a {{Code|<csv>}} root element.* Rows are represented via {{Code|<record>}} elements.* Fields are represented via {{Code|<entry>}} elements. The value of a field is represented as text node.* If the {{Code|header}} option is set to {{Code|true}}, the first text line is parsed as table header, and the {{Code|<entry>}} elements are replaced with the field names:** Empty names are represented by a single underscore ({{Code|_}}), and characters that are not valid in element names are replaced with underscores or (when invalid as first character of an element name) prefixed with an underscore.** If the {{Code|laxformat}} option is set to {{Code|falsedirect}}, invalid characters will be rewritten to an underscore and the character’s four-digit Unicodefield names are encoded, as described in the [[Conversion Module#Keys|Conversion Module]], and underscores will be represented used as two underscores ({{Code|__}}). The resulting element names may be less readable, but can always be converted back to the original field names.* If * Otherwise, if {{Code|format}} is set to {{Code|attributes}}, the field names will be stored in name attributes.
'''A little advice''': in the Database Creation dialog of the GUI, if you select CSV Parsing and switch to the ''Parsing'' tab, you can see the effects of some of the conversion options.
===XQuery===
With the {{MarkCode|Introduced with Version 9.0xquery}}format, CSV records are converted to a sequence of arrays:
CSV * The resulting value will be a map with a {{Code|records can be converted to }} entry and an optional {{Code|names}} entry.* Records are organized as a plain sequence of arrays:. An array contains the entries of a single record.* The {{Code|names}} entry contains an array with the column names. It is generated if the {{Code|header}} option is set to {{Code|true}}.
* The resulting value will be a CSV map with a {{Code|records}} and an optional {{Code|names}} key.* Records are organized as a sequence of arrays. A single array contains the entries of a single record.* The column names will be available if {{Code|header}} option is set to {{Code|true}}. The resulting data can e.g. be accessed as follows:
* <code>$csv?records[5]</code> returns all entries of the 5th record (row)
* <code>$csv?records?(2)</code> returns all entries of the 2nd field (column)
* <code>$csv?names?*</code> returns the names of all fields (if available)
* Return enumerated strings for all records:
<pre classlang="brush:'xquery"'>
for $record at $pos in $csv?records
return $pos || ". " || string-join($record?*, ', ')
</pre>
The resulting representation consumes less memory than XML-based formats, and values can be directly accessed without conversion. Thus, it is recommendable for very large inputs and for efficient ad-hoc processing. Before {{Version|9.0}}, a {{Code|map}} format was available, which was now replaced with the more flexible and light-weight {{Code|xquery}} format.
==Options==
In the following table, all available options are listed. The Excel column indicates what are the preferred lists recommended options for data that is to be imported, processed with Excel or has been exported from ExcelOpen/Libre Office.
{| class="wikitable sortable" width="100%"
|- valign="top"
! width="140" | Option! width="50%" | Description
! Allowed
! Default
! Direction
! Excel
|- valign="top"
| {{Code|comma}}, {{Code|semicolon}}, {{Code|colon}}, {{Code|tab}}, {{Code|space}} or a ''single character''
| {{Code|comma}}
| ''parse'', ''serialize''| {{Code|semicolon}}or {{Code|comma}}, depending on the region
|- valign="top"
| {{Code|header}}
| {{Code|yes}}, {{Code|no}}
| {{Code|no}}
| ''parse'', ''serialize''
|
|- valign="top"
| {{Code|format}}
| Specifies the format of the XML for converting CSV data:<br/>* With {{Code|direct}} conversion, field names are represented as element names* With {{Code|attributes}} conversion, field names are stored in {{Code([[#Conversion Formats|name}} attributes* With {{Code|xquery}} conversion, the input is converted to an XQuery mapsee above]]).| {{Code|direct}}, {{Code|attributes}}, {{Code|mapxquery}}
| {{Code|direct}}
| ''parse'', ''serialize''
|
|- valign="top"
| {{Code|lax}}
| Specifies if a [[Conversion Module#Keys|lax approach is conversion rules]] are used to convert QNames to JSON names.
| {{Code|yes}}, {{Code|no}}
| {{Code|yes}}
| ''parse'', ''serialize''
| {{Code|no}}
|- valign="top"
| Specifies how quotes are parsed:
* Parsing: If the option is enabled, quotes at the start and end of a value will be treated as control characters. Separators and newlines within the quotes will be adopted without change.
* Serialization: If the option is enabled, the value will be wrapped with quotesif it contains characters that might be treated as control characters. A quote character in the value will be encoded according to the rules of the {{Code|backslashes}} option.
| {{Code|yes}}, {{Code|no}}
| {{Code|yes}}
| ''parse'', ''serialize''
| {{Code|yes}}
|- valign="top"
| {{Code|yes}}, {{Code|no}}
| {{Code|no}}
| ''parse'', ''serialize''
| {{Code|no}}
|- valign="top"
| {{Code|skip-empty}}
| {{Announce|Version 11:}} Indicates if empty fields are included in the result. Only gets effective for the formats {{Code|direct}} or {{Code|attribute}}, and if the {{Code|header}} option is enabled. Please note that if this option is used and the data is serialized again, the resulting CSV header may be incomplete.
| {{Code|yes}}, {{Code|no}}
| {{Code|no}}
| ''parse''
| {{Code|no}}
|- valign="top"
| {{Code|allow}}
| In Excel, a value will be evaluated if it starts with the character {{Code|-}}, {{Code|+}}, <code>=</code>, {{Code|@}}, {{Code|\t}} or {{Code|\r}}. A regular expression can be specified to reject data that will be handled differently than expected by an application, or that may be malicious (see https://owasp.org/www-community/attacks/CSV_Injection for more details).
| ''string''
|
| ''serialize''
| <code><nowiki>[^-+=\t\r].*|[-+]\d*([,.]\d+)?</nowiki></code>
|}
=Functions=
 
==csv:doc==
 
{| width='100%'
|- valign="top"
| width='120' | '''Signature'''
|<pre>csv:doc(
$href as xs:string?,
$options as map(*)? := map { }
) as item()?</pre>
|- valign="top"
| '''Summary'''
|Fetches the CSV document referred to by the given {{Code|$href}} and converts it to an XQuery value. The {{Code|$options}} argument can be used to control the way the input is converted.
|- valign="top"
| '''Errors'''
|{{Error|parse|#Errors}} the specified input cannot be parsed as CSV document.<br/>{{Error|options|#Errors}} the specified options are conflicting.
|}
==csv:parse==
{| width='100%'
|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|csv:parse|$input as xs:string|document-node(element(csv))}}<br/pre>{{Func|csv:parse|( $input value as xs:string?, $options as map(xs*)? :string, item(= map { }))|as item()}}?</pre>|-valign="top"
| '''Summary'''
|Converts the CSV data specified by {{Code|$inputvalue}} to an XML document or a mapXQuery value. The {{Code|$options}} argument can be used to control the way the input is converted.|-valign="top"
| '''Errors'''
|{{Error|BXCS0001parse|#Errors}} the specified input cannot be parsedas CSV document.
|}
{| width='100%'
|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|csv:serialize|$input as node()|xs:string}}<br/pre>{{Func|csv:serialize|( $input as nodeitem()?, $options as map(xs*)? :string, item(= map { }))|as xs:string}}</pre>|-valign="top"
| '''Summary'''
|Serializes the node specified by {{Code|$input}} as CSV data, and returns using the result as specified {{Code|xs:string$options}}, and returns the result as string. Items Values can also be serialized as JSON if CSV with the standard [[Serialization|Serialization Parameter]] feature of XQuery:* The parameter {{Code|method}} is needs to be set to {{Code|csv}}.<br/>The , and* the options presented in this article need to be assigned to the {{Code|$optionscsv}} argument can be used to control the way the input is serializedparameter.|-valign="top"
| '''Errors'''
|{{Error|BXCS0002serialize|#Errors}} the input cannot be serialized.
|}
'''Input''' {{Code|addressbook.csv}}:
<pre classlang="brush:xml">
Name,First Name,Address,City
Huber,Sepp,Hauptstraße 13,93547 Hintertupfing
'''Query:'''
<pre classlang="brush:'xquery"'>
let $text := file:read-text('addressbook.csv')
return csv:parse($text, map { 'header': true() })
'''Result:'''
<pre classlang="brush:xml">
<csv>
<record>
'''Query:'''
<pre classlang="brush:'xquery"'>
let $options := map { 'lax': false() }
let $input := file:read-text('some-data.csv')
'''Query:'''
<pre classlang="brush:'xquery"'>
let $text := ``[Name,City
Jack,Chicago
]``
let $options := map { 'format': 'xquery', 'header': true() }
let $data csv := csv:parse($text, $options)
return (
'Distinct values:',
let $records := $data?csv('records') for $name at $pos in $data?csv('names')?*
let $values := $records?($pos)
return (
'''Result:'''
<pre classlang="brush:'xquery"'>
Distinct values:
* Name: Jack, John
! width="110"|Code
|Description
|-valign="top"|{{Code|BXCS0001parse}}
| The input cannot be parsed.
|-valign="top"|{{Code|BXCS0002serialize}}
| The node cannot be serialized.
|}
=Changelog=
 
;Version 11
* Added: [[#Options|Options]]: <code>skip-empty</code> option.
 
;Version 9.7
* Added: [[#Options|Options]]: <code>allow</code> option.
 
;Version 9.4
* Added: {{Function||csv:doc}}
 
; Version 9.1
* Updated: {{Function||csv:parse}} can be called with empty sequence.
;Version 9.0
* Added: {{Code|xquery}} option
* Removed: {{Code|map}} option
* Updated: error codes updated; errors now use the module namespace
;Version 8.6
;Version 7.8
* Updated: [[#csv:parse{{Function||csv:parse]] }} now returns a document node instead of an element, or an XQuery map if {{Code|format}} is set to {{Code|map}}.
* Added: {{Code|format}} and {{Code|lax}} options
The module was introduced with Version 7.7.2.
Bureaucrats, editor, reviewer, Administrators
13,554

edits

Navigation menu