Changes

This [[Module Library|XQuery Module]] extends the [http://www.w3.org/TR/xpath[Full-full-text-10 W3C Full Text Recommendation] with some useful functions] features of BaseX: The index can be directly accessed, full-text results can be marked with additional elements, or the relevant parts can be extracted. Moreover, the score value, which is generated by the <code>{{Code|contains text</code> }} expression, can be explicitly requested from items. All functions are introduced with the <code>ft:</code> prefix, which is linked to the statically declared <code>http://basex.org/modules/ft</code> namespace.

=Conventions= All functions and errors in this module are assigned to the <code><nowiki>http://basex.org/modules/ft</nowiki></code> namespace, which is statically bound to the {{Code|ft}} prefix.<br/> =Database Functions=

{{Mark|Updated with Version 7.2.2:}} second argument generalized, third parameter added. {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|<code><bpre>ft:search</b>( $db as item(), $terms as item()*) as text()*</code><br/><code><b>ftxs:search</b>($db as item()string, $terms as item()*, $options as itemmap(*)? := map { }) as text()*</codepre>|-| valign='"top' "| '''Summary'''|Returns all text nodes from the full-text index of the database <code>[[Database Module#Database Argument{{Code|$db]]</code> }} that contain the specified {{MonoCode|$terms}}.<br/>The options used for tokenizing the input and building the full-text will also be applied to the search terms. As an example, if the index terms have been stemmed, the search string will be stemmed as well.The {{MonoCode|$options}} argument can be used to overwrite control full-text processing. The following options are supported (the introduction on [[Full-Text]] processing gives you equivalent expressions in the XQuery Full-Text notation):* {{Code|mode}}: determine the mode how tokens are searched. Allowed values are {{Code|any}}, {{Code|any word}}, {{Code|all}}, {{Code|all words}}, and {{Code|phrase}}. {{Code|any}} is the default search mode.* {{Code|wildcards}}: turn wildcard querying on or off. Allowed values are {{Code|true}} and {{Code|false}}. By default, wildcard querying is turned off.* {{Code|fuzzy}}: turn fuzzy querying on or off. Allowed values are {{Code|true}} and {{Code|false}}. By default, fuzzy querying is turned off.* {{Code|errors}}: control the maximum number of tolerated errors for fuzzy querying. By default full, {{Code|0}} is assigned (see [[Full-text Text#Fuzzy_Querying|Fuzzy Querying]] for more details).* {{Code|ordered}}: indicate if all tokens must occur in the order in which they are specified. Allowed values are {{Code|true}} and {{Code|false}}. The default is {{Code|false}}.* {{Code|content}}: specify that the matched tokens need to occur at the beginning or end of a searched string, or need to cover the entire string. Allowed values are {{Code|start}}, {{Code|end}}, and {{Code|entire}}. By default, the option is turned off.* {{Code|scope}}: define the scope in which tokens must be located. The option has following sub options:** {{Code|same}}: can be set to {{Code|true}} or {{Code|false}}. It specifies if tokens need to occur in the same or different units.** {{Code|unit}}: can be specified as{{Code|sentence}} or {{Code|paragraph}}. It specifies the unit for finding tokens.* {{MonoCode|element(options)window}}: <code>&lt;options/&gt;</code> set up a window in which all tokens must be used as root elementlocated. By default, and the parameters are specified as child nodesoption is turned off. It has following sub options:** {{Code|size}}: specify the size of the window in terms of ''units''.** {{Code|unit}}: can be {{Code|sentences}}, with {{Code|sentences}} or {{Code|paragraphs}}. The default is {{Code|words}}.* {{Code|distance}}: specify the element name representing distance in which tokens must occur. By default, the key and option is turned off. It has following sub options:** {{Code|min}}: specify the text node representing minimum distance in terms of ''units''. The default is {{Code|0}}.** {{Code|max}}: specify the valuemaximum distance in terms of ''units''. The default is {{Code|∞}}.** {{Code|unit}}:can be {{Code|words}}, {{Code|sentences}} or {{Code|paragraphs}}. The default is {{Code|words}}.|- valign="top"| '''Errors'''|{{Error|db:get|Database Module#Errors}} The addressed database does not exist or could not be opened.<br/>{{Error|db:no-index|Database Module#Errors}} the index is not available.<br />{{Error|options|#Errors}} the fuzzy and wildcard option cannot be both specified.<pre class|- valign="brushtop"| '''Examples'''|* {{Code|ft:xmlsearch(">DB", "QUERY")}}: Return all text nodes of the database {{Code|DB}} that contain the term {{Code|QUERY}}.* Return all text nodes of the database {{Code|DB}} that contain the numbers {{Code|2010}} and {{Code|2020}}:<optionsbr/> <keycode>valueft:search("DB", ("2010", "2020"), map { 'mode': 'all' })</keycode> ...* Return text nodes that contain the terms {{Code|A}} and {{Code|B|}} in a distance of at most 5 words:</optionspre lang='xquery'>ft:search("db", ("A", "B"), map { "mode": "all words", "distance": map { "max": "5", "unit": "words" }})

* [[Map Module|map structure]]: all parameters can be directly represented as key/value pairs:<br /><code>map { "key" := "value", ... </code>}<br/>This variant is more compact, but please note that the W3C’s specification of maps in XQuery is still work in progress.The following keys are supported:* {{Mono|mode}}: determines the search mode (also called [http://www.w3.org/TR/xpath-full-text-10/#ftwords AnyAllOption]). Allowed values are {{Mono|any}}, {{Mono|any word}}, {{Mono|all}}, {{Mono|all words}}, and {{Mono|phrase}}. {{Mono|any}} is the default search mode.* {{Mono|fuzzy}}: turns fuzzy querying on or off. Allowed values are an empty string or {{Mono|true}}, or {{Mono|false}}. By default, fuzzy querying is turned off.* {{Mono|wildcards}}: turns wildcard querying on or off. Allowed values are an empty string or {{Mono|true}}, or {{Mono|false}}. By default, wildcard querying is turned off.|-| valign='top' | '''Errors'''|'''[[XQuery Errors#BaseX Errors|BASX0001]]''' is raised if the full-text index is not available, or if the selected option is not supported by the existing index.<br/>'''[[XQuery Errors#BaseX Errors|BASX0002]]''' is raised if a referenced node is not stored in a database (i.e., references a main-memory XML fragment).<br/>'''[[XQuery Errors#BaseX Errors|BASX0021]]''' is raised if the specified full-text option is unknown.<br/>'''[[XQuery Errors#BaseX Errors|BASX0022]]''' is raised if both fuzzy and wildcard querying has been selected.|-| valign='top' | '''Examples'''|* <code>ft:search("DB", "QUERY")</code> returns all text nodes of the database {{Mono|DB}} that contain the term {{Mono|QUERY}}.* <code>ft:search("DB", (2010,2011), map { 'mode':='all' })</code><br/>returns all text nodes of the database {{Mono|DB}} that contain the numbers {{Mono|2010}} and {{Mono|20111}}.* The last example iterates Iterate over five three databases and returns return all elements containing terms similar to {{MonoCode|Hello World}} in the text nodes:<pre classlang="brush:'xquery"'>

return ft:search($dbname, $terms, map { 'fuzzy': $optionsfuzzy })/..

==ft:marktokens== {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|<code><bpre>ft:mark</b>tokens( $nodes db as node()*) as node()*</code><br /><code><b>ftxs:mark</b>($nodes as node()*string, $tag prefix as xs:string := ()) as nodeelement(value)*</codepre>|-| valign='"top' "| '''Summary'''|Puts a marker element around the resulting <code>$nodes</code> of a Returns all full-text tokens stored in the index request.<br />The default tag name of the marker element is <code>mark</code>. An alternative tag name can be chosen via the optional <code>database {{Code|$tag</code> argumentdb}}, along with their numbers of occurrences.<br />Note that If {{Code|$prefix}} is specified, the XML node returned nodes will be refined to be transformed must be an internal "database" nodethe strings starting with that prefix. The <code>transform</code> expression can prefix will be used tokenized according to apply the method to a mainfull-memory fragment (see example)text used for creating the index.|-| valign='"top' "| '''Errors'''|'''[[XQuery Errors{{Error|db:get|Database Module#BaseX Errors|BASX0002]]''' is raised if a referenced node is }} The addressed database does not exist or could not stored in a database (i.e., references a main-memory XML fragment)be opened.<br />'''[[XQuery Errors{{Error|db:no-index|Database Module#Functions Errors|FOCA0002]]''' }} the full-text index is raised if <code>$name</code> is no valid QNamenot available.|-| valign='"top' "| '''Examples'''|* The following query returns <code>&lt;XML&gt;&lt;mark&gt;hello&lt;/mark&gt; world&lt;/XML&gt;</code>Returns the number of occurrences for a single, if one text node of the database <code>DB</code> has the value "hello world"specific index entry:<pre classlang="brush:'xquery"'>let $term := ft:marktokenize(db$term)return number(ft:opentokens('DBdb', $term)//*[text() contains text 'hello'. = $term]/@count)

=General Functions= ==ft:extractcontains== {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|<code><bpre>ft:extract</b>contains( $nodes input as nodeitem()*) as node()*</code><br /><code><b>ft:extract</b>(, $nodes terms as nodeitem()*, $tag options as xs:string) as nodemap(*)*</code><br /><code><b>ft? :extract</b>($nodes as node(= map { })*, $tag as xs:string, $length as xs:integer) as node()*boolean</codepre>|-| valign='"top' "| '''Summary'''|Extracts and returns relevant parts of full-text results. It puts a marker element around Checks if the resulting <code>specified {{Code|$nodes</code> of a full-text index request and chops irrelevant sections of input}} items contain the resultspecified {{Code|$terms}}.<br />The default tag name of function does the same as the marker element is <code>mark</code>. An alternative tag name [[Full-Text]] expression {{Code|contains text}}, but options can be chosen via specified more dynamically. The {{Code|$options}} are the optional <code>$tag</code> argumentsame as for {{Function||ft:search}}, and the following ones exist:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|lower}}.<br />The By default length of the returned text , search is <code>150</code> case-insensitive.* {{Code|diacritics}}: determines how diacritical charactersare processed. Allowed values are {{Code|insensitive}} and {{Code|sensitive}}. An alternative length can be specified via the optional <code>$length</code> argumentBy default, search is diacritical insensitive.* {{Code|stemming}}: determines is tokens are stemmed. Note that the effective text length may differ from the specified text due to formatting Allowed values are {{Code|true}} and readibility issues{{Code|false}}. By default, stemming is turned off.* {{Code|-language}}: determines the language. This option is relevant for stemming tokens. All language codes are supported. The default language is {{Code|en}}.| - valign='"top' "| '''Errors'''|'''[[XQuery Errors#BaseX Errors{{Error|options|BASX0002]]''' is raised if a referenced node is not stored in a database (i.e., references a main-memory XML fragment).<br />'''[[XQuery Errors#Functions Errors|FOCA0002]]''' is raised if <code>$name</code> is no valid QName}} specified options are conflicting.|-| valign='"top' "| '''Examples'''

* The following query may return <code>&lt;XML&gt;...&lt;b&gt;hello&lt;/b&gt;...&lt;XML&gt;</code> Checks if a text node of {{Code|jack}} or {{Code|john}} occurs in the database input string {{Code|John Doe}}:<codepre lang='xquery'>DBft:contains("John Doe", ("jack", "john"), map { "mode": "any" })</codepre> contains * Calls the string "hello world"function with stemming turned on and off:<pre classlang="brush:'xquery"'>ft:extract(db:opentrue('DB')//*[text, false() ) ! ft:contains text ("Häuser", "Haus", map { 'hellostemming']: ., 'blanguage', 1:'de' })

 {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|<code><bpre>ft:count</b>( $nodes as node()*) as xs:integer</codepre>|-| valign='"top' "| '''Summary'''

|-| valign='"top' | '''Errors'''|'''[[XQuery Errors#BaseX Errors|BASX0002]]''' is raised if a referenced node is not stored in a database (i.e., references a main-memory XML fragment).|-"| valign='top' | '''Examples'''

* <code>{{Code|ft:count(//*[text() contains text 'QUERY'])</code> }} returns the <code>{{Code|xs:integer</code> }} value <code>{{Code|2</code> }} if a document contains two occurrences of the string "QUERY".

 {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|<code><bpre>ft:score</b>( $item as item()*) as xs:double*</codepre>|-| valign='"top' "| '''Summary'''|Returns the score values (0.0 - 1.0) that have been attached to the specified items. <code>{{Code|0</code> }} is returned a value if no score was attached.|-valign="top"| '''Examples'''|* {{Code|ft:score('a' contains text 'a')}} returns the {{Code| xs:double}} value {{Code|1}}.|} ==ft:tokenize== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>ft:tokenize( $string as xs:string?, $options as map(*)? := map { }) as xs:string*</pre>|- valign="top"| '''Summary' ''|Tokenizes the given {{Code|$string}}, using the current default full-text options or the {{Code|$options}} specified as second argument, and returns a sequence with the tokenized string. The following options are available:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|lower}}. By default, search is case insensitive.* {{Code|diacritics}}: determines how diacritical characters are processed. Allowed values are {{Code|insensitive}} and {{Code|sensitive}}. By default, search is diacritical insensitive.* {{Code|stemming}}: determines is tokens are stemmed. Allowed values are {{Code|true}} and {{Code|false}}. By default, stemming is turned off.* {{Code|language}}: determines the language. This option is relevant for stemming tokens. All language codes are supported. The default language is {{Code|en}}.The {{Code|$options}} argument can be used to control full-text processing.|- valign="top"| '''Examples'''

* <code>ft:scoretokenize('a' contains text 'a'"No Doubt")</code> returns the two strings {{Code|no}} and {{Code|doubt}}.* <code>xsft:tokenize("École", map { 'diacritics':double'sensitive' })</code> value returns the string {{Code|école}}.* <code>1declare ft-option using stemming; ft:tokenize("GIFTS")</code>returns a single string {{Code|gift}}.

==ft:tokensnormalize== {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|<pre>ft:normalize( $string as xs:string?, $options as map(*)? := map { }) as xs:string</pre>|- valign="top"| '''Summary'''|Normalizes the given {{Code|$string}}, using the current default full-text options or the {{MonoCode|$options}} specified as second argument. The function accepts the same arguments as {{Function||ft:tokenize}}; special characters and separators will be preserved.|- valign="top"|'''Examples'''|* <bcode>ft:tokensnormalize("Häuser am Meer", map { 'case': 'sensitive' })</bcode> returns the string {{Code|Hauser am Meer}}.|} ==ft:thesaurus== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>ft:thesaurus( $db node as itemnode()) , $term as xs:string, $options as elementmap(value*)? := map { })as xs:string*}}<br/pre>|- valign="top"| '''Summary'''|Looks up a {{Code|$term}} in a [[Full-Text#Thesaurus|Thesaurus Structure]] supplied by {{Code|$node}}. The following {{MonoCode|$options}} exist:* {{Code|relationship}}: determines the relationship between terms* {{Code|levels}}: determines the maximum number of levels to traverse|- valign="top"| '''Examples'''| Returns {{Code|happy}} and {{Code|lucky}}:<bpre lang='xquery'>ft:tokensthesaurus( <thesaurus> <entry> <term>happy</term> <synonym> <term>lucky</bterm> <relationship>RT</relationship> </synonym> </entry> </thesaurus>, 'happy')</pre>|} =Highlighting Functions= ==ft:mark== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>ft:mark( $db nodes as itemnode()*, $prefix name as xs:string := ()) as elementnode(value)*}}</pre>|-| valign='"top' "| '''Summary'''|Returns all Puts a marker element around the resulting {{Code|$nodes}} of a full-text tokens stored in the index of the database <code>[[Database Module#Database Argument|$db]]request.<br/code>, along with their numbers The default name of occurrencesthe marker element is {{Code|mark}}. An alternative name can be chosen via the optional {{MonoCode|$dbname}} may either argument.<br/>Please note that:* The full-text expression that computes the token positions must be an specified as argument of the <code>xsft:stringmark()</code>function, denoting as all position information is lost in subsequent processing steps. You may need to specify more than one full-text expression if you want to use the database namefunction in a FLWOR expression, or as shown in Example 2.* The supplied node must be a node stored [[Database Module#Database Node|Database Node]]. As shown in the database.<br/>If Example 3, {{Code|update}} or {{MonoCode|$prefixtransform}} is specified, the returned nodes will can be refined utilized to the strings starting with that prefix. The prefix will be tokenized according convert a fragment to the full-text used for creating the indexrequired internal representation.|-| valign='"top' "| '''ErrorsExamples'''|'''[[XQuery Errors#BaseX Errors|BASX0001]]Example 1''' is raised : The following query returns {{Code|&lt;XML&gt;&lt;mark&gt;hello&lt;/mark&gt; world&lt;/XML&gt;}}, if one text node of the database {{Code|DB}} has the full-value "hello world":<pre lang='xquery'>ft:mark(db:get('DB')//*[text() contains text index is not available.'hello'])<br/pre>'''[[XQuery Errors#BaseX Errors|BASX0002]]Example 2''': The following expression loops through the first ten full-text results and marks the results in a second expression:<pre lang='xquery'>let $start := 1let $end := 10let $term := 'welcome' is raised if let $test := function($node) {$node/text() contains text {Mono|$dbterm }} references a node that is not stored for $ft in a database (idb:get('DB')//*[$test(.e)])[position() = $start to $end]return ft:mark($ft[$test(., references a main-memory XML fragment).])<br/pre>'''Example 3''': The following expression returns <code>&lt;xml>hello &lt;b&gt;word&lt;/b&gt;&lt;/xml&gt;</code>:<pre lang='xquery'>copy $p := <xml>hello world</xml>modify ()return ft:mark($p[[XQuery Errors#BaseX Errors|BASX0003]text() contains text 'word'], 'b'' is raised if the addressed database cannot be opened.)</pre>

==ft:tokenizeextract== {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|{{Mono|<bpre>ft:tokenize</b>extract( $nodes as node()*, $input name as xs:string := () , $length as xs:stringinteger := ()) as node()*}}</pre>|-| valign='"top' "| '''Summary'''|Tokenizes Extracts and returns relevant parts of full-text results. It puts a marker element around the given resulting {{MonoCode|$inputnodes}} of a full-text index request and chops irrelevant sections of the result.<br/>The default element name of the marker element is {{Code|mark}} string, using . An alternative element name can be chosen via the current optional {{Code|$name}} argument.<br/>The default full-length of the returned text is {{Code|150}} characters. An alternative length can be specified via the optional {{Code|$length}} argument. Note that the effective text length may differ from the specified text optionsdue to formatting and readibility issues.<br/>For more details on this function, please have a look at {{Function||ft:mark}}.|-| valign='"top' "| '''Examples'''

* <code>ft:tokenize("No Doubt")</code> returns the two strings The following query may return {{MonoCode|no&lt;XML&gt;...&lt;b&gt;hello&lt;/b&gt;...&lt;XML&gt;}} and if a text node of the database {{MonoCode|doubtDB}}.contains the string "hello world":* <codepre lang='xquery'>declare ft-option using stemming; ft:tokenizeextract(db:get('DB')//*[text("GIFTS")contains text 'hello'], 'b', 1)</codepre> returns a single string |} =Errors= {| class="wikitable" width="100%"! width="110"|Code|Description|- valign="top"|{{MonoCode|giftoptions}}|Both wildcards and fuzzy search have been specified as search options.

===; Version 79.2===6* Added: {{Function||ft:thesaurus}}* Updated: {{Function||ft:search}}, {{Function||ft:contains}}: new {{Code|errors}} option. ; Version 9.1* Updated: {{Function||ft:tokenize}} and {{Function||ft:normalize}} can be called with empty sequence. ;Version 9.0* Updated: error codes updated; errors now use the module namespace ;Version 8.0* Added: {{Function||ft:contains}}, {{Function||ft:normalize}}* Updated: Options added to {{Function||ft:tokenize}}

;Version 7.8* UpdatedAdded: [[#{{Function||ft:searchcontains}}* Updated: Options added to {{Function||ft:search]] (second argument generalized, third parameter added)}}

===;Version 7.1===7* Updated: the functions no longer accept [[Database Module#Database Nodes|Database Nodes]] as reference. Instead, the name of a database must now be specified.

;Version 7.2* AddedUpdated: [[#ft:tokens{{Function||ft:tokens]]search}} (second argument generalized, [[#ft:tokenize|ft:tokenize]]third parameter added)

[[Category;Version 7.1* Added:XQuery]]{{Function||ft:tokens}}, {{Function||ft:tokenize}}