Changes

This [[Module Library|XQuery Module]] extends the [http://www.w3.org/TR/xpath[Full-full-text-10 W3C Full Text Recommendation] with some useful functions] features of BaseX: The index can be directly accessed, full-text results can be marked with additional elements, or the relevant parts can be extracted. Moreover, the score value, which is generated by the <code>{{Code|contains text</code> }} expression, can be explicitly requested from items.

All functions and errors in this module are assigned to the <code><nowiki>http://basex.org/modules/ft</nowiki></code> namespace, which is statically bound to the <code>{{Code|ft</code> }} prefix.<br/>All errors are assigned to the <code>http://basex.org/errors</code> namespace, which is statically bound to the <code>bxerr</code> prefix.

=Database Functions=

{{Mark|Updated with Version 7.2.2:}} second argument generalized, third parameter added. {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|<code><bpre>ft:search</b>( $db as item(), $terms as item()*) as text()*</code><br/><code><b>ftxs:search</b>($db as item()string, $terms as item()*, $options as itemmap(*)? := map { }) as text()*</codepre>|-| valign='"top' "| '''Summary'''|Returns all text nodes from the full-text index of the [[Database Module#Database Nodesdatabase {{Code|database node]] <code>$db</code> }} that contain the specified {{MonoCode|$terms}}.<br/>The options used for tokenizing the input and building the full-text will also be applied to the search terms. As an example, if the index terms have been stemmed, the search string will be stemmed as well.The {{MonoCode|$options}} argument can be used to overwrite control full-text processing. The following options are supported (the introduction on [[Full-Text]] processing gives you equivalent expressions in the XQuery Full-Text notation):* {{Code|mode}}: determine the mode how tokens are searched. Allowed values are {{Code|any}}, {{Code|any word}}, {{Code|all}}, {{Code|all words}}, and {{Code|phrase}}. {{Code|any}} is the default fullsearch mode.* {{Code|wildcards}}: turn wildcard querying on or off. Allowed values are {{Code|true}} and {{Code|false}}. By default, wildcard querying is turned off.* {{Code|fuzzy}}: turn fuzzy querying on or off. Allowed values are {{Code|true}} and {{Code|false}}. By default, fuzzy querying is turned off.* {{Code|errors}}: control the maximum number of tolerated errors for fuzzy querying. By default, {{Code|0}} is assigned (see [[Full-text Text#Fuzzy_Querying|Fuzzy Querying]] for more details).* {{Code|ordered}}: indicate if all tokens must occur in the order in which they are specified. Allowed values are {{Code|true}} and {{Code|false}}. The default is {{Code|false}}.* {{Code|content}}: specify that the matched tokens need to occur at the beginning or end of a searched string, or need to cover the entire string. Allowed values are {{Code|start}}, {{Code|end}}, and {{Code|entire}}. By default, the option is turned off.* {{Code|scope}}: define the scope in which tokens must be located. The option has following sub options:** {{Code|same}}: can be set to {{Code|true}} or {{Code|false}}. It specifies if tokens need to occur in the same or different units.** {{Code|unit}}: can be specified as{{Code|sentence}} or {{Code|paragraph}}. It specifies the unit for finding tokens.* {{MonoCode|element(options)window}}: <code>&lt;options/&gt;</code> set up a window in which all tokens must be used as root elementlocated. By default, and the parameters are specified as child nodesoption is turned off. It has following sub options:** {{Code|size}}: specify the size of the window in terms of ''units''.** {{Code|unit}}: can be {{Code|sentences}}, with {{Code|sentences}} or {{Code|paragraphs}}. The default is {{Code|words}}.* {{Code|distance}}: specify the element name representing distance in which tokens must occur. By default, the key and option is turned off. It has following sub options:** {{Code|min}}: specify the text node representing minimum distance in terms of ''units''. The default is {{Code|0}}.** {{Code|max}}: specify the valuemaximum distance in terms of ''units''. The default is {{Code|∞}}.** {{Code|unit}}:can be {{Code|words}}, {{Code|sentences}} or {{Code|paragraphs}}. The default is {{Code|words}}.|- valign="top"| '''Errors'''|{{Error|db:get|Database Module#Errors}} The addressed database does not exist or could not be opened.<br />{{Error|db:no-index|Database Module#Errors}} the index is not available.<br/>{{Error|options|#Errors}} the fuzzy and wildcard option cannot be both specified.<pre class|- valign="brushtop"| '''Examples'''|* {{Code|ft:xmlsearch(">DB", "QUERY")}}: Return all text nodes of the database {{Code|DB}} that contain the term {{Code|QUERY}}.* Return all text nodes of the database {{Code|DB}} that contain the numbers {{Code|2010}} and {{Code|2020}}:<optionsbr/> <keycode>valueft:search("DB", ("2010", "2020"), map { 'mode': 'all' })</keycode> ...* Return text nodes that contain the terms {{Code|A}} and {{Code|B|}} in a distance of at most 5 words:</optionspre lang='xquery'>ft:search("db", ("A", "B"), map { "mode": "all words", "distance": map { "max": "5", "unit": "words" }})

* [[Map Module|map structure]]: all parameters can be directly represented as key/value pairs:<br /><code>map { "key" := "value", ... </code>}<br/>This variant is more compact, but please note that the W3C’s specification of maps in XQuery is still work in progress.The following keys are supported:* {{Mono|mode}}: determines the search mode (also called [http://www.w3.org/TR/xpath-full-text-10/#ftwords AnyAllOption]). Allowed values are {{Mono|any}}, {{Mono|any word}}, {{Mono|all}}, {{Mono|all words}}, and {{Mono|phrase}}. {{Mono|any}} is the default search mode.* {{Mono|fuzzy}}: turns fuzzy querying on or off. Allowed values are an empty string or {{Mono|true}}, or {{Mono|false}}. By default, fuzzy querying is turned off.* {{Mono|wildcards}}: turns wildcard querying on or off. Allowed values are an empty string or {{Mono|true}}, or {{Mono|false}}. By default, wildcard querying is turned off.|-| valign='top' | '''Errors'''|'''[[Database Module#Errors|BXDB0004]]''' is raised if the full-text index is not available.<br/>'''[[XQuery Errors#BaseX Errors|BASX0022]]''' is raised if both fuzzy and wildcard querying has been selected.|-| valign='top' | '''Examples'''|* <code>ft:search("DB", "QUERY")</code> returns all text nodes of the database {{Mono|DB}} that contain the term {{Mono|QUERY}}.* <code>ft:search("DB", (2010,2011), map { 'mode':='all' })</code><br/>returns all text nodes of the database {{Mono|DB}} that contain the numbers {{Mono|2010}} and {{Mono|20111}}.* The last example iterates Iterate over five three databases and returns return all elements containing terms similar to {{MonoCode|Hello World}} in the text nodes:<pre classlang="brush:'xquery"'>

return ft:search($dbname, $terms, map { 'fuzzy': $optionsfuzzy })/..

==ft:marktokens== {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|<code><bpre>ft:mark</b>tokens( $nodes db as node()*) as node()*</code><br /><code><b>ftxs:mark</b>($nodes as node()*string, $tag prefix as xs:string := ()) as nodeelement(value)*</codepre>|-| valign='"top' "| '''Summary'''|Puts a marker element around the resulting <code>$nodes</code> of a Returns all full-text tokens stored in the index request.<br />The default tag name of the marker element is <code>mark</code>. An alternative tag name can be chosen via the optional <code>database {{Code|$tag</code> argumentdb}}, along with their numbers of occurrences.<br />Note that If {{Code|$prefix}} is specified, the XML node returned nodes will be refined to be transformed must be an internal "database" nodethe strings starting with that prefix. The <code>transform</code> expression can prefix will be used tokenized according to apply the method to a mainfull-memory fragment (see example)text used for creating the index.|-| valign='"top' "| '''ExamplesErrors'''|* {{Error|db:get|Database Module#Errors}} The following query returns <code>&lt;XML&gt;&lt;mark&gt;hello&lt;/mark&gt; world&lt;/XML&gt;</code>, if one text node of the addressed database does not exist or could not be opened.<code>DB<br/code> has {{Error|db:no-index|Database Module#Errors}} the value full-text index is not available.|- valign="hello worldtop"| '''Examples'''|Returns the number of occurrences for a single, specific index entry:<pre classlang="brush:'xquery"'>let $term := ft:marktokenize(db$term)return number(ft:opentokens('DBdb', $term)//*[text() contains text 'hello'. = $term]/@count)

=General Functions= ==ft:extractcontains== {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|<code><bpre>ft:extract</b>contains( $nodes input as nodeitem()*) as node()*</code><br /><code><b>ft:extract</b>(, $nodes terms as nodeitem()*, $tag options as xs:string) as nodemap(*)*</code><br /><code><b>ft? :extract</b>($nodes as node(= map { })*, $tag as xs:string, $length as xs:integer) as node()*boolean</codepre>|-| valign='"top' "| '''Summary'''|Extracts and returns relevant parts of full-text results. It puts a marker element around Checks if the resulting <code>specified {{Code|$nodes</code> of a full-text index request and chops irrelevant sections of input}} items contain the resultspecified {{Code|$terms}}.<br />The default tag name of function does the same as the marker element is <code>mark</code>. An alternative tag name [[Full-Text]] expression {{Code|contains text}}, but options can be chosen via specified more dynamically. The {{Code|$options}} are the same as for {{Function||ft:search}}, and the optional <code>$tag</code> argumentfollowing ones exist:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|lower}}.<br />The By default length of the returned text , search is <code>150</code> case-insensitive.* {{Code|diacritics}}: determines how diacritical charactersare processed. Allowed values are {{Code|insensitive}} and {{Code|sensitive}}. By default, search is diacritical insensitive.* {{Code|stemming}}: determines is tokens are stemmed. Allowed values are {{Code|true}} and {{Code|false}}. An alternative length can be specified via By default, stemming is turned off.* {{Code|language}}: determines the optional <code>$length</code> argumentlanguage. This option is relevant for stemming tokens. All language codes are supported. Note that the effective text length may differ from the specified text due to formatting and readibility issuesThe default language is {{Code|en}}.|-valign="top"| '''Errors'''|{{Error|options|#Errors}} specified options are conflicting.|- valign='"top' "| '''Examples'''

* The following query may return <code>&lt;XML&gt;...&lt;b&gt;hello&lt;/b&gt;...&lt;XML&gt;</code> Checks if a text node of {{Code|jack}} or {{Code|john}} occurs in the database input string {{Code|John Doe}}:<codepre lang='xquery'>DBft:contains("John Doe", ("jack", "john"), map { "mode": "any" })</codepre> contains * Calls the string "hello world"function with stemming turned on and off:<pre classlang="brush:'xquery"'>ft:extract(db:opentrue('DB')//*[text, false() ) ! ft:contains text ("Häuser", "Haus", map { 'hellostemming']: ., 'blanguage', 1:'de' })

 {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|<code><bpre>ft:count</b>( $nodes as node()*) as xs:integer</codepre>|-| valign='"top' "| '''Summary'''

|-| valign='"top' "| '''Examples'''

* <code>{{Code|ft:count(//*[text() contains text 'QUERY'])</code> }} returns the <code>{{Code|xs:integer</code> }} value <code>{{Code|2</code> }} if a document contains two occurrences of the string "QUERY".

 {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|<code><bpre>ft:score</b>( $item as item()*) as xs:double*</codepre>|-| valign='"top' "| '''Summary'''|Returns the score values (0.0 - 1.0) that have been attached to the specified items. <code>{{Code|0</code> }} is returned a value if no score was attached.|-| valign='"top' "| '''Examples'''

* <code>{{Code|ft:score('a' contains text 'a')</code> }} returns the <code>{{Code|xs:double</code> }} value <code>{{Code|1</code>}}.

==ft:tokenstokenize== {|width='100%'|-valign="top"| width='120' | '''Signature'''|<pre>ft:tokenize( $string as xs:string?, $options as map(*)? := map { }) as xs:string*</pre>| - valign="top"| 'top' width='90Summary''' |Tokenizes the given {{Code|$string}}, using the current default full-text options or the {{Code|$options}} specified as second argument, and returns a sequence with the tokenized string. The following options are available:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|lower}}. By default, search is case insensitive.* {{Code|diacritics}}: determines how diacritical characters are processed. Allowed values are {{Code|insensitive}} and {{Code|sensitive}}. By default, search is diacritical insensitive.* {{Code|stemming}}: determines is tokens are stemmed. Allowed values are {{Code|true}} and {{Code|false}}. By default, stemming is turned off.* {{Code|language}}: determines the language. This option is relevant for stemming tokens. All language codes are supported. The default language is {{Code|en}}.The {{Code|$options}} argument can be used to control full-text processing.|- valign="top"| '''SignaturesExamples'''|* <code>ft:tokenize("No Doubt")</code> returns the two strings {{MonoCode|no}} and {{Code|doubt}}.* <code>ft:tokenize("École", map { 'diacritics': 'sensitive' })</code> returns the string {{Code|école}}.* <bcode>declare ft-option using stemming; ft:tokenstokenize("GIFTS")</bcode> returns a single string {{Code|gift}}.|} ==ft:normalize== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>ft:normalize( $string as xs:string?, $db options as itemmap(*)? := map { }) as element(value)*}}xs:string<br/pre>|- valign="top"| '''Summary'''|Normalizes the given {{Code|$string}}, using the current default full-text options or the {{MonoCode|$options}} specified as second argument. The function accepts the same arguments as {{Function||ft:tokenize}}; special characters and separators will be preserved.|- valign="top"| '''Examples'''|* <bcode>ft:tokensnormalize("Häuser am Meer", map { 'case': 'sensitive' })</bcode> returns the string {{Code|Hauser am Meer}}.|} ==ft:thesaurus== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>ft:thesaurus( $db node as itemnode(), $prefix term as xs:string) , $options as elementmap(value*)? := map { })as xs:string*}}</pre>|-| valign='"top' "| '''Summary'''|Returns all full-text tokens stored Looks up a {{Code|$term}} in the index of the a [[Database ModuleFull-Text#Database NodesThesaurus|database nodeThesaurus Structure]] supplied by {{Code|$node}}. The following {{Code|$options}} exist:* {{Code|relationship}}: determines the relationship between terms* {{Code|levels}}: determines the maximum number of levels to traverse|- valign="top"| '''Examples'''| Returns {{Code|happy}} and {{Code|lucky}}:<pre lang='xquery'>ft:thesaurus( <thesaurus> <entry> <term>happy</term> <synonym> <term>lucky</term> <relationship>RT</relationship> </synonym> </entry> </thesaurus>, 'happy')<code/pre>|} =Highlighting Functions= ==ft:mark== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>ft:mark( $nodes as node()*, $dbname as xs:string := ()) as node()*</codepre>, along with their numbers |- valign="top"| '''Summary'''|Puts a marker element around the resulting {{Code|$nodes}} of occurrencesa full-text request.<br/>If The default name of the marker element is {{MonoCode|mark}}. An alternative name can be chosen via the optional {{Code|$prefixname}} is specified, argument.<br/>Please note that:* The full-text expression that computes the returned nodes will token positions must be refined to specified as argument of the strings starting with that prefix<code>ft:mark()</code> function, as all position information is lost in subsequent processing steps. The prefix will be tokenized according You may need to the specify more than one full-text used for creating expression if you want to use the indexfunction in a FLWOR expression, as shown in Example 2.* The supplied node must be a [[Database Module#Database Node|Database Node]]. As shown in Example 3, {{Code|-update}} or {{Code|transform}} can be utilized to convert a fragment to the required internal representation.| - valign="top"| '''Examples''top' | '''ErrorsExample 1''': The following query returns {{Code|&lt;XML&gt;&lt;mark&gt;hello&lt;/mark&gt; world&lt;/XML&gt;}}, if one text node of the database {{Code|DB}} has the value "hello world":|<pre lang='xquery'>ft:mark(db:get('DB')//*[[Database Module#Errors|BXDB0004]text() contains text 'hello'])</pre>'''Example 2''' is raised if : The following expression loops through the first ten full-text index is not availableresults and marks the results in a second expression:<pre lang='xquery'>let $start := 1let $end := 10let $term := 'welcome'let $test := function($node) { $node/text() contains text { $term } }for $ft in (db:get('DB')//*[$test(.)])[position() = $start to $end]return ft:mark($ft[$test(.)])</pre>'''Example 3''': The following expression returns <code>&lt;xml>hello &lt;b&gt;word&lt;/b&gt;&lt;/xml&gt;</code>:<pre lang='xquery'>copy $p := <xml>hello world</xml>modify ()return ft:mark($p[text() contains text 'word'], 'b')</pre>

==ft:tokenizeextract== {|width='100%'|-| valign='"top' "| width='90120' | '''SignaturesSignature'''|{{Mono|<bpre>ft:tokenize</b>extract( $nodes as node()*, $input name as xs:string := () , $length as xs:stringinteger := ()) as node()*}}</pre>|-| valign='"top' "| '''Summary'''|Tokenizes Extracts and returns relevant parts of full-text results. It puts a marker element around the given resulting {{MonoCode|$inputnodes}} of a full-text index request and chops irrelevant sections of the result.<br/>The default element name of the marker element is {{Code|mark}} string, using . An alternative element name can be chosen via the current optional {{Code|$name}} argument.<br/>The default full-length of the returned text is {{Code|150}} characters. An alternative length can be specified via the optional {{Code|$length}} argument. Note that the effective text length may differ from the specified text optionsdue to formatting and readibility issues.<br/>For more details on this function, please have a look at {{Function||ft:mark}}.|-| valign='"top' "| '''Examples'''

* <code>ft:tokenize("No Doubt")</code> returns the two strings The following query may return {{MonoCode|no&lt;XML&gt;...&lt;b&gt;hello&lt;/b&gt;...&lt;XML&gt;}} and if a text node of the database {{MonoCode|doubtDB}}.contains the string "hello world":* <codepre lang='xquery'>declare ft-option using stemming; ft:tokenizeextract(db:get('DB')//*[text("GIFTS")contains text 'hello'], 'b', 1)</codepre> returns a single string |} =Errors= {| class="wikitable" width="100%"! width="110"|Code|Description|- valign="top"|{{MonoCode|giftoptions}}|Both wildcards and fuzzy search have been specified as search options.

===; Version 79.2===6* Added: {{Function||ft:thesaurus}}* Updated: {{Function||ft:search}}, {{Function||ft:contains}}: new {{Code|errors}} option. ; Version 9.1* Updated: {{Function||ft:tokenize}} and {{Function||ft:normalize}} can be called with empty sequence. ;Version 9.0* Updated: error codes updated; errors now use the module namespace ;Version 8.0* Added: {{Function||ft:contains}}, {{Function||ft:normalize}}* Updated: Options added to {{Function||ft:tokenize}}

;Version 7.8* UpdatedAdded: [[#{{Function||ft:searchcontains}}* Updated: Options added to {{Function||ft:search]] (second argument generalized, third parameter added)}}

===;Version 7.1===7* Updated: the functions no longer accept [[Database Module#Database Nodes|Database Nodes]] as reference. Instead, the name of a database must now be specified.

;Version 7.2* AddedUpdated: [[#ft:tokens{{Function||ft:tokens]]search}} (second argument generalized, [[#ft:tokenize|ft:tokenize]]third parameter added)

[[Category;Version 7.1* Added:XQuery]]{{Function||ft:tokens}}, {{Function||ft:tokenize}}