Changes

This [[Module Library|XQuery Module]] extends the [http://www.w3.org/TR/xpath[Full-full-text-10 W3C Full Text Recommendation] with some useful functions] features of BaseX: The index can be directly accessed, full-text results can be marked with additional elements, or the relevant parts can be extracted. Moreover, the score value, which is generated by the {{Code|contains text}} expression, can be explicitly requested from items.

All functions and errors in this module are assigned to the {{Code|<code><nowiki>http://basex.org/modules/ft}} </nowiki></code> namespace, which is statically bound to the {{Code|ft}} prefix.<br/>All errors are assigned to the {{Code|http://basex.org/errors}} namespace, which is statically bound to the {{Code|bxerr}} prefix.

=Database Functions=

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|ft:search|$db as xs:string, $terms as item()*|text()*}}<br/pre>{{Func|ft:search|( $db as xs:string, $terms as item()*, $options as itemmap(*)|? := map { }) as text()*}}</pre>|-valign="top"

The {{Code|$options}} argument can be used to control full-text processing. Options can be either specified<br/>* as children of an {{Code|&lt;options/&gt;}} element, e.g.:<pre class="brush:xml"><options> <key1 value='value1'/> ...</options></pre>* as map, which contains all key/value pairs:<pre class="brush:xml">{ "key1": "value1", ... }</pre>The following options are supported (the introduction on [[Full-Text]] processing gives you equivalent expressions in the XQuery Full-Text notation):* {{Code|mode}}: determines determine the mode how tokens are searched. Allowed values are {{Code|any}}, {{Code|any word}}, {{Code|all}}, {{Code|all words}}, and {{Code|phrase}}. {{Code|any}} is the default search mode.* {{Code|fuzzywildcards}}: turns fuzzy turn wildcard querying on or off. Allowed values are an empty string or {{Code|true}}, or and {{Code|false}}. By default, fuzzy wildcard querying is turned off.* {{Code|wildcardsfuzzy}}: turns wildcard turn fuzzy querying on or off. Allowed values are an empty string or {{Code|true}}, or and {{Code|false}}. By default, wildcard fuzzy querying is turned off.The following options have been added in * {{VersionCode|7errors}}: control the maximum number of tolerated errors for fuzzy querying.8By default, {{Code|0}}:is assigned (see [[Full-Text#Fuzzy_Querying|Fuzzy Querying]] for more details).* {{Code|ordered}}: requires that indicate if all tokens must occur in the order in which they are specified. Allowed values are {{Code|true}} and {{Code|false}}. The default is {{Code|false}}.* {{Code|content}}: specifies specify that the matched tokens need to occur at the beginning or end of a searched string, or need to cover the entire string. Allowed values are {{Code|start}}, {{Code|end}}, and {{Code|entire}}. By default, the option is turned off.* {{Code|scope}}: defines define the scope in which tokens must be located. The option has following sub options:

* {{Code|window}}: sets set up a window in which all tokens must be located. By default, the option is turned off. It has following sub options:** {{Code|size}}: specifies specify the size of the window in terms of ''units''.** {{Code|unit}}: can be {{Code|sentences}}, {{Code|sentences}}, or {{Code|paragraphs}}. The default is {{Code|words}}.* {{Code|windowdistance}}: specifies specify the distance in which tokens must occur. By default, the option is turned off. It has following sub options:** {{Code|min}}: specifies specify the minimum distance in terms of ''units''. The default is {{Code|0}}.** {{Code|max}}: specifies specify the maximum distance in terms of ''units''. The default is {{Code|∞}}.** {{Code|unit}}: can be {{Code|words}}, {{Code|sentences}}, or {{Code|paragraphs}}. The default is {{Code|words}}.|-valign="top"

|{{Error|BXDB0002db:get|XQuery ErrorsDatabase Module#BaseX Errors}} The addressed database does not exist or could not be opened.<br/>{{Error|BXDB0004db:no-index|Database Module#Errors}} the full-text index is not available.<br/>{{Error|BXFT0001options|#Errors}} both the fuzzy and wildcard querying was selectedoption cannot be both specified.|-valign="top"

* {{Code|ft:search("DB", "QUERY")}} returns : Return all text nodes of the database {{Code|DB}} that contain the term {{Code|QUERY}}.* Return all text nodes of the database {{Code|DB}} that contain the numbers {{Code|2010}} and {{Code|2020}}:<br/><code>ft:search("DB", ("2010","20112020"), map { 'mode': 'all' })</code><br/>returns all * Return text nodes of that contain the database terms {{Code|DBA}} that contain the numbers and {{Code|2010B|}} and in a distance of at most 5 words:<pre lang='xquery'>ft:search("db", ("A", "B"), map { "mode": "all words", "distance": map {Code|2011 "max": "5", "unit": "words" }}.)</pre>* The last example iterates Iterate over five three databases and returns return all elements containing terms similar to {{Code|Hello World}} in the text nodes:<pre classlang="brush:'xquery"'>

return ft:search($dbname, $terms, map { 'fuzzy': $optionsfuzzy })/..

==ft:marktokens== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|ft:mark|$nodes as node()*|node()*}}<br /pre>{{Func|ft:mark|tokens( $nodes db as node()*xs:string, $tag prefix as xs:string|node := ()) as element(value)*}}</pre>|-valign="top"

|Puts a marker element around Returns all full-text tokens stored in the index of the resulting database {{Code|$nodesdb}} , along with their numbers of a full-text index requestoccurrences.<br />The default tag name of the marker element is If {{Code|mark$prefix}}is specified, the returned nodes will be refined to the strings starting with that prefix. An alternative tag name can The prefix will be chosen via tokenized according to the full-text used for creating the optional index.|- valign="top"| '''Errors'''|{{CodeError|db:get|$tagDatabase Module#Errors}} argumentThe addressed database does not exist or could not be opened.<br />Please note that:* the XML node to be transformed must be an internal "database" node. The {{CodeError|db:no-index|transformDatabase Module#Errors}} expression can be used to apply the method to a main-memory fragment, as shown in Example 2.* the full-text expression, which computes the token positions, must be specified within <code>ft:mark()</code> function, as all position information index is lost in subsequent processing steps. You may need to specify more than one full-text expression if you want to use the function in a FLWOR expression, as shown in Example 3not available.|-valign="top"

|'''Example 1''': The following query returns {{Code|&lt;XML&gt;&lt;mark&gt;hello&lt;/mark&gt; world&lt;/XML&gt;}}Returns the number of occurrences for a single, if one text node of the database {{Code|DB}} has the value "hello world"specific index entry:<pre classlang="brush:'xquery">ft:mark(db:open('DB')//*[text() contains text 'hello'])</pre>'''Example 2''': The following expression returns {{Code|&lt;p&gt;&lt;b&gt;word&lt;/b&gt;&lt;/p&gt;}}:<pre class="brush:xquery">copy let $p term := &lt;p&gt;word&lt;/p&gt;modify ()return ft:marktokenize($p[text(term) contains text 'word'], 'b')</pre>'''Example 3''': The following expression loops through the first ten full-text results and marks the results in a second expression:<pre class="brush:xquery">let $start := 1let $end := 10let $term := 'welcome'for $return number(ft in (db:opentokens('DBdb')//*[text() contains text { , $term }])[position() . = $start to $end]return element hit { ft:mark($ft[text() contains text { $term }]/@count)}

=General Functions= ==ft:extractcontains== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|ft:extract|$nodes as node()*|node()*}}<br /pre>{{Func|ft:extract|contains( $nodes input as nodeitem()*, $tag terms as xs:string|nodeitem()*}}<br />{{Func|ft:extract|, $nodes options as nodemap(*)*, $tag as xs? :string, $length = map { }) as xs:integer|node()*}}boolean</pre>|-valign="top"

|Extracts and returns relevant parts of full-text results. It puts a marker element around Checks if the resulting specified {{Code|$nodesinput}} of a full-text index request and chops irrelevant sections of items contain the resultspecified {{Code|$terms}}.<br />The default tag name of function does the same as the marker element is [[Full-Text]] expression {{Code|markcontains text}}. An alternative tag name , but options can be chosen via specified more dynamically. The {{Code|$options}} are the optional same as for {{Function||ft:search}}, and the following ones exist:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|$taglower}} argument.<br />The By default length of the returned text , search is case-insensitive.* {{Code|150diacritics}} : determines how diacritical charactersare processed. An alternative length can be specified via the optional Allowed values are {{Code|insensitive}} and {{Code|sensitive}}. By default, search is diacritical insensitive.* {{Code|$lengthstemming}} argument: determines is tokens are stemmed. Note that the effective text length may differ from the specified text due to formatting Allowed values are {{Code|true}} and readibility issues{{Code|false}}.<br />For more details on this functionBy default, please have a look at [[#ftstemming is turned off.* {{Code|language}}:markdetermines the language. This option is relevant for stemming tokens. All language codes are supported. The default language is {{Code|en}}.|- valign="top"| '''Errors'''|{{Error|options|ft:mark]]#Errors}} specified options are conflicting.|-valign="top"

* The following query may return Checks if {{Code|&lt;XML&gt;...&lt;b&gt;hello&lt;/b&gt;...&lt;XML&gt;jack}} if a text node of the database or {{Code|DBjohn}} contains occurs in the input string "hello world"{{Code|John Doe}}:<pre classlang="brush:'xquery"'>ft:extractcontains(db"John Doe", ("jack", "john"), map { "mode": "any" })</pre>* Calls the function with stemming turned on and off:open(<pre lang='DBxquery'>(true()//*[text, false() ) ! ft:contains text ("Häuser", "Haus", map { 'hellostemming']: ., 'blanguage':', 1de' })

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:count|( $nodes as node()*|) as xs:integer}}</pre>|-valign="top"

|-valign="top"

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:score|( $item as item()*|) as xs:double*}}</pre>|-valign="top"

|-valign="top"

==ft:tokenize== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>ft:tokenize( $string as xs:string?, $options as map(*)? := map { }) as xs:string*</pre>|- valign="top"| '''Summary'''|Tokenizes the given {{Code|$string}}, using the current default full-text options or the {{Code|$options}} specified as second argument, and returns a sequence with the tokenized string. The following options are available:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|lower}}. By default, search is case insensitive.* {{Code|diacritics}}: determines how diacritical characters are processed. Allowed values are {{Code|insensitive}} and {{Code|sensitive}}. By default, search is diacritical insensitive.* {{Code|stemming}}: determines is tokens are stemmed. Allowed values are {{Code|true}} and {{Code|false}}. By default, stemming is turned off.* {{Code|language}}: determines the language. This option is relevant for stemming tokens. All language codes are supported. The default language is {{Code|en}}.The {{Code|$options}} argument can be used to control full-text processing.|- valign="top"| '''Examples'''|* <code>ft:tokenize("No Doubt")</code> returns the two strings {{Code|no}} and {{Code|doubt}}.* <code>ft:tokenize("École", map { 'diacritics': 'sensitive' })</code> returns the string {{Code|école}}.* <code>declare ft-option using stemming; ft:tokenize("GIFTS")</code> returns a single string {{Code|gift}}.|} ==ft:normalize== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>ft:normalize( $string as xs:string?, $options as map(*)? := map { }) as xs:string</pre>|- valign="top"| '''Summary'''|Normalizes the given {{Code|$string}}, using the current default full-text options or the {{Code|$options}} specified as second argument. The function accepts the same arguments as {{Function||ft:tokenize}}; special characters and separators will be preserved.|- valign="top"| '''Examples'''|* <code>ft:normalize("Häuser am Meer", map { 'case': 'sensitive' })</code> returns the string {{Code|Hauser am Meer}}.|} ==ft:thesaurus== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>ft:thesaurus( $node as node(), $term as xs:string, $options as map(*)? := map { }) as xs:string*</pre>|- valign="top"| '''Summary'''|Looks up a {{Code|$term}} in a [[Full-Text#Thesaurus|Thesaurus Structure]] supplied by {{Code|$node}}. The following {{Code|$options}} exist:* {{Code|relationship}}: determines the relationship between terms* {{Code|levels}}: determines the maximum number of levels to traverse|- valign="top"| '''Examples'''| Returns {{Code|happy}} and {{Code|lucky}}:<pre lang='xquery'>ft:thesaurus( <thesaurus> <entry> <term>happy</term> <synonym> <term>lucky</term> <relationship>RT</relationship> </synonym> </entry> </thesaurus>, 'happy')</pre>|} =Highlighting Functions= ==ft:mark== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:tokens|mark( $db nodes as xs:string|elementnode(value)*}}<br/>{{Func|ft:tokens|, $db name as xs:string, $prefix := ()) as xs:string|elementnode(value)*}}</pre>|-valign="top"

|Returns all full-text tokens stored in the index of Puts a marker element around the database resulting {{Code|$dbnodes}}, along with their numbers of occurrencesa full-text request.<br/>If The default name of the marker element is {{Code|mark}}. An alternative name can be chosen via the optional {{Code|$prefixname}} is argument.<br/>Please note that:* The full-text expression that computes the token positions must be specifiedas argument of the <code>ft:mark()</code> function, the returned nodes will be refined as all position information is lost in subsequent processing steps. You may need to specify more than one full-text expression if you want to use the strings starting with that prefixfunction in a FLWOR expression, as shown in Example 2. * The prefix will supplied node must be a [[Database Module#Database Node|Database Node]]. As shown in Example 3, {{Code|update}} or {{Code|transform}} can be tokenized according utilized to convert a fragment to the full-text used for creating the indexrequired internal representation.|-valign="top"| '''ErrorsExamples'''|'''Example 1''': The following query returns {{ErrorCode|BXDB0002|XQuery Errors#BaseX Errors&lt;XML&gt;&lt;mark&gt;hello&lt;/mark&gt; world&lt;/XML&gt;}} The addressed , if one text node of the database does not exist or could not be opened.<br/>{{ErrorCode|BXDB0004|Database Module#ErrorsDB}} has the value "hello world":<pre lang='xquery'>ft:mark(db:get('DB')//*[text() contains text 'hello'])</pre>'''Example 2''': The following expression loops through the first ten full-text index is not availableresults and marks the results in a second expression:<pre lang='xquery'>let $start := 1let $end := 10let $term := 'welcome'let $test := function($node) { $node/text() contains text { $term } }for $ft in (db:get('DB')//*[$test(.)])[position() = $start to $end]return ft:mark($ft[$test(.)])</pre>'''Example 3''': The following expression returns <code>&lt;xml>hello &lt;b&gt;word&lt;/b&gt;&lt;/xml&gt;</code>:<pre lang='xquery'>copy $p := <xml>hello world</xml>modify ()return ft:mark($p[text() contains text 'word'], 'b')</pre>

==ft:tokenizeextract== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:tokenize|extract( $nodes as node()*, $input name as xs:string| := (), $length as xs:stringinteger := ()) as node()*}}</pre>|-valign="top"

|Tokenizes Extracts and returns relevant parts of full-text results. It puts a marker element around the given resulting {{Code|$inputnodes}} string, using of a full-text index request and chops irrelevant sections of the result.<br/>The default element name of the marker element is {{Code|mark}}. An alternative element name can be chosen via the current optional {{Code|$name}} argument.<br/>The default full-length of the returned text is {{Code|150}} characters. An alternative length can be specified via the optional {{Code|$length}} argument. Note that the effective text length may differ from the specified text optionsdue to formatting and readibility issues.<br/>For more details on this function, please have a look at {{Function||ft:mark}}.|-valign="top"

* The following query may return {{Code|ft:tokenize("No Doubt")&lt;XML&gt;...&lt;b&gt;hello&lt;/b&gt;...&lt;XML&gt;}} returns if a text node of the two strings database {{Code|noDB}} and {{Code|doubt}}.contains the string "hello world":<pre lang='xquery'>* {{Code|declare ft-option using stemming; ft:tokenizeextract(db:get('DB')//*[text("GIFTS")}} returns a single string {{Code|gift}}.contains text 'hello'], 'b', 1)</pre>

|-valign="top"|{{Code|BXFT0001options}}

 * Updated: [[#ft:search{{Function||ft:search]] }} (second argument generalized, third parameter added)

 * Added: [[#ft:tokens{{Function||ft:tokens]]}}, [[#ft:tokenize{{Function||ft:tokenize]] [[Category:XQuery]]}}