Changes

This [[Module Library|XQuery Module]] extends the [http://www.w3.org/TR/xpath[Full-full-text-10 W3C Full Text Recommendation] with some useful functions] features of BaseX: The index can be directly accessed, full-text results can be marked with additional elements, or the relevant parts can be extracted. Moreover, the score value, which is generated by the {{Code|contains text}} expression, can be explicitly requested from items.

All functions and errors in this module are assigned to the {{Code|<code><nowiki>http://basex.org/modules/ft}} </nowiki></code> namespace, which is statically bound to the {{Code|ft}} prefix.<br/>All errors are assigned to the {{Code|http://basex.org/errors}} namespace, which is statically bound to the {{Code|bxerr}} prefix.

=Database Functions=

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|ft:search|$db as xs:string, $terms as item()*|text()*}}<br/pre>{{Func|ft:search|( $db as xs:string, $terms as item()*, $options as itemmap(*)|? := map { }) as text()*}}</pre>|-valign="top"

The {{Code|$options}} argument can be used to control full-text processing. Options can be either specified<br/>* as children of an {{Code|&lt;options/&gt;}} element, e.g.:<pre class="brush:xml"><options> <key1 value='value1'/> ...</options></pre>* as map, which contains all key/value pairs:<pre class="brush:xml">{ "key1": "value1", ... }</pre>The following options are supported (the introduction on [[Full-Text]] processing gives you equivalent expressions in the XQuery Full-Text notation):* {{Code|mode}}: determines determine the mode how tokens are searched. Allowed values are {{Code|any}}, {{Code|any word}}, {{Code|all}}, {{Code|all words}}, and {{Code|phrase}}. {{Code|any}} is the default search mode.* {{Code|fuzzywildcards}}: turns fuzzy turn wildcard querying on or off. Allowed values are {{Code|true}} and {{Code|false}}. By default, fuzzy wildcard querying is turned off.* {{Code|wildcardsfuzzy}}: turns wildcard turn fuzzy querying on or off. Allowed values are {{Code|true}} and {{Code|false}}. By default, wildcard fuzzy querying is turned off.The following options have been added in * {{VersionCode|7errors}}: control the maximum number of tolerated errors for fuzzy querying.8By default, {{Code|0}}:is assigned (see [[Full-Text#Fuzzy_Querying|Fuzzy Querying]] for more details).* {{Code|ordered}}: requires that indicate if all tokens must occur in the order in which they are specified. Allowed values are {{Code|true}} and {{Code|false}}. The default is {{Code|false}}.* {{Code|content}}: specifies specify that the matched tokens need to occur at the beginning or end of a searched string, or need to cover the entire string. Allowed values are {{Code|start}}, {{Code|end}}, and {{Code|entire}}. By default, the option is turned off.* {{Code|scope}}: defines define the scope in which tokens must be located. The option has following sub options:

* {{Code|window}}: sets set up a window in which all tokens must be located. By default, the option is turned off. It has following sub options:** {{Code|size}}: specifies specify the size of the window in terms of ''units''.

* {{Code|distance}}: specifies specify the distance in which tokens must occur. By default, the option is turned off. It has following sub options:** {{Code|min}}: specifies specify the minimum distance in terms of ''units''. The default is {{Code|0}}.** {{Code|max}}: specifies specify the maximum distance in terms of ''units''. The default is {{Code|∞}}.

|-valign="top"

|{{Error|BXDB0002db:get|XQuery ErrorsDatabase Module#BaseX Errors}} The addressed database does not exist or could not be opened.<br/>{{Error|BXDB0004db:no-index|Database Module#Errors}} the full-text index is not available.<br/>{{Error|BXFT0001options|#Errors}} the fuzzy and wildcard option cannot be both specified.|-valign="top"

* Return all text nodes of the database {{Code|DB}} that contain the numbers {{Code|2010}} and {{Code|20112020}}:<br/><code>ft:search("DB", ("2010","20112020"), map { 'mode': 'all' })</code>

<pre classlang="brush:'xquery"'>ft:search("db", ("A", "B"), map {

"distance": map {

<pre classlang="brush:'xquery"'>

return ft:search($dbname, $terms, map { 'fuzzy': $optionsfuzzy })/..

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|ft:contains|$input as item()*, $terms as item()*|xs:boolean}}<br/pre>{{Func|ft:contains|( $input as item()*, $terms as item()*, $options as itemmap(*)|? := map { }) as xs:boolean}}</pre>|-valign="top"

|Checks if the specified {{Code|$input}} items contain the specified {{Code|$terms}}.<br/>The function does the same as the [[Full-Text]] expression {{Code|contains text}}, but options can be specified more dynamically. The {{Code|$options}} are the same as for [[#ft:search{{Function||ft:search]]}}, and the following ones in additionexist:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|lower}}. By default, search is case -insensitive.

|-valign="top"

|{{Error|BXFT0001options|#Errors}} the fuzzy and wildcard option cannot be both specifiedoptions are conflicting.|-valign="top"

<pre classlang="brush:'xquery"'>ft:contains("John Doe", ("jack", "john"), map { "mode": "any" })

<pre classlang="brush:'xquery"'>(true(), false()) ! ft:contains("Häuser", "Haus", map { 'stemming': ., 'language':'de' })

==ft:markcount== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|ft:mark|$nodes as node()*|node()*}}<br /pre>{{Func|ft:mark|count( $nodes as node()*, $tag ) as xs:string|node()*}}integer</pre>|-valign="top"

|Puts a marker element around Returns the resulting {{Code|$nodes}} number of a full-text index request.<br />The default tag name occurrences of the marker element is {{Code|mark}}. An alternative tag name can be chosen via the optional {{Code|$tag}} argument.<br />Please note that:* the XML node to be transformed must be an internal "database" node. The {{Code|transform}} expression can be used to apply the method to a main-memory fragment, as shown in Example 2.* the full-text expression, which computes the token positions, must be search terms specified within <code>ft:mark()</code> function, as all position information is lost in subsequent processing steps. You may need to specify more than one a full-text expression if you want to use the function in a FLWOR expression, as shown in Example 3.|-valign="top"

|'''Example 1''': The following query returns * {{Code|&lt;XML&gt;&lt;mark&gt;hello&lt;/mark&gt; world&lt;/XML&gt;}}, if one text node of the database {{Code|DB}} has the value "hello world":<pre class="brush:xquery">ft:mark(db:opencount('DB')//*[text() contains text 'helloQUERY'])</pre>'''Example 2'''}} returns the {{Code|xs: The following expression returns integer}} value {{Code|&lt;p&gt;&lt;b&gt;word&lt;/b&gt;&lt;/p&gt;2}}:<pre class="brush:xquery">copy $p := &lt;p&gt;word&lt;/p&gt;modify ()return ft:mark($p[text() if a document contains text 'word'], 'b')</pre>'''Example 3''': The following expression loops through two occurrences of the first ten full-text results and marks the results in a second expression:<pre class=string "brush:xqueryQUERY">let $start := 1let $end := 10let $term := 'welcome'for $ft in (db:open('DB')//*[text() contains text { $term }])[position() = $start to $end]return element hit { ft:mark($ft[text() contains text { $term }])}</pre>.

==ft:extractscore== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|ft:extract|$nodes as node()*|node()*}}<br /pre>{{Func|ft:extract|score( $nodes item as nodeitem()*, $tag ) as xs:string|node()double*}}<br /pre>{{Func|ft:extract|$nodes as node()*, $tag as xs:string, $length as xs:integer|node()*}}|-valign="top"

|Extracts and returns relevant parts of full-text resultsReturns the score values (0. It puts a marker element around the resulting {{Code|$nodes}} of a full0 -text index request and chops irrelevant sections of the result1.<br />The default tag name of 0) that have been attached to the marker element is {{Code|mark}}specified items. An alternative tag name can be chosen via the optional {{Code|$tag0}} argument.<br />The default length of the is returned text is {{Code|150}} characters. An alternative length can be specified via the optional {{Code|$length}} argument. Note that the effective text length may differ from the specified text due to formatting and readibility issues.<br />For more details on this function, please have a look at [[#ft:mark|ft:mark]]value if no score was attached.|-valign="top"

* The following query may return {{Code|&lt;XML&gt;...&lt;b&gt;hello&lt;/b&gt;...&lt;XML&gt;ft:score('a' contains text 'a')}} if a text node of returns the database {{Code|DBxs:double}} contains the string "hello world":<pre class="brush:xquery">ft:extract(db:open('DB')//*[text() contains text 'hello'], 'b', value {{Code|1)</pre>}}.

==ft:counttokenize== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:count|tokenize( $string as xs:string?, $nodes options as nodemap(*)*|? := map { }) as xs:integer}}string*</pre>|-valign="top"

|Returns Tokenizes the number of occurrences of given {{Code|$string}}, using the current default full-text options or the search terms {{Code|$options}} specified in as second argument, and returns a sequence with the tokenized string. The following options are available:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|lower}}. By default, search is case insensitive.* {{Code|diacritics}}: determines how diacritical characters are processed. Allowed values are {{Code|insensitive}} and {{Code|sensitive}}. By default, search is diacritical insensitive.* {{Code|stemming}}: determines is tokens are stemmed. Allowed values are {{Code|true}} and {{Code|false}}. By default, stemming is turned off.* {{Code|language}}: determines the language. This option is relevant for stemming tokens. All language codes are supported. The default language is {{Code|en}}.The {{Code|$options}} argument can be used to control full-text expressionprocessing.|-valign="top"

* <code>ft:tokenize("No Doubt")</code> returns the two strings {{Code|no}} and {{Code|doubt}}.* <code>ft:counttokenize(//*[text() contains text "École", map { 'diacritics': 'QUERYsensitive']})}} </code> returns the string {{Code|xs:integerécole}} value .* <code>declare ft-option using stemming; ft:tokenize("GIFTS")</code> returns a single string {{Code|2gift}} if a document contains two occurrences of the string "QUERY".

==ft:scorenormalize== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:score|normalize( $string as xs:string?, $item options as itemmap(*)*|? := map { }) as xs:double*}}string</pre>|-valign="top"

|Returns Normalizes the score values (0.0 given {{Code|$string}}, using the current default full- 1.0) that have been attached to text options or the {{Code|$options}} specified itemsas second argument. The function accepts the same arguments as {{CodeFunction||0ft:tokenize}} is returned a value if no score was attached; special characters and separators will be preserved.|-valign="top"

* <code>ft:normalize("Häuser am Meer", map { 'case': 'sensitive' })</code> returns the string {{Code|Hauser am Meer}}.|} ==ft:thesaurus== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>ft:scorethesaurus( $node as node(), $term as xs:string, $options as map(*)? := map { }) as xs:string*</pre>|- valign="top"| '''Summary'a' contains text '|Looks up a {{Code|$term}} in a')[[Full-Text#Thesaurus|Thesaurus Structure]] supplied by {{Code|$node}}. The following {{Code|$options}} exist:* {{Code|relationship}} returns : determines the relationship between terms* {{Code|xslevels}}:doubledetermines the maximum number of levels to traverse|- valign="top"| '''Examples'''| Returns {{Code|happy}} value and {{Code|1lucky}}.:<pre lang='xquery'>ft:thesaurus( <thesaurus> <entry> <term>happy</term> <synonym> <term>lucky</term> <relationship>RT</relationship> </synonym> </entry> </thesaurus>, 'happy')</pre>

=Highlighting Functions= ==ft:tokensmark== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:tokens|mark( $db nodes as xs:string|elementnode(value)*}}<br/>{{Func|ft:tokens|, $db name as xs:string, $prefix := ()) as xs:string|elementnode(value)*}}</pre>|-valign="top"

|Returns all full-text tokens stored in Puts a marker element around the index of the database resulting {{Code|$dbnodes}}, along with their numbers of occurrencesa full-text request.<br/>If The default name of the marker element is {{Code|mark}}. An alternative name can be chosen via the optional {{Code|$prefixname}} is specified, argument.<br/>Please note that:* The full-text expression that computes the returned nodes will token positions must be refined to specified as argument of the strings starting with that prefix<code>ft:mark()</code> function, as all position information is lost in subsequent processing steps. The prefix will be tokenized according You may need to the specify more than one full-text used for creating expression if you want to use the indexfunction in a FLWOR expression, as shown in Example 2.* The supplied node must be a [[Database Module#Database Node|-| '''Errors'''|Database Node]]. As shown in Example 3, {{Error|BXDB0002Code|XQuery Errors#BaseX Errorsupdate}} The addressed database does not exist or could not be opened.<br/>{{ErrorCode|BXDB0004|Database Module#Errorstransform}} can be utilized to convert a fragment to the full-text index is not availablerequired internal representation.|-valign="top"

|Finds '''Example 1''': The following query returns {{Code|&lt;XML&gt;&lt;mark&gt;hello&lt;/mark&gt; world&lt;/XML&gt;}}, if one text node of the database {{Code|DB}} has the number of occurrences for a single, specific index entry value "hello world":<pre lang='xquery'>ft:mark(db:get('DB')//*[text() contains text 'hello'])</pre>'''Example 2''': The following expression loops through the positional predicate speeds up retrieval)first ten full-text results and marks the results in a second expression:<pre classlang="brush:'xquery"'>let $start := 1let $end := 10let $term := ft'welcome'let $test :tokenize= function($node) { $node/text() contains text { $term)} }return datafor $ft in ((ftdb:tokensget('dbDB', )//*[$termtest(.)])[position()= $start to $end]return ft:mark($ft[$test(. )])</pre>'''Example 3''': The following expression returns <code>&lt;xml>hello &lt;b&gt;word&lt;/b&gt;&lt;/xml&gt;</code>:<pre lang= 'xquery'>copy $term]p := <xml>hello world</xml>modify ()return ft:mark($p[1text() contains text 'word']/@count, 'b')

==ft:tokenizeextract== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:tokenize|extract( $nodes as node()*, $input name as xs:string| := (), $length as xs:stringinteger := ()) as node()*}}</pre>|-valign="top"

|Tokenizes Extracts and returns relevant parts of full-text results. It puts a marker element around the given resulting {{Code|$inputnodes}} string, using of a full-text index request and chops irrelevant sections of the result.<br/>The default element name of the marker element is {{Code|mark}}. An alternative element name can be chosen via the current optional {{Code|$name}} argument.<br/>The default full-length of the returned text is {{Code|150}} characters. An alternative length can be specified via the optional {{Code|$length}} argument. Note that the effective text length may differ from the specified text optionsdue to formatting and readibility issues.<br/>For more details on this function, please have a look at {{Function||ft:mark}}.|-valign="top"

* The following query may return {{Code|ft:tokenize("No Doubt")&lt;XML&gt;...&lt;b&gt;hello&lt;/b&gt;...&lt;XML&gt;}} returns if a text node of the two strings database {{Code|noDB}} and {{Code|doubt}}.contains the string "hello world":<pre lang='xquery'>* {{Code|declare ft-option using stemming; ft:tokenizeextract(db:get('DB')//*[text("GIFTS")}} returns a single string {{Code|gift}}.contains text 'hello'], 'b', 1)</pre>

|-valign="top"|{{Code|BXFT0001options}}

 * Added: [[#ft:contains{{Function||ft:contains]]}}* Updated: Options added to [[#ft:search{{Function||ft:search]]}}

 * Updated: [[#ft:search{{Function||ft:search]] }} (second argument generalized, third parameter added)

 * Added: [[#ft:tokens{{Function||ft:tokens]]}}, [[#ft:tokenize{{Function||ft:tokenize]] [[Category:XQuery]]}}