Changes

 This [[Module Library|XQuery Module]] extends the [http://www.w3.org/TR/xpath-full-text[Full-10 W3C Full Text Recommendation] with some useful functions] features of BaseX: The index can be directly accessed, fulltext full-text results can be marked with additional elements, or the relevant parts can be extracted. Moreover, the score value, which is generated by the {{Code|contains text}} expression, can be explicitly requested from items.

=Database Functions=

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|ft:search|$db as xs:string, $terms as item()*|text()*}}<br/pre>{{Func|ft:search|( $db as xs:string, $terms as item()*, $options as map(xs*)? :string, item(= map { }))|as text()*}}</pre>|-valign="top"

* {{Code|mode}}: determines determine the mode how tokens are searched. Allowed values are {{Code|any}}, {{Code|any word}}, {{Code|all}}, {{Code|all words}}, and {{Code|phrase}}. {{Code|any}} is the default search mode.* {{Code|fuzzywildcards}}: turns fuzzy turn wildcard querying on or off. Allowed values are {{Code|true}} and {{Code|false}}. By default, fuzzy wildcard querying is turned off.* {{Code|wildcardsfuzzy}}: turns wildcard turn fuzzy querying on or off. Allowed values are {{Code|true}} and {{Code|false}}. By default, wildcard fuzzy querying is turned off.* {{Code|errors}}: control the maximum number of tolerated errors for fuzzy querying. By default, {{Code|0}} is assigned (see [[Full-Text#Fuzzy_Querying|Fuzzy Querying]] for more details).* {{Code|ordered}}: requires that indicate if all tokens must occur in the order in which they are specified. Allowed values are {{Code|true}} and {{Code|false}}. The default is {{Code|false}}.* {{Code|content}}: specifies specify that the matched tokens need to occur at the beginning or end of a searched string, or need to cover the entire string. Allowed values are {{Code|start}}, {{Code|end}}, and {{Code|entire}}. By default, the option is turned off.* {{Code|scope}}: defines define the scope in which tokens must be located. The option has following sub options:

* {{Code|window}}: sets set up a window in which all tokens must be located. By default, the option is turned off. It has following sub options:** {{Code|size}}: specifies specify the size of the window in terms of ''units''.

* {{Code|distance}}: specifies specify the distance in which tokens must occur. By default, the option is turned off. It has following sub options:** {{Code|min}}: specifies specify the minimum distance in terms of ''units''. The default is {{Code|0}}.** {{Code|max}}: specifies specify the maximum distance in terms of ''units''. The default is {{Code|∞}}.

|-valign="top"

|{{Error|db:openget|Database Module#Errors}} The addressed database does not exist or could not be opened.<br/>{{Error|db:no-index|Database Module#Errors}} the index is not available.<br/>{{Error|options|#Errors}} the fuzzy and wildcard option cannot be both specified.|-valign="top"

<pre classlang="brush:'xquery"'>

<pre classlang="brush:'xquery"'>

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|ft:contains|$input as item()*, $terms as item()*|xs:boolean}}<br/pre>{{Func|ft:contains|( $input as item()*, $terms as item()*, $options as map(xs*)? :xstring, item(= map { }))|as xs:boolean}}</pre>|-valign="top"

|Checks if the specified {{Code|$input}} items contain the specified {{Code|$terms}}.<br/>The function does the same as the [[Full-Text]] expression {{Code|contains text}}, but options can be specified more dynamically. The {{Code|$options}} are the same as for [[#ft:search{{Function||ft:search]]}}, and the following ones in additionexist:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|lower}}. By default, search is case -insensitive.

|-valign="top"

|-valign="top"

<pre classlang="brush:'xquery"'>

<pre classlang="brush:'xquery"'>

==ft:markcount== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|ft:mark|$nodes as node()*|node()*}}<br /pre>{{Func|ft:mark|count( $nodes as node()*, $name ) as xs:string|node()*}}integer</pre>|-valign="top"

|Puts a marker element around Returns the resulting {{Code|$nodes}} number of a full-text index request.<br />The default name occurrences of the marker element is {{Code|mark}}. An alternative name can be chosen via the optional {{Code|$name}} argument.<br />Please note that:* the full-text expression that computes the token positions must be search terms specified as argument of the <code>ft:mark()</code> function, as all position information is lost in subsequent processing steps. You may need to specify more than one a full-text expression if you want to use the function in a FLWOR expression, as shown in Example 2.* the XML node to be transformed must be an internal |- valign="databasetop" node. The {{Code|transform}} expression can be used to apply the method to a main-memory fragment, as shown in Example 3.|-

|'''Example 1''': The following query returns * {{Code|&lt;XML&gt;&lt;mark&gt;hello&lt;/mark&gt; world&lt;/XML&gt;}}, if one text node of the database {{Code|DB}} has the value "hello world":<pre class="brush:xquery">ft:markcount(db:open('DB')//*[text() contains text 'helloQUERY'])</pre>'''Example 2''': The following expression loops through the first ten full-text results and marks }} returns the results in a second expression:<pre class="brush:xquery">let $start := 1let $end := 10let $term := 'welcome'for $ft in (db:open('DB')//*[text() contains text { $term }])[position() = $start to $end]return element hit { ftCode|xs:mark($ft[text() contains text { $term integer}])}</pre>'''Example 3''': The following expression returns value {{Code|&lt;p&gt;&lt;b&gt;word&lt;/b&gt;&lt;/p&gt;2}}:<pre class=if a document contains two occurrences of the string "brush:xqueryQUERY">copy $p := &lt;p&gt;word&lt;/p&gt;modify ()return ft:mark($p[text() contains text 'word'], 'b')</pre>.

==ft:extractscore== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|ft:extract|$nodes as node()*|node()*}}<br /pre>{{Func|ft:extract|score( $nodes item as nodeitem()*, $name ) as xs:string|node()double*}}<br /pre>{{Func|ft:extract|$nodes as node()*, $name as xs:string, $length as xs:integer|node()*}}|-valign="top"

|Extracts and returns relevant parts of full-text resultsReturns the score values (0. It puts a marker element around the resulting {{Code|$nodes}} of a full0 -text index request and chops irrelevant sections of the result1.<br />The default element name of 0) that have been attached to the marker element is {{Code|mark}}specified items. An alternative element name can be chosen via the optional {{Code|$name0}} argument.<br />The default length of the is returned text is {{Code|150}} characters. An alternative length can be specified via the optional {{Code|$length}} argument. Note that the effective text length may differ from the specified text due to formatting and readibility issues.<br />For more details on this function, please have a look at [[#ft:mark|ft:mark]]value if no score was attached.|-valign="top"

* The following query may return {{Code|&lt;XML&gt;...&lt;b&gt;hello&lt;/b&gt;...&lt;XML&gt;ft:score('a' contains text 'a')}} if a text node of returns the database {{Code|DBxs:double}} contains the string "hello world":<pre class="brush:xquery">ft:extract(db:open('DB')//*[text() contains text 'hello'], 'b', value {{Code|1)</pre>}}.

==ft:counttokenize== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:count|tokenize( $string as xs:string?, $nodes options as nodemap(*)*|? := map { }) as xs:integer}}string*</pre>|-valign="top"

|Returns Tokenizes the number of occurrences of given {{Code|$string}}, using the current default full-text options or the search terms {{Code|$options}} specified in as second argument, and returns a sequence with the tokenized string. The following options are available:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|lower}}. By default, search is case insensitive.* {{Code|diacritics}}: determines how diacritical characters are processed. Allowed values are {{Code|insensitive}} and {{Code|sensitive}}. By default, search is diacritical insensitive.* {{Code|stemming}}: determines is tokens are stemmed. Allowed values are {{Code|true}} and {{Code|false}}. By default, stemming is turned off.* {{Code|language}}: determines the language. This option is relevant for stemming tokens. All language codes are supported. The default language is {{Code|en}}.The {{Code|$options}} argument can be used to control full-text expressionprocessing.|-valign="top"

* <code>ft:tokenize("No Doubt")</code> returns the two strings {{Code|no}} and {{Code|doubt}}.* <code>ft:counttokenize(//*[text() contains text "École", map { 'diacritics': 'QUERYsensitive']})}} </code> returns the string {{Code|xs:integerécole}} value .* <code>declare ft-option using stemming; ft:tokenize("GIFTS")</code> returns a single string {{Code|2gift}} if a document contains two occurrences of the string "QUERY".

==ft:scorenormalize== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:score|normalize( $string as xs:string?, $item options as itemmap(*)*|? := map { }) as xs:double*}}string</pre>|-valign="top"

|Returns Normalizes the score values (0.0 given {{Code|$string}}, using the current default full- 1.0) that have been attached to text options or the {{Code|$options}} specified itemsas second argument. The function accepts the same arguments as {{CodeFunction||0ft:tokenize}} is returned a value if no score was attached; special characters and separators will be preserved.|-valign="top"

* {{Code|<code>ft:scorenormalize("Häuser am Meer", map { 'acase' contains text : 'asensitive'})}} </code> returns the string {{Code|xs:double}} value {{Code|1Hauser am Meer}}.

==ft:tokensthesaurus== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:tokens|thesaurus( $db node as xs:string|elementnode(value)*}}<br/>{{Func|ft:tokens|, $db term as xs:string, $prefix options as map(*)? := map { }) as xs:string|element(value)*}}</pre>|-valign="top"

|Returns all full-text tokens stored in the index of the database Looks up a {{Code|$dbterm}}, along with their numbers of occurrences.<br/>If in a [[Full-Text#Thesaurus|Thesaurus Structure]] supplied by {{Code|$prefixnode}} is specified, the returned nodes will be refined to the strings starting with that prefix. The prefix will be tokenized according to the full-text used for creating the index.|-| '''Errors'''|following {{ErrorCode|db:open|Database Module#Errors$options}} The addressed database does not exist or could not be opened.<br/>:* {{ErrorCode|dbrelationship}}:no-indexdetermines the relationship between terms* {{Code|Database Module#Errorslevels}} : determines the full-text index is not available.maximum number of levels to traverse|-valign="top"

|Returns the number of occurrences for a single, specific index entry{{Code|happy}} and {{Code|lucky}}:<pre classlang="brush:'xquery"'>let $term := ft:tokenizethesaurus($ <thesaurus> <entry> <term>happy</term)>return number(ft:tokens('db', $ <synonym> <term)[. = $>lucky</term]> <relationship>RT</relationship> </synonym> </@countentry> </thesaurus>, 'happy')

=Highlighting Functions= ==ft:tokenizemark== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:tokenize|mark( $input nodes as xs:string|xs:stringnode()*}}<br/>{{Func|ft:tokenize|, $input name as xs:string, $options as map(xs :xstring, item= ()) as node()|xs:string*}}</pre>|-valign="top"

|Tokenizes Puts a marker element around the given resulting {{Code|$inputnodes}} string, using the current default of a full-text options or the {{Code|$options}} specified as second argumentrequest. <br/>The following options are available:* {{Code|case}}: determines how character case is processed. Allowed values are {{Code|insensitive}}, {{Code|sensitive}}, {{Code|upper}} and {{Code|lower}}. By default, search name of the marker element is case insensitive.* {{Code|diacriticsmark}}: determines how diacritical characters are processed. Allowed values are An alternative name can be chosen via the optional {{Code|insensitive}} and {{Code|sensitive$name}}argument. By default, search is diacritical insensitive.<br/>Please note that:* {{Code|stemming}}The full-text expression that computes the token positions must be specified as argument of the <code>ft: determines mark()</code> function, as all position information is tokens are stemmedlost in subsequent processing steps. Allowed values are {{Code|true}} and {{Code|false}}. By defaultYou may need to specify more than one full-text expression if you want to use the function in a FLWOR expression, stemming is turned offas shown in Example 2.* {{CodeThe supplied node must be a [[Database Module#Database Node|language}}: determines the languageDatabase Node]]. This option is relevant for stemming tokens. All language codes are supported. The default language is As shown in Example 3, {{Code|enupdate}}.The or {{Code|$optionstransform}} argument can be used utilized to control full-text processingconvert a fragment to the required internal representation.|-valign="top"

|* <code>ft'''Example 1''':tokenize("No Doubt")</code> The following query returns the two strings {{Code|no&lt;XML&gt;&lt;mark&gt;hello&lt;/mark&gt; world&lt;/XML&gt;}} and , if one text node of the database {{Code|doubtDB}}.has the value "hello world":* <codepre lang='xquery'>ft:tokenizemark(db:get("École", map { 'diacriticsDB': )//*[text() contains text 'sensitivehello' }])</codepre> returns '''Example 2''': The following expression loops through the first ten full-text results and marks the string results in a second expression:<pre lang='xquery'>let $start := 1let $end := 10let $term := 'welcome'let $test := function($node) {$node/text() contains text {Code|école$term }}for $ft in (db:get('DB')//*[$test(.)])[position() = $start to $end]* return ft:mark($ft[$test(.)])</pre>'''Example 3''': The following expression returns <code>declare ft-option using stemming&lt;xml>hello &lt;b&gt;word&lt;/b&gt;&lt;/xml&gt; </code>:<pre lang='xquery'>copy $p := <xml>hello world</xml>modify ()return ft:tokenizemark($p[text("GIFTS")contains text 'word'], 'b')</codepre> returns a single string {{Code|gift}}.

==ft:normalizeextract== 

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|<pre>ft:normalize|extract( $input nodes as xs:string|xs:stringnode()*}}<br/>{{Func|ft:normalize|, $input name as xs:string := (), $options length as map(xs:xstring, iteminteger := ()) as node()|xs:string*}}</pre>|-valign="top"

|Normalizes Extracts and returns relevant parts of full-text results. It puts a marker element around the given resulting {{Code|$inputnodes}} string, using the current default of a full-text options or index request and chops irrelevant sections of the result.<br/>The default element name of the marker element is {{Code|mark}}. An alternative element name can be chosen via the optional {{Code|$optionsname}} argument.<br/>The default length of the returned text is {{Code|150}} characters. An alternative length can be specified as second via the optional {{Code|$length}} argument. The Note that the effective text length may differ from the specified text due to formatting and readibility issues.<br/>For more details on this function expects the same arguments as [[#ft:tokenize, please have a look at {{Function||ft:tokenize]]mark}}.|-valign="top"

* The following query may return {{Code|&lt;XML&gt;...&lt;b&gt;hello&lt;/b&gt;...&lt;XML&gt;}} if a text node of the database {{Code|DB}} contains the string "hello world":<codepre lang='xquery'>ft:tokenizeextract("Häuser am Meer", map { db:get('DB')//*[text() contains text 'casehello': ], 'sensitiveb' }, 1)</codepre> returns the string {{Code|Hauser am Meer}}.

|-valign="top"|{{Code|conflictoptions}}

 * Updated: error codes updatesupdated; errors now use the module namespace

 * Added: [[#ft:contains{{Function||ft:contains]]}}, [[#ft:normalize{{Function||ft:normalize]]}}* Updated: Options added to [[#ft:tokenize{{Function||ft:tokenize]]}}

 * Added: [[#ft:contains{{Function||ft:contains]]}}* Updated: Options added to [[#ft:search{{Function||ft:search]]}}

 * Updated: [[#ft:search{{Function||ft:search]] }} (second argument generalized, third parameter added)

 * Added: [[#ft:tokens{{Function||ft:tokens]]}}, [[#ft:tokenize{{Function||ft:tokenize]]}}