Changes

This is a simple example for a basic full-text expression:

"This is YOUR World" contains text "your world"

It yields {{Code|true}}, because the search string is ''tokenized'' before it is compared with the tokenized input string. In the tokenization process, several normalizations take place. Many of those steps can hardly be simulated with plain XQuery: as an example, upper/lower case and diacritics (umlauts, accents, etc.) are removed and an optional, language-dependent stemming algorithm is applied. Beside that, special characters such as whitespaces and punctuation marks will be ignored. Thus, this query also yields true:

"Well... Done!" contains text "well, done"

The {{Code|occurs}} keyword comes into play when more than one occurrence of a token is to be found:

"one and two and three" contains text "and" occurs at least 2 times

Various range modifiers are available: {{Code|exactly}}, {{Code|at least}}, {{Code|at most}}, and {{Code|from ... to ...}}.

In the given example, curly braces are used to combine multiple keywords:

for $country in doc('factbook')//country

where $country//religions[text() contains text { 'Sunni', 'Shia' } any]

return $country/name

The query will output the names of all countries with a religion element containing {{Code|sunni}} or {{Code|shia}}. The {{Code|any}} keyword is optional; it can be replaced with:

The keywords {{Code|ftand}}, {{Code|ftor}} and {{Code|ftnot}} can also be used to combine multiple query terms. The following query yields the same result as the last one does:

doc('factbook')//country[descendant::religions contains text 'sunni' ftor 'shia']/name

The keywords {{Code|not in}} are special: they are used to find tokens which are not part of a longer token sequence:

for $text in ("New York", "new conditions")

return $text contains text "New" not in "New York"

Due to the complex data model of the XQuery Full Text spec, the usage of {{Code|ftand}} may lead to a high memory consumption. If you should encounter problems, simply use the {{Code|all}} keyword:

doc('factbook')//country[descendant::religions contains text { 'Christian', 'Jewish'} all]/name

==Positional Filters==

A popular retrieval operation is to filter texts by the distance of the searched words. In this query…

<xml>

<text>There is some reason why ...</text>

<text>The reason why some people ...</text>

</xml>//text[. contains text { "some", "reason" } all ordered distance at most 3 words]

…the two first texts will be returned as result, because there are at most three words between {{Code|some}} and {{Code|reason}}. Additionally, the {{Code|ordered}} keyword ensures that the words are found in the specified order, which is why the third text is excluded. Note that {{Code|all}} is required here to guarantee that only those hits will be accepted that contain all searched words.

The {{Code|window}} keyword is related: it accepts those texts in which all keyword occur within the specified number of tokens. Can you guess what is returned by the following query?

("A C D", "A B C D E")[. contains text { "A", "E" } all window 3 words]

Sometimes it is interesting to only select texts in which all searched terms occur in the {{Code|same sentence}} or {{Code|paragraph}} (you can even filter for {{Code|different}} sentences/paragraphs). This is obviously not the case in the following example:

'Mary told me, “I will survive!”.' contains text { 'will', 'told' } all words same sentence

By the way: In some examples above, the {{Code|words}} unit was used, but {{Code|sentences}} and {{Code|paragraphs}} would have been valid alternatives.

* If {{Code|case}} is insensitive, no distinction is made between characters in upper and lower case. By default, the option is {{Code|insensitive}}; it can also be set to {{Code|sensitive}}:

"Respect Upper Case" contains text "Upper" using case sensitive

* If {{Code|diacritics}} is insensitive, characters with and without diacritics (umlauts, characters with accents) are declared as identical. By default, the option is {{Code|insensitive}}; it can also be set to {{Code|sensitive}}:

"'Äpfel' will not be found..." contains text "Apfel" using diacritics sensitive

* If {{Code|stemming}} is activated, words are shortened to a base form by a language-specific stemmer:

"catch" contains text "catches" using stemming

* With the {{Code|stop words}} option, a list of words can be defined that will be ignored when tokenizing a string. This is particularly helpful if the full-text index takes too much space (a standard stopword list for English texts is provided in the directory {{Code|etc/stopwords.txt}} in the full distributions of BaseX, and available online at http://files.basex.org/etc/stopwords.txt):

"You and me" contains text "you or me" using stop words ("and", "or"),

"You and me" contains text "you or me" using stop words at "http://files.basex.org/etc/stopwords.txt"

* Related terms such as synonyms can be found with the sophisticated [[#Thesaurus|Thesaurus]] option.

* <code>.{min,max}</code> matches ''min''–''max'' number of characters.

"This may be interesting in the year 2000" contains text { "interest.*", "2.{3,3}" } using wildcards

This was a quick introduction to XQuery Full Text; you are invited to explore the numerous other features of the language!

A list of all language codes that are available on your system can be retrieved as follows:

declare namespace locale = "java:java.util.Locale";

distinct-values(locale:getAvailableLocales() ! locale:getLanguage(.))

By default, unless the languages codes <code>ja</code>, <code>ar</code>, <code>ko</code>, <code>th</code>, or <code>zh</code> are specified, a tokenizer for Western texts is used:

The following two queries, which both return <code>true</code>, demonstrate that stemming depends on the selected language:

"Indexing" contains text "index" using stemming,

"häuser" contains text "haus" using stemming using language "German"

==Scoring==

The scoring model of BaseX takes into consideration the number of found terms, their frequency in a text, and the length of a text. The shorter the input text is, the higher scores will be:

(: Score values: 1 0.62 0.45 :)

for $text in ("A", "A B", "A B C")

order by $score descending

return <hit score='{ format-number($score, "0.00") }'>{ $text }</hit>

This simple approach has proven to consistently deliver good results, in particular when little is known about the structure of the queried XML documents.

Scoring values can be further processed to compute custom values:

let $terms := ('a', 'b')

let $scores := ft:score($terms ! ('a b c' contains text { . }))

return avg($scores)

let $string := 'a b'

for $n score $s in ft:search('factbook', 'orthodox')

==Thesaurus==

using thesaurus default

'computers' contains text 'hardware'

==Fuzzy Querying==

'''Document 'doc.xml'''':

<doc>

<a>house</a>

<a>haus</a>

</doc>

'''Query:'''

//a[text() contains text 'house' using fuzzy]

'''Result:'''

<a>house</a>

<a>hous</a>

=Mixed Content=

When working with so-called narrative XML documents, such as HTML, [https://tei-c.org/ TEI], or [https://docbook.org/ DocBook] documents, you typically have ''mixed content'', i.e., elements containing a mix of text and markup, such as:

<p>This is only an illustrative <hi>example</hi>, not a <q>real</q> text.</p>

Since the logical flow of the text is not interrupted by the child elements, you will typically want to search across elements, so that the above paragraph would match a search for “real text”. For more examples, see [https://www.w3.org/TR/xpath-full-text-10-use-cases/#Across XQuery and XPath Full Text 1.0 Use Cases].

To enable this kind of searches, it is recommendable to:

(: Structure is ignored; no highlighting: :)

ft:mark(//p[. contains text 'real'])

(: Single text nodes are addressed: results will be highlighted: :)

ft:mark(//p[.//text() contains text 'real'])

return db:create(

'index-db',

map { 'ftindex': true() }

)

=Functions=

* If a default collation is specified, it applies to all collation-dependent string operations in the query. The following expression yields <code>true</code>:

declare default collation 'http://basex.org/collation?lang=de;strength=secondary';

'Straße' = 'Strasse'

* Collations can also be specified in {{Code|order by}} and {{Code|group by}} clauses of FLWOR expressions. This query returns {{Code|à plutôt! bonjour!}}:

for $w in ("bonjour!", "à plutôt!") order by $w collation "?lang=fr" return $w

* Various string function exists that take an optional collation as argument: The following functions give us {{Code|a}} and {{Code|1 2 3}} as results:

distinct-values(("a", "á", "à"), "?lang=it-IT;strength=primary"),

index-of(("a", "á", "à"), "a", "?lang=it-IT;strength=primary")

If the [http://site.icu-project.org/download ICU Library] is added to the classpath, the full [https://www.w3.org/TR/xpath-functions-31/#uca-collations Unicode Collation Algorithm] features become available:

(: returns 0 (both strings are compared as equal) :)

compare('a-b', 'ab', 'http://www.w3.org/2013/collation/UCA?alternate=shifted')

=Changelog=

; Version 9.2:

* Added: Arabic stemmer.

; Version 8.0:

* Updated: [[#Scoring|Scores]] will be propagated by the {{Code|and}} and {{Code|or}} expressions and in predicates.

; Version 7.7:

* Added: [[#Collations|Collations]] support.

; Version 7.3:

* Removed: Trie index, which was specialized on wildcard queries. The fuzzy index now supports both wildcard and fuzzy queries.

* Removed: TF/IDF scoring was discarded in favor of the internal scoring model.