Changes

Please read the separate [[Indexes#Full-Text Index|Full-Text Index]] section in our documentation if you want to learn how to evaluate full-text requests on large databases within milliseconds.

This is a simple example for a basic full-text expression:

"This is YOUR World" contains text "your world"

</pre>

It yields {{Code|true}}, because the search string is ''tokenized'' before it is compared with the tokenized input string. In the tokenization process, several normalizations take place. Many of those steps can hardly be simulated with plain XQuery: as an example, upper/lower case and diacritics (umlauts, accents, etc.) are removed and an optional, language-dependent stemming algorithm is applied. Beside that, special characters such as whitespaces and punctuation marks will be ignored. Thus, this query also yields true:

"Well... Done!" contains text "well, done"

</pre>

The {{Code|occurs}} keyword comes into play when more than one occurrence of a token is to be found:

"one and two and three" contains text "and" occurs at least 2 times

</pre>

In the given example, curly braces are used to combine multiple keywords:

for $country in doc('factbook')//country

where $country//religions[text() contains text { 'Sunni', 'Shia' } any]

The keywords {{Code|ftand}}, {{Code|ftor}} and {{Code|ftnot}} can also be used to combine multiple query terms. The following query yields the same result as the last one does:

doc('factbook')//country[descendant::religions contains text 'sunni' ftor 'shia']/name

</pre>

The keywords {{Code|not in}} are special: they are used to find tokens which are not part of a longer token sequence:

for $text in ("New York", "new conditions")

return $text contains text "New" not in "New York"

Due to the complex data model of the XQuery Full Text spec, the usage of {{Code|ftand}} may lead to a high memory consumption. If you should encounter problems, simply use the {{Code|all}} keyword:

doc('factbook')//country[descendant::religions contains text { 'Christian', 'Jewish'} all]/name

</pre>

A popular retrieval operation is to filter texts by the distance of the searched words. In this query…

<xml>

<text>There is some reason why ...</text>

The {{Code|window}} keyword is related: it accepts those texts in which all keyword occur within the specified number of tokens. Can you guess what is returned by the following query?

("A C D", "A B C D E")[. contains text { "A", "E" } all window 3 words]

</pre>

Sometimes it is interesting to only select texts in which all searched terms occur in the {{Code|same sentence}} or {{Code|paragraph}} (you can even filter for {{Code|different}} sentences/paragraphs). This is obviously not the case in the following example:

'Mary told me, “I will survive!”.' contains text { 'will', 'told' } all words same sentence

</pre>

* If {{Code|case}} is insensitive, no distinction is made between characters in upper and lower case. By default, the option is {{Code|insensitive}}; it can also be set to {{Code|sensitive}}:

"Respect Upper Case" contains text "Upper" using case sensitive

</pre>

* If {{Code|diacritics}} is insensitive, characters with and without diacritics (umlauts, characters with accents) are declared as identical. By default, the option is {{Code|insensitive}}; it can also be set to {{Code|sensitive}}:

"'Äpfel' will not be found..." contains text "Apfel" using diacritics sensitive

</pre>

* If {{Code|stemming}} is activated, words are shortened to a base form by a language-specific stemmer:

"catch" contains text "catches" using stemming

</pre>

* With the {{Code|stop words}} option, a list of words can be defined that will be ignored when tokenizing a string. This is particularly helpful if the full-text index takes too much space (a standard stopword list for English texts is provided in the directory {{Code|etc/stopwords.txt}} in the full distributions of BaseX, and available online at http://files.basex.org/etc/stopwords.txt):

"You and me" contains text "you or me" using stop words ("and", "or"),

"You and me" contains text "you or me" using stop words at "http://files.basex.org/etc/stopwords.txt"

* <code>.{min,max}</code> matches ''min''–''max'' number of characters.

"This may be interesting in the year 2000" contains text { "interest.*", "2.{3,3}" } using wildcards

</pre>

A list of all language codes that are available on your system can be retrieved as follows:

declare namespace locale = "java:java.util.Locale";

distinct-values(locale:getAvailableLocales() ! locale:getLanguage(.))

The basic JAR file of BaseX comes with built-in stemming support for English, German, Greek and Indonesian. Some more languages are supported if the following libraries are found in the [[Startup#Distributions|classpath]]:

The JAR files are included in the ZIP and EXE distributions of BaseX.

The following two queries, which both return <code>true</code>, demonstrate that stemming depends on the selected language:

"Indexing" contains text "index" using stemming,

"häuser" contains text "haus" using stemming using language "German"

The scoring model of BaseX takes into consideration the number of found terms, their frequency in a text, and the length of a text. The shorter the input text is, the higher scores will be:

(: Score values: 1 0.62 0.45 :)

order by $score descending

return <hit score='{ format-number($score, "0.00") }'>{ $text }</hit>

</pre>

</pre>

==Thesaurus==

using thesaurus default

</pre>

'computers' contains text 'hardware'

</pre>

==Fuzzy Querying==

'''Document 'doc.xml'''':

<doc>

<a>house</a>

'''Query:'''

//a[text() contains text 'house' using fuzzy]

</pre>

'''Result:'''

<a>house</a>

<a>hous</a>

</pre>

=Mixed Content=

<p>This is only an illustrative <hi>example</hi>, not a <q>real</q> text.</p>

</pre>

To enable this kind of searches, it is recommendable to:

(: Structure is ignored; no highlighting: :)

ft:mark(//p[. contains text 'real'])

</pre>

return db:create(

'index-db',

|-

| {{Code|decomposition}}

|}

* If a default collation is specified, it applies to all collation-dependent string operations in the query. The following expression yields <code>true</code>:

declare default collation 'http://basex.org/collation?lang=de;strength=secondary';

'Straße' = 'Strasse'

* Collations can also be specified in {{Code|order by}} and {{Code|group by}} clauses of FLWOR expressions. This query returns {{Code|à plutôt! bonjour!}}:

for $w in ("bonjour!", "à plutôt!") order by $w collation "?lang=fr" return $w

</pre>

* Various string function exists that take an optional collation as argument: The following functions give us {{Code|a}} and {{Code|1 2 3}} as results:

distinct-values(("a", "á", "à"), "?lang=it-IT;strength=primary"),

index-of(("a", "á", "à"), "a", "?lang=it-IT;strength=primary")

</nowiki></pre>

=Changelog=

; Version 9.2:

* Added: Arabic stemmer.

; Version 8.0:

* Updated: [[#Scoring|Scores]] will be propagated by the {{Code|and}} and {{Code|or}} expressions and in predicates.

; Version 7.7:

* Added: [[#Collations|Collations]] support.

; Version 7.3:

* Removed: Trie index, which was specialized on wildcard queries. The fuzzy index now supports both wildcard and fuzzy queries.

* Removed: TF/IDF scoring was discarded in favor of the internal scoring model.