Bureaucrats, editor, reviewer, Administrators
13,554
edits
Changes
Jump to navigation
Jump to search
==Full-Text Features== ===Options===
===Languages===
===Scoring===
===Thesaurus===
===Fuzzy Querying===
==Mixed Content==
==Functions==
==Collations==
====Examples====
=Changelog=
no edit summary
Full-text retrieval is an essential query feature for working with XML documents, and BaseX was the first query processor that fully supported the [http://www.w3.org/TR/xpath-full-text-10/ W3C XQuery Full Text 1.0] Recommendation. This page lists some singularities and extensions of the BaseX implementation.
=Features=Query Evaluation== BaseX offers different evaluation strategies for XQFT queries, the choice of whichdepends on the input data and the existence of a full text index. The query compiler triesto optimize and speed up queries by applying a full text index structure wheneverpossible and useful. Three evaluation strategies are available: the standard sequentialdatabase scan, a full-text index based evaluation and a hybrid one, combining both strategies (see [http://www.inf.uni-konstanz.de/gk/pubsys/publishedFiles/GrGaHo09.pdf XQuery Full Text implementation in BaseX]). Query optimization and selection of the most efficient evaluation strategy is donein a full-fledged automatic manner. The output of the query optimizer indicates whichevaluation plan is chosen for a specific query. It can be inspected by activating verbosequerying (Command: <code>SET VERBOSE ON</code>) or opening the Query Info in the GUI.The message <code>Applying full-text index</code> suggests that the full-text index is applied to speed up query evaluation.A second message <code>Removing path with no index results</code> indicates that the index does not yield any results for the specified term andis thus skipped. If index optimizations are missing, it sometimes helps to givethe compiler a second chance and try different rewritings of the same query.
The available full-text index can handle various combinations of the match options defined in the XQuery Full Text Recommendation. By default, most options are disabled. The GUI dialogs for creating new databases or displaying the database properties contain a tab for choosing between all available options. On the command-line, the <code>SET</code> command can be used to activate full-text indexing or creating a full-text index for existing databases:
* '''Stopword List''': a stop word list can be defined to reduce the number of indexed tokens (<code>SET STOPWORDS [filename]</code>).
The chosen language determines how the input text will be tokenized and stemmed. The basic code base and <code>jar</code> file of BaseX comes with built-in support for English and German. More languages are supported if the following libraries are found in the classpath:
</pre>
The XQuery Full Text Recommendation allows for the usage of scoring models
Terms will be ranked higher if they are found in short text nodes.
BaseX supports full-text queries using thesauri, but it does not provide a default thesaurus. This is why query such as
The format of the thesaurus files must be the same as the format of the thesauri provided by the [http://dev.w3.org/2007/xpath-full-text-10-test-suite XQuery and XPath Full Text 1.0 Test Suite]. It is an XML with structure defined by an [http://dev.w3.org/cvsweb/~checkout~/2007/xpath-full-text-10-test-suite/TestSuiteStagingArea/TestSources/thesaurus.xsd?rev=1.3;content-type=application%2Fxml XSD Schema].
In addition to the official recommendation, BaseX supports fuzzy querying.
“house” and “hous”.
=Performance= ==Index Processing==Performance BaseX offers different evaluation strategies for XQFT queries, the choice of whichdepends on the input data and the existence of a full text index. The query compiler triesto optimize and speed up queries by applying a full text index structure wheneverpossible and useful. Three evaluation strategies are available: the standard sequentialdatabase scan, a full-text index based evaluation and a hybrid one, combining both strategies (see [http://www.inf.uni-konstanz.de/gk/pubsys/publishedFiles/GrGaHo09.pdf XQuery Full Text implementation in BaseX]). Query optimization and selection of the most efficient evaluation strategy is donein a full-fledged automatic manner. The output of the query optimizer indicates whichevaluation plan is chosen for a specific query. It can be inspected by activating verbosequerying (Command: <code>SET VERBOSE ON</code>) or opening the Query Info in the GUI.The message <code>Applying full-text index</code> suggests that the full-text index is applied to speed up query evaluation.A second message <code>Removing path with no index results</code> indicates that the index does not yield any results for the specified term andis thus skipped. If index optimizations are missing, it sometimes helps to givethe compiler a second chance and try different rewritings of the same query. ==FTAnd==
The internal XQuery Full Text data model is pretty complex and may consume more main
</pre>
When working with so-called narrative XML documents, such as HTML, [http://tei-c.org/ TEI], or [http://docbook.org DocBook] documents, you typically have ''mixed content'', i.e., elements containing a mix of text and markup, such as:
As BaseX does not support the ignore option, it raises error [[XQuery_Errors#Full-Text_Errors|FTST0007]] when it encounters <code>without content</code> in a full-text <code>contains</code> expression.
Some additional [[Full-Text Module|Full-Text Functions]] have been added to BaseX to extend the official language recommendation with useful features, such as explicitly requesting the score value of an item, marking the hits of a full-text request, or directly accessing the full-text index with the default index options.
{{Mark|Introduced with Version 7.7:}}
|}
If a default collation is specified, it applies to all collation-dependent string operations in the query. The following expression yields <code>true</code>:
</nowiki></pre>
; Version 7.7:
