Bureaucrats, editor, reviewer, Administrators
13,554
edits
Changes
Jump to navigation
Jump to search
m
==Introduction==The lexer was contributed by Toshio HIRAI.
The lexical analysis of Japanese documents is performed by [http://igo.sourceforge.jp/ Igo]. Igo is a ''morphological analyser'',and some of the advantages and reasons for using Igo are:* compatible with the results of a prominent morphological analyzer "MeCab"* it can use the dictionary distributed by the Project MeCab* the morphological analyzer is implemented in Java and is relatively fast=Introduction=
In addition to * Compatible with the library, one results of a prominent morphological analyzer "MeCab".* It can use the following dictionary files must either be unzipped into distributed by the current directory, or into the <code>etc</code> sub-directory of the project’s [[Configuration#Home Directory|Home Directory]]:* IPA Dictionary: http://files.basex.org/etc/ipadicProject MeCab.zip* NAIST Dictionary: http://files.basex.org/etc/naistdicThe morphological analyzer is implemented in Java and is relatively fast.zip
=Japanese tokenization will be activated in BaseX if Igo is found in the classpath. [https://osdn.net/projects/igo/releases/ igo-0.4.3.jar] of Igo is currently included in all distributions of BaseX. In addition to the library, one of the following dictionary files must either be unzipped into the current directory, or into the <code>etc</code> subdirectory of the project’s [[Configuration#Home Directory|Home Directory]]: * IPA Dictionary: https://files.basex.org/etc/ipadic.zip* NAIST Dictionary: https://files.basex.org/etc/naistdic.zip =Lexical Analysis==
==Parsing==
==Token Processing== "Fullwidth" and "Halfwidth" (which is defined by [https://unicode.org/Public/UNIDATA/EastAsianWidth.txt East Asian Width Properties]) are not distinguished (this is the so-called ZENKAKU/HANKAKU problem).
"Fullwidth" and "Halfwidth" (which is defined by[http://unicode.org/Public/UNIDATA/EastAsianWidth.txt East Asian Width Properties])are not distinguished (this is the so-called ZENKAKU/HANKAKU problem).For example, <code>ＸＭＬ</code> and <code>XML</code> will be treatedas the same word. If documents are ''hybrid'', i.e. written in multiple languages,this is also helpful for some other options of the XQuery Full Text Specification,such as the [httphttps://www.w3.org/TR/xpath-full-text-10/#ftcaseoption Case] or the[httphttps://www.w3.org/TR/xpath-full-text-10/#ftdiacriticsoption Diacritics] Optionoption.
==Stemming==
==Wildcards==
Text replacement - "<syntaxhighlight lang="xquery">" to "<pre lang='xquery'>"
This article is linked from the [[Full-Text]] page. It gives some insight into the implementation of the full-text features for Japanese text corpora. The Japanese version is [httphttps://files.basex.org/etc/ja-ft.pdf also available as PDF].Thank you to [http://blog.infinite.jp Toshio HIRAI] for integrating the lexer in BaseX!
The lexical analysis of Japanese tokenization will be activated in BaseX if Igo documents is found in theclasspath. performed by [httphttps://enigo.sourceforgeosdn.jp/projects/igo/releases/ igo-0Igo].4.3.jar]of Igo is currently included in all distributions a ''morphological analyser'', and some of BaseX.the advantages and reasons for using Igo are:
The example sentence "私は本を書きました。(I wrote a book.)"
is analyzed as follows.
<pre>私は本を書きました。
私 名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ
は 助詞,係助詞,*,*,*,*,は,ハ,ワ
The Morpheme component is built as follows:
<pre>品詞,品詞細分類1,品詞細分類2,品詞細分類3,活用形,活用型,原形,読み,発音
(POS, subtyping POS 1, subtyping POS 2, subtyping POS 3, inflections, use type, prototype, reading, pronunciation)
</pre>
morpheme are used in indexing and stemming.
During indexing and parsing, the input strings are split into single ''tokens''.
for each token.
Stemming in Japanese means to analyze the results of morphological analysis
is returned for the following two types of queries:
<pre classlang="brush:'xquery"'>
'私は本を書いた' contains text '書く' using stemming using language 'ja'
'私は本を書く' contains text '書いた' using stemming using language 'ja'
</pre>
The Wildcard option in XQuery Full-Text is available for Japanese as well.
queries both return <code>true</code>:
<pre classlang="brush:'xquery"'>
'芥川龍之介' contains text '.之介' using wildcards using language 'ja'
'芥川竜之介' contains text '.之介' using wildcards using language 'ja'
query will yield <code>false</code>:
<pre classlang="brush:'xquery"'>
'芥川龍之介' contains text '芥川.之介' using wildcards using language 'ja'
</pre>
an additional whitespaces as word boundary:
<pre classlang="brush:'xquery"'>
'芥川龍之介' contains text '芥川 .之介' using wildcards using language 'ja'
</pre>
As an alternative, you may modify the query as follows:
<pre classlang="brush:'xquery"'>
'芥川龍之介' contains text '芥川' ftand '.之介' using wildcards using language 'ja'
</pre>
