Full-Text: Japanese - Revision history

CG: Text replacement - "" to "

2023-12-01T16:36:11Z

Text replacement - "<syntaxhighlight lang="xquery">" to "<pre lang='xquery'>"

CG: Text replacement - "" to ""

2023-12-01T16:32:10Z

Text replacement - "</syntaxhighlight>" to "</pre>"

CG: Text replacement - "ub-director" to "ubdirector"

2022-07-25T13:59:29Z

Text replacement - "ub-director" to "ubdirector"

CG at 10:16, 15 February 2021

2021-02-15T10:16:56Z

CG at 11:56, 2 July 2020

2020-07-02T11:56:32Z

CG: /* Introduction */

2020-07-02T11:37:04Z

Introduction

CG: /* Token Processing */

2020-07-02T11:35:23Z

Token Processing

CG: /* Introduction */

2020-07-02T11:22:20Z

Introduction

CG: Text replacement - "[http://files.basex.org/" to "[https://files.basex.org/"

2020-07-02T10:45:47Z

Text replacement - "[http://files.basex.org/" to "[https://files.basex.org/"

CG: Text replacement - "[http://www.w3.org/TR/xpath" to "[https://www.w3.org/TR/xpath"

2020-07-02T10:34:13Z

Text replacement - "[http://www.w3.org/TR/xpath" to "[https://www.w3.org/TR/xpath"

@@ Line 86: / Line 86: @@
 is returned for the following two types of queries:
-<syntaxhighlight lang="xquery">
+<pre lang='xquery'>
 '私は本を書いた' contains text '書く' using stemming using language 'ja'
 '私は本を書く' contains text '書いた' using stemming using language 'ja'
@@ Line 98: / Line 98: @@
 queries both return <code>true</code>:
-<syntaxhighlight lang="xquery">
+<pre lang='xquery'>
 '芥川龍之介' contains text '.之介' using wildcards using language 'ja'
 '芥川竜之介' contains text '.之介' using wildcards using language 'ja'
@@ Line 106: / Line 106: @@
 query will yield <code>false</code>:
-<syntaxhighlight lang="xquery">
+<pre lang='xquery'>
 '芥川龍之介' contains text '芥川.之介' using wildcards using language 'ja'
 </pre>
@@ Line 114: / Line 114: @@
 an additional whitespaces as word boundary:
-<syntaxhighlight lang="xquery">
+<pre lang='xquery'>
 '芥川龍之介' contains text '芥川　.之介' using wildcards using language 'ja'
 </pre>
@@ Line 120: / Line 120: @@
 As an alternative, you may modify the query as follows:
-<syntaxhighlight lang="xquery">
+<pre lang='xquery'>
 '芥川龍之介' contains text '芥川' ftand '.之介' using wildcards using language 'ja'
 </pre>

@@ Line 89: / Line 89: @@
 '私は本を書いた' contains text '書く' using stemming using language 'ja'
 '私は本を書く' contains text '書いた' using stemming using language 'ja'
-</syntaxhighlight>
+</pre>
 =Wildcards=
@@ Line 101: / Line 101: @@
 '芥川龍之介' contains text '.之介' using wildcards using language 'ja'
 '芥川竜之介' contains text '.之介' using wildcards using language 'ja'
-</syntaxhighlight>
+</pre>
 However, there is a special case that requires attention. The following
@@ Line 108: / Line 108: @@
 <syntaxhighlight lang="xquery">
 '芥川龍之介' contains text '芥川.之介' using wildcards using language 'ja'
-</syntaxhighlight>
+</pre>
 This is because the next word boundary metacharacters
@@ Line 116: / Line 116: @@
 <syntaxhighlight lang="xquery">
 '芥川龍之介' contains text '芥川　.之介' using wildcards using language 'ja'
-</syntaxhighlight>
+</pre>
 As an alternative, you may modify the query as follows:
@@ Line 122: / Line 122: @@
 <syntaxhighlight lang="xquery">
 '芥川龍之介' contains text '芥川' ftand '.之介' using wildcards using language 'ja'
-</syntaxhighlight>
+</pre>

@@ Line 13: / Line 13: @@
 Japanese tokenization will be activated in BaseX if Igo is found in the classpath. [https://osdn.net/projects/igo/releases/ igo-0.4.3.jar] of Igo is currently included in all distributions of BaseX.
-In addition to the library, one of the following dictionary files must either be unzipped into the current directory, or into the <code>etc</code> sub-directory of the project’s [[Configuration#Home Directory|Home Directory]]:
+In addition to the library, one of the following dictionary files must either be unzipped into the current directory, or into the <code>etc</code> subdirectory of the project’s [[Configuration#Home Directory|Home Directory]]:
 * IPA Dictionary: https://files.basex.org/etc/ipadic.zip

@@ Line 23: / Line 23: @@
 is analyzed as follows.
-<syntaxhighlight>
+<pre>
 私は本を書きました。
 私      名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ
@@ Line 33: / Line 33: @@
 た      助動詞,*,*,*,特殊・タ,基本形,た,タ,タ
 。      記号,句点,*,*,*,*,。,。,。
-</syntaxhighlight>
+</pre>
 The element of the decomposed part is called "Surface",
@@ Line 39: / Line 39: @@
 The Morpheme component is built as follows:
-<syntaxhighlight>
+<pre>
 品詞,品詞細分類1,品詞細分類2,品詞細分類3,活用形,活用型,原形,読み,発音
 (POS, subtyping POS 1, subtyping POS 2, subtyping POS 3, inflections, use type, prototype, reading, pronunciation)
-</syntaxhighlight>
+</pre>
 Of these, the surface is used as a token. Also, The contents of analysis of a
@@ Line 75: / Line 75: @@
 can be led back to the same prototype by analyzing their verb:
-<syntaxhighlight>
+<pre>
 書く    動詞,自立,*,*,五段・カ行イ音便,基本形,[書く],カク,カク
 書い    動詞,自立,*,*,五段・カ行イ音便,連用タ接続,[書く],カイ,カイ
 た      助動詞,*,*,*,特殊・タ,基本形,た,タ,タ
-</syntaxhighlight>
+</pre>
 Because the "auxiliary verb" is always excluded from the tokens, there is

@@ Line 1: / Line 1: @@
 This article is linked from the [[Full-Text]] page. It gives some insight into the implementation of the full-text features for Japanese text corpora. The Japanese version is [https://files.basex.org/etc/ja-ft.pdf also available as PDF].
-Thank you to [http://blog.infinite.jp Toshio HIRAI] for integrating the lexer in BaseX!
 =Introduction=
-The lexical analysis of Japanese documents is performed by [http://igo.osdn.jp/ Igo]. Igo is a ''morphological analyser'', and some of the advantages and reasons for using Igo are:
+The lexical analysis of Japanese documents is performed by [https://igo.osdn.jp/ Igo]. Igo is a ''morphological analyser'', and some of the advantages and reasons for using Igo are:
 * Compatible with the results of a prominent morphological analyzer "MeCab".

@@ Line 63: / Line 63: @@
 =Token Processing=
-"Fullwidth" and "Halfwidth" (which is defined by
+"Fullwidth" and "Halfwidth" (which is defined by [https://unicode.org/Public/UNIDATA/EastAsianWidth.txt East Asian Width Properties]) are not distinguished (this is the so-called ZENKAKU/HANKAKU problem).
-[http://unicode.org/Public/UNIDATA/EastAsianWidth.txt East Asian Width Properties])
-are not distinguished (this is the so-called ZENKAKU/HANKAKU problem).
+For example, <code>ＸＭＬ</code> and <code>XML</code> will be treated as the same word. If documents are ''hybrid'', i.e. written in multiple languages, this is also helpful for some other options of the XQuery Full Text Specification, such as the [https://www.w3.org/TR/xpath-full-text-10/#ftcaseoption Case] or the [https://www.w3.org/TR/xpath-full-text-10/#ftdiacriticsoption Diacritics] option.
 =Stemming=

@@ Line 5: / Line 5: @@
 The lexical analysis of Japanese documents is performed by
-[http://igo.sourceforge.jp/ Igo]. Igo is a ''morphological analyser'',
+[http://igo.osdn.jp/ Igo]. Igo is a ''morphological analyser'',
 and some of the advantages and reasons for using Igo are:
 * compatible with the results of a prominent morphological analyzer "MeCab"
@@ Line 16: / Line 16: @@
 In addition to the library, one of the following dictionary files must either be unzipped into the current directory, or into the <code>etc</code> sub-directory of the project’s [[Configuration#Home Directory|Home Directory]]:
-* IPA Dictionary: http://files.basex.org/etc/ipadic.zip
+* IPA Dictionary: https://files.basex.org/etc/ipadic.zip
-* NAIST Dictionary: http://files.basex.org/etc/naistdic.zip
+* NAIST Dictionary: https://files.basex.org/etc/naistdic.zip
 =Lexical Analysis=

← Older revision		Revision as of 10:45, 2 July 2020
Line 1:		Line 1:
−	This article is linked from the [[Full-Text]] page. It gives some insight into the implementation of the full-text features for Japanese text corpora. The Japanese version is [~~http~~://files.basex.org/etc/ja-ft.pdf also available as PDF].	+	This article is linked from the [[Full-Text]] page. It gives some insight into the implementation of the full-text features for Japanese text corpora. The Japanese version is [https://files.basex.org/etc/ja-ft.pdf also available as PDF].
	Thank you to [http://blog.infinite.jp Toshio HIRAI] for integrating the lexer in BaseX!		Thank you to [http://blog.infinite.jp Toshio HIRAI] for integrating the lexer in BaseX!