Storage Layout - Revision history

CG: Text replacement - "
" to "
"

2023-03-09T08:17:24Z

Text replacement - "<br />" to "<br/>"

CG at 13:42, 25 July 2022

2022-07-25T13:42:21Z

CG at 08:20, 21 July 2022

2022-07-21T08:20:54Z

James Ball: Added information regarding pth.basex and idp.basex

2020-12-12T22:52:28Z

Added information regarding pth.basex and idp.basex

CG: /* Value Indexes: {{Code|txtl}}, {{Code|txtr}}, {{Code|atvl}}, {{Code|atvr}} */

2016-02-05T17:06:18Z

Value Indexes: {{Code|txtl}}, {{Code|txtr}}, {{Code|atvl}}, {{Code|atvr}}

CG at 14:10, 19 January 2016

2016-01-19T14:10:42Z

CG: Text replace - "\[\[Category:Internals\]\]" to ""

2016-01-17T13:59:15Z

Text replace - "\[\[Category:Internals\]\]" to ""

James Ball: Provided a link to the Index File Structure page giving more details of the layout of those files.

2014-08-22T14:51:51Z

Provided a link to the Index File Structure page giving more details of the layout of those files.

Prakash at 13:16, 15 August 2014

2014-08-15T13:16:12Z

Prakash: basex-core is added in github link

2014-08-15T13:12:38Z

basex-core is added in github link

@@ Line 48: / Line 48: @@
 |- valign="top"
 | '''1. Metadata'''
-| 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br/>&nbsp; &bull; Examples: {{Code|FNAME}}, {{Code|TIME}}, {{Code|SIZE}}, ...<br />&nbsp; &bull; {{Code|PERM}} → Number of users ({{Type|Num}}), and name/password/permission values for each user ({{Type|Token}}/{{Type|Token}}/{{Type|Num}})<br/>2. Empty key as finalizer
+| 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br/>&nbsp; &bull; Examples: {{Code|FNAME}}, {{Code|TIME}}, {{Code|SIZE}}, ...<br/>&nbsp; &bull; {{Code|PERM}} → Number of users ({{Type|Num}}), and name/password/permission values for each user ({{Type|Token}}/{{Type|Token}}/{{Type|Num}})<br/>2. Empty key as finalizer
 |- valign="top"
 | '''2. Main memory indexes'''
-| 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br />&nbsp; &bull; {{Code|TAGS}} → Element Name Index<br />&nbsp; &bull; {{Code|ATTS}} → Attribute Name Index<br />&nbsp; &bull; {{Code|PATH}} → Path Index<br />&nbsp; &bull; {{Code|NS}} → Namespaces<br />&nbsp; &bull; {{Code|DOCS}} → Document Index<br/>2. Empty key as finalizer
+| 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br/>&nbsp; &bull; {{Code|TAGS}} → Element Name Index<br/>&nbsp; &bull; {{Code|ATTS}} → Attribute Name Index<br/>&nbsp; &bull; {{Code|PATH}} → Path Index<br/>&nbsp; &bull; {{Code|NS}} → Namespaces<br/>&nbsp; &bull; {{Code|DOCS}} → Document Index<br/>2. Empty key as finalizer
 |- valign="top"
 | '''2 a) Name Index'''<br/>Element/attribute names
-| 1. Token set, storing all names ({{Type|TokenSet}})<br />2. One StatsKey instance per entry:<br/>2.1. Content kind ({{Type|Num}}):<br />2.1.1. Number: min/max ({{Type|Doubles}})<br />2.1.2. Category: number of entries ({{Type|Num}}), entries ({{Type|Tokens}})<br />2.2. Number of entries ({{Type|Num}})<br />2.3. Leaf flag ({{Type|Boolean}})<br />2.4. Maximum text length ({{Type|Double}}; legacy, could be {{Type|Num}})
+| 1. Token set, storing all names ({{Type|TokenSet}})<br/>2. One StatsKey instance per entry:<br/>2.1. Content kind ({{Type|Num}}):<br/>2.1.1. Number: min/max ({{Type|Doubles}})<br/>2.1.2. Category: number of entries ({{Type|Num}}), entries ({{Type|Tokens}})<br/>2.2. Number of entries ({{Type|Num}})<br/>2.3. Leaf flag ({{Type|Boolean}})<br/>2.4. Maximum text length ({{Type|Double}}; legacy, could be {{Type|Num}})
 |- valign="top"
 | '''2 b) Path Index'''

@@ Line 40: / Line 40: @@
 The following tables illustrate the layout of the BaseX database files. All files are suffixed with {{Code|.basex}}.
-==Meta Data, Name/Path/Doc Indexes: {{Code|inf}}==
+==Metadata, Name/Path/Doc Indexes: {{Code|inf}}==
 {| class="wikitable"
@@ Line 47: / Line 47: @@
 ! Format
 |- valign="top"
-| '''1. Meta Data'''
+| '''1. Metadata'''
 | 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br/>&nbsp; &bull; Examples: {{Code|FNAME}}, {{Code|TIME}}, {{Code|SIZE}}, ...<br />&nbsp; &bull; {{Code|PERM}} → Number of users ({{Type|Num}}), and name/password/permission values for each user ({{Type|Token}}/{{Type|Token}}/{{Type|Num}})<br/>2. Empty key as finalizer
 |- valign="top"

@@ Line 5: / Line 5: @@
 The following data types are used for specifying the storage layout:
-{| class="wikitable" width="100%"
+{| class="wikitable"
-|-
+|- valign="top"
 ! Type
 ! Description
 ! Example (native → hex integers)
-|-
+|- valign="top"
 | {{Type|Num}}
 | Compressed integer (1-5 bytes), specified in [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/util/Num.java Num.java]
 | {{Code|15}} → {{Code|0F}}; {{Code|511}} → {{Code|41 FF}}<br/>
-|-
+|- valign="top"
 | {{Type|Token}}
 | Length ({{Type|Num}}) and bytes of UTF8 byte representation
 | {{Code|Hello}} → {{Code|05 48 65 6c 6c 6f}}
-|-
+|- valign="top"
 | {{Type|Double}}
 | Number, stored as token
 | {{Code|123}} → {{Code|03 31 32 33}}
-|-
+|- valign="top"
 | {{Type|Boolean}}
 | Boolean (1 byte, {{Code|00}} or {{Code|01}})
 | {{Code|true}} → {{Code|01}}
-|-
+|- valign="top"
 | {{Type|Nums}}, {{Type|Tokens}}, {{Type|Doubles}}
 | Arrays of values, introduced with the number of entries
 | {{Code|1,2}} → {{Code|02 01 31 01 32}}
-|-
+|- valign="top"
 | {{Type|TokenSet}}
 | Key array ({{Type|Tokens}}), next/bucket/size arrays (3x {{Type|Nums}})
@@ Line 42: / Line 42: @@
 ==Meta Data, Name/Path/Doc Indexes: {{Code|inf}}==
-{| class="wikitable" width="100%"
+{| class="wikitable"
-|-
+|- valign="top"
 ! Description
 ! Format
-! Method
+|- valign="top"
 | '''1. Meta Data'''
 | 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br/>&nbsp; &bull; Examples: {{Code|FNAME}}, {{Code|TIME}}, {{Code|SIZE}}, ...<br />&nbsp; &bull; {{Code|PERM}} → Number of users ({{Type|Num}}), and name/password/permission values for each user ({{Type|Token}}/{{Type|Token}}/{{Type|Num}})<br/>2. Empty key as finalizer
-| [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/data/DiskData.java DiskData()]<br/>[https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/data/MetaData.java MetaData()]<br/>[https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/core/Users.java Users()]
+|- valign="top"
 | '''2. Main memory indexes'''
 | 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br />&nbsp; &bull; {{Code|TAGS}} → Element Name Index<br />&nbsp; &bull; {{Code|ATTS}} → Attribute Name Index<br />&nbsp; &bull; {{Code|PATH}} → Path Index<br />&nbsp; &bull; {{Code|NS}} → Namespaces<br />&nbsp; &bull; {{Code|DOCS}} → Document Index<br/>2. Empty key as finalizer
-| [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/data/DiskData.java DiskData()]
+|- valign="top"
 | '''2 a) Name Index'''<br/>Element/attribute names
 | 1. Token set, storing all names ({{Type|TokenSet}})<br />2. One StatsKey instance per entry:<br/>2.1. Content kind ({{Type|Num}}):<br />2.1.1. Number: min/max ({{Type|Doubles}})<br />2.1.2. Category: number of entries ({{Type|Num}}), entries ({{Type|Tokens}})<br />2.2. Number of entries ({{Type|Num}})<br />2.3. Leaf flag ({{Type|Boolean}})<br />2.4. Maximum text length ({{Type|Double}}; legacy, could be {{Type|Num}})
-| [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/index/Names.java Names()]<br/>[https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/util/hash/TokenSet.java TokenSet.read()]<br/>[https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/index/StatsKey.java StatsKey()]
+|- valign="top"
 | '''2 b) Path Index'''
 | 1. Flag for path definition ({{Type|Boolean}}, always {{Code|true}}; legacy)<br/>2. PathNode:<br/>2.1. Name reference ({{Type|Num}})<br/>2.2. Node kind ({{Type|Num}})<br/>2.3. Number of occurrences ({{Type|Num}})<br/>2.4. Number of children ({{Type|Num}})<br/>2.5. {{Type|Double}}; legacy, can be reused or discarded<br/>2.6. Recursive generation of child nodes (→ 2)
-| [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/index/path/PathSummary.java PathSummary()]<br/>[https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/index/path/PathNode.java PathNode()]
+|- valign="top"
 | '''2 c) Namespaces'''
 | 1. Token set, storing prefixes ({{Type|TokenSet}})<br/>2. Token set, storing URIs ({{Type|TokenSet}})<br/>3. NSNode:<br/>3.1. pre value ({{Type|Num}})<br/>3.2. References to prefix/URI pairs ({{Type|Nums}})<br/>3.3. Number of children ({{Type|Num}})<br/>3.4. Recursive generation of child nodes (→ 3)
-| [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/data/Namespaces.java Namespaces()]<br/>[https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/data/NSNode.java NSNode()]
+|- valign="top"
 | '''2 d) Document Index'''
 | Array of integers, representing the distances between all document pre values ({{Type|Nums}})
 |}

@@ Line 93: / Line 93: @@
 For a more detailed discussion and examples of these file formats please see [[Index File Structure]].
 ==Full-Text Fuzzy Index: {{Code|ftxx}}, {{Code|ftxy}}, {{Code|ftxz}}==
 ...may soon be reimplemented.

@@ Line 90: / Line 90: @@
 * {{Code|txtl}}: Heap file with ID lists.
 * {{Code|txtr}}: Index file with references to ID lists.
-The '''Attribute Index''' is contained in the files {{Code|atvl}} and {{Code|atvr}}; it uses the same layout.
+The '''Attribute Index''' is contained in the files {{Code|atvl}} and {{Code|atvr}}, the '''Token Index''' in {{Code|tokl}} and {{Code|tokr}}. All have the same layout.
 For a more detailed discussion and examples of these file formats please see [[Index File Structure]].

← Older revision		Revision as of 13:59, 17 January 2016
Line 97:		Line 97:

	...may soon be reimplemented.		...may soon be reimplemented.
−
−	~~[[Category:Internals]]~~

@@ Line 54: / Line 54: @@
 | '''2. Main memory indexes'''
 | 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br />&nbsp; &bull; {{Code|TAGS}} → Tag Index<br />&nbsp; &bull; {{Code|ATTS}} → Attribute Name Index<br />&nbsp; &bull; {{Code|PATH}} → Path Index<br />&nbsp; &bull; {{Code|NS}} → Namespaces<br />&nbsp; &bull; {{Code|DOCS}} → Document Index<br/>2. Empty key as finalizer
-| [https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/data/DiskData.java DiskData()]
+| [https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/basex/data/DiskData.java DiskData()]
 |-
 | '''2 a) Name Index'''<br/>Tag/attribute names