Difference between revisions of "Storage Layout"
Jump to navigation
Jump to search
Meta Data, Name/Path/Doc Indexes:
Node Table:
Texts:
Value Indexes:
Full-Text Fuzzy Index:
Full-Text Trie Index:
m (Text replace - "| valign='top' | " to "| ") |
m (Text replace - "{{Mono|" to "{{Code|") |
||
| Line 13: | Line 13: | ||
| {{Type|Num}} | | {{Type|Num}} | ||
| Compressed integer (1-5 bytes), specified in [https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/util/Num.java Num.java] | | Compressed integer (1-5 bytes), specified in [https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/util/Num.java Num.java] | ||
| − | | {{ | + | | {{Code|15}} → {{Code|0F}}; {{Code|511}} → {{Code|41 FF}}<br/> |
|- | |- | ||
| {{Type|Token}} | | {{Type|Token}} | ||
| Length ({{Type|Num}}) and bytes of UTF8 byte representation | | Length ({{Type|Num}}) and bytes of UTF8 byte representation | ||
| − | | {{ | + | | {{Code|Hello}} → {{Code|05 48 65 6c 6c 6f}} |
|- | |- | ||
| {{Type|Double}} | | {{Type|Double}} | ||
| Number, stored as token | | Number, stored as token | ||
| − | | {{ | + | | {{Code|123}} → {{Code|03 31 32 33}} |
|- | |- | ||
| {{Type|Boolean}} | | {{Type|Boolean}} | ||
| − | | Boolean (1 byte, {{ | + | | Boolean (1 byte, {{Code|00}} or {{Code|01}}) |
| − | | {{ | + | | {{Code|true}} → {{Code|01}} |
|- | |- | ||
| {{Type|Nums}}, {{Type|Tokens}}, {{Type|Doubles}} | | {{Type|Nums}}, {{Type|Tokens}}, {{Type|Doubles}} | ||
| Arrays of values, introduced with the number of entries | | Arrays of values, introduced with the number of entries | ||
| − | | {{ | + | | {{Code|1,2}} → {{Code|02 01 31 01 32}} |
|- | |- | ||
| {{Type|TokenSet}} | | {{Type|TokenSet}} | ||
| Line 38: | Line 38: | ||
=Database Files= | =Database Files= | ||
| − | The following tables illustrate the layout of the BaseX database files. All files are suffixed with {{ | + | The following tables illustrate the layout of the BaseX database files. All files are suffixed with {{Code|.basex}}. |
| − | ==Meta Data, Name/Path/Doc Indexes: {{ | + | ==Meta Data, Name/Path/Doc Indexes: {{Code|inf}}== |
{| class="wikitable" width="100%" | {| class="wikitable" width="100%" | ||
| Line 49: | Line 49: | ||
|- | |- | ||
| '''1. Meta Data''' | | '''1. Meta Data''' | ||
| − | | 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br/> • Examples: {{ | + | | 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br/> • Examples: {{Code|FNAME}}, {{Code|TIME}}, {{Code|SIZE}}, ...<br /> • {{Code|PERM}} → Number of users ({{Type|Num}}), and name/password/permission values for each user ({{Type|Token}}/{{Type|Token}}/{{Type|Num}})<br/>2. Empty key as finalizer |
| [https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/data/DiskData.java DiskData()]<br/>[https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/data/MetaData.java MetaData()]<br/>[https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/core/Users.java Users()] | | [https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/data/DiskData.java DiskData()]<br/>[https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/data/MetaData.java MetaData()]<br/>[https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/core/Users.java Users()] | ||
|- | |- | ||
| '''2. Main memory indexes''' | | '''2. Main memory indexes''' | ||
| − | | 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br /> • {{ | + | | 1. Key/value pairs, in no particular order ({{Type|Token}}/{{Type|Token}}):<br /> • {{Code|TAGS}} → Tag Index<br /> • {{Code|ATTS}} → Attribute Name Index<br /> • {{Code|PATH}} → Path Index<br /> • {{Code|NS}} → Namespaces<br /> • {{Code|DOCS}} → Document Index<br/>2. Empty key as finalizer |
| [https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/data/DiskData.java DiskData()] | | [https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/data/DiskData.java DiskData()] | ||
|- | |- | ||
| Line 61: | Line 61: | ||
|- | |- | ||
| '''2 b) Path Index''' | | '''2 b) Path Index''' | ||
| − | | 1. Flag for path definition ({{Type|Boolean}}, always {{ | + | | 1. Flag for path definition ({{Type|Boolean}}, always {{Code|true}}; legacy)<br/>2. PathNode:<br/>2.1. Name reference ({{Type|Num}})<br/>2.2. Node kind ({{Type|Num}})<br/>2.3. Number of occurrences ({{Type|Num}})<br/>2.4. Number of children ({{Type|Num}})<br/>2.5. {{Type|Double}}; legacy, can be reused or discarded<br/>2.6. Recursive generation of child nodes (→ 2) |
| [https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/index/path/PathSummary.java PathSummary()]<br/>[https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/index/path/PathNode.java PathNode()] | | [https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/index/path/PathSummary.java PathSummary()]<br/>[https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/index/path/PathNode.java PathNode()] | ||
|- | |- | ||
| Line 73: | Line 73: | ||
|} | |} | ||
| − | ==Node Table: {{ | + | ==Node Table: {{Code|tbl}}, {{Code|tbli}}== |
| − | * {{ | + | * {{Code|tbl}}: Main database table, stored in blocks. |
| − | * {{ | + | * {{Code|tbli}}: Database directory, organizing the database blocks. |
Some more information on the [[Node table storage]] is available. | Some more information on the [[Node table storage]] is available. | ||
| − | ==Texts: {{ | + | ==Texts: {{Code|txt}}, {{Code|atv}}== |
| − | * {{ | + | * {{Code|txt}}: Heap file for text values (document names, string values of texts, comments and processing instructions) |
| − | * {{ | + | * {{Code|atv}}: Heap file for attribute values. |
| − | ==Value Indexes: {{ | + | ==Value Indexes: {{Code|txtl}}, {{Code|txtr}}, {{Code|atvl}}, {{Code|atvr}}== |
'''Text Index:''' | '''Text Index:''' | ||
| − | * {{ | + | * {{Code|txtl}}: Heap file with ID lists. |
| − | * {{ | + | * {{Code|txtr}}: Index file with references to ID lists. |
| − | The '''Attribute Index''' is contained in the files {{ | + | The '''Attribute Index''' is contained in the files {{Code|atvl}} and {{Code|atvr}}; it uses the same layout. |
| − | ==Full-Text Fuzzy Index: {{ | + | ==Full-Text Fuzzy Index: {{Code|ftxx}}, {{Code|ftxy}}, {{Code|ftxz}}== |
...will soon be reimplemented. | ...will soon be reimplemented. | ||
| − | ==Full-Text Trie Index: {{ | + | ==Full-Text Trie Index: {{Code|ftxa}}, {{Code|ftxb}}, {{Code|ftxc}}== |
...will soon be dismissed. | ...will soon be dismissed. | ||
Revision as of 15:14, 26 May 2012
This article is part of the Advanced User's Guide. It presents some low-level details on how data is stored in the database files.
Contents
Data Types
The following data types are used for specifying the storage layout:
| Type | Description | Example (native → hex integers) |
|---|---|---|
Num
|
Compressed integer (1-5 bytes), specified in Num.java | 15 → 0F; 511 → 41 FF |
Token
|
Length (Num) and bytes of UTF8 byte representation
|
Hello → 05 48 65 6c 6c 6f
|
Double
|
Number, stored as token | 123 → 03 31 32 33
|
Boolean
|
Boolean (1 byte, 00 or 01)
|
true → 01
|
Nums, Tokens, Doubles
|
Arrays of values, introduced with the number of entries | 1,2 → 02 01 31 01 32
|
TokenSet
|
Key array (Tokens), next/bucket/size arrays (3x Nums)
|
Database Files
The following tables illustrate the layout of the BaseX database files. All files are suffixed with .basex.
Meta Data, Name/Path/Doc Indexes: inf
| Description | Format | Method |
|---|---|---|
| 1. Meta Data | 1. Key/value pairs, in no particular order (Token/Token):• Examples: FNAME, TIME, SIZE, ...• PERM → Number of users (Num), and name/password/permission values for each user (Token/Token/Num)2. Empty key as finalizer |
DiskData() MetaData() Users() |
| 2. Main memory indexes | 1. Key/value pairs, in no particular order (Token/Token):• TAGS → Tag Index• ATTS → Attribute Name Index• PATH → Path Index• NS → Namespaces• DOCS → Document Index2. Empty key as finalizer |
DiskData() |
| 2 a) Name Index Tag/attribute names |
1. Token set, storing all names (TokenSet)2. One StatsKey instance per entry: 2.1. Content kind ( Num):2.1.1. Number: min/max ( Doubles)2.1.2. Category: number of entries ( Num), entries (Tokens)2.2. Number of entries ( Num)2.3. Leaf flag ( Boolean)2.4. Maximum text length ( Double; legacy, could be Num)
|
Names() TokenSet.read() StatsKey() |
| 2 b) Path Index | 1. Flag for path definition (Boolean, always true; legacy)2. PathNode: 2.1. Name reference ( Num)2.2. Node kind ( Num)2.3. Number of occurrences ( Num)2.4. Number of children ( Num)2.5. Double; legacy, can be reused or discarded2.6. Recursive generation of child nodes (→ 2) |
PathSummary() PathNode() |
| 2 c) Namespaces | 1. Token set, storing prefixes (TokenSet)2. Token set, storing URIs ( TokenSet)3. NSNode: 3.1. pre value ( Num)3.2. References to prefix/URI pairs ( Nums)3.3. Number of children ( Num)3.4. Recursive generation of child nodes (→ 3) |
Namespaces() NSNode() |
| 2 d) Document Index | Array of integers, representing the distances between all document pre values (Nums)
|
DocIndex() |
Node Table: tbl, tbli
tbl: Main database table, stored in blocks.tbli: Database directory, organizing the database blocks.
Some more information on the Node table storage is available.
Texts: txt, atv
txt: Heap file for text values (document names, string values of texts, comments and processing instructions)atv: Heap file for attribute values.
Value Indexes: txtl, txtr, atvl, atvr
Text Index:
txtl: Heap file with ID lists.txtr: Index file with references to ID lists.
The Attribute Index is contained in the files atvl and atvr; it uses the same layout.
Full-Text Fuzzy Index: ftxx, ftxy, ftxz
...will soon be reimplemented.
Full-Text Trie Index: ftxa, ftxb, ftxc
...will soon be dismissed.
