GNI - Struct

Struct

NOTE: this document describes traditional ISIS and early OpenIsis proposals. For updated information please refer to Malete structures

structuring ISIS records using subfields or subrecords

structures

The means by which an Isis record can be structured into "data elements" ("A defined unit of information", Z39.2 a.k.a. ISO2709) fall in one of two broad categories (citing Z39.2):

subfields
"A data element considered as a component of a field." In *ML (SGML,HTML,XML...), subfields correspond to a node's attributes. In MIME, subfields correspond to attributes of a MIME header value.
subrecords
"A group of fields within a record that may be treated as a logical entity. (When a record describes more than one entity, the descriptions of individual entities may be treated as subrecords.)" In *ML, subrecords correspond to a node's childs. In MIME, subrecords correspond to multipart body parts.

subfields

Since a field value can actually be anything, including XML text or a serialized (textual or binary) Isis record, it can be arbitrarily structured according to a regular expression or some other grammar (machine parseable or not).

The term subfield, however, is used for a range of characters in the value which is identified by rather simple means:

fixed
if all (or all but the last) subfields have a fixed length and are neither optional nor repeatable, then each subfield can be found at a fixed position.
delimited with optional identifier
this is the proper Z39.2 notion of a subfield.

If a special delimiter character is found in the field, it breaks the field into subfields. Z39.2, and thus MARC, use the character 31 as delimiter (hex 1F, CTRL-_, ASCII "unit separator" US). Traditional Isis uses the caret '^'.
OpenIsis permits any character, including the horizontal TAB and semicolon. More precisely, OpenIsis reverts Z39.2's notion that "every subfield is INTRODUCED by a delimiter, unless it isn't" to the principle that for every data element, it is specified how it's end is detected, including by fixed length or varying delimiters.

The initial n characters of a subfield are used to identify the subfield. Z39.2 permits any (small) fixed value for n, including 0, i.e. not identified. The MARC family of standards uses n=1. OpenIsis allows for any value, including variable length identifiers, which are themselves delimited by some character like a '=' (see below).

Z39.2 states that if identifiers are used, each must be preceeded by a delimiter, and every data element, including the first, must be identified that way. However, an initial range of m characters (i.e. preceeding the first delimiter) in every field may serve as "indicator", which is not regarded a "data element". Again, m is a small fixed number; MARC uses m=2. Traditional Isis has no special support for indicators. OpenIsis allows to access whatever is before the first delimiter.

Different subfielding methods can be mixed or nested. Typical cases are:

mixed fixed/delimited
After some initial fixed subfields, following subffields are delimited. This can be used to describe MARC's fixed indicators.
nested delimited/fixed
A delimited subfield has itself a fixed substructure. Actually the leading identifier in a subfield can be regarded as fixed part in a mixed substructure.
nested unidentified delimited
A delimited subfield has itself a delimited structure. This can be used to model variable length identifiers.

In other words, identifiers are themselves nothing but subfields used as keys on some level of nesting. On the other hand, any subfield could serve as a key for it's parent. This is used e.g. to select a field by a subfield indicating a language (see below for keyed subrecords).
If you look at the plaintext representation of an Isis record, actually the whole record is a newline delimited value, the whole database is a blankline (double newline) delimited value and each field has it's tag as initial tab-delimited subfield.

In the future, OpenIsis will add support for a wide variety of subfielding techniques such as defined by regular expressions, MIME headers or produced in typical "character/comma separated values" files (opionally using quotes).
Since splitting subfields is mostly and can always be done on the application level (i.e. a database server rarely needs to care), "support" essentially boils down to the definition of appropriate meta data.

subrecords

A subrecord consists of a typically continuous range of fields within a record, started by some field to introduce the subrecord. Some variants, however, like keyed subfields, can be freely scattered and don't need a "header" field.

There are basically four ways to denote the boundaries of structures:

embraced
where a special field is used to denote the structures end. This resembles SGML-style notations, where each opening tag is matched by a closing tag. This is relatively easy and recommended for every day use.
marked
where the fields of the child structure are marked as such. This is sort of the opposite approach of embracing. Marking comes in several powerful flavours, see below for a more detailled discussion.
counted
where the number of fields (not childs) belonging to the structure is given in (any leading digits of) the initial field. This allows for safe embedding regardless of the structure's contents and is thus used in contexts where full generality is needed like when embedding result records within a server's response. A variant of this is negative counting, where the TAG of the initial field is the negative number of fields including the initial one.
implicit
where the number of childs is fixed. An example of this is the parse tree of a query, where the structure "AND" has exactly two childs (which in turn might be structures). This is used mostly for internal structures like parsed queries or formats, which are not meant to be exchanged.

The field introducing a subrecord might have any subfields just like other fields, similar to the attributes that might be assigned to a tag in SGML applications like HTML.
However, the first subfield (unidentified initial characters) of a field opening an embraced or counted subrecord is reserved as indicator:

a plus sign '+' as first character
indicates explicity opening a subrecord
a minus sign '-' as first character
indicates an empty subrecord (containing no childs)
an empty value
indicates explicity closing a subrecord (similar to the closing blank line used in several protocols)
an initial numeric value
(of decimal digits) gives the number of fields to follow.
an initial character @A-Z
gives the number of childs to follow (@=0,A=1,B=2...) (rarely used)

Auxiliary information about the child, like an embedded records row number and type, are stored in subfields of the parent.

conventions

While the intented usage of subrecords might be specified in more detail in the table metadata , the schema can also be used standalone (without referring to metadata), if some conventions on tag ranges are followed.
The extend of subrecords by length or braces can be safely determined if you just know that you want the given field to be regarded as subrecord.
For subrecords of fixed number of childs (meant for internal use), it is necessary to recognize whether a following field is itself a structure. If they are used at all, the tag range -1..-99 should be reserved for this purpose.
In this context, typically one of two modes is used:

the MIME processing mode for processing list-style content, assumes that negative tags denote structures, while positive contain plain data.
in XML processing mode, everything but the 0 tag (text node) is a structure.

If a parent has a subfield ^0, that should contain the childs identity as dbname or mfn or dbname.mfn. If the parents indicator is delimited by a tab instead of a ^, the next tab-delimited subfield is interpreted that way (where applicable).

marked structures

There is a wide variety of techniques for marking fields as "childs" of other fields. Marking techniques work especially well for a single level of substructuring; for nested structures, some restrictions apply.
We give some commonly used examples:

quoting
is done by prefixing every child field value with a special string, which is not used as prefix outside the child fields. However, at least for a single level of quoting, it does not impose a problem if the child fields themselves started with the same prefix: Still, the original value is retrieved by stripping the (first) prefix. This even works for multiple levels, as long as the record was properly constructed, i.e. the quoting prefix is not used outside childs. Examples are the output of the diff command (which is driving the RCS/CVS revision control system very reliably) and the '>' quoting used in e-mail replies.
tagging
Instead of the field value, of course also the field tag can be used as child mark. In some situations it might be possible to choose appropriate reserved tags for the childs. In other situations, where some given child tag must be kept, it can be stored as prefix in the field value according to the canonical Serialized plain text format.
keying
If the mark used is dependent on an attribute of the parent field, the childs can be determined even if non-continuous. With some more cooperation of the childs, the mark might be an attribute (subfield) instead of a prefix (indicator). That way, childs and parents are linked together rather logically than "physically" by a common key just like in relational databases. This easily extends to multiple levels using segmented keys (consisting of several attributes/subfields). While this scheme only works with well behaved childs and may waste some space by replicating keys, it is simple and robust and gives convenient access to the childs without inspecting the structure.

childs vs. attributes

Every information that can be represented using an attribute, can also be represented using a child. From that point of view, attributes are a redundant "language" construct and one might deem a model using only childs as the simpler one. We call such an attributeless model "canonical verbose" representation. It's a little bit similar to the "everything is an object" approach of pure OO languages like Smalltalk.

But then, having a richer language isn't always such a bad thing, if you know how to use it appropriately. (This "if" is the core of almost any serious criticism of rich languages, but for now, let's assume we know what we're doing). Appropriate use basically boils down to choosing the language construct that was just made for your situation, i.e. not the most general one, but quite to the opposite the most specific (restricted) one. That way you will not only have the most efficient representation, but also express additional information about what's going on.

In short, a "canonical compact" modelling can be based upon the principle "Use attributes wherever possible".
Some logical property of a logical structure can be represented by means of attributes, if

it is simple,
i.e. one single string value.
or at least flat,
i.e. itself a structure that can be represented based on attributes that do not interfere with the parents attributes. In the latter case, the property will show up as several logically interrelated attributes of the parent. However, such a flat group of attributes might be a candidate for a child under some circumstances.
it is not repeatable.
Although OpenIsis supports repeated subfields as used by some MARCs, XML/SGML attributes can not be repeated. (Technically, they can, but there neither is defined semantics for repeated attributes nor is access supported by parsers or the DOM). Moreover, traditional CDS/ISIS implementations do not support repeated subfields, so it's probably a good idea to not use them without a pretty good reason.

Basically, when you think C, one field's attributes take everything that goes into a simple struct, without using arrays or pointers.

The detailled modelling should also take into account the intended usage.
For example, one might devise some attribute candidates to childs, if

they are likely to be accessed or modified together but independent of other properties
they are candidates to be inherited or overridden as a group in a PatchWork
the parent would otherwise become very large

variant structures

The C language construct of a "union" is frequently used in bibliographic databases. The typical form resembles the PASCAL "variant record", using an initial field as indicator for the usage of the given field. Sometimes, however, the more liberal C practice is used, where the intented interpretation is specified somewhere in the record, somehow.
A similar construct is used in ALGOL-derived OO languages like C++ or Java, where the indicator (of what object is this ?) is out-of-band data (i.e. cannot be modified or inspected like any other data).

In Isis records, fields always have a tag (and subfields commonly have an identifier) indicating the kind of data. Therefore, there is little need to introduce another level of switches. A canonically decomposed model

would not reuse fields or subfields with different structure
would not contain rules like
"if subfield a has value b then subfield c must be present"

However, on the other hand, full decomposition might be tedious and even hide relationships. Moreover, from a given point of view, tags and identifiers are just ordinary subfields on some level.

In general, if the same tag is used for variants of a field, the risk of misinterpretation of data should be minimized by not reusing the same subfields with different structure. After all, defining another indicator and ignoring an unexpected subfield or moaning on the lack of an expected one is cheaper and more robust and clear than verifying an expected structure based on other subfield values.

examples

A typical HTML table definition starting with

<table width="100%" cellpadding="0" cellspacing="0"
  marginwidth="0" marginheight="0" topmargin="0" leftmargin="0" border="0">
<tr>
<td valign="top" width="160">
this is the textbody <br/> of the td node
</td>
</tr>
...

will be compacted to, say,

100	+^w100%^p0^s0^m0^h0^t0^l0^b0
101	+
102	+^vtop^w160
0	this is the textbody
103	-
0	of the td node
102
101
...

For a detailed description of the transformation, see the XML-ISIS doku
A six field result record might be embedded within a response like

908	6	cds.47
24	Hydrological achievements and social problems
...

Assuming we gave tag -20 to "OR" (and 0 to a literal), the query "plant OR water" might be parsed to

-20	B
0	plant
0	water

"frog AND (plant OR water)" might look like, if -21 is assigned to "AND"

-21	B
0	frog
-20	B
0	plant
0	water

For implicit tags, the number of childs is redundant (fixed per tag in a given use) and will typically be omitted.

$Id: Struct.txt,v 1.10 2004/06/10 12:52:29 kripke Exp $