";
break;
case '-':
if ( l.startsWith("---") ) { // rule
l = "") + "" + l.substring(tab);
if ( 40 > len ) // break after short heading
l += ""+name+" ";
break;
case '.':
if ( l.startsWith("...") ) { // continued
notes = l.substring(3).trim();
continue;
}
break;
case '!': // pre
br = true;
case ';': // comment
continue;
}
if ( inli && !isli )
o.write( " " );
o.write( l );
if ( br && ! l.endsWith( "OpenIsis - <%=name%>
<%-- left: the index --%>
<%-- middle: the content --%>
<% } // sides %>
<%= name %>
<% right = mikidoc( out, file ); %>
<% if ( sides ) { %>
<%
if ( null != right ) {
%>
<%-- right: notes --%>
<%= right %>
<% mikidoc( out, right+".txt" ); %>
<%
}
%>
<% } // sides %>
doc/.index 0100644 0001130 0001130 00000001652 07632113415 011416 0 ustar krip krip !
> OverView pgp
- Gabi Rohmann pgp
- Ingo Struck pgp
- Thomas Sonnemann
OpenIsis.org is sponsored by
> http://www.sferix.com/
$
should be split into three subfields using any subfield code, say '>'.
The advantage of the pre-split approach is that it makes both
display and indexing easier.
The disadvantage is, that you'll loose some information,
albeit not too much.
* searching
Searching is also relatively easy, as long as you ask simple questions ;).
To get a list of the mfns with a field 24 containing Twain, use
$
SELECT DISTINCT(mfn) FROM Mst WHERE val = 'Twain' AND tag = 24
$
For this to work efficiently, you have to put an index on val.
You may than search for 'Twain' with or without the restriction
on a certain tag, just like in ISIS.
You can also use right truncation and restrict on a couple of tags:
$
SELECT DISTINCT(mfn) FROM Mst WHERE val LIKE 'Twain%' AND tag IN (201,401)
$
Right truncation should still use the index on val,
thus be reasonable efficient (depending on how smart your RDB is).
With the pre-split structure as above, this applies to any subfield.
If you want to find Twain anywhere within the field's values, you need
$
SELECT DISTINCT(mfn) FROM Mst WHERE val LIKE '%Twain%'
$
This will perform a pretty costly full table scan,
so you may go fetch yourself a coffee meanwhile.
While you may use SQLs OR predicate to combine several terms as
alternatives, the ISIS AND does NOT map to SQLs AND,
but to an INTERSECTion on the mfn lists.
It's actually pretty much the same as ISIS does internally (but more costly).
* building an index
Still, this is lightyears away from the possibilities of an ISIS db.
The index on val does not do word splits, does not use stopwords,
is not case insensitive, does not apply character or word conversion
and so on.
To come anywhere near the features of ISIS, you have to do just
what ISIS does: set up a separate "Inverted File".
$
CREATE TABLE Index (
val varchar(30), -- index entry
mfn number, -- record containing entry
tag number, -- tag of field containing entry
occ number, -- field is occ'th occurence of field tag in record
pos number -- entry is pos'th word in field
)
$
To fill this table, you could of course go and reimplement
all the features of an ISIS .FST in, say, Oracle's PL/SQL
and recreate one record's entries from within an INSERT/UPDATE trigger.
Much easier, given you had a true ISIS database in the first place,
is to load the .LK1 and .LK2 text files created during full index update.
Some ISIS tools also allow you to create these files separately.
Searching is performed against the Index table
(which of course needs an RDB index at least on val),
not the Mst table.
The latter is only used to retrieve the full content for a given mfn,
or for full table scans.
* so why then use ISIS in the first place?
There is a long list of reasons, here go a few of them:
- You don't want to reimplement the .FST, the full formatting language,
all the nifty details of the various indexing techniques and so on.
Actually, if you DO want, you're very welcome in the OpenIsis team ;)
- Not only the index building, but also data entry is done much more
comfortable using ISIS tools.
- Any queries but the most simple examples given above will not only
be somewhat complex to write in SQL, but will also execute much slower
than in a native ISIS implementation (consider publishing with OpenIsis).
- Especially the NEAR operators can't easily be expressed in SQL.
Although the index structure given above contains the necessary data
in occ and pos, advanced SQL features like nested queries are needed
in the appropriate SQL query. You need to have an advanced (read:
expensive) RDBMS and a deep and thorough understanding of it's
query optimizer and hinting system for this to run in any
reasonable time.
- So why then use a RDBMS in the first place?
See also
> whatabout a comparision of ISIS and other database systems
and
> unirec notes on the flexibility of the ISIS data model.
---
$Id: RdbConv.txt,v 1.4 2002/11/26 16:57:19 kripke Exp $
doc/SeeIt.txt 0100664 0001130 0001130 00000003415 07633345152 012067 0 ustar krip krip
* demo databases
the very simplistic
> ../Demo?search=development standard Demo
searches the small (160 record) cds database
or the larger (~ 58.000 records) unesb.
Based on the OpenIsis java binding, the whole interface
is done with only 200 lines of code.
There is also a
> http://galileo.or.cr/php_isis/ PHP version
available at galileo foundation of Costa Rica, written by
> mailto:braulio@openisis.org Braulio José Solano Rojas
.
a special sample was set up to search for
> ../unicode?search=&prefix=1&action=search unicode characters
. You can also list the
> ../unicode?search=&prefix=1&action=search&terms=1 index
. Note that the ability to display fonts like arabic or georgian
depends on your browser. See
> /doc/charsets.html notes on character sets.
* real world examples
> http://databases.unesco.org/xtrans/xtra-form.html UNESCO Index Translationum
, the International Bibliography of Translations
with more than 1,300,000 references.
> http://isisonline.lib.itb.ac.id/ IsisOnline databases
by Institut Teknologi Bandung of Indonesia.
> http://catalog.dml.or.id/credit.php (Credit)
The
> http://knjiznice.ffzg.hr/search/Search.cgi WebPAC
of the Zagreb University's faculty of philosophy.
See also Dobrica Pavlinusic's
> http://webpac.sourceforge.net/ paper
Library of the
> ../idm Institut für Didaktik der Mathematik
of the University Bielefeld, Germany.
> http://www.bv.umsanet.edu.bo/home.htm Biblioteca Virtual Universitaria
de Bolivia ("Este sitio ha sido desarrollado principalmente con OpenIsis").
> http://www.funde.org/cendoc.html Fundación Nacional para el Desarollo
, El Salvador (still using very early and slow version, but works).
> http://www.nied.edu.na/resource%20centre/openisis_lookup.dwt the National Institute for Educational Development
of Namibia.
doc/Selbstdarstellung.txt 0100664 0001130 0001130 00000010077 07567201043 014555 0 ustar krip krip Der OpenIsis Förderverein für open source Informationssysteme
Die digitale Informationsvermittlung und -archivierung sind im
Zeitalter der sogenannten Wissensgesellschaft zur Basis
des Zugangs zu den Wissensvorräten der Welt geworden.
Der OpenIsis Verein will mit der Entwicklung und Förderung
von open source Informationssystemen einen Beitrag
zu einem gerechten Zugang zu Wissen und Information leisten,
besonders durch Unterstützung von Projekten im öffentlichen,
sozialen und kulturellen Bereich.
Dazu vermittelt der OpenIsis Verein Kenntnisse über den Einsatz von open source
Informationssystemen auf der Basis bekannter Studien
wie auch eigener Forschungsarbeiten.
Mit Publikationen und der Präsenz auf Kongressen und Tagungen betreibt
der OpenIsis Verein intensive Öffentlichkeitsarbeit.
Er engagiert sich in weltweiter Kooperation mit öffentlichen und
gemeinnützigen Institutionen und Organisationen.
In Deutschland arbeitet der OpenIsis Verein bisher
mit dem Tübinger open source Projekt "Open Community for Science" (OC4) und
Isisnetz e.V., dem Deutschland-Distributor der UNESCO für das
Bibliotheksprogramm CDS/ISIS.
Der OpenIsis Verein unterstützt beispielsweise BibliothekarInnen in Argentinien,
Costa Rica, Indonesien und Kroatien in der Entwicklung von
Informationssystemen. Das erste große Projekt des Vereins
ist die Entwicklung und der Einsatz des gleichnamigen
Bibliotheks- und Dokumentationsprogramms OpenIsis.
* Das Programm OpenIsis
OpenIsis ist eine freie Software für Bibliotheken und andere
Einrichtungen und Organisationen, die Wissensbestände archivieren und
verfügbar machen wollen. Es knüpft an das seit über 30 Jahren entwickelte System
CDS/ISIS an, das auf internationalen bibliothekarischen Standards basiert,
und bis heute von der UNESCO weltweit vertrieben wird.
Der Quellcode von CDS/ISIS ist nicht zugänglich,
eine offene Weiterentwicklung oder Anpassung daher nicht möglich.
Seit April 2001 arbeitet eine Gruppe, gegründet von den Berliner Programmierern
Klaus Ripke und Erik Grziwotz, an einer freien Version von CDS/ISIS.
Der Quellcode von OpenIsis ist offen.
Wie bei Linux kann jede und jeder an dieser Software mitarbeiten;
AnwenderInnen überall auf der Welt können das Programm an ihre Bedürfnisse
bei der Archivierung und Digitalisierung von Bibliotheksbeständen anpassen.
Weiterentwicklungen sind schon in Costa Rica, Indonesien und Kroatien entstanden.
* OpenIsis - Digitale Spaltung - Wissensgesellschaft
CDS/ISIS wird von UNESCO, GTZ und anderen als wichtiger Schritt in Richtung
eines chancengleichen Zugangs zu Informationen und zur Vernetzung von
Wissensbeständen gefördert. Mit abnehmenden Budgets der Bibliotheken
wächst die Bedeutung der Fernleihe und damit auch des Zugriffs
auf andere Kataloge über das Internet oder durch Datenaustausch.
Für digitale Katalogisierung in Lateinamerika oder Afrika kommt aber
nur ganz oder nahezu kostenfreie Software in Betracht.
Außerdem müssen lokale Versionen - nicht nur in den Kolonialsprachen -
von den AnwenderInnen selbst erstellt werden können.
OpenIsis leistet diese Wissensvernetzung und ist gleichzeitig kostenfrei.
Die Software trägt damit dazu bei, die digitale Spaltung zwischen Nord und Süd
zu überwinden und steht für eine Demokratisierung des Wissens.
* Förderung von open source und freiem Wissen
Wir freuen uns über jede weitere Unterstützung, im technischen und vor allem
im bibliothekarischen, archivarischen oder dokumentarischen Bereich um unser
Know-how zu ergänzen.
Dem Verein sind bisher Programmierer, WebdesignerInnen, Sozial- und
WirtschaftwissenschaftlerInnen und JournalistInnen beigetreten, die aktiv an
der Entwicklung, Verbreitung und Anwendung von OpenIsis arbeiten.
Mit Ihrem Beitritt können Sie persönlich einen besonderen Beitrag
für die Entwicklung von freiem Wissen leisten.
Nähere Information über den Verein und das Programm OpenIsis finden Sie unter
http://openisis.de/
Gerne beraten wir im deutschsprachigen Raum Einrichtungen im öffentlichen und
gemeinnützigen Bereich in Fragen der Verwaltung, Digitalisierung und
Archivierung ihrer Dokumentationen sowie im Wissensmanagement.
- Stand: November 2002
doc/Serialized.txt 0100644 0001130 0001130 00000020574 07644300503 013145 0 ustar krip krip ISIS records serialized
Serialization means to convert the internal representation
of an ISIS record to a sequence of bytes ("octets") suitable
to be stored in a file or transferred via a network.
The serialization format described here is used by OpenIsis
for both database master files and network communications.
* design goals
The serialization format should be
- easy to use
for programmers and tool writers
- efficient
in execution time and space used
- robust
a broken masterfile should be fixed using a text editor
- versatile
can be used for a variety of applications using a variety of tools
- without limits
in number and size of records and fields
* basic format
In general, a record is serialized by
- serializing meta information
- serializing the fields in order
- appending a blank line
Fields are serialized as
- the field tag printed using ASCII decimal digits
(optionally preceeded by a minus sign, if negative tags are allowed)
- a (horizontal) TAB character (ASCII value 9, ^I)
- the field value
- a newline character (ASCII value 10, ^J)
Metadata is serialized in the same way,
using special tags according to the needs of the environment.
Two situations are distinguished:
- "soft" metadata, which may and should be accessible as part of the record.
This is encoded by convention using negative tags.
An example of this is HTTP and other MIME-style communication,
where the MIME headers like "User-agent" or "Date" are encoded
in such a way, while content data like GET or POST parameters
should be mapped to positive IDs.
- "hard" metadata, which must not interfere with the record contents
in order for the environment to work properly.
This is encoded using a single non-digit character instead of tag digits.
An example of this is the MFN in a master file and information
regarding record deletion or update.
The final blankline may be omitted,
where only a single record is contained in an otherwise delimited byte sequence.
A reader should support a lazy mode,
allowing the TAB to be omitted, where unambigous
(the field value does not start with a digit or a TAB).
Writers, however, are strongly urged to write the TAB.
Tags with leading zeros are allowed (typically with %03d)
and must not be interpreted as octal using atoi.
* newline conventions
Two ways are supported to deal with newline characters in field values:
- in "text mode", newlines are replaced with vertical tabs (ASCII 11, ^K)
on serialization and vice versa on deserialization.
- in "binary mode", newlines are replaced as newline-TAB sequences.
The advantages of text mode over binary mode are
- it is slightly faster than the binary translation
- the serialized records do not need more space
than the internal representation
(whereas the binary serialization might need nearly twice as much
in worst case)
- it is easily used with line-oriented utilities like grep or sed,
since each field is contained within one line
The binary mode (which resembles MIME continuation lines) has the
advantage of not loosing vertical tab characters that might have
been contained in the original field values.
It is fully transparent and can be used to store any binary data like images
with an average overhead of 0.4%
(as compared to +33% needed by BASE64 encoding).
The OpenIsis server automatically detects binary mode,
if the client uses a continuation line.
* masterfile format
A basic masterfile consists of a blank-line separated series of records.
The MFNs are assigned implicitly in order, starting from 1.
There is no distinction between
empty (consecutive blanklines) and deleted records.
There is absolutely no redundant information contained.
Masterfile compression creates this state
(however, it may choose to use special meta lines for long ranges
of deleted records, see below -- but those are inefficient in
the Xref, anyway).
Such a masterfile can be very easily created by any tool (like Perl and such).
The Xref file can be easily, fast and reliably recreated.
When writing to the database, information is ALWAYS appended to the end.
There is NO OVERWRITING of any data, ever, period.
That way data CANT BE DESTROYED by any operation
(one could advise the operating system to set a mandatory read lock on that),
and all changes are easily traced using tail -f.
A binary mode masterfile starts with a "broken" continuation line
containing a single TAB.
* basic mode writing
In metalines, all numbers are given in decimal digits
and multiple items are separated by TABs.
The optional timestamps are an arbitrary prefix of generalized time format
YYYYMMDDhhmmssttt... as of 'date +%Y%m%d%H%M' plus milliseconds ttt,
optionally followed by more digits to create a unique request id.
Write operations use the following meta line (preceeding record data):
- W mfn [oldpos [timestamp]]
followed by new record data denotes a write of record mfn.
Depending on the needs of the environment,
the byte offset of the last version might be added in order to
support access to old versions (e.g. for delayed index update).
A W with no data following is mostly equivalent to a delete.
If the deleted mfn is the implicit MFN counter (i.e. the highest mfn),
that is decremented.
A lazy reader should not require meta lines to be followed by a blank line,
where unambigous.
For writers, however, the blank line is strongly recommended.
* advance mode
For advanced space efficiency at the cost of read increased access time,
the following lines may be used:
- D mfn [oldpos [timestamp]]
a record entry consisting of only a D line denotes the deletion of
record number mfn.
Basically equivalent to a write with no data following,
but a little bit more explicit.
- I mfn [timestamp]
(set id) override the implicit MFN counter,
e.g. after a series of deleted rows or just to be explicit.
Basically equivalent to a write with no old pos,
but a little bit more explicit.
Recommended when appending records after some deletes.
- C mfn oldpos [timestamp]
introduces a series of patch commands specifying how record mfn
was changed.
Software writing the masterfile will typically choose to
write full updates and MUST provide a switch to forcedly do so,
in order to be compatible with basic mode readers.
However, the patch command language is particularily useful
with server operations, to avoid the need for read-write
sequences with some sort of locking.
* the patch language
The patch commands are lines starting with special
characters like +, -, ~ and so on,
followed by (an optional TAB and) field addresses, TAB and field data.
The simplest case is the '+' command,
meaning that it's data is to be appended to the record.
The + and TAB may be omitted
(both, in order to not be confused with a continuation line).
In other words, the add command may look exactly like an ordinary
field line.
A series of '=' commands works exactly like the set operation in an
OpenIsis Tcl record. Especially, field indexes and subfields are supported.
The '-' command resembles the del operation.
A detailled description of the "patch language" is to be done.
example
$
C 1234
= 24 foo
= 24 bar
25 baz
$
changes record 1234 by setting the first to occurences of
field 24 to foo and baz, respectively, deleting any other occurences
of field 24, and appending a field 25 with value baz.
* the xref file
The Xref file is an array of n-Byte (n >= 6) entries,
the ith entry referencing mfn i.
The n bytes specify two or three numbers (in native byte order)
of up to 8 bytes each:
- the first k bytes (k >= 4) give the position of the record
(or it's last update or change entry)
- the next l bytes (l >= 2) give the length of the record
(excluding the terminating blank line)
- the final m bytes (m >= 0) give the number of fields.
If m is 0, or all bits in a field number are set (for large records),
the reader has to determine the number fo fields by inspecting the record.
The first six bytes of the first entry describe the detailled layout.
Four bytes are the "magic number" containing the ASCII characters "ISIX".
Two bytes are the number (m*256 + l*16 + k) in native byte order.
The minimum case k=4, l=2 imposes the limits of traditional ISIS.
Recommended is at least k=5, l=3.
However, when any of the limits is reached,
or an unsupported combination or byte order is found,
the Xref can easily be recreated with greater values.
The number of fields is redundant, but as an optimization
may make live a little bit easier for a reader.
---
$Id: Serialized.txt,v 1.3 2003/04/07 13:12:42 kripke Exp $
doc/Server.txt 0100644 0001130 0001130 00000013526 07623223671 012326 0 ustar krip krip operating an OpenIsis server with one or more threads
* server interface
From the OpenIsis API, all db IO calls can be redirected to a server.
However, only the basic database access is done by the server,
convenience functions like formatting are processed locally.
There would be no point in processing stream IO or record utilities remotely.
While database control calls are generally NOT accessible via the wire,
some of them have counterparts for managing the server session.
* the multisession server
A multisession server deploys multiple threads to process requests
from multiple sessions in parallel. Each session occupies at most one
thread at a time (and, of course, vice versa), but threads and sessions
are not (necessarily) bound to each other during their lifetime.
Rather there is a dispatcher monitoring all session's input streams
and allocating threads as needed.
However, there is one thread bound to the default session,
which is thus not subject to dispatching.
* principles of communication
The server communicates with it's clients by sending messages over streams.
While the server's IO multiplexer engine is designed to allow for a variety
of applications, the database server API's messages are OpenIsis records.
There are field numbers and subfield codes assigned to each
parameter of a request. Parameters which are records themselves
are encapsulated transparently, i.e. by giving their length in advance.
In basic communication mode (i.e. not using a wrapper protocol like HTTP),
these records are then transmitted as lines consisting of a field number
printed using ASCII decimal digits, followed by a TAB, followed
by the field's contents and a newline (NL).
Each NL within the contents is converted to NL-TAB and
on the receiving side, each line starting with a TAB is treated
as continuation of the previous field.
Transmission ends with an empty line.
* session streams
The streams connected to a session are usually backed by some kind of socket.
Since requests carry a session id,
sessions may be multiplexed onto one or more sockets,
possibly using one connection per request or even UDP datagrams.
Each response, however, is sent on the socket where the corresponding
request was received on and in the order request went in.
The default session is always bound 1:1 to a communication channel,
not subject to multiplexing
(i.e. session id 0 is implied by and allowed only on this channel).
Yet this channel may be a named pipe or localhost or unix socket,
which does not have a permanent client connection
and may be controlled by the multiplexer.
Typically, it's client side is a Tcl belonging to the same process,
either a tclsh as controller of a multisession server
or a wish as the sole client within a standalone application.
On Unix, this server thread is most easily connected to by
a pair of pipes (using Tcl_CreateFileHandler).
Under Windows, things are much more complicated ...
In general, the marshalling to a stream might of course
be skipped when using inter thread communication.
Either side might just prepare the record to the session's
request or response buffer, resp., and then ring some bell
like Tcl_ThreadQueueEvent and Tcl_ThreadAlert.
* the server's heart
A server enters multisession mode by starting up the multiplexer.
The multiplexer may run in a separate thread or in the same
one as the default session, depending on the means by which
the default session is connected to it's controlling input.
The multiplexer starts up some number of worker threads,
listens on a TCP port and then sits in a loop
- selecting the server socket and all already open connections
- accepting new connections
- writing available responses to writable sockets
- reading the first few bytes from readable sockets
(from which it not already has a partly request)
to check for session ids (SID).
If a request does not start with a SID,
it belongs to (i.e. optionally creates) the session bound to this socket.
- deciding from which (readable) sockets more should be read.
Input belonging to sessions already in process is left alone.
- reading available input, checking for complete requests
- if the multiplexer handles default session's input and there
is a request available, it is processed in this thread
- queing complete requests for processing
- checking the response queue for completed requests,
collecting responses and freeing threads.
Requests producing large amounts of output
(similar to piggybacked Z39.50 present requests)
may create partly responses before beeing finished.
- assigning from the request queue to available worker threads
Note that this multiplexer architecture is in no way special
to OpenIsis database access, but suited and intended to drive
a session oriented, but otherwise multi-purpose application server.
In the context of serving an OpenIsis database,
the server is essentially stateless.
It may at any time decide to drop inactive sessions,
unless it was talked into (configured for) promising to keep result sets.
* the single threaded server
The single threaded version of the server has only one thread
and typically only one session, i.e. the default session.
Note that single threaded means there is only one thread accessing
the database. There may still be other threads in the process,
e.g. to care for the user interface.
A single session server is not very useful in itself.
However, it allows some user interface to be programmed using
the server API regardless of whether a local or remote DB is accessed,
and to discouple even local processing from the UI.
Another variant of a single threaded server is multisession,
i.e. running a session multiplexer, but not dispatching jobs
to a worker thread pool.
This may be useful as minimal server on platforms with no
or broken multithreading support, especially Win 9x/Me
(which lacks the NT call SignalObjectAndWait).
---
$Id: Server.txt,v 1.3 2003/02/14 17:30:33 kripke Exp $
doc/Status.txt 0100644 0001130 0001130 00000004340 07642065016 012333 0 ustar krip krip status of OpenIsis development
* stable version
Version 0.8.6 marks the first version of OpenIsis
to contain basic implementations of all major features
required for web based publishing of catalogues:
- simple queries as typically composed by web search forms
- basic formatting (field access and modes) with detailled
rendering left to the web frontend code (JSP, PHP ...)
All features present in 0.8.6 are and will be supported
with the same syntax.
In most situations you should use the most recent version,
but not rely on the new features.
If in doubt, please contact us.
*devel developer version: use with caution!
Since summer 2002 we are developing the various 0.8.7.x versions.
They include
- writing capability
- new B-L-Tree index
- server side sessions
- threadsafe buffered IO
- preliminary Tcl binding
Most new features where planned carefully and prepared by a
lot of paperwork.
However, as long as the number is 0.8.7.*, all features introduced
since 0.8.6 are considered experimental and might still need to be changed.
So before you are going to base a lot of new work on these features,
it's recommended that you contact us in order to learn
detailled up-to-date status.
Update: as of March 03, we have the core of a multi-threaded
server engine (request dispatcher, session handling) running,
a Tcl/Tk GUI demo using TSPs (Tcl Server Pages) and Java code
to convert to/from XML.
While basically all is as planned, we may remove the Sessions
from the API before blessing it 0.8.8.
* plans for 2003
(... not necessarily in exactly that order)
- within short, the new index should support immediate update
(DONE) based on basic formatting (TODO)
- by end of winter, a local implementation of the server interface
should be basis for a first Tk based GUI (Demo DONE)
- in spring, a single threaded but multi-session server with
GUI client should give proof of all those concepts,
marked by version 0.9 (Tests DONE)
- during summer, the server should actually run multithreaded
and Java, Perl and PHP will get a client binding
- once the formatting has gotten a complete overhaul,
we may consider OpenIsis "grown-up" and put version 1.0
under the christmas tree
... ToDo
---
$Id: Status.txt,v 1.4 2003/03/31 16:17:18 kripke Exp $
doc/Struct.txt 0100644 0001130 0001130 00000007644 07644300503 012341 0 ustar krip krip embedded structures in ISIS records
* structures
In order to support substructures (i.e. embedded records),
we introduce a new field type with symbolic name S and code 9.
A structure consists of a continous range of fields within a
record, started by some field to introduce the structure.
There are basically three ways to denote the boundaries of structures:
- embraced
where a special field is used to denote the structures end.
This resembles SGML-style notations,
where each opening tag is matched by a closing tag.
This is relatively easy and recommended for every day use.
- by length
where the number of fields (not childs) belonging to the
structure is given in the initial field.
This allows for safe embedding regardless of the
structure's contents and is thus used in contexts where
full generality is needed like when embedding result records
within a server's response.
- implicit
where the number of childs is fixed.
An example of this is the parse tree of a query,
where the structure "AND" has exactly two childs
(which in turn might be structures).
This is used mostly for internal structures like parsed
queries or formats, which are not meant to be exchanged.
Where used, Implicit fields and their number of childs are typically
recognized by the field tag range -1..-99 reserved for this purpose.
The field introducing a structure might have any subfields
just like other fields, similar to the attributes that might
be assigned to a tag in SGML applications like HTML.
The first (unnamed) subfield ("indicator" in MARC speech) however is reserved:
- a plus sign '+' as first character
indicates explicity opening a structure
- a minus sign '-' as first character
indicates an empty structure (containing no childs)
- an empty value
indicates explicity closing a structure
- an initial numeric value
(of decimal digits) gives the number of fields to follow.
- an initial character @A-Z
gives the number of childs to follow (@=0,A=1,B=2...)
The opening sign or digit may be followed by auxiliary information.
An at sign @ followed by a name (alphanum) indicates the structure
to be of the type name, typically described in file name.fdt.
* conventions
While the intented usage of structures might be specified in
more detail in the
> Meta Field Definition Table
, the schema is meant to be as standalone as possible.
The extend of structures by length or braces can be safely
determined if you just know that you want the given field
to be regarded as structure.
For structures of fixed number of childs (meant for internal use),
it is necessary to recognize whether a following field is itself a structure.
In this context, typically one of two modes is used:
* the MIME processing mode for processing list-style content,
assumes that negative tags denote structures,
while positive contain plain data.
* in XML processing mode, everything but the 0 tag (text node) is a structure.
* examples
A typical HTML table definition starting with
$
this is the textbody
...
$
will be compacted to, say,
$
100 +^w100%^p0^s0^m0^h0^t0^l0^b0
101 +
102 +^vtop^w160
0 this is the textbody
103 -
0 of the td node
102
101
...
$
For a detailed description of the transformation, see
> xmlisis the XML-ISIS doku
A six field result record might be embedded within a response like
$
4 6@cds
24 Hydrological achievements and social problems
...
$
Assuming we gave tag -20 to "OR" (and 0 to a literal),
the query "plant OR water" might be parsed to
$
-20 B
0 plant
0 water
$
"frog AND (plant OR water)" might look like, if -21 is assigned to "AND"
$
-21 B
0 frog
-20 B
0 plant
0 water
$
For implicit tags, the number of childs is redundant
(fixed per tag in a given use) and will typically be omitted.
---
$Id: Struct.txt,v 1.5 2003/04/07 13:12:42 kripke Exp $
doc/Syspar.txt 0100644 0001130 0001130 00000011403 07607554653 012342 0 ustar krip krip the system and database parameters of Winisis
(collected from various sources)
* syspar.par
- 0 redirection path to other syspar file
- 1 program path pascal files
- 2 menus path the MNlltt menu databases and ISISA/UC.TAB
- 3 msg path the MSll message databases
- 4 work path for temporary files
- 5 db path
permanent DB files ANY, CNT, IFP, L01, L02, N01, N02, MST, XRF,
FDT, FST, FMT, PFT, STW and SRT; c.f. the dbname.par
- 6 default db
- 7 language (DOS)
- 8 repeatable fields sep
string to separate reps on entry; default '%'
- 9 startup sequence (DOS)
e.g. "scds^M" to select CSD database on startup
- 10 editor insert mode (DOS) (bool); default 0
- 11,12 line drawing characters (DOS)
- 13 extended memory (DOS); values 0,64k,128k,192k
- 14 network mode
default 0 (single user); 1 is full network, 2 restricted network
- 101 language; default en
- 102 menu (tt in db name); default DF (SH has short version)
- 103 display search progress (bool); default 0
- 104 restrict dictionary (bool)
default 0; display only terms occuring in selected field
- 105 auto display results (bool)
default 0; immediately open result display after search
- 106 name of DOS to Windows ("OemToAnsi") character conversion table
file in the menu dir with 256 codes, one per line.
use supplied file TXT_TXT.TAB for identitiy conversion.
- 107 name of Windows to DOS character conversion table
- 108 right to left script (bool)
default 0; set 1 for arabic, hebrew ...
- 109 field edit font
format fontname[,size] like Arial Narrow,24
- 110 formatting defaults
a PFT expression (not file name) containing any of
fonts,cols,cf,f,fs to preselect fonts and colours
- 111 format edit font; like 109 for the format editor
- 112 printer setup
last used printer name/port; auto saved by Winisis
- 113 print index separator; default ", "
- 115 small font used by dictionary, search expressions
- 116 dialog font for message dialog windows
- 117 system font for some other texts
- 120 display toolbar (bool); default 1
- 121 display toolbar help (bool); default 1
- 122 last main window position; auto saved by Winisis
- 123 show empty fields in worksheet (bool); default 0
- 124 restricted shell mode (bool) and pos
mode[,left,top,right,bot]; mode 1 restricts to default db w/o toolbar
- 125 restricted databases (bool)
default 0; restricts access to databases in fields 1001-1005
- 130 auto IF update (bool); default 0
- 131 word length to paginate non-latin scripts
- 132 display pascal error (bool); default 0
- 133 windows "large fonts" (bool); default 0
- 134 data entry field width (pixel)
- 135 search results statusbar (0..20); default 0
- 136 window zoom (80..160); default 120
- 137 confirm close (bool); default 1
- 138 allow dictionary dump
- 140 display log. del. recs (bool); default 1
- 141 background bitmaps dir
- 142 highlight fgcolor[,bgcolor]
- 144 force tabular display (bool); default 0; see 980
- 148 plugins dir
- 150 date format; YMD or DMY; default is M-D-Y
- 152 close print dialog (bool); default 0
- 155 deny repeated subfields (bool); default 0
- 156 validate even unmodified records (bool); default 0
- 160 startup plugin name
- 161 exit plugin name
- 170 username for GetUsername() format; max 9 chars
- 902 hypertext debugger (bool); default 0
- 905 status for new/upd records
default 0,0 (???)
- 918 export separators
replace (ISO) export separators 29 (GS/RT for records)
and 30 (RS/FT for fields). Value is two printable chars literally.
- 939 logfile name
- 980 tabular display on multiple results (bool); default 0
- 981 pft name for search tips
- 1001-1005 recently used masterfiles
filenames may be followed by *** and a descriptive text
The DOS version also allows to assign character sequences to function
keys using syntax Kn=seq, where n is between 59 and 118.
See section 02-2-15 (PARÁMETROS DE DEFINICIÓN DE TECLAS DE FUNCIÓN)
of the manual.
* dbname.par
This file is searched for where the masterfile is expected,
i.e. in the db path, and may specify locations for various database files:
- 1 XRF
- 2 MST
- 3 CNT
- 4 N01
- 5 N02
- 6 L01
- 7 L02
- 8 IFP
- 9 ANY
- 10 others:
FDT, FST, FMT, PFT, STW and SRT
- 0 network switch like syspar.par field 14
* dbname.exp
This one is also searched in the db path.
It contains settings to finetune data display.
- 500 pft name for right results panel
- 502 pft expression for left results panel
- 504 descending mfns (bool); default 1
- 506 allow pft selection (bool); default 1
- 550 FST tags prefix (repeatable)
each value is "tag prefix", e.g. "24 TI=" to hide prefix from terms list
- 560 FST tags descriptions (repeatable)
each value is "tag text", e.g. "240 Words from title" for FST tags.
Language sensitive runs of description may be introduced by "0 ll",
where ll is the language code like EN, FR.
---
$Id: Syspar.txt,v 1.1 2003/01/10 14:33:15 kripke Exp $
doc/Tcl.txt 0100644 0001130 0001130 00000013567 07612767461 011617 0 ustar krip krip design of the Tcl interface
* sessions and records
The `objects´ in the OpenIsis Tcl interface are tcl commands using the
same generic syntax like "array", "info", "interp" or the Tk widgets:
$
name option ?arg arg ...?
$
Like slave interpreters created by "interp create" or widgets created by
"button", "entry" and the like, the name of the command can be choosen freely.
There are two basic `objects´ to consider when using OpenIsis:
- a record is a command giving access to it's fields.
It provides access to single fields, a selection of fields,
formatting or looping over the records contents.
- a session creates new records from scratch or by
fetching them from a database.
A session also implements all record commands,
giving access to it's configuration.
Moreover a session has two embedded records,
the request and the result.
Finally, new sessions may be created.
There is one builtin command introduced by OpenIsis giving access
to the default session: the command "openIsis".
If you do not intend to work with multiple sessions,
you can switch the command openIsis to a remote session.
* synchronous vs. event based processing
Sessions within this context are always client sessions,
and all but the default session (which is local) are connected to
a remote server doing the actual work. Unlike the corresponding
sessions on the server(s), which are dispatched to multiple threads
in order to process multiple client's requests in parallel,
the client sessions do essentially nothing but communicate with
the server, which is handled pseudo-asynchronously within one thread
by means of fileevents, compare
> Client client basics.
A session may be configured for event based processing
by configuring a command to be executed upon request completion.
Client sessions are asynchronous in nature and the server socket
can (on Unix) easily be bound to a Tcl filehandler.
If no command is specified, each request will block until completed.
The default session, on the other hand, runs in the same process.
Although it is possible to detach processing to a slave thread,
integration is more demanding and system dependent and may not
be available until some later version. Nonetheless,
you may specify a command which will be executed after the completion.
* the record options
Following is a short outline of the options of the record command.
Wherever a field parameter is used, it may be specified as either
a decimal number or, if a dictionary is available for this record,
as a field name. Field names may also be preceeded by a dash
for an option style look.
- del ?field ...?
All values for all or the specified fields are removed.
- get ?field ...?
Returns a list of all or the specified field's values.
- add field value ?field value ...?
Adds the specified pairs of fields and values.
- set field ?value field value ...?
With only a single field specified,
returns the first value of the single specified field.
If values are given, each field is set to the given value,
overriding any previous value in the same place.
In detail, for every sequence of the same field in the list,
the values are assigned to existing occurences of this field
in the record. Remaining occurences in the record are deleted.
Remaining occurences in the parameter list are added.
- do ?options? varname body
loops body over fields of the record,
assigning each value in turn to varname.
body may use break and continue.
options include:
-fieldid varname to assign each field id to varname,
-fieldname varname to assign each field name to varname,
-field field to select only occurences of field,
-fields fieldlist to select only occurences of fields in fieldlist
- create ?options? newname ?field value ...?
clone this record to newname (which must not be a valid option).
If newname is "-", some name starting with "openIsis" is choosen.
Following field value pairs are applied as with set.
options include:
-empty to make an empty clone (sharing the same dictionary)
- fmt ?options? format
apply ("print") format.
- done
have the record go away. same effect as with Tcl's rename.
- .path ?option arg ...?
access embedded record specified by path,
where each component is of the form field or field[occ].
Without parameters, return 0 or 1 depending on availability of embedded rec.
Example: rec .4[2] set -foo bar
* the session options
Note that a session supports the record options,
applied to it's configuration.
TODO ...
- open ?-cfg val ...?
While it is not strictly necessary for a session to "open"
a database on a server before issuing calls against it,
opening serves three purposes:
- you know that the specified database is available on the server,
- the FDT meta data is fetched allowing for the use of field names,
- the given database and connection parameters are stored as
defaults in the session.
- req ?-param val?
Apply the -param val settings to the request record (as with set),
then send it to the server. See the
> Client common client description
for an overview of parameters.
- recv
Force wait for an response. Does nothing on a synchronous connection
or if the response is already there.
- ?-db db? new
create new record (for db)
- .req
access request record
- .res ?option ...?
("response/result") With further parameters,
apply them to the result record (error, if not response there).
Without parameters, return 0 or 1 depending on availability of response.
- dict option ?arg ...?
access metadata ... TODO
- create newname ?-cfg val?
create new session
* the session configuration
TODO config
- server
Hostname
- port
- command
command to be executed upon completion (async, if possible)
- encoding
encoding used by the server (default under windows is Cp850)
- db (database)
Name of default database on the server to be used for calls
that don't specify one.
- url
specify server, port and db in url style
- loglevel
- logfile
---
$Id: Tcl.txt,v 1.4 2003/01/20 12:04:21 kripke Exp $
doc/ToDo.txt 0100664 0001130 0001130 00000001046 07632113434 011714 0 ustar krip krip underway ...
- rework most routines to take a session parameter,
bind thread, memory and other resource usage to sessions
- cleanup I/O layer,
enhance streaming
- finish Tcl binding (Matthias)
- create Tk-based GUI
* next steps
(not necessarily in that order)
- update language bindings Perl, Java ...
- utilities for
> unirec nested records
- rework formatting based on structured representation of format as a record
- rework query processing based on structured representation of query
- XML I/O-conversion utilities
- implement server version
doc/UeberSicht.txt 0100644 0001130 0001130 00000005625 07622711025 013110 0 ustar krip krip Der OpenIsis Förderverein für open source Informationssysteme
wurde im Oktober 2002 gegründet. Wir unterstützen Bibliotheken,
Archive und Bildungseinrichtungen in Deutschland und anderen Ländern
beim qualifizierten Informationsmanagement mit
Programmierung und der Bereitstellung von Servern.
Unser zentrales Projekt ist das Datenbanksystem
> OpenIsis
, seit Mai 2001 entwickelt von den Vereinsgründern.
OpenIsis ist die freie ("open source") Variante im Kreis der
> IsisIntro CDS/ISIS
> CdsIsis software Familie
.
* wer benötigt "Informationssysteme"?
Jeder, der qualitativ hochwertige Informationen hat!
Bibliothekare investieren seit Alexandria ihr Wissen und ihre Erfahrung,
damit das gesuchte Buch gefunden wird. Die großen Bibliotheken
nutzen und entwickeln bibliographische Datenbanken seit langem;
diese Möglichkeiten sind aber vielen kleineren Archiven wie denen
von Wohlfahrtsorganisationen, Gewerkschaften oder sozialen Bewegungen
nicht zugänglich.
Selbst im reinen Online-Bereich (Web-Archive, Newsgroups, Mailing-Listen)
weicht angesichts der zunehmenden Masse die Begeisterung darüber,
etwas zu finden, dem Verständnis für die Notwendigkeit,
das Gesuchte zuverlässig zu finden,
auch und gerade wenn es selten referenziert wird.
Das "Dublin Core" Metadaten-Schema und das Resource Description Framework
(RDF) sind Beispiele neuerer Ansätze auf der Basis bibliographischer Prinzipien.
* wer benötigt "open source" Informationssysteme?
Die Kosten für brauchbare kommerzielle Systeme im fünf- bis sechsstelligen
Euro-Bereich sind nur für wenige Institutionen tragbar.
Nicht ohne Grund also ist das
> http://www.unesco.org/webworld/isis/ CDS/ISIS
System der UNESCO (gegen Schutzgebühr) dort sehr weit verbreitet,
wo die Mittel knapp sind.
Unsere
> http://www.gnu.org/philosophy/free-sw.html freie Software
ist nicht nur kostenlos, sondern vor allem frei von Beschränkungen.
Die Nutzer können darauf aufbauen, sie verbessern oder an ihre
Bedürfnisse anpassen, ohne auf die ursprünglichen Entwickler
(uns) angewiesen zu sein. Wir sind überzeugt, dass ein wesentlicher
Beitrag zur Minderung des "digital gap" nur geleistet werden kann,
wenn nicht nur Zugang zu Wissen durch ein Informationssystem,
sondern auch Zugang zu dem System selbst gewährleistet wird.
* was hat das mit der "Informationsgesellschaft" zu tun?
Auch die Bibliothekare und Techniker an den gut ausgestatteten
Universitäten in den USA und Europa finden ja keineswegs die
Unterstützung der Industrie, wenn sie an neuen Wegen arbeiten,
der alten Idee des
> http://www.informatik.uni-leipzig.de/~graebe/projekte/imd-mat.html Gemeinguts Wissen
neues Leben einzuhauchen, wie etwa die
> http://www.openarchives.org/ Open Archives Initiative
oder unsere Freunde von der Tübinger
> http://oc4s.org/ Open Community for Science
. Ein Pfad zu gesellschaftlicher Entwicklung jenseits von
reinen Verwertungsinteressen wird vielleicht von der
"Linux Ökonomie" vorgezeichnet ...
... News
doc/WritePerf.txt 0100664 0001130 0001130 00000005272 07631510202 012754 0 ustar krip krip writing performance test:
copying the 58970 records of unesb (~ 500 byte each)
with simple full index (-idxall)
making one entry for every field (total of 585334 fields)
system:
$
cat /proc/cpuinfo
...
model name : Pentium III (Coppermine)
cpu MHz : 801.832
cache size : 256 KB
free
total used free shared buffers cached
Mem: 256252 150428 105824 0 640 70560
-/+ buffers/cache: 79228 177024
Swap: 265032 7668 257364
$
initial write to non existing db db/test/t,
creating a 30MB .mst and 18MB .oxi
$
time ./openisis -db db/unesb/unesb -append db/test/t -fmt mfn -idxall >/dev/null
real 0m20.304s
user 0m15.680s
sys 0m3.600s
./openisis -db db/test/t -fmt mfn | wc -l
58970
./openisis -db db/test/t -ifdump | wc -l
585334
$
that is about 2,900 rows + 29,000 index entries per second.
(without indexing, it writes nearly 15,000 rows a second).
successive append are substantially slower, as IO cache is exhausted.
* the never reading system
On a 650 MHz system with *lots* of cache memory
(it actually *never* reads a block),
a test of five successive appends yields pretty much constants times
around 29 seconds per pass (~ 2,000 rows + 20,000 entries / sec),
although the .mst stacks up to 294850 records and the index then has
2926670 entries.
That same system does on that fat db
one million random reads in 10 seconds (~ rows 100,000 / sec)
and an ifchk (252465 keys in 50596 blks of 1K depth 4) in 5.18 secs,
accessing 48,738 per second.
* dual CPU use
Using both CPUs of this box,
four parallel writers (copying unesb to four different targets)
take a total of 60 seconds, giving a combined throughput of
more than 3,900 rows + 39,000 index entries / sec.
* comparision
While these numbers are all derived with single threaded debugging code,
they yet outperform the (more or less) "relational" databases
by one or even two orders of magnitude.
For example, MySQL tested 350768 inserts without key in in 619*0.62 seconds,
yielding 914 rows/sec as compared 15,000 with OpenIsis.
Insert with key (one, probably) did less than 35 rows per second,
compared to around 3,000 (with ten keys each).
Select (probably reading all rows) did 20,000 in 39*0.54 seconds,
that is 950 per second -- compared to 100,000 with OpenIsis.
I couldn't figure out what kind of system was used for these
> http://www.mysql.com/information/benchmark-results/result-db2,informix,ms-sql,mysql,solid,sybase-relative.html MySQL benchmarks,
however, it probably was not much less than our 800 MHz Pentium.
Most other databases are even slower, and that, by the way,
is also our experience when working with Oracle and the like.
doc/XReunion.txt 0100664 0001130 0001130 00000003556 07622000465 012624 0 ustar krip krip OpenIsis joined the X. regional meeting of CDS/ISIS distributors and experts
in Latin America and the Caribbean, Feb. 3.-5. in la Havana, Cuba.
* Overview
Among the more than 50 participants where Davide Storti,
chief CDS/ISIS manager in UNESCO HQ Paris,
Adalberto Tardelli, senior developer at BIREME,
and known expert Ernesto Spinak from Uruguay.
Davide announced that there are new developments underway
to build a much advanced successor of WinISIS,
which shall finally be released under some open source license.
OpenIsis was welcomed in the "CDS/ISIS family"
and possibly may make some contributions to further developments.
* Presentations
Just to pick a few among a couple of interesting speeches:
- Adalberto explained how BIREME is focussing their next software generation
on XML storage to enhance, for example, the
> http://www.scielo.org/ SCIELO platform.
- Ernesto introduced IsisMarc, a windows app for data entry with strong
support for any MARC standard, including the highly complex MARC21.
- Elysio Mira Soares from Brasil developed the library automation system
> http://www.elysio.com.br/eng_index.htm PHL
, which is under GPL, but users are asked to contribute their contents
to a pool.
- A group of Cuban developers from INFOMED are working on CLABEL
(Catálogo en linea para automatizar bibliotecas electrónicas),
which shall be a KOHA replacement based on CDS/ISIS (under GPL, of course),
while their colleagues are developing
> http://webservices.sld.cu/test/isisearch/ XML-RPC interfaces for ISIS.
- Paulo Cattelan from Porto Alegre presented the ISIS
information and helpdesk system
> http://www.oraculo.inf.br/ ORACULO.
- From Argentina we saw the LGPLd
> http://www.bl.fcen.uba.ar/openopac.php OpenOPAC.
OpenIsis prepared bilingual (english/espanol) papers on
> Cuba state and roadmap of our development
and
> CubaPHP the OpenIsis PHP extension
by Braulio.
doc/btree.txt 0100644 0001130 0001130 00000032611 07623223622 012151 0 ustar krip krip OpenIsis deploys a B-Tree structure similar to that of CDS/ISIS.
There are several differences to the original CDS/ISIS file layout:
- The complete index is kept in one file, including
control information (.CNT), inner nodes (.N0x) and leave nodes (.L0x)
of any key length and "inverted file pointers" (.IFP, see below)
- The maximum key length is configurable up to 255 bytes (CDS/ISIS: 30)
- The maximum mfn is configurable up to 6 bytes (CDS/ISIS: 3)
- The keys are not blank padded.
- The blocks have no fixed spread factor (CDS/ISIS: 10),
but a fixed size of mK (where m is 1,2,4 or 8),
allowing for about 200 keys of 30 bytes.
- A custom string comparison function may be used,
allowing for proper unicode collation.
- To enhance safe high concurrency, it is a B-Link-Tree as used in postgres
(see Lehmann/Yao 1982), i.e. each block has a link to it's right sibling.
Advantages:
- The number of necessary I/O operations is dramatically reduced,
typically to 1. Since the invention of B-Tree's by Bayer, the
ratio of available main memory to disk storage is ~ 1:100.
Since the ratio of the total size of the inner nodes to the total
index size is aproximately equal to the spread factor, it is crucial
that this is at least 100, allowing all the inner nodes to be in RAM.
- The price paid for this, is that, since the inner nodes locate the
leaf wanted less precise (up to a block of 200 entries instead of 10),
there is more unwanted data read from disk.
However, all I/O is in mK blocks on mK boundaries,
matching the "page size" of most modern operating systems.
This is the basic idea of B-Trees in the first place.
- The maximum database size is much raised with regard to the mfn.
This allows for an index spanning several databases, where the
higher mfn bytes indicate the relevant master file.
Limits:
The theoretical maximum number of terms is also somewhat raised from 20G
(2G leaf blocks per 10 terms) to about 400G (4G blocks per 200 entries).
Without support for large files (> 2G), however,
the limit of the one-file design may actually even be lower,
since the 2G file limits the total size of terms + IFPs.
* Basic operation:
The B-L-Tree relates keys to "inverted file pointers" (IFP).
The key is a byte sequence of a length between 0 and 255.
The IFPs are 8 to 11 bytes long, with 3 to 6 bytes for an mfn (rowid),
and 5 bytes describing where the key occurs within the given record:
2 field tag, 1 occ(repetition of field), 2 word position.
These numbers are in big-endian byte order for easy sorting.
The mfn/value length is fixed for a given B-Tree.
File structure:
The index file consists of consecutive mK blocks,
with block number n at file offset n*mK.
Block 0 is the root, Block 1 the leftmost leaf.
Block structure:
The inner structure of a block is similar to an ISIS record:
some header fields followed by a dictionary of entries,
which is an array of positions (offset into block) and lengths.
The dictionary is sorted according to the entry's keys.
The actual entry data is located at the end of the block,
growing towards the dictionary as entries are added.
Entry structure:
Each entry starts with len bytes key as stated in the dictionary.
For an inner node, it is optionally followed by an IFP,
as indicated by a dictionary flag, then by a 4 byte block number.
For a leaf node, it is followed by a sorted array of IFPs.
Searching:
Since it is not uncommon for a key to relate to more distinct IFPs
than would fit within one block, we actually consider the pair of
key and IFP as the key when searching the tree.
When looking for all IFPs for a key, we start with empty (nulled) IFP.
Upon modification (insert/delete), we use the given IFP as second key segment.
Where the IFPs for one key span several leaf nodes,
the separating inner node entry will have it's key augmented
with the IFP, so we may quickly locate the proper leaf node.
Deleting:
The entry is searched and deleted, no restructuring.
(Background cleanup may be implemented some day).
Inserting:
If a new entry fit's within the target leaf, everything is fine.
Else either the new entry, or one or more other entries may have
to be shifted to the right sibling node.
If the right sibling is non-existent or too full
(or we didn't yet implement checking it),
a new right sibling is created.
The new separating key, which is lower than the one we stopped
at while locating our node, is then updated or inserted in the parent.
Multi-Process Concurrency:
There *must not* be more than 1 process having the index open for writing.
Readers, however, should work correctly, even in the presence of a writer.
Multi-Thread Concurrency:
Within a multi-threaded process, all access to internal structures
is interlocked within a monitor. The monitor is released during I/O,
with the block marked as being read or written from/to disk.
A thread wishing to access a block being read, or wishes to write
to a block while it is written to disk, must wait on the monitor.
Lehmann/Yao Concurrency:
While this design should make sure that each block is always
in a consistent state, the interrelation between blocks may be
disturbed by node splitting or shifting:
The key may be found to be no longer in the expected block.
Thus, while the search key is greater than the greatest key
in the current block, one has to try the right sibling.
* Concurrency Details:
Within a multi-threaded process, all access to internal structures
is interlocked within a monitor. The monitor is released during I/O,
with the block marked as being read or written from/to disk.
This design will give the best possible over-all utilization on a
single-CPU system: letting a given thread run exclusively while it can
utilize the CPU. (I.e. unless it's waiting for I/O, swapping shouldn't
be an issue, if you're seriously running a db server).
If you have a multi-CPU system for nothing but OpenIsis,
run secondary read-only servers, each bound to one CPU.
This will be by far more efficient than sharing multiple CPUs
by one processe's threads.
Actually, we deploy one mutex (LOCK/UNLOCK) and one condition
(WAIT/WAKE) bound to it. Optionally a set of conditions bound
to the same mutex may be used.
All access to btree blocks is via a common cache, so that there
is never more than one copy of a block in memory.
During I/O, the cache block is flagged as being read or written, resp.
The read flag is held only temporarily during the I/O;
it marks the block contents as being asynchronously changed
(by the read call) and thus completely useless.
Any thread wishing to access the block has to wait on it.
After the read call returns, the thread initiating the read will
re-acquire the monitor, clear the flag and wake any waiting threads.
The write flag, on the other hand, acts as a WRITE LOCK on the block.
Threads wishing read-only access to the block completely ignore it,
since the block content is valid and will NOT asynchronously change:
it is changed by the writing thread only while it has the monitor,
thus no other thread will ever observe a change.
A thread wishing to modify the block, however, has to wait,
since a modification during an ongoing write could corrupt data.
Moreover, the lock need not be released immediatly after the write
call returns, but can be held until a second block is also written.
* The block reading sequence is as follows:
$
r0 find the block in the cache.
if it is not found, set it reading, release the monitor, read.
else, if it is being read, increase waiter count, wait (releases the monitor).
else use it (skip next state).
[r1] SUSPENDED
wait for either our own read to return or to be waked by another reader.
if we return from read, re-acquire the monitor.
(if we are awaked, the monitor is re-aqcuired implicitly).
r2 if we return from read, clear the reading flag, wake waiting threads.
if we are awaked, decrease waiter count.
(Since we might have less different conditions than blocks,
we have to check the reading flag to see whether we were meant
and possibly wait again).
we have the block wanted.
on searching, if it's not the leaf we wanted, start over.
$
NOTE:
- the thread starting the read on a block is always the first to get it,
since waiters can proceed only after this thread cleared the flag.
- for a block that already is in cache, the reader proceeds without
releasing the monitor. This especially holds for blocks being written.
These properties can be used for a save background cleanup.
- a reader does not need to hold a block in the cache while it is
suspended in I/O: all processing is immediately after reading.
For a secondary reader on a database being written,
at least leaf blocks should be invalidated asap.
(Otherwise it will obviously miss some updates).
* The overall search algorithm:
$
s1 starting from the root, locate the first entry with key
greater than our key, read the block it pointed to by the previous entry.
if there is no such entry found, and the block has a right sibling,
check it for possible split or shift (Lehmann/Yao).
s2 when reaching a matching entry in a leaf, read subsequent right siblings
according to the search range criteria and buffer.
$
NOTE:
- due to the right sibling checking, it does not lead to inaccurate
search results, if inner nodes have missing or too high keys
for some childs, it only has a performance penalty
(dropping the tree altogether means linear search).
* The general block writing sequence:
$
w0 reading "for update" as r0, but with additional condition:
if it has the WRITE LOCK flag set, wait, else use it (skip next state).
[w1] SUSPENDED
wait for read or to be waked by other writer.
on return from read, re-acquire the monitor.
w2 if we are awaked and there's a WRITE LOCK, wait again.
(the original reader or some other waiting thread might have set it).
after returning from read, set the WRITE LOCK flag, wake waiting readers.
if we need to write another block, start over.
process block(s).
release the monitor, start write call on first block to write.
[w3] SUSPENDED
wait for the write call to return.
repeat while there are more blocks to be written.
re-acquire the monitor.
w4 clear the WRITE LOCK flag on some blocks, wake waiting threads.
if there are more blocks to be processed, start over.
$
* The insert algorithm:
$
i0 locate the leaf L where to insert the entry.
if it has enough room, process it and we're done.
i1 if it has a right sibling S, read it for update.
if this hasn't enough room to take some entries from L,
release the write lock and use a new block S as L's right sibling instead.
shift entries to S, insert the new entry into L (or S).
i2 write first S, then L to disk.
release the write lock on L.
i3 read the parent P of S for update,
i.e. the parent of L or some block to the right.
if we used old S, delete it's old entry.
insert the new lower bound on the keys of S.
if the parent hasn't enough room, start over at i1 (with P for L).
$
when unfolded to synchronized/unsynchronized steps,
a simplified version of this might look like:
$
read your way down to L until it's read for update...
u0 not enough space in L, read S for update
[u1] wait for S (L locked)
v2 modify both blocks, initiate read of P for update
[v3] wait for P, write S then L (L,S locked)
v4 release lock on L and S, update P or go to u1 (P locked)
$
NOTE:
- this is deadlock-free, since locking is ordered left to right,
bottom to top.
- by always keeping one lock, we avoid conflicts with other writers
for the same entries, since those have to follow the same lock path,
and thus work always in a serializable manner.
- if we are the original reader for P,
other threads wanting to read P have to wait for three I/Os.
The detailled variant below avoids this.
- reading the sibling and therefore the [u1] suspend is optional.
depending on the key distribution, it might not hurt much
or even be more efficient to always create a new block.
- the only inconsistency a reader thread might notice is during v3
(or a later suspend in order to lock P's sibling):
there are new versions of L and S, but P is in the old state.
The key of S (or L's former sibling) is too high and thus an entry
now in S may be searched for in L. In this situation,
the reader has to follow L's right link to find the key.
There is a slightly more complex variant of insert,
wich allows for somewhat enhanced concurreny
but with higher synchronization effort:
$
u2 modify both blocks (L,S locked)
[u3] write S then L
u4 release lock on L (S locked), read P for update
[u5] wait for P
u6 release lock on S, update P or go to u1 (P locked)
$
* Multi-Process NOTEs:
- we have to assume, that block writes
* are atomic (seen by others completely or not at all) and
* are ordered (if a later write is seen, an earlier is also).
This does NOT hold for the file state on disk,
but should hold for the operating system's file cache.
- a reader in another process might also read the new version of
S but the old version of L and thus has to deal with duplicate entries
when searching ranges.
- if a reader is caching, and thus write order is not in effect,
it might miss entries completely.
This does still give correct results, if
* no leaves are cached
* for all cached blocks, the parent is older
This is ok for one-shot readers, or if caching is strictly from top
with forced re-read after cache miss on search path.
Deleting an entry is trivial, since it affects one block only (if we're lazy).
In a single-process multi-threaded environment, however,
we can take advantage of certain properties to provide
relatively simple and save cleanup synchronously or in background.
doc/charsets.html 0100664 0001130 0001130 00000014450 07566146164 013027 0 ustar krip krip
ISIS charsets and unicode
some notes on the use of charsets with ISIS
what are charsets?
Since computers can store nothing but numbers, but we want them to store
characters, there has to a table telling which character is stored as which
number, or, vice versa, which number is to display and print as which character.
such tables are called charsets .
one-byte-charsets .
CJK
) scripts with several thousand characters, it's not enough. The modern
vietnamese
script is based on latin letters but needs a vast amount of accented letters,
so 256 isn't enough. Those scripts don't get by with one byte per character,
so they need multi-byte-charsets , where two or more bytes are needed
to encode one character.what is UNICODE
UNICODE
is a big multi-byte-charset designed to include all
characters
needed in the world (over 40.000 by now), even for some ancient languages.
The problems having several charsets are a) you have to know which charset
is used in a given text, b) computer systems need to be aware of all possible
charsets and c) it's not possible to have a text or database contain characters
which are encoded in different charsets. Having all text in unicode solves
those problems. Check out
this sample page
- with a 21st century browser like Mozilla 5 (Netscape 6) you will see most
or all of the letters.ASCII-compatible charsets and encodings
Many charsets use the numbers 0 to 127 in the same way: to represent the
basic set of latin characters defined by
ASCII
. Whenever there's a byte with a number in that range, this byte has the
meaning of the corresponding ASCII-character. For example, the number 43 always
is a plus sign +, which is important if a query expression is scanned for
such characters.ISO-8859-x
charsets are ASCII-compatible. Older
Cyrillic
charsets are NOT compatible with ASCII. Some of the eastern multi-byte-charsets
are, some are not.encodings , that is,
there is only one table mapping numbers to letters, but distinct ways to use
multiple bytes to express such a number, some of which use the numbers in
the ASCII-range only for ASCII characters, others don't. UNICODE has two widely
used encodings, UTF-8
and UTF-16 (UCS-2). UTF-8 is ASCII-compatible , UTF-16 is not.so what about ISIS
the ISIS database format itself is capable of storing anything and
thus can store text in any charset/encoding.
the ISIS query and formatting language depends on special ASCII-characters
having special meaning and therefore will require an ASCII-compatible
encoding . All the ISO-8859-x charsets will do as will UTF-8 encoded
unicode (although some care must be taken when multiple bytes representing
one character are cut off in the midth). At least in theory, mx and
wwwisis are able to search for records in any ASCII-compatible
encoding including UTF-8 unicode (given carefull web-programming).
winisis doesn't know about the possibility of one character
having multiple bytes. It will work with any ASCII-compatible one-byte-charset
, as long as it doesn't have to know what it does. That is, if your computer
has some preferred charset installed, you will see all characters displayed
according to that charset, and a character possibly entered as the german
ä could show up as greek delta :). No support for multi-byte-charsets,
especially not unicode .Like any Java software,
JavaISIS
unicode and most of the other charsets.
Some limitations may result from the underlying wwwisis. In practice, version
3.5 claims to give "Multi-language encoding support", but unfortunately it's
in beta since March 2001 (sources made available in Feb 2002).
openisis supports any charset and with it's Java-binding,
especially unicode and all the conversions. openisis alone can
do it on the web, and in combination with JavaISIS (once new sources are
available) also with a winisis-like interface.
some other resources on unicode
To see all those characters, you need fonts to tell your display
or printer how they look like.
Here's a
very fine page
on how to acquire and install those fonts (and some more advice).
If you for some reason have to waste your time with M$ products,
you may want to check out
this page .
Especially there's the one-size(23 MB)-fits-all fat font
Arial Unicode MS (TM, (c), ... expect the worst)
containing nearly all unicode glyphs, which is also included
with newer Windoze and/or Ophice versions.
doc/encoding.txt 0100644 0001130 0001130 00000012242 07572664272 012651 0 ustar krip krip 95 single-byte and 45 multi-byte encodings as supported by Java rt+i18n.jar
$
ASCII American Standard Code for Information Interchange
Cp037 USA, Canada (Bilingual, French), Netherlands, Portugal, Brazil, Australia
Cp273 IBM Austria, Germany
Cp277 IBM Denmark, Norway
Cp278 IBM Finland, Sweden
Cp280 IBM Italy
Cp284 IBM Catalan/Spain, Spanish Latin America
Cp285 IBM United Kingdom, Ireland
Cp297 IBM France
Cp420 IBM Arabic
Cp424 IBM Hebrew
Cp437 MS-DOS United States, Australia, New Zealand, South Africa
Cp500 EBCDIC 500V1
Cp737 PC Greek
Cp775 PC Baltic
Cp838 IBM Thailand extended SBCS
Cp850 MS-DOS Latin-1
Cp852 MS-DOS Latin-2
Cp855 IBM Cyrillic
Cp856 IBM Hebrew
Cp857 IBM Turkish
Cp858 Variant of Cp850 with Euro character
Cp860 MS-DOS Portuguese
Cp861 MS-DOS Icelandic
Cp862 PC Hebrew
Cp863 MS-DOS Canadian French
Cp864 PC Arabic
Cp865 MS-DOS Nordic
Cp866 MS-DOS Russian
Cp868 MS-DOS Pakistan
Cp869 IBM Modern Greek
Cp870 IBM Multilingual Latin-2
Cp871 IBM Iceland
Cp874 IBM Thai
Cp875 IBM Greek
Cp918 IBM Pakistan (Urdu)
Cp921 IBM Latvia, Lithuania (AIX, DOS)
Cp922 IBM Estonia (AIX, DOS)
Cp1006 IBM AIX Pakistan (Urdu)
Cp1025 IBM Multilingual Cyrillic: Bulgaria, Bosnia, Herzegovinia, Macedonia (FYR)
Cp1026 IBM Latin-5, Turkey
Cp1046 IBM Arabic - Windows
Cp1097 IBM Iran (Farsi)/Persian
Cp1098 IBM Iran (Farsi)/Persian (PC)
Cp1112 IBM Latvia, Lithuania
Cp1122 IBM Estonia
Cp1123 IBM Ukraine
Cp1124 IBM AIX Ukraine
Cp1140 Variant of Cp037 with Euro character
Cp1141 Variant of Cp273 with Euro character
Cp1142 Variant of Cp277 with Euro character
Cp1143 Variant of Cp278 with Euro character
Cp1144 Variant of Cp280 with Euro character
Cp1145 Variant of Cp284 with Euro character
Cp1146 Variant of Cp285 with Euro character
Cp1147 Variant of Cp297 with Euro character
Cp1148 Variant of Cp500 with Euro character
Cp1149 Variant of Cp871 with Euro character
Cp1250 Windows Eastern European
Cp1251 Windows Cyrillic
Cp1252 Windows Latin-1
Cp1253 Windows Greek
Cp1254 Windows Turkish
Cp1255 Windows Hebrew
Cp1256 Windows Arabic
Cp1257 Windows Baltic
Cp1258 Windows Vietnamese
ISO8859_1 ISO 8859-1, Latin alphabet No. 1
ISO8859_2 ISO 8859-2, Latin alphabet No. 2
ISO8859_3 ISO 8859-3, Latin alphabet No. 3
ISO8859_4 ISO 8859-4, Latin alphabet No. 4
ISO8859_5 ISO 8859-5, Latin/Cyrillic alphabet
ISO8859_6 ISO 8859-6, Latin/Arabic alphabet
ISO8859_7 ISO 8859-7, Latin/Greek alphabet
ISO8859_8 ISO 8859-8, Latin/Hebrew alphabet
ISO8859_9 ISO 8859-9, Latin alphabet No. 5
ISO8859_13 ISO 8859-13, Latin alphabet No. 7
ISO8859_15_FDIS ISO 8859-15, Latin alphabet No. 9
KOI8_R KOI8-R, Russian
MacArabic Macintosh Arabic
MacCentralEurope Macintosh Latin-2
MacCroatian Macintosh Croatian
MacCyrillic Macintosh Cyrillic
MacDingbat Macintosh Dingbat
MacGreek Macintosh Greek
MacHebrew Macintosh Hebrew
MacIceland Macintosh Iceland
MacRoman Macintosh Roman
MacRomania Macintosh Romania
MacSymbol Macintosh Symbol
MacThai Macintosh Thai
MacTurkish Macintosh Turkish
MacUkraine Macintosh Ukraine
MS874 Windows Thai
TIS620 TIS620, Thai
Big5 Big5, Traditional Chinese
Cp930 Japanese Katakana-Kanji mixed with 4370 UDC, superset of 5026
Cp933 Korean Mixed with 1880 UDC, superset of 5029
Cp935 Simplified Chinese Host mixed with 1880 UDC, superset of 5031
Cp937 Traditional Chinese Host miexed with 6204 UDC, superset of 5033
Cp939 Japanese Latin Kanji mixed with 4370 UDC, superset of 5035
Cp942 IBM OS/2 Japanese, superset of Cp932
Cp942C Variant of Cp942
Cp943 IBM OS/2 Japanese, superset of Cp932 and Shift-JIS
Cp943C Variant of Cp943
Cp948 OS/2 Chinese (Taiwan) superset of 938
Cp949 PC Korean
Cp949C Variant of Cp949
Cp950 PC Chinese (Hong Kong, Taiwan)
Cp964 AIX Chinese (Taiwan)
Cp970 AIX Korean
Cp1381 IBM OS/2, DOS People's Republic of China (PRC)
Cp1383 IBM AIX People's Republic of China (PRC)
Cp33722 IBM-eucJP - Japanese (superset of 5050)
EUC_CN GB2312, EUC encoding, Simplified Chinese
EUC_JP JIS X 0201, 0208, 0212, EUC encoding, Japanese
EUC_KR KS C 5601, EUC encoding, Korean
EUC_TW CNS11643 (Plane 1-3), EUC encoding, Traditional Chinese
GBK GBK, Simplified Chinese
ISO2022CN ISO 2022 CN, Chinese (conversion to Unicode only)
ISO2022CN_CNS CNS 11643 in ISO 2022 CN form, Traditional Chinese (conversion from Unicode only)
ISO2022CN_GB GB 2312 in ISO 2022 CN form, Simplified Chinese (conversion from Unicode only)
ISO2022JP JIS X 0201, 0208 in ISO 2022 form, Japanese
ISO2022KR ISO 2022 KR, Korean
JIS0201 JIS X 0201, Japanese
JIS0208 JIS X 0208, Japanese
JIS0212 JIS X 0212, Japanese
JISAutoDetect Detects and converts from Shift-JIS, EUC-JP, ISO 2022 JP (conversion to Unicode only)
Johab Johab, Korean
MS932 Windows Japanese
MS936 Windows Simplified Chinese
MS949 Windows Korean
MS950 Windows Traditional Chinese
SJIS Shift-JIS, Japanese
UnicodeBig Sixteen-bit Unicode Transformation Format, big-endian byte order, with byte-order mark
UnicodeBigUnmarked Sixteen-bit Unicode Transformation Format, big-endian byte order
UnicodeLittle Sixteen-bit Unicode Transformation Format, little-endian byte order, with byte-order mark
UnicodeLittleUnmarked Sixteen-bit Unicode Transformation Format, little-endian byte order
UTF8 Eight-bit Unicode Transformation Format
UTF-16 Sixteen-bit Unicode Transformation Format, byte order specified by a mandatory initial byte-order mark
$
doc/formatting.txt 0100644 0001130 0001130 00000023644 07566146164 013243 0 ustar krip krip * supported features
As of version 0.8.6, openisis supports only a *very* limited
subset of the formatting language.
It should however be sufficient for the most important purposes:
building indexes and basic preprocessing.
- all kinds of literals (',",|)
- all modes (P,D,H)
- Vn[]^c[] field selector including repeated subfields
- implicit and explicit loops
The rest of this text describes sort of a merger of formatting
features from both WinISIS and CISIS/wwwisis,
which openisis may attempt to support one day.
* formatting basics
Formatting in openisis is separated into two tasks,
that used to be mixed up in traditional ISIS software.
- record processing
In openisis, execution of a ("print"-)format actually transforms
one or more records into one new ("output") record.
It loops and selects fields, applies Mxx modes, REFs other records,
but screen formatting directives are simply added as special fields.
In terms of relational databases, a format defines a view.
- screen rendering
It is then the task of separate rendering engines to turn those
printformat fields into well-indented plaintext or HTML or Postscript
or TeX or Windows GDI commands or you name it.
* elements of formatting
A formatting expression is a series of literals and functions.
Funtions use zero, one or more data items as parameters.
Functions that expect one of the parameters to the left are called operators.
* input types
The expected type of a function parameter can be one of:
- s string (auto-concatenated auto-stringified any)
- n numeric expression (including string, boolean)
- o output (auto-stringified, but not concatenated)
- v variable or iterator
- r row iterator (list of rowids)
Immediatly after the function name, the following can be used:
- c a single character
- a alphanumeric bareword (identifier in the C language)
- i integer literal
- x anything
* output types
The output of an expression is zero, one or more fields,
which can be string, numeric or value.
The type of the last field determines which operators are recognized.
Field tags are positive for values (i.e. the output of value iterators),
zero for literals and negative in the range -1 .. -999 for printformats.
Other negative tags are only temporary:
value iterators are finally evaluated to emit values,
conditional literals and numbers are later eliminated or
changed to zero, resulting in a literal.
* context
during processing, we have the following context:
o output record (not changed)
r input record/s, db and loop; changed by REF, LR and loops
f format, mode and variables; changed by @, Mcc
x frame: function, signature, parameter type and position
new frames are opened for functions, operators and blocks
* show stoppers: ( ) , .. ]
During processing the format string is scanned left to right
and fields are appended to the ouput record as encountered.
Function "calls" can be explicit (parameter list is enclosed in parentheses),
or implicit, taking only one parameter (operators and literals like V24, X3).
An opening parentheses following an operator or literal function makes the
frame explicit. A parameter type of i or a is changed to n or s, resp.
Otherwise a ( starts
- in numeric context: an anonymous (n) arithmetic parentheses
- in string context: an S(s_) function
- in output context inside a loop: an indentation function
- otherwise starts an explicit loop
A closing parenthes closes whatever was opened by the last opening one.
A comma ',', or range '..' closes a parameter,
which closes an operator or saturated implicit loop.
If there was no parameter to close, an empty string or 0 number is emitted,
thus [..] is equivalent to [0..0].
* functions and type coercion
Every function has a signature which denotes the types of the
expected parameters.
Functions and switches are put on the stack, the type expected
from the following expressions is set according to their signature.
Whenever a comma or the closing parentheses is encountered,
the fields added for this parameter are converted to the expected type.
This is not much work for numeric or boolean types,
since expressions are evaluated while parsing.
Where a string is expected, we might have a record (i.e. multiple strings),
which is then collapsed as with the S function:
all printformats are discarded and other strings are concatenated.
When a function sees it's closing ')', it replaces the parameters
pushed on the output record by the function value calculated from them.
The return value is tagged with the first data tag encountered,
even from records evaluating to empty.
The signature is denoted by a string, where
- the 1st char gives the left hand operand ('_' for none)
- the 2nd char is a digit of required params or '_' for specials
- the following chars give types of each parameter in turn
- an trailing '_' denotes, that the last type may be repeated
* record loops: "x" |x| + (s_) WHILE n (s_) CONTINUE BREAK OCC IOCC
A loop is either started explicitly by a '(' in record context,
or implicitly by a conditional literal or a V value iterator.
An explicit loop is closed by the matching ')',
while an implicit one ends when saturated (after first iterator),
with a comma ',' or any other V iterator.
The loop content is then repeatedly executed while incrementing
the OCC counter from 1 (and possibly subject to a preceding WHILE).
On each turn there is a "last" flag (initially true), which is cleared by any
iterator that expects to have more fields (this may even be true if
there was no OCCth field, e.g. if the OCCth field didn't have some subfield),
consequently, if there are no iterators, the first turn is the last.
Iterators also set or clear a "had" state (initially unknown).
During each execution of the loop, we're then in string type context:
- a "" conditional emits it's contents only on OCC=1
if before the first iterator, else on last=true
- a || conditional emits it's contents on had!=false
- a + undoes an immediatly preceding conditional on OCC==1
and sets had to false if last is true
- an iterator adds the OCCth of (the selected) occurences (see below)
- if an iterator has no OCCth occurence, an immediatly preceding
|| is undone and had is set to false
- other tokens are processed normally
* field iterator and operators: Vi Di Ni [n_] ^c
A field clause is always evaluated in a loop over OCC.
The iterator can be modified by the range and subfield operators.
What are the occurences in question, depends on the selected options:
field occ range, subfield selector and subfield range.
The integer list n_ may be given as x..y, where a missing x is 1 and
missing y or the keyword LAST means up to last occurence.
If no range is specified, that means all like [..].
If a subfield is specified, that subfield is selected.
If a subfield range is given, those occurences of the subfield are used:
if no range is specified for the field, subfield occs are relative
to the record, else relative to each individual field occ,
so use V71[..]^a[1] to get the first occ of a in each occ of v71.
With 71=^afoo^ax, 71=^abar^ay and 71=^abaz^az,
V71^a[1..3] will give the first three total occurences of subfield a,
i.e. foo, x and bar, whereas in CISIS it would give foo, bar and baz.
* other values and operators on values: Ei Si :=
* operators on string: *n .n (n) (n,n)
The * and . string operators modify the top (the last pushed field)
by removing the first n chars or all but the first n chars, resp.
An () indentation operator is a somewhat late indentation printformat,
which exchanges itself with the previous field.
* arithmetic operators on numbers: * / + -
* relational operators on numbers: = <> < <= > >=
* relational operators on strings: = <> < <= > >= :
* boolean operators on numbers: AND NOT OR
* literals: 123 123.45 'x' "x" |x| /*x*/ !cxc
* switches: IF n THEN o ELSE o FI SELECT s CASE s: o ELSECASE o ENDSEL
* format: @a MPL MPU MHL MHU MDL MDU
* db: DB MSTNAME MFN[i] L(s) NPST(s) NPOST(s) LR(s) LR(s[,n,n]) REF(r,o)
* xternal db: L->a(s) LR->a(s) NPOST->a(s) REF->a(r,o)
* printformats
# / % { } !cxc () QC QJ B I UL Ci Xi Fi FSi CLi NEWLINE(s) LW(n) PICT(s)
M(n[,n]) TAB[i] BOX[i] NP[i] NC[i] BPICT(s[,n])
FONTS(a_) COLS(s_) LINK(s_)
* other functions
&a(s_) CAT(s) GETENV(s) PUTENV(s) SYSTEM(s) DATE[i] PROC(s)
DATETIME DATEONLY VAL(s) RMAX(n_) RMIN(n_) RSUM(n_) RAVR(n_)
LEFT(s,n) RIGHT(s,n) SS(n,n,s) MID(s,n,n) REPLACE(s,s,s) INSTR(s,s) SIZE(s)
F(n) F(n,n) F(n,n,n) TYPE(s) TYPE(s,s) S(s_) LAST
NOCC(s_) P(s_) A(s_)
* unsupported syntax
L([s]s) NPOST([s]s) REF([s]j,...) -- hopeless, use winisis notation
* list of tokens by syntax
- empty tokens
stopper: , .. ) ] THEN ELSE FI CASE ELSECASE ENDSEL CONTINUE BREAK
state: MPL MPU MHL MHU MDL MDU
values: DB MSTNAME OCC IOCC # / % { } QC QJ B I UL DATETIME DATEONLY LAST
- immediate literal
i MFN[i] DATE[i] TAB[i] BOX[i] NP[i] NC[i] Ci Xi Fi FSi CLi Vi Di Ni Ei Si
@a L->a(s) LR->a(s) NPOST->a(s) REF->a(r,o)
^c 'x' "x" |x| /*x*/ !cxc
- syntax blocks (jump & run)
o ( o ) WHILE n ( o ) REF( r, o )
IF n THEN o ELSE o FI SELECT s CASE s: o ELSECASE o ENDSEL
- operators
[n..n] ^c (n[,n]) := . * / + - = <> < <= > >= : AND OR NOT
- all others are properly braced functions
- ambiguous tokens
S F might be Si, Fi or S(), F()
/ + * might be arithmetic or string operators or a newline
= <> < <= > >= might compare strings or numbers
:= assigns number to Ei, string else
( opens one or the other frame ...
* tokenizing & processing
* read a token and literal
- get (longest matching) token
- if token accepts a literal, get the literal
- if token accepts an opening (, get it
- resolve S/F ambiguity syntactically depending on presence of i literal
* process
- if token is possibly an operator of higher precedence,
check operator ambiguities,
coerce field according to operators wishes
and go opening the operator frame
- else
coerce field according to frame context
- if we had a field in o-context or token is a stopper,
close the parameter
- if the frame is implicit and saturated or token is a frame closer,
close the frame and start over processing
- add the token or literal
doc/fst.txt 0100644 0001130 0001130 00000003022 07566146164 011651 0 ustar krip krip fst lines read
"tag tec mode,fmt",
where
- tag is the field number used in the index posting
- tec is the indexing technique (see below)
- mode is a Mmc formatting mode or, for tec 5..8, a prefixing format
- fmt is some format expression, typically a single field like v24
* formatting modes
indicated by Mmc, where m is a character indicating some translations,
and c is a character indicating the handling of case; default MPL
m
- P proof mode: no changes applied
- H heading mode: angle brackets are removed,
^x is replaced as ';' for x=a, ',' for x=b..i, '.' for others
- D data mode: like heading mode plus '. ' after each field
c
- L lower case: no changes applied
- U upper case: (ASCII-)characters are converted to "uppercase" as listed
in the file ISISUC.TAB (a textfile containing 256 replacement byte values
like ... '096 065 066' ... in 16 lines of 16 numbers each)
* indexing techniques
- 0 one entry per line (format should produce appropriate newlines with /)
- 1 one entry per line or subfield (use proof mode MPx to retain ^x)
- 2 one entry per each term in (use proof mode)
- 3 one entry per each term /between slashes/
- 4 one term per "word":
a word is a sequence of 'alphabetical characters' as listed in the file
ISISAC.TAB (a textfile containing numbers like ... '098 099 100 101' ...).
"stopwords" (listed in the db's .stw-file) are ignored
- 5..8 like 1..4, but instead of applying a mode Mmc,
every entry with delimiter 'd' is given a prefix 'p' like
1 8 '/TI=/',v24 (prefix //-entries with TI=)
doc/gizmo.txt 0100664 0001130 0001130 00000004217 07607552054 012206 0 ustar krip krip from the Winisis "reference" manual:
$
i. Gizmo conversion table
This field may contain the name of a Gizmo conversion table to be
used for import. This facility is provided to facilitate
interchange of text containing diacritical marks between different
hardware and/or software platforms. A gizmo conversion table is an
ASCII text file, which you may create using any familiar text
editor, and is defined as follows.
The first character of the first line contains the diacritical
prefix (normally the @ sign for CDS/ISIS). Each subsequent line
has the following format:
dxyxyxyxyxy....xy
where : d is the diacritical code (e.g. 7 for acute accent), x is
the character to be accented, and y is the accented character.
A sample gizmo conversion table is given below:
@
6aàeè
7aáeé
(([
))]
CDS/ISIS uses this table to provide encoded diacriticals on export
(e.g. é will be converted to @7e), and to decode encoded
diacriticals on import (e.g. @6a will be converted to à). When a
Gizmo conversion table is used for import, CDS/ISIS will simply
strip the diacritical for all those characters not listed in the
table. For example, using the table given above, the input string
@7e. will be converted to é, whereas the string @7y will be
converted to y, because y is not listed in line 7.
Note: this feature is not implemented in this version.
$
at some (?) other source i found:
$
a gizmo is a masterfile containing text replacements.
gizmo records have fields with tags 1 and 2.
every occurence of the contents of field 1 is replaced
by the contents of the associated field 2.
gizmo translation can be used
- to perform charset translation, e.g. Cp850 to ISO-8859-1
- to translate special chars like <> with something nicer like /
- to htmlify output like with < to <
- to replace certain keywords that might be used in field contents
if a gizmo record has a field 11,
and it contains the three characters:
- asc then field 1 is used as decimal number giving a char value
- hex then field 1 is used as hex number (e.g. d7) giving a char value
in the same way field 21 may control interpretation of field 2.
$
doc/limits.txt 0100644 0001130 0001130 00000001071 07607550746 012361 0 ustar krip krip $
lengths in bits
record length: 15 (32K)
mfrl signed short in MST
DOS version has 8K
mfn: 24 bit (16M)
3 byte in IFP
mfn long in MST
db size(record position): 29 bit (512M)
xrmfb/xrmfp 20+9 in XRF
IFP occurence: 8 bit
IFP word count: 16 bit
#fields in FDT: 200
#lines in FST: 600
#stopwords: 799
#characters PFT: 10000
#characters PFT result: 64000
proposed limits:
record length: 24 bits (16M)
db size: 40 bit (1T)
index size: 32+13 bit (4G*8K blocks = 32T)
mfn: 32+8bit (256 databases per 4G records)
this needs 64 bit xrf (40pos+24len)
and 80 bit ifp
$
doc/longidx.txt 0100644 0001130 0001130 00000005225 07566146164 012530 0 ustar krip krip
With the emerging use of unicode in ISIS databases,
the 30 byte maximum on keys might become a somewhat
narrow limit for languages using 2 or 3 UTF-8 bytes per
character while still having lots of characters per word.
Where UTF-16/UCS-2 is used, all languages would be
equally affected by an effective maximum key of
15 characters.
Therefore, there is a clear need for a new index
structure supporting longer keys for some environments.
This would in itself not raise any compatibility issues,
since an index is always redundant and can be reconstructed
by any plattform in a suitable format for this plattform
(of course according to the plattform's builtin limits).
While it would be straightforward to just create some n03/l03
pair of files or change 01/02 parameters, and some tools
might even care for the control-file contents and work without
modification using such indexes, some other tools wouldn't.
In this situation, one may want to spend some time thinking
about how an index should be organised. For programmers having
coped with ISAM and databases in theory and practice,
the reasons for some design decisions of traditional
ISIS index structures are less than obvious.
* blank padding
a typical index-block lookup routine does not benefit in speed
from blank-padding. while the fixed-array organisation allows
for very easy Pascal implementation (where you can't cast
some part of a block to a struct), C code doesn't care.
instead, the additional space accounts for
additional IO and therefore slows down index access.
considering that many DBs even allow for compressed index files,
where every key is stored as a number of leading bytes it
shares with it's predecessor followed by the trailing difference,
one should expect some performance improvement from at least
eliminating all those blanks.
* block alignment
the node and leaves files are made up of blocks not aligned
at any page boundaries. Every B*tree in textbooks as well as
in real-world databases uses page-aligned blocks for best IO
performance. potential reader/writer concurrency problems
(readers reading partially updated blocks) are also reduced
for most systems when IO occurs at filesystem page boundaries.
* multiple files
The separation of one index structures nodes and leaves is
due to the fact that they do have different block sizes.
The obvious advantage of having two index file structures is to
reduce the space wasted by blank padding.
Using page-aligned blocks of fix size with unpadded keys
would allow the complete index to go into one single file.
Moreover, the postings could be stored together with the
leaves, further reducing IO.
---
$Id: longidx.txt,v 1.2 2002/11/13 18:52:55 kripke Exp $
doc/mx.txt 0100664 0001130 0001130 00000005524 07600076133 011477 0 ustar krip krip mx overview
running mx with no params yields:
=========
CISIS Interface v3.33/GC/M/30000/10/30/I - Utility MX
Copyright (c) BIREME/PAHO 1997. All rights reserved.
mx [cipar=] [{mfrl|fmtl}=]
{|{in|cgi|seq|iso[=]}=|tmp} [ [...]]
options:
[bool=]{|@}
text[/show]=
{from|to|loop|count|tell|load}= [now]
gizmo=[,] [gizp=] [decod=]
join=[,]={|@} [jmax=]
jchk=[+]={|@} [actab=]
proc={|@}
fst={|@[]} [stw=@[]] [ln{1|2}= [+fix[/m]]]
{+|-}{control|leader|xref|dir|fields|all}
pft={|@[]} [lw=]
sys[/show]={|@}
[mono|mast|full] {create|copy|append|merge|updatf}=
ifupd[/create][/dict]=
{iso[=]|fix}=
=========
* db to use
basename of dbfiles, e.g. db/cds/cds
tmp or null denotes a pseudo-db with an unlimited no of empty records
iso= full name of iso2709-file, e.g. db/cds/cds.iso
seq=[] plaintext input, fields separated by char (default |)
* general options
now proceed without prompting (nowait)
-all suppress printing record contents (but print errors and tell=)
tell= print notice every records (usually with -all)
in= read params, one per line, from textfile
* record selection
from= start at MFN (default 1)
to= run up to MFN (default last)
loop= select every th record (default 1)
count= select records (default all or as by to)
[bool=] select by , e.g. bool='water*plants'
[bool=]@ read query from
text[/show]= fulltext scan for (more output with /show)
* processing options
gizmo=[,tags] apply gizmo translation to all or given fields
tag numbers are seperated by , tagrange by / (e.g. n/m is n to m)
join= join additional db by key (.... TODO)
proc= process records according to (see below)
proc=@ read processing cmd from
* key generation
fst= use field selection table
fst=@[] read fst from (default .fst)
stw=@[] read stopwords from (default .stw)
ln= create linkfile (1 or 2) as (see below)
+fix[/m] select linkfile format suitable for sorting (see below)
* controlling output format
pft= use printformat string , e.g. mfn/v24/v70
pft=@ use printformat from
lw= set linewidth to (default 78)
* linkfile format
linkfiles are textfiles with one line per posting,
containing the key, mfn, tag, occ and cnt.
standard format: mfn tag occ cnt key (blank separated)
+fix: key mfn tag occ cnt (blank padded to max len)
+fix/m: key mfn (blank padded to max len)
doc/sfindex.html 0100644 0001130 0001130 00000004614 07615234152 012640 0 ustar krip krip
OpenIsis and other ISIS related projects at sourceforge
OpenIsis
By now, OpenIsis is mainly used for web publishing of Isis databases.
Web developers use the Java, Perl or PHP binding for this.
developing OpenIsis
openisis.org
doc/threading.txt 0100644 0001130 0001130 00000007007 07623223632 013017 0 ustar krip krip
the benefit of multithreading depends heavily on two factors:
- whether the db is in the filesystem cache or needs real disk I/O
- whether you use Java or pure C
if there is real disk I/O, then multithreading gives a big
performance enhancement: 8 threads outperform a single one
by a factor of 4.2 (C) or 3.3 (Java) respectively,
and C outperforms Java only by a factor of
1.3 (single-threaded) or 1.6 (8 threads).
if there is sufficient memory available to cache the whole db,
so that the process is mostly CPU bound,
then 8 threads give only a 9% throughput advantage over one in C,
and an 18% performance degradation (!) in Java.
C outperforms Java by a factor of 50 (yes, 5000% percent !)
with 8 threads or 38 single-threaded.
======
conclusion:
For a mostly I/O-bound server, you might well use threaded Java,
it is only about 40% slower than multithreaded C,
but still beats single-threaded C by a factor of 2.5 .
If, on the other hand, you want to go for real performance,
get the memory needed and use C.
As long as you are not concerned too much with fairness and there
is no risk of some too lengthy operations, it may as well be
single threaded.
======
the following numbers were derived without any thread interlocking (mutex)
on a single processor
some numbers from the C multithreaded crashtest (make crash)
on the unesb db for prefix "Z" running on an 800MHz PIII with 1/4G RAM
under Linux 2.4.13
409 terms
4205 postings max mfn 58880
* with lots of memory available for caching:
$
sequential read 58880 rows in 0.618 seconds 95275 rows per sec
8 threads 82384 rows per sec
2 threads 79600 rows per sec
1 threads 75544 rows per sec
4 threads 81916 rows per sec
real 0m3.040s
user 0m1.860s
sys 0m0.920s
$
* with nearly all memory locked down by root:
$
sequential read 58880 rows in 6.720 seconds 8761 rows per sec
8 threads 1199 rows per sec
2 threads 467 rows per sec
1 threads 288 rows per sec
4 threads 624 rows per sec
real 5m6.422s
user 0m1.940s
sys 0m2.230s
$
some numbers from the Java multithreaded crashtest (make jcrash)
on the same db and machine
409 terms
4205 postings max mfn 58880
* with lots of memory available for caching:
$
sequential read 58880 rows in 30.508 seconds 1929 rows per sec
8 threads 1645 rows per sec
2 threads 1836 rows per sec
1 threads 2013 rows per sec
4 threads 1785 rows per sec
129.00user 7.40system 2:20.80elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (4978major+7237minor)pagefaults 0swaps
$
* with nearly all memory locked down by root:
$
sequential read 58880 rows in 58.827 seconds 1000 rows per sec
8 threads 738 rows per sec
2 threads 380 rows per sec
1 threads 225 rows per sec
4 threads 558 rows per sec
123.68user 3.87system 8:16.98elapsed 25%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (5286major+7366minor)pagefaults 0swaps
$
* on a 360MHz Mob. PII with 40MB and Linux 2.2.12,
numbers differ even more:
without disk activity,
a single thread gives about 50% more throughput than 8
with lots of disk I/O on the other hand,
8 threads nearly quadruple the single thread throughput
* the mobile now with an 2.4.18 kernel:
$
enough memory:
8 threads 54558 rows per sec
4 threads 49822 rows per sec
2 threads 43575 rows per sec
1 threads 37656 rows per sec
with mutex:
8 threads 49525 rows per sec
4 threads 45174 rows per sec
2 threads 41859 rows per sec
1 threads 36262 rows per sec
no memory (mutex):
8 threads 696 rows per sec
4 threads 367 rows per sec
2 threads 258 rows per sec
1 threads 150 rows per sec
$
doc/trad-fmt.txt 0100664 0001130 0001130 00000014006 07600076166 012572 0 ustar krip krip the status quo:
===============
notations:
UPPERCASE letters are verbatim
a,b,x string literals
c,d,e single characters
f,g,h formats producing a single string value
i,j,k,l,m,n,o numbers (formats processed in integer context)
ll list of numbers (pushed by an expression)
* mode
Mcd c = P, H or D, d = U or L
P proof mode: no changes applied
H heading mode:
^x is replaced as ';' for x=a, ',' for x=b..i, '.' for others
angle brackets are removed (>< replaced by '; '):
gives b in index, a else
gives nothing in index, a else
D data mode: like heading mode plus '. ' after each field not followed
by a suffix. no '.', if field ends in "punctuation" character.
L lower case: no changes applied
U upper case: characters are converted to "uppercase" as listed
in the file ISISUC.TAB
* field selector s
Vn^c[i..j]
Vn[i..j]^c select field n (0 = all), optional subfield c (* = first),
optional range of occ i to j (j may be ommited or LAST),
Dn^c select presence (optionally of subfield c)
Nn^c select absence (optionally of subfield c)
NOCC(s) total number of occurences of field s (NUMERIC)
P(s) true, if s is present (BOOLEAN, see IF)
A(s) true, if s is absent (BOOLEAN, see IF)
["x" a ["x" a ...]][|x|[+]]s[*o][.l][(k[,m])][[+]|x|]["x"]
here [] are not literal, but indicate optional parts.
x are strings not containing their respective delimiter
a arbitrary formatting or mode commands conditionally executed
repeated prefix or suffix in || output for any repetition,
with + first prefix / last suffix is ommited,
optional string offset o (0=start) and length l,
optional indentation k,m (see formatting)
* structural elements
they do not themselves generate output,
but rather control execution of formats u,v,w
, separator (end some construct)
/*...*/ comment
'a' unconditional literal
@a, include format file a
&a(f) call external function a
(u) repeat group (over IOCC, only in record or string mode)
BREAK break a repeat group or REF
CONTINUE continue a repeat group
SELECT f CASE g: u CASE h: v ... ENDSEL
SELECT f CASE g: u CASE h: v ... ELSECASE w ENDSEL
REF(m,u) apply format to row number m
REF->a(ll,u)
REF([f]ll,u) apply format to row number m in db
En:=m assignment to numerical variable
Sn:=(f) assignment to string variable
IF t THEN u FI
IF t THEN u ELSE v FI
WHILE t (u)
t a boolean expression using:
fxg comparision, where x is one of = <> < <= > >=
AND NOT OR () logical combination
f:g true iff f contains g case insensitiv
P(s) A(s) (see field selector)
f format not empty
* calculation (NUMERIC)
calculation is performed on longs until a decimal point is seen
or RAVR is used.
statistical functions run over format giving a list of numbers,
separated by ; or blank or any other non-numerical characters
+-*/() arithmetic
123 integer literal
123.45 float literal
VAL(f) numeric value of string
RMAX(ll) maximum
RMIN(ll) minimum
RSUM(ll) sum
RAVR(ll) average (always float)
En numerical variable n
* database functions
OCC
IOCC current occurence (1..) in repeat
MFN current mfn (with 6 digits) (NUMERIC)
MFN(n) current mfn (with n digits)
DB
MSTNAME current masterfile (possibly changed by REF)
L(f) mfn of 1st posting for key f (NUMERIC)
L->a(f)
L([g]f) mfn of 1st posting for key f in db (NUMERIC)
LR((f))
LR((f),m,n) list of mfns (postings m through n)
NPOST(f)
NPST(f)
NPOST([g]f) number of postings for f (in db) (NUMERIC)
* system functions
CAT(f) contents of file f
DATE date as yyyymmdd hhmmss w nnn (w day of week, nnn julian)
DATE(DATETIME) dd/mm/yy hh:mm:ss
DATE(DATEONLY) dd/mm/yy
DATE(n) mm-dd-yy (n=2), hh:mm:ss (n=3) or both (n=1)
GETENV(f) value of env variable f
PUTENV(f) read f as x=y, set env var x to y
SYSTEM(f) execute system command
* string functions
LEFT(f,n) substring (first n chars of f)
RIGHT(f,n) substring (last n chars of f)
SS(n,m,f)
MID(f,n,m) substring (m chars starting at n)
REPLACE(f,g,h) replace g in f with h
F(f)
F(f,n)
F(f,n,m) number to string
S(f)*n.m stringify format f, apply optional offset n, length m
INSTR(f,g) pos of g in f (0 = none) (NUMERIC)
SIZE(f) length of string f (NUMERIC)
TYPE(f) A (if alphabetic), N (if numeric) or X (else)
TYPE(?a?,f) 0 or 1, if f matches pattern .... ????
TYPE(n,f) 0 or 1, if f is alphanumeric (n=1), alpha(n=2), digits(n=3),
integer, incl. sign(n=4), number, incl. E-notation(n=5)
Sn string variable n
* changing records
PROC(f) apply changes resulting from f:
D* delete all fields
Dn delete all fields n
Dn/m delete occ m of field n
Ancac add string a (delim by any c) as new field n
Hn m a add string a of length m as new field n
* formatting
# newline unconditional
/ newline if not on line start
% reset blank lines
{
} open/close formatting scope
Vn..(m) or Vn..(m,o) indentation in columns for one field selector
NEWLINE(f) change newline to emit f (\r,\n are recognized in f)
Cm mv to column m
Xm m blanks
M(n)
M(n,m) indentation in twips (twentieth point = 1/1440 inch)
TAB mv to some (?) position
TAB(m) mv to position in twips
LW(n) linewidth
QC center
QJ justify
BOX
BOX(n) box (with color n)
NP
NP(n) newpage (if less than n lines left)
NC
NC(n) new column (if less than n lines left)
FONTS((a,b),...) setup fonts table, each font with family a and name b
COLS((i,j,k),...) setup color table, each color with RGB-values i,j,k
Fn select font n
FSn select fontsize n
B bold
I italic
UL underline
CLn select color n
!cac print ESC a
PICT(f) picture
BPICT(f) background picture
BPICT((f),m) background picture in mode m
LINK((f),g,...) link text f to command g, where g is:
* link commands
TEXTBOX f open box with text f
TEXTBOXCHILD f same, as child of DB window
TEXTBOXRCHILD f same, as child of record window
TEXTBOX[[R]CHILD]LOAD f[,a,b] display format f, optionally changing a to b
TEXTBOX[[R]CHILD]IMG f display img file f
TEXTBOX[i]... with literal [] identify window number i
OPENFILE f open according to filetype
CMD f run external prog f
GOTO m display record with mfn m
LGOTO f display record with key f
GOBACK ... :)
FORMAT f[,a,b] switch format to f, optionally changing a to b
BROWSE a[,m[,f]] browse DB a, mfn m, format f
\,{ and } "must be escaped with a \" (???)
doc/unirec.txt 0100644 0001130 0001130 00000007406 07644300511 012335 0 ustar krip krip
The universal ISIS record
Users of CDS/ISIS are accustomed to stuffing not only bibliographic,
but all sorts of data into ISIS records. The 1994 edition of ANSI/NISO
Z39.2 ("Information Interchange Format" alias ISO2709), after which ISIS
records are modelled (more or less), contained a "reduction of references
to 'bibliographic' data, because the standard is used for many other types".
For example, CDS/ISIS uses a ISIS database to hold the various texts
for language specific versions. Various control files like the syspar,
FDT and FST are well suited for storage in ISIS records.
Probably some implementations of CDS/ISIS internally use ISIS records
to hold that data, but none seems to be able to read/write those to/from
ISIS master files, ISO2709 or a choice of textformats, including XML.
* nesting
But then, there is more. I already mentioned that e-mail and many
simple XML structures are conveniently stored in ISIS records.
But even data which does not all that obviously follow a tag-value-list
scheme can be fit easily into such a record. Consider what is happening
when ASN.1/BER-encoded structures are sent down the wire to a Z39.50 server,
for example structured Type-1 queries to locate bibliographic information:
They are turned into a series of tags and values. "Twain OR Clemens" is
sent as an "OR" field followed by two term fields valued Twain and Clemens.
We can do that same serialization trick, of course:
embedding one structure (i.e. record i.e. tag-value-list)
into another by simple inserting the fields.
That way we can achieve nesting of arbitrary depth no less than with XML.
One problem that comes to mind: how do we tell the boundaries?
- for structures with a fixed number of fields, like the "OR" node
having two childs, boundaries are implicit.
- the length (number of childs) may be given with the opening field.
This may be inconvenient and/or error-prone, if not computed automatically.
- a closing item like "" may be used, e.g. a reserved field tag.
These approaches are now
> Struct discussed in more detail.
Based on such a schema, not only queries can be expressed and stored
as records, but also formats, with proper nesting of IFs, loops and so on.
This approach has a couple of advantages:
- formats may be specified in any of a couple of external representations
including XML
- the variants of the CDS/ISIS formatting language with different
names for the same functions can be supported using input filters
- formats can be stored, retrieved and exchanged using standard means
On the other hand, the formatting language could be augmented
to support substructures. A straightforward and relatively easy to
use and implement extension would be a PASCAL-style WITH r DO.
The current OpenIsis bindings, especially Tcl as preferred formatting language,
contain such support.
* external representation
Besides CDS/ISIS master files and ISO2709 files,
there are a couple of text based formats suitable
to store or exchange ISIS records.
Most follow a name=value style and are using separators like '=',
':' and linebreaks, with different quoting rules.
Among these are
- RFC 822 emails
- Java properties
- Windows-style .INI files
- character or tabulator separated values (tsv/csv)
(think of the TAB as subfield delimiter)
Then there are XML/HTML/SGML and finally freestyle languages,
like the query or formatting language, where item boundaries
are determined depending on context.
Conversion would typically be based on one or more FDTs,
mapping between names and numbers.
Such a mapping, when used with formats, could also enable the
use of symbolic field names like author instead of v24.
The "plain" representation as preferrably used by OpenIsis
is described in the papaer on
> Serialized
records.
-----
$Id: unirec.txt,v 1.6 2003/04/07 13:12:43 kripke Exp $
doc/whatabout.txt 0100644 0001130 0001130 00000032321 07623223671 013050 0 ustar krip krip * what makes ISIS ISIS ?
Andrew Giles-Peters raised the important question
"What is it about ISIS that makes it ISIS?"
So here are some thougts on this topic from the OpenIsis team:
- As a database used for bibliographic data (among other),
ISIS must be able to store and retrieve records as exchanged via
ISO2709 efficiently and with no or minimal loss of information.
- Besides the ability to retrieve records by number,
ISIS must support an indexing mechanism which is essentially
"function based", that is, index entries are not the immediate
field values, but rather the values of a "view" derived by
some computation are indexed.
- ISIS must efficiently support typical query elements commonly
used on bibliographic databases, like looking up a value without
regard for the field or in several fields at once and specifying
a distance within search terms should occur.
Since these are minimal requirements,
they would not stop anybody from adding tons of features on top.
For example, it's relatively easy to store ISO2709 data in a
relational database like Sybase (used by OCLC/Pica),
each record covering several rows (mfn, field number, field occ, value),
then compute a second similar table for the index and so on.
However, there is the word "efficiently",
which practically turns out to put some restrictions on the
feature-load, especially when combined with:
- ISIS must be widely usable even in the face of *very* low budgets.
Therefore, not only the software itself must be available for at most
a nominal fee, but it also must not require very new, very powerful
or otherwise expensive hardware and system.
Even very large catalogs should get by with moderate system costs.
The OCLC/Pica system for example requires one to spend
hundreds of thousands of dollars for powerful Sun machines.
* end of story ?
Still, it would be very nice if more areas of application could
be explored for ISIS, both for the librarians in order to be able
to use their favourite DB (i.e. ISIS) for a broader range of
tasks and also to expand the user community, possibly leading
to more support for everybody.
One important question is whether ISIS needs some fundamental changes
deep in it's guts, or whether it already has everything that's needed
to build a broad range of sophisticated solutions on top of it.
As you might expect, we are pretty well convinced of the latter.
* file formats
Just like it doesn't harm a database much to be exported to and
imported from ISO2709, there is not much of a problem with different
file formats, as long as there do exist conversion tools.
As you know, CISIS/Unix-DBs are incompatible to WinIsis/DOS-DBs,
but may be converted via ISO files.
As long as the basic data structures are the same,
lossless conversion is just a matter of tools.
It's even less of a problem if the software itself can read
several file formats (like openisis does).
You won't care much whether your wordprocessor is reading
a .doc or .rtf file, would you?
We did an interesting and very successful study implementing
an ISIS-like DB in pure Java using a plaintext masterfile
very similar to the Mbox mailfolder format
(hope to be able to release the code soon).
Likewise there is no reason why one should not be able to read
directly from an ISO2709 file.
Besides convertible masterfile formats, one might well use other
formats for xref and index, which always can be reconstructed as needed.
There are several reasons like improved performance or robustness to do so.
So I don't think ISIS is defined in terms of detailled file formats,
but rather in terms of the basic data structures.
One problem that might come to mind when talking about file formats
are the limits. While the maximum number of records per DB as well
as the maximum total file sizes are bypassed relatively easy
by logically joining several databases, the maximum record size of
about 32K is a limit which might be unacceptable for some applications.
(Although it can partly be resolved by deploying external files
like OCLC/Pica does to circumvent Sybase's varchar limits).
Raising this limit would clearly restrict lossless conversion to one way,
from small to large DB. Where a large DB model is needed,
all parties developing ISIS software should agree on one format
to allow for as-painless-as-possible interoperability.
* so what kind of database is ISIS ?
Classical database theory basically distinguishes ISAM,
network, hierarchical and relational database systems.
ISIS is strongly related to ISAM DBs, however it's flexible
indexing is rarely paralleled by any of these systems
and it's non-flat data model is targeted by hierarchical DBs
only (in greater generality and with much higher costs).
- Although direct joins by MFN shouldn't be too costly,
ISIS is not the database of choice when several records
typically need to be combined in queries or transactions.
However, in many application cases, only one ISIS record is
needed as opposed to several relational table rows.
In such situations, ISIS is even an excellent and efficient
transaction (OLTP) database (since save writing of an ISIS
record is much simpler than other DB's undo/redo logs).
- ISIS is not the database of choice when records are updated
by the hour. However, where only about 10% of records are
changed between two (monthly, weekly or daily) runs of backup
and compactification, the space overhead is not a big problem.
Where old versions of data need to be retained anyway
(as often needed and supported, for example, by postgres history),
you would hardly find a more efficient solution.
- ISIS is not the database of choice when it comes to high volume online
analytical processing (querying statistics on several dimensions, OLAP).
However, after reading some database books and Oracle manuals,
one learns that OLAP requires a well designed ("star schema")
database separate from the transactional one, anyway.
- ISIS does not, in itself, provide any concurrency control
(actual implementations do, to some extend). This doesn't
hurt when running a read-only multi-user catalogue,
a stand-alone application and in some insert-only situations.
For distributed multi-client update, there are mechanisms based
on timestamps or stored procedures that need to be supported
by some ISIS server to come.
While these data models are strongly tied to the logical
nature and physical organisation of the data,
newer notions like that of an 'object oriented' or 'XML'
database rather describe a way to use and access a database.
Actually OO or XML DBs are usually based on one of
the above mentioned systems (mostly relational ones).
For the most part, using a DB as OO or XML storage does require nothing
but some libraries and optionally precompilers for C++ or Java
-- these can be build on top of existing ISIS without changing it,
and ISIS will be an excellent choice for many applications.
Some aspects of increased functionality and performance will
require sort of "stored procedures" running inside the database.
In the case of a XML DB they are used for example to decomposite structures,
in the OO case they might need some sort of "magic switch" (method
overriding) to perform differently for some records than for others.
We believe that all this magic can be achieved based on ISIS.
The concepts of an ISIS database server and a scripting language as an
alternative to formatting exits are to be discussed elsewhere ...
First we want to shed some more light on the great flexibility the
ISIS database system has by it's very nature.
* ISIS is a mail database
Looking at http://www.faqs.org/rfcs/rfc822.html (or its updates)
one will find many similarities between ISO2709 records and internet mails,
which are, after all, essentially a series of header names and values.
After assigning numbers to the 100 or 200 most commonly used headers
and some sort of subfield encoding (e.g. "^nname^vvalue",
"namevalue" or simply "name: value") to store other header lines
with a special field number, mails are easily and very efficiently
stored in an ISIS database. Given the enormous number of communication,
groupware and workflow systems that are nowadays built upon standard plain
internet mails (typically using a set of special mail headers),
this is a very large area to be served by ISIS databases.
The above mentioned Mbox-style implementation of ISIS tends towards
that direction, building upon the javax.mail standard.
IMAP mail servers could greatly benefit from the powerful indexing
and retrieval system of ISIS databases.
If also the mail sending application allows to select special headers
from an entry form prepared by a skilled librarian with thesauri
and systematics, an institution or company could really come to a new
way of using mail as a system of qualified, living information.
* ISIS is a multimedia database
After all the mail not only has got headers, but also a body.
A plaintext body of reasonable length (some KB, like sent by nice people),
fits without problem in a field whose number means "body".
A multipart body is easily decomposed to a series of body fields.
Wether larger or non-plaintext bodies are stored within or outside
the masterfile is a matter of the actual implementation and doesn't
need to be discussed here, both approaches have their pros and cons.
Anyway, the MIME standard, up and running since 1982,
allows for storage and transmission of anything that uses bytes,
and is easily integrated with ISIS databases
(we partly did it, code to be released).
* ISIS is a XML database
Likewise XML, which basically is text, can be stored in an ISIS database
(with respect to the implementation's maximum record length).
Add some formatting exits to address the XML node content via
a DOM-style a.b.c notation as used in javascript, use them in your FST
and you will for sure have one of the world's best indexed and fastest
XML database -- most others are using a relational DB as basis.
So indexing, retrieving and displaying XML data is more or less
simply a matter of some formatting functions.
However, when thinking about data entry forms, for example,
the dark side of the force shows up:
Even with a very sophisticated database system with the ability to make
sense out of XML DTDs, it is anyway potentially much more complicated.
XML was meant to provide arbitrary complexity in the first place.
And when it comes to DTDs like that of XHTML, which will carry just about
the same content as any HTML page, one easily understands that reasonable
automatic processing becomes nearly impossible -- that's the reason why
HTML pages are largely beefed up with headers (Dublin Core and others).
If you really desperately need it, it's good to have it,
but else using it might be looking for trouble.
When having to work with XML structures for one or the other reason,
typically because they should be imported or exported,
one should think of a mapping between XML and ISIS structures.
In many situations XML structures are shallow and can be ISIfied by
simply mapping the first level of sister nodes to ISIS fields and the
second level to subfields (may require repeated subfield support).
In other situations a closer look at the data structure may reveal
that it is not well designed with regard to Ockham's razor but contains
totally unnecessary depth which may be collapsed to the first case.
Actually, during several years of work with XML structures as
suggested by several "standards", I rarely found a reasonable
structure which can not be mapped to a field-subfield-schema.
But even if you really need XML structures "as is",
they can be stored very
> Struct efficiently
in ISIS, with all the benefits of the flexible index
(c.f.
> unirec the universal ISIS record)
.
Anyway, Dublin Core metadata or other RDF (resource description framework)
headers are conveniently stored in ISIS just like mail headers.
Maybe, as this schema was created to suit the needs of the very
old science of bibliographic knowledge management, much of that
experience was built into it.
On the other hand, XML's ancestor SGML was conceived for a document's body,
not the head, and I guess there still is it's place in spite of
programming industry's hype. The use of XML for structuring documents
that are ment to be read by humans rather than machines of course
is perfectly reasonable. Transparent access to file based data associated
with a record and a XML add-on to the formatting language could aid
in converting extracts of document contents to metadata accessible
in the ISIS database and/or it's index.
To wrap it up, I'd suggest to look at XML as an optional add-on to ISIS
rather than an integral part. ISIS already has all the functionality
needed to support any reasonable use of XML. ISIS data can much more
efficiently contain XML structures than the other way round.
* ISIS is a database for document/content management systems
It follows that ISIS may very well support the needs of
systems for XML documents or website content in XML or HTML.
With increasing experience with such systems, people tend to
understand that content metadata should be organized according
to bibliographic principles. (Not that surprising, is it)?
In cooperation with the oc4science.org there are projects at german
universities to integrate publishing, document management and website CMS,
based on an (Open)ISIS DB and directed by the librarian.
-----------
$Id: whatabout.txt,v 1.8 2003/02/14 17:30:33 kripke Exp $
doc/writing.txt 0100644 0001130 0001130 00000004505 07566146165 012550 0 ustar krip krip Since version 0.8.7 (October 2002) openisis is able to store
new or updated records in the masterfile.
Programmers may call the function openIsisWrite,
see openisis.c for sample usage.
From the command line, the openisis executable accepts
a new -write option which specifies a database,
where all retrieved records are written to.
This works with all record selections like dump, search, scan, mfnlist
and the -stream option for reading records from text files.
examples:
$
# dump (copy) the whole unesb database to cds
openisis -db db/unesb/unesb -write db/cds/cds
# copy only records found for azteca
openisis -db db/unesb/unesb -search azteca -write db/cds/cds
# add the syspar.par as new record to the cds db
openisis -stream -write db/cds/cds $BASE/boot.log
$
doc/xmlisis.txt 0100644 0001130 0001130 00000006317 07644300512 012541 0 ustar krip krip * some notes on the relation of XML and ISIS
XML is in widespread use as a lingua franca
for glueing software components together.
Several tools for this can be found at xml.apache.org.
What is missing here is an efficient, easy to use
way of storing XML data. Only the most trivial cases
are easily mapped onto the relational data model,
which uses flat records, consisting of a fixed number
of fields. The data structures modelled in XML
typically have a variable number of childs.
Hierarchical databases like ADABAS C are well suited
and actually used by SoftwareAG in their Tamino XML DB,
but aren't widely and freely available.
* ISIS to XML
ISIS records can be easily and canonically converted to XML.
Anything up to the first subfield delimiter is the body,
subfields are attributes
(strictly XML-ish this is ok only for non-repeated subfields).
Other special subdivisions of field content like the typical
<key word> may split to real child nodes.
The result (as generated by make pdemo) may look like:
$
Educational Psychology
universities
Kenya
Okatcha, F.M.M.O.
Personal statement
Symposium on the Psychological Bases of Programmed Learning
$
Instead of tag numbers and subfield characters,
symbolic names from the FDT may be used.
* XML to ISIS
XML data structures can be
easily and efficiently mapped to the data model of ISO2709.
The general conversion (based on a SAX parser) works as follows:
* when encountering an opening tag, look up it's name in the FDT.
If there is no FDT provided, create one on the fly.
If the FDT does not contain the tag name,
create a new entry using tag number max(100,1+highest tag in FDT).
Create a field using the tag number found and field value '+'.
* when encountering an attribute, look up it's name in the FDT
(using either
> Meta extended FDT
entries or the FDT21 as of ISISMARC).
Create a new subfield entry if needed using code 'a'
or 1+highest code used (for this tag).
Append a subfield using the code found.
* When encountering a self-closer (the current field ends with />),
change the starting '+' to '-'.
* When encountering a text node, add a field using tag number 0
with the node's body as value.
* When enceountering a closing tag, look up it's name as for opening tags,
add a field with an empty value.
For example look at RDF (
> http://www.w3.org/RDF
,
> http://archive.dstc.edu.au/RDU/reports/RDF-Idiot
).
A structure like
$
Dr Jacky J Crystal
Director
jacky@dstc.com.au
Researcher
$
canonically maps to
$
100 +^aResource
101 +
0 Dr Jacky J Crystal
101
102 +
...
$
.
If they had made an attribute what can be an attribute
(not substructered, not repeatable) instead of a child,
it would read (with explicitly assigned subfield codes)
much more efficiently like
$
100 ^pResource^fDr Jacky J Crystal^tDirector^ejacky@dstc.com.au^rResearcher
$
Also see
> unirec
and
> Struct
---
$Id: xmlisis.txt,v 1.5 2003/04/07 13:12:43 kripke Exp $
foo.oip 0100755 0001130 0001130 00000000515 07635353345 011050 0 ustar krip krip #!/opt/openisis/oip
<%! int i;
static void foo () {
/* as an extension over common JSP-behaviour,
a global %! block *ending* with !% introduces
literal-printing code to be added outside of main
*/
!%>
lala
<%!
}
%>
<%
for ( i=0; i<3; i++ ) {
%>
bla <%=i%> blub <%="%f",i/3.%>
<%
}
foo();
%>
<%!
int j;
%>
fulltext 0100755 0001130 0001130 00000001547 07626500561 011346 0 ustar krip krip #!/usr/bin/perl
# read files for fulltext index
# filelist is on stdin (e.g. by find)
# index entries go to stdout
# usage:
# find /foo -name \*.html | ./fulltext >/tmp/idx 2>/tmp/mst
# find /usr/share/doc -type f -a \! -name \*.htm\* | ./fulltext >/tmp/idx 2>/tmp/mst
# sort -o /tmp/idx /tmp/idx
# time ./openisis -write db/test/ft -stream -fmt mfn ) {
chomp;
$f = $_;
next unless open( F, $f );
$fn++;
print STDERR "100\t$f\n";
$line = 0;
while ( ) {
next if /^\s*$/;
last if 255 < ++$line;
chomp;
$w = 0;
for $word (split /\W+/) {
next unless $word;
printf "%s\t%d\t%d\t%d\t%d\n", uc($word), $fn, 800, $line, ++$w;
}
}
print STDERR "\f\n";
}
iAPI.txt 0100755 0001130 0001130 00000120551 07355467174 011107 0 ustar krip krip /*
this file composed by Klaus Ripke from the iAPI files by Robert Janusz.
this is the test app "ix.c" with all the includes slurped in,
see // #include "isomething.c"
you may run this with:
cp iAPI.txt iAPI.c
gcc iAPI.c
./a.out
*/
/* ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
iAPI software: The Example of ISIS (UNIX v.3) Database Handle
Copyright (C) 2000 by Robert Janusz
E-mail: rj@jezuici.krakow.pl
Address: Robert Janusz, ul. Kopernika 26, 31-501 Krakow, Poland
tel: (0048-12) 4294416/432 ; fax: (0048-12) 4295003.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ */
// Linux gcc Compiler conventions !
#include
#include
#include
#include
// Global variables +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
static
char DBpath [71];
char UpCase [256];
#define NFIND 3 // let's keep it for future
#define PACKED __attribute__((packed)) // it shoud work on Linux gcc ;-)
// #include "idef.c"
/* ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
iAPI software: The Type Definitions for Unix-ISIS databases
Copyright (C) 2000 by Robert Janusz
E-mail: rj@jezuici.krakow.pl
Address: Robert Janusz, ul. Kopernika 26, 31-501 Krakow, Poland
tel: (0048-12) 4294416/432 ; fax: (0048-12) 4295003.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ */
// gcc Linux Compiler conventions !
// *** ONLY UNIX ISIS = the difference between DOS/UNIX structures
// *************************** TYPE DEFINITIONS *****************************
// Cross Reference File XRF structure +++++++++++++++++++++++++++++++++++++++
typedef /* Packed */ struct // /* Packed */ and order are necessary for block reading
{
long int xpos PACKED;
// Block nr: 1, 2...; < 0: last block
long int xrec [127] PACKED;
// References to Master File records...
// xrec [...] = 2048 *xrmfb [21 bit] & xrmfp [11 bit]
// rec. deleted log. < 0 & > 0 (restore: -xrmfb)
// rec. deleted phis. =-1 & = 0 (ref@MST.ctrl)
// rec. does not exist = 0 & = 0
// new rec. to invert bit 1024 = 1
// rec. edited to inv. (add & del post.) bit 512 = 1
}
xrfBlock;
// Master File (MST) record structures... +++++++++++++++++++++++++++++++++++
typedef /* Packed */ struct // The first record in Master File
{
long int ctlmfn PACKED // Always 0
, nxtmfn PACKED // Next Master File Number
, nxtmfb PACKED; // Next block in Master File, the 1st has 1
short int nxtmfp PACKED // Next position in the last block
, mftype PACKED; // 0 for user's bases; 1 for messages
long int reccnt PACKED // Reserved for future
, mfcxx1 PACKED // Reserved for future
, mfcxx2 PACKED // LAN, > 0: inversion, update not possible
, mfcxx3 PACKED; // LAN, = 1: exists exclusive user
}
mstControl;
typedef /* Packed */ struct // Fixed part of record header that must be...
{ // included in the same block;
// the header can begin only on 0, 2, 4...498
long int mfn PACKED; // MFN number
short int mfrl PACKED; // Length of record (allways eaven)...
// LAN: < 0 ==> record blocked
long int mfbwb PACKED; // Block Pointer | If the record waits for inv...
short int mfbwp PACKED; // Position Pointer | ...ref. to old location
// mfbwb, mfbwp: = 0 at creation;
// after modif. = ref to the old record.
// xrmfp bit 1024 = 1 at creation
// (= waits to be inverted);
// xrmfp bit 512 = 1 at edit
// (=waits to be inv. & delete old references)
}
mstHdrFix;
typedef /* Packed */ struct
{
short int base PACKED // Position of data fields in the record
, nvf PACKED // Number of fields in the record
, status PACKED; // 0 = active, 1 = deleted logicaly (- xrfb < 0)
}
mstHdrTxt;
typedef /* Packed */ struct
{
mstHdrFix mhf PACKED; // This header is allways in MST file block
mstHdrTxt mht PACKED;
}
RecHeaders;
typedef /* Packed */ struct // Index for every field in the record
{ // dimension = 6 *nvf; base = 18 + 6 *nvf
short int tag PACKED // Label of the field
, pos PACKED // Position of the first char in data part: 0, 1...
, len PACKED; // Length of the field
}
mstIndex;
// Inverted File structures +++++++++++++++++++++++++++++++++++++++++++++++++
// Control File (CNT) structure..............................................
typedef /* Packed */ struct
{
short int idtype PACKED // B*tree type; 1 = N01/L01, 2 = N02/L02
, ordn PACKED // = 5; a half of size of key-record in N0x
, ordf PACKED // = 5; a half of size of key-record in L0x
, n PACKED // = 15; number of buffers for nodes
, k PACKED // = 5; number of buffers for 1st index level (k < n)
, lev PACKED; // Current number of levels in index; -1: no N0x
long int posrx PACKED // Reference to the root in N0x
, nmaxpos PACKED // Next record in N0x
, fmaxpos PACKED; // Next record in L0x
short int abnormal PACKED; // 0: N0x has only the root
}
cntBlock;
// Index of Dictionary (N0x) structures......................................
typedef /* Packed */ struct
{
long int pos PACKED; // Number of record
short int ock PACKED // Number of active keys in the record: 1, 2, 3...2*ordn
, it PACKED; // B*tree type
}
nodeHeader;
typedef /* Packed */ struct
{
char key [10] PACKED;
long int ref PACKED; // > 0: reference to next node;
// < 0: ref. to leaf: idx [1st].key = key
// = 0: inactive
}
nodeIndex1;
typedef /* Packed */ struct
{
char key [30] PACKED;
long int ref PACKED;
}
nodeIndex2;
typedef /* Packed */ struct
{
nodeHeader hdr PACKED;
nodeIndex1 idx [10] PACKED; // Vector of 2*ordn elements
}
nodeBlock1;
typedef /* Packed */ struct
{
nodeHeader hdr PACKED;
nodeIndex2 idx [10] PACKED; // Vector of 2*ordn elements
}
nodeBlock2;
typedef /* Packed */ union
{
nodeBlock1 nb1 PACKED;
nodeBlock2 nb2 PACKED;
}
nodeBlock;
// Leaf File (L0x) structures (all index terms)..............................
typedef /* Packed */ struct
{
long int pos PACKED; // Number of leaf record: 1, 2...
short int ock PACKED // Number of active keys in the record: 1..2*ordf
, it PACKED; // = 1 for L01; = 2 for L02
long int ps PACKED; // Next record in the order: key [ock] < ps^ idx [1st].key
}
leafHeader;
typedef /* Packed */ struct
{
char key [10] PACKED;
short int xxx PACKED; // *** ONLY UNIX ISIS;
long int infb PACKED // Reference to IFP segment *** != ISIS.MANUAL ***
, infp PACKED;
}
leafKey1;
typedef /* Packed */ struct
{
char key [30] PACKED;
short int xxx PACKED; // *** ONLY UNIX ISIS;
long int infb PACKED // Reference to IFP segment *** != ISIS.MANUAL ***
, infp PACKED;
}
leafKey2;
typedef /* Packed */ struct
{
leafHeader hdr PACKED;
leafKey1 idx [10] PACKED; // 2*ordf keys
}
leafBlock1;
typedef /* Packed */ struct
{
leafHeader hdr PACKED;
leafKey2 idx [10] PACKED; // 2*ordf keys
}
leafBlock2;
typedef /* Packed */ union
{
leafBlock1 lb1 PACKED;
leafBlock2 lb2 PACKED;
}
leafBlock;
// Inverted File Posting structures..........................................
// ordained postings organized in segments: Index --> Master File
typedef /* Packed */ struct
{
long int ifblk PACKED // Number of block
, ifrec [127] PACKED; // Vector of long int *** != ISIS.MANUAL ***
// In 1st block ifrec [1st/2nd] = next pos. in IFP
}
ifpBlock;
typedef /* Packed */ struct // This header and 1st posting - always in IFP block
{
long int ifpnxtb PACKED // Pointer to the next segment: 1, 2... | = 0: last seg.
, ifpnxtp PACKED // 0, 1... | = 0: last seg.
, ifptotp PACKED // Tot. number of postings (o.k. only in the 1st segment)
// = Sum (segment=1, nseg; ifpsegp)
, ifpsegp PACKED // Number of postings in this segment
, ifpsegc PACKED; // Max. number of postings in this segment (<= 32768)
}
ifpHeader;
// ifpHeader & 1st ifpPosting always in the same ifpBlock
// Postings in Segment are always in chain of IFP blocks
typedef /* Packed */ struct // Stored left-->right + 0 --> 8 char
{
long int pmfn PACKED; // First 3 bytes --> Master File
short int ptag PACKED; // Field ident - according to File Selection Table (FST)
char pocc PACKED; // Number of occurence of the field
short int pcnt PACKED; // Counter of the termin inside the field
}
ifpPosting;
typedef struct
{
int opened; // Which files are opened (bit spec.)...
FILE *mst; // 1
FILE *xrf; // 2
FILE *ifp; // 4
FILE *n01; // 8
FILE *l01; // 16
FILE *n02; // 32
FILE *l02; // 64
mstControl mctrl; // MST ctrl record
cntBlock cnt1, cnt2; // Inv. Files ctrl records
long int ibm, ipm; // IFP new Segment Bock/Pos
char name [72]; // Database name
}
BaseCtrl;
typedef struct // This structer must be allways corelated with BaseCtrl
{
char st [31]; // Max. string to find
short int v; // Field to find; 0: all fields
short int it; // Type of B*Tree <== strlen (st)
leafBlock lbl; // Block L0x with index terms
short int ilk; // Position of term in lbl...idx
char *term; // The term in lbl...idx
long int ib, ip; // Number of block/position for IFP
ifpBlock ibl; // Block IFP with posting/s
ifpHeader ihd; // Header for Segment of postings
long int iiseg; // Number of posting in current Segment (index)
long int iitot; // Number of posting in total (index)
long int iall; // Number of posting in all Segments
char *ipst; // Ref. to last posting string [8 chars]
cntBlock *cntr; // BaseCtrl.cntr1/2 <== it
FILE *n0x; // N01 or N02 <== it, = BaseCtrl.n0x
FILE *l0x; // L01 or L02 <== it, = BaseCtrl.l0x
}
TableQuery;
#define RECLENGTH 8192
typedef union
{
char buf [RECLENGTH];
RecHeaders hdrs;
}
RecCtrl;
// *** MAIN DATABASE FILE RELATIONS (MST <--> XRF) ***
// 1 2 3
// MST |-mblk-|-mblk-|-mblk-|-mblk-|-mblk-|-mblk-|-mblk-|-mblk-|...
// ctrl |----record----|
// |--->|
// 1 2 3 | pointer: B/P (P in [0, 2, 4... 498])
// XRF |-xblk-|-xblk-|-xblk-|...
// MFN= 1... 128... 255...
// Record |-->-pos->--|
// |-mhf-mht-|-idx[1]...idx[nvf]-|---text...---|
// |-------------- record buffer --------------|
// *** INVERTED FILE RELATIONS (N0x <--> L0x <-->IFP) ***
// N0x |-nbl-|-nbl-|-nbl-|-nbl-|-nbl-|-nbl-|...
// -->-root->-| |-->--next-->--| |->-->-| r<0
// |
// L0x |--lbl--|--lbl--|--lbl--|--lbl--|--lbl--|--lbl--|...
// |
// |-<--------------<-------------<-| r<0
// *** POSTING *** |
// IFP |---ibl---|---ibl---|---ibl---|---ibl---|---ibl---|---ibl---|...
// |next |-ihd-|---Segment---| |-ihd-|---Segment---|
// |Seg. |-->------>------>-next->--| |||...
// MFN...
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
// #include "idef.c"
// Let's specify non int header ;-) just to keep order
void ShowRec (RecCtrl *rc);
void setQuery (BaseCtrl *bc, TableQuery *tq, char *sx, short int fld);
void isis ();
// #include "iutil.c"
/* ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
iAPI software: The UTIL procedures for Unix-ISIS databases
Copyright (C) 2000 by Robert Janusz
E-mail: rj@jezuici.krakow.pl
Address: Robert Janusz, ul. Kopernika 26, 31-501 Krakow, Poland
tel: (0048-12) 4294416/432 ; fax: (0048-12) 4295003.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ */
// gcc Linux Compiler conventions !
// *** ONLY UNIX ISIS = the difference between DOS/UNIX structures
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int ReadMem (void *ptr, FILE *f, long int nr, long int size)
{ // Read "nr"'s block of size "size" from file "f" into "ptr"...
if (fseek (f, nr * size, SEEK_SET) != 0) return (-1);
if (fread (ptr, size, 1, f) == 0) return (-2);
return (0); // O.K.
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void doUpcase (char *s, char *st, int slen)
{ int i;
for (i = 0; i < slen; i++)
{ if (i < strlen (st)) s [i] = UpCase [(int) st [i]]; else s [i] = ' '; }
s [slen] = 0;
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int sComp (char *s1, char *s2, int len)
{ int i;
for (i = 0; i < len; i++)
{ if (s1 [i] > s2 [i]) return (1); // .gt.
if (s1 [i] < s2 [i]) return (-1); // .lt.
}
return (0); // .eq.
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int bfOpen (FILE **f, BaseCtrl *bc, char *ext, char *opt)
{ char st [81];
if (strlen (bc->name) + strlen (ext) > 80) return (-2);
strcpy (st, bc->name); strcat (st, ext);
if ((*f = fopen (st, opt)) == NULL) return (-1);
return (0);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int InitBase (BaseCtrl *bc)
// Error return: -1 base file/s; -2 inv. file/s; -3 mst. ctrl rec/s;
// -4 cnt param/s; -5 ifp param/s
{ ifpBlock ib;
bc->opened = 0;
if (bfOpen (&bc->mst, bc, ".MST", "rb") != 0) return (-1);
bc->opened += 1; // MST opened
if (bfOpen (&bc->xrf, bc, ".XRF", "rb") != 0) return (-1);
bc->opened += 2; // XRF opened
if (bfOpen (&bc->ifp, bc, ".CNT", "rb") != 0) return (-2);
if (fread (&bc->cnt1, sizeof (cntBlock), 1, bc->ifp) == 0 ||
fread (&bc->cnt2, sizeof (cntBlock), 1, bc->ifp) == 0) return (-2);
if (fclose (bc->ifp) != 0) return (-3);
if (bfOpen (&bc->ifp, bc, ".IFP", "rb") != 0) return (-2);
bc->opened += 4; // IFP opened
if (bfOpen (&bc->n01, bc, ".N01", "rb") != 0) return (-2);
bc->opened += 8; // N01 opened
if (bfOpen (&bc->l01, bc, ".L01", "rb") != 0) return (-2);
bc->opened += 16; // L01 opened
if (bfOpen (&bc->n02, bc, ".N02", "rb") != 0) return (-2);
bc->opened += 32; // N02 opened
if (bfOpen (&bc->l02, bc, ".L02", "rb") != 0) return (-2);
bc->opened += 64; // L02 opened
if (fread (&bc->mctrl, sizeof (mstControl), 1, bc->mst) == 0) return (-3);
if (bc->cnt1.ordn != 5 || bc->cnt1.ordf != 5 || bc->cnt1.n != 15 ||
bc->cnt1.k != 5 || bc->cnt2.ordn != 5 || bc->cnt2.ordf != 5 ||
bc->cnt2.n != 15 || bc->cnt2.k != 5) return (-4);
if (fread (&ib, sizeof (ifpBlock), 1, bc->ifp) == 0) return (-5);
if (ib.ifblk != 1) return (-5);
bc->ibm = ib.ifrec [0]; bc->ipm = ib.ifrec [1];
return (0);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void Done (BaseCtrl *bc) // Close the database files
{
if (bc->opened && 1) fclose (bc->mst);
if (bc->opened && 2) fclose (bc->xrf);
if (bc->opened && 4) fclose (bc->ifp);
if (bc->opened && 8) fclose (bc->n01);
if (bc->opened && 16) fclose (bc->l01);
if (bc->opened && 32) fclose (bc->n02);
if (bc->opened && 64) fclose (bc->l02);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
char UpMe (char c) // ^A = ^a etc.
{
if (c > 96) c = c - 32;
return (c);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void outS (char *st)
{ char i;
if (st == NULL) return;
for (i = 0; i < strlen (st); i++)
// c = tblconvert [st [i]];
if (st [i] == (char) 145) printf ("<"); // This is HTML <
else if (st [i] == (char) 146) printf (">"); // and that is >
else printf ("%c", st [i]);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void outC (char ch)
{ char st [2];
st [0] = ch; st [1] = 0; outS (st);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void outD (double x, int n)
{ char s [80];
gcvt (x, n, s);
outS (s);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void outN (long int x) // long int -> sting @ base=10
{
// char st [33];
// ltoa (x, st, 10); outS (st);
printf ("%li", x);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
// #include "imasterf.c"
/* ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
iAPI software: The Master File access to Unix-ISIS databases
Copyright (C) 2000 by Robert Janusz
E-mail: rj@jezuici.krakow.pl
Address: Robert Janusz, ul. Kopernika 26, 31-501 Krakow, Poland
tel: (0048-12) 4294416/432 ; fax: (0048-12) 4295003.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ */
// gcc Linux Compiler conventions !
// *** ONLY UNIX ISIS = the difference between DOS/UNIX structures
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int GetRec (BaseCtrl *bc, long int mfnr, RecCtrl *rc, int bufvol)
{ char mblk [512]; // Master File block;
long int xmfb, xmfp; // Pointer to Master File (last read) record
// xMfB < 0 : record deleted (status = 1), -xMfB --> block;
// xMfP = 0: deleted phisicaly
// = 0 & = 0: does not exist
// 1st block: xMfB = 1, xMfP = 0; pointer = xMfB * 2048 + xMfP
// ==> mst < 500 MBy;
long int xn, xb;
int ret, nch;
xrfBlock xblk;
mstHdrFix *hf = &rc->hdrs.mhf;
mstHdrTxt *ht = &rc->hdrs.mht;
char *b = rc->buf;
if ((mfnr < 1) || (mfnr >= bc->mctrl.nxtmfn)) return (-2); // No record
// Find the address in XRF...
xn = mfnr - 1; xb = xn / 127;
if (ReadMem (&xblk, bc->xrf, xb, sizeof (xrfBlock)) != 0) return (-1);
xn = xblk.xrec [xn % 127]; // = address in MST
xmfb = xn / 2048; // Master File Block
xmfp = xn % 512; // Master File Position
// Find the record in MST...
if (xmfb < 1) return (1); // Record deleted
ret = 0; // Record o.k.
if (ReadMem (&mblk, bc->mst, xmfb - 1, sizeof (mblk)) != 0) return (-1);
// Copy headers into record buffer...
memcpy (b, &mblk [xmfp], sizeof (mstHdrFix));
if (hf->mfn != mfnr) return (-2); // Database corrupted
if (hf->mfrl >= bufvol) return (-3); // Buffer too small
// Get the rest of the record
xmfp += sizeof (mstHdrFix); // Posinion in block
xb = sizeof (mstHdrFix); // Position in buf
xn = labs (hf->mfrl) - xb; // Number of chars to read
if (xn <= 0) return (4); // No fields ??
for ( ; ; ) // We have some chars
{ // Find number of chars to copy from this block
if (xn <= (nch = sizeof (mblk) - xmfp)) nch = xn;
memcpy (b + xb, &mblk [xmfp], nch);
xb += nch; xn -= nch; xmfp = 0;
if (xn <= 0) break; // Nothing more to read
if (fread (&mblk, sizeof (mblk), 1, bc->mst) == 0) return (-1);
}
if (ht->status != 0) ret = 1; // Record deleted logicaly
if (hf->mfbwb != 0 || hf->mfbwp != 0) ret = 2; // Record not inv.
if (hf->mfrl < 0) ret = 3; // Record blocked
return (ret);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int IsF (RecCtrl *rc, short int v, char ch, char *st)
{ short int n, k, kk, flag;
char *b = rc->buf;
mstHdrTxt *mht = &rc->hdrs.mht;
mstIndex *ind = (mstIndex *) &rc->buf [sizeof (RecHeaders)];
flag = 0;
for (n = 0; n < mht->nvf; n++)
{
if (ind->tag == v) // there is a tag
{
if (ch == ' ') // field found
{ flag = 1; break; }
else // there is sub-field
{ flag = 1;
kk = 0; k = mht->base + ind->pos;
while (b [k] != '^' || b [k + 1] != ch) // convention ;-)
{ k += 1; kk += 1;
if (kk >= ind->len) // out of field
return (0);
}
break;
}
}
ind += 1; // next index ref.
}
if (flag == 0) return (0); // false
if (st == NULL) return (1); // true
else
if (sComp (&b [mht->base + ind->pos], st, ind->len) == 0)
return (1);
else return (0);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void RepF (RecCtrl *rc, char nf, ...)
// Repeated "nf" fields in record "rc", the left label appears only if:
// - begins with "\n", - there was another field and there is defined
// filed; the right label appears only if ther was defined field.
{ va_list pinfo; // for ... parameter list
int i, c, flag, n, k, kk, v;
int hit = 0;
char *sl, *sr;
char *b = rc->buf;
mstHdrTxt *mht = &rc->hdrs.mht;
mstIndex *ind = (mstIndex *) &rc->buf [sizeof (RecHeaders)];
for (n = 0; n < mht->nvf; n++)
{
va_start (pinfo, nf);
for (i = 0; i < nf; i++)
{
kk = 0;
k = mht->base + ind->pos;
sl = va_arg (pinfo, char *); // left
v = va_arg (pinfo, short int); // field
c = va_arg (pinfo, char); // sub-field
c = UpMe (c);
sr = va_arg (pinfo, char *); // right
if (v != ind->tag) // another Tag
break;
if (c != ' ') // sub-field
{ flag = 1;
while (b [k] != '^' || UpMe (b [k + 1]) != c) // convention ;-)
{ k += 1; kk += 1;
if (kk >= ind->len)
{ flag = 0; break; }
}
if (flag) { k += 2; kk += 2; }
}
else flag = 1;
if (flag)
{ // Reference to NULL = NW error !!!
if (sl == NULL) flag = 0;
else if (*sl == '\n') flag = 1; else flag = 0;
if (hit != 0 || flag) outS (sl);
hit += 1;
if (c == ' ')
while (kk < ind->len)
{
if (b [k] != '<' && b [k] != '>') // HTML special ;-)
outC (b [k]);
k += 1; kk += 1;
}
else
while (kk < ind->len && b [k] != '^') // convention ;-)
{
if (b [k] != '<' && b [k] != '>') // HTML special ;-)
outC (b [k]);
k += 1; kk += 1;
}
outS (sr);
}
}
va_end (pinfo);
ind += 1;
}
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
// #include "iinvterm.c"
/* ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
iAPI software: The Inv. File Terms access for Unix-ISIS databases
Copyright (C) 2000 by Robert Janusz
E-mail: rj@jezuici.krakow.pl
Address: Robert Janusz, ul. Kopernika 26, 31-501 Krakow, Poland
tel: (0048-12) 4294416/432 ; fax: (0048-12) 4295003.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ */
// gcc Linux Compiler conventions !
// *** ONLY UNIX ISIS = the difference between DOS/UNIX structures
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int FindFirst (TableQuery *tq)
{
short int i, j // Indexes for idx []
, srec, klen, sidx; // Size (rec), key length, size (idx)
long int r; // Reference (record number); < 0: --> L0x --> IFP
char *ref;
leafHeader *lhdr = &tq->lbl.lb1.hdr; // = ...lb2.hdr
nodeBlock nbl;
nodeHeader *nhdr = &nbl.nb1.hdr; // = ...nb2.hdr
if (tq->it == 1)
{ klen = 10; srec = sizeof (nodeBlock1); sidx = sizeof (nodeIndex1); }
else
{ klen = 30; srec = sizeof (nodeBlock2); sidx = sizeof (nodeIndex2); }
r = tq->cntr->posrx; // Root in N0x
while (r > 0L)
{
if (ReadMem (&nbl, tq->n0x, r - 1, srec) != 0) return (-1);
j = nhdr->ock;
if (nhdr->pos != r || nhdr->it != tq->it ||
j < 1 || j > 10) return (-3); // 10 = 2*ordn
ref = (char *) &nbl.nb1.idx; // Reference to first elem. of index
ref += sidx; // The first key is allways <= st, skip it
for (i = 1; i < j; i++)
if (sComp (ref, tq->st, klen) > 0) break;
else ref += sidx;
ref -= sidx; // Correct value
r = *(long int *) (ref + klen); // We have the "ref" value
if (r > tq->cntr->nmaxpos) return (-3); // Out of N0x
}
// Now find the term in L0x index
if (tq->it == 1)
{ srec = sizeof (leafBlock1); sidx = sizeof(leafKey1); }
else
{ srec = sizeof(leafBlock2); sidx = sizeof(leafKey2); }
r = - r;
if (ReadMem (&tq->lbl, tq->l0x, r - 1, srec) != 0) return (-1);
j = lhdr->ock;
if (lhdr->pos != r || lhdr->it != tq->it ||
j < 1 || j > 10) return (-2); // 10 = 2*ordf
tq->ilk = 0;
ref = (char *) &tq->lbl.lb1.idx; // Reference to first elem. of index
ref += sidx; // The first key is allways <= s, skip it
for (i = 1; i < j; i++)
if (sComp (ref, tq->st, klen) > 0) break;
else { ref += sidx; tq->ilk += 1; }
ref -= sidx;
tq->term = ref; // We have the term address
if (tq->it == 1)
ref += 12; // += 10; *** NOT UNIX
else ref += 32; // += 30; *** NOT UNIX
tq->ib = *(long int *) ref; // = infb
ref += sizeof (long int);
tq->ip = *(long int *) ref; // = infp
return (0); // O.K.
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int FindNext (TableQuery *tq)
{ int srec; // Block size
long int r; // Reference (record number); < 0: --> L0x --> IFP
leafHeader *lhdr = &tq->lbl.lb1.hdr; // = ...lb2.hdr
if (tq->it == 1) srec = sizeof (leafBlock1);
else srec = sizeof (leafBlock2);
tq->ilk += 1; // Next key in the index
if (tq->ilk >= lhdr->ock) // Out of index
{ r = lhdr->ps; // Next L0x record in order
if (r <= 0)
{ tq->ilk -= 1; return (1); } // Nothing more, end of index
if (ReadMem (&tq->lbl, tq->l0x, r - 1, srec) != 0) return (-1);
if (lhdr->pos != r || lhdr->it != tq->it ||
lhdr->ock < 1 || lhdr->ock > 10) return (-2); // 10 = 2*ordf
tq->ilk = 0;
}
srec = tq->ilk;
if (tq->it == 1)
{ tq->term = tq->lbl.lb1.idx [srec].key;
tq->ib = tq->lbl.lb1.idx [srec].infb;
tq->ip = tq->lbl.lb1.idx [srec].infp;
}
else
{ tq->term = tq->lbl.lb2.idx [srec].key;
tq->ib = tq->lbl.lb2.idx [srec].infb;
tq->ip = tq->lbl.lb2.idx [srec].infp;
}
return (0); // We have the new key
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int Find (BaseCtrl *bc, TableQuery tabq [], int ifind, long int *npg
, RecCtrl *rc, long int recvol)
{ int i;
if (ifind <= 0 || ifind > NFIND) return (-1);
for (i = 0; i < ifind; i++)
if (FindFirst (&tabq [i]) < 0) return (-1);
i = Postings (bc, tabq, ifind, npg, rc, recvol);
return (i);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int IndexPrep (BaseCtrl *bc, TableQuery *tq1, TableQuery *tq2, char *st)
{ char sx [31]; // tqx->v contains the status of L0x index
int i;
if (st == NULL)
for (i = 0; i < 30; i++) sx [i] = ' ';
else
for (i = 0; i < 30; i++)
if (i < strlen (st)) sx [i] = st [i]; else sx [i] = ' ';
sx [30] = 0;
setQuery (bc, tq2, sx, 0); // We have a long term in tq2; v := 0
if (FindFirst (tq2) < 0) return (-1);
sx [10] = 0;
setQuery (bc, tq1, sx, 0); // We have a short term in tq1; v := 0
if (FindFirst (tq1) < 0) return (-1);
while (sComp (tq1->term, tq1->st, 10) < 0)
if ((tq1->v = FindNext (tq1)) != 0) break; // At the end or error
if (tq1->v < 0) return (-1);
while (sComp (tq2->term, tq2->st, 30) < 0)
if ((tq2->v = FindNext (tq2)) != 0) break; // At the end or error
if (tq2->v < 0) return (-1);
return (IndexTest (tq1, tq2));
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int IndexTest (TableQuery *tq1, TableQuery *tq2)
{
if (tq1->v == 1) // At the end of 1st
if (tq2->v == 1) // At the end of 2nd
return (0); // All done
else return (2); // We have only 2nd
else
if (tq2->v == 1) return (1); // We have only 1st
else if (sComp (tq1->term, tq2->term, 10) <= 0) return (1);
else return (2);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int IndexTerm (int i, TableQuery *tq1, TableQuery *tq2)
{
if (i == 1) tq1->v = FindNext (tq1); // The term was in 1st
else if (i == 2) tq2->v = FindNext (tq2); // The term was in 2nd
else return (-1);
return (IndexTest (tq1, tq2));
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void setQuery (BaseCtrl *bc, TableQuery *tq, char *sx, short int fld)
{ // Must be activated AFTER the initialization of database
tq->v = fld;
if (strlen (sx) <= 10)
{ tq->it = 1; doUpcase (tq->st, sx, 10);
tq->cntr = &bc->cnt1; tq->n0x = bc->n01; tq->l0x = bc->l01;
}
else
{ tq->it = 2; doUpcase (tq->st, sx, 30);
tq->cntr = &bc->cnt2; tq->n0x = bc->n02; tq->l0x = bc->l02;
}
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
// #include "ipost.c"
/* ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
iAPI software: The Inv. File Postings access for Unix-ISIS databases
Copyright (C) 2000 by Robert Janusz
E-mail: rj@jezuici.krakow.pl
Address: Robert Janusz, ul. Kopernika 26, 31-501 Krakow, Poland
tel: (0048-12) 4294416/432 ; fax: (0048-12) 4295003.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ */
// gcc Linux Compiler conventions !
// *** ONLY UNIX ISIS = the difference between DOS/UNIX structures
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int PostSeg (BaseCtrl *bc, TableQuery *tq)
{
if (ReadMem (&tq->ibl, bc->ifp, tq->ib - 1, sizeof (ifpBlock)) != 0)
return (-1);
if (labs (tq->ibl.ifblk) - tq->ib != 0) return (-2);
memcpy (&tq->ihd, &tq->ibl.ifrec [tq->ip], sizeof (ifpHeader));
tq->ip += 3; // = long int.s / (header - posting) ***!= ISIS.MANUAL ***
tq->iiseg = 0;
return (0);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int PostNext (BaseCtrl *bc, TableQuery *tq)
{
if (tq->iitot >= tq->iall) return (1); // No more postings
if (tq->iiseg >= tq->ihd.ifpsegp) // No more postings in this seg.
{ // Get next Segment...
tq->ib = tq->ihd.ifpnxtb; tq->ip = tq->ihd.ifpnxtp;
if (PostSeg (bc, tq) != 0) return (-2);
}
tq->iiseg += 1; tq->iitot += 1;
tq->ip += 2; // = Number of long int / posting ***!= ISIS.MANUAL ***
if (tq->ip > 125) // ***!= ISIS.MANUAL ***
{ // Get next block for this segment; we HAVE TO find the right block!
tq->ib += 1; tq->ip = 0;
if (ReadMem (&tq->ibl, bc->ifp, tq->ib - 1, sizeof (ifpBlock)) != 0)
return (-1);
if (labs (tq->ibl.ifblk) != tq->ib) return (-3);
}
tq->ipst = (char *) &tq->ibl.ifrec [tq->ip];
return (0);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void PostMake (long int mfn, short int tag, char occ, short int pcnt
, char *s, char *slen)
{ ifpPosting rnp;
char *p = (char *) &rnp;
rnp.pmfn = mfn; rnp.ptag = tag; rnp.pocc = occ; rnp.pcnt = pcnt;
if (tag == 0) *slen = 3; // Only MFN
else *slen = 5; // MFN and TAG
s [2] = p [0]; s [1] = p [1]; s [0] = p [2]; // = pmfn
s [4] = p [4]; s [3] = p [5]; s [5] = p [6]; s [7] = p [7]; s [6] = p [8];
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void PostInv (ifpPosting *ifpp, void *ps)
{ char *p = (char *) ifpp;
char *s = (char *) ps;
// Posting string: |0 a |1 b |2 c |3 d |4 e |5 f |6 g |7 h | ps *s
// ifpPosting: |0 c|1 b |2 a |3 0 |4 e |5 d |6 f |7 h |8 g | ifpp *p
// |pmfn |ptag |pocc|pcnt |
p [0] = s [2]; p [1] = s [1]; p [2] = s [0]; p [3] = 0; // = pmfn
p [4] = s [4]; p [5] = s [3]; p [6] = s [5]; p [7] = s [7]; p [8] = s [6];
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
int main (int argc, char **argv) // char *argv []
{ char c;
printf ("ix \n");
printf (" ::= UNIX-ISIS (v.3) MF without .MST\n");
printf ("ix ver. 0.3, Copyright (C) 2000 by Robert Janusz\n");
printf ("ix comes with ABSOLUTELY NO WARRANTY\n");
printf ("License: GPL; see http://www.gnu.org/ for details\n");
for (c = 'A'; c <= 'z'; c++) // Let's prepare it, we use API, not an example
UpCase [c] = UpMe (c);
if (argv [1] != NULL) {
strcpy (DBpath, argv [1]); // Database; no test ! ;-) it's only an example
isis ();
}
return (0);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void isis ()
{
int j, v, k;
long int nrc;
BaseCtrl bc;
RecCtrl rc;
char st [31];
TableQuery tabq [NFIND];
strcpy (bc.name, DBpath);
InitBase (&bc);
// ShowRec but not all...
printf ("\n*** ShowRec ***\n\n");
//for (nrc = 1L; nrc < bc.mctrl.nxtmfn; nrc++) {
for (nrc = 10L; nrc < 20L; nrc++) {
if ((j = GetRec (&bc, nrc, &rc, RECLENGTH)) == 0) {
ShowRec (&rc);
}
}
// ShowIndex
printf ("\n*** ShowIndex: %s ***\n\n", st);
strcpy (st, "TEST"); // just an Inv. File term
v = 0; // term counter
k = IndexPrep (&bc, &tabq [0], &tabq [1], st); // prepare indexes
while (k > 0) {
j = 20 * k - 10; // We have the term length ;-)
memcpy (&st, &tabq [k - 1].term [0], j); // keep it in st
st [j] = 0;
if ((v += 1) > 12) // we want 12 terms in a list
{
printf ("->%s\n", st); // the next term
break;
}
printf ("%s\n", st);
k = IndexTerm (k, &tabq [0], &tabq [1]); // Next term
}
// Find a term & show records
printf ("\n*** Search for: %s ***\n\n", st); // Let's use the last term
setQuery (&bc, &tabq [0], st, 0); // String to search in a filed 0 = any
j = Find (&bc, tabq, 1, &nrc, &rc, RECLENGTH); // 1 term to search for
// Clean all
printf ("\nO.K.\n");
Done (&bc);
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void ShowRec (RecCtrl *rc)
{ long int nr;
short int n, k, kk;
char st [256];
char *b = rc->buf;
mstHdrTxt *mht = &rc->hdrs.mht;
mstIndex *ind = (mstIndex *) &rc->buf [sizeof (RecHeaders)];
nr = rc->hdrs.mhf.mfn; // Record header
printf ("=== %li\n", nr);
for (n = 0; n < mht->nvf; n++)
{ // Show fileds without formating
printf ("*** [%li] ", (long int) ind->tag);
k = mht->base + ind->pos;
for (kk = 0; kk < ind->len; kk++)
printf ("%c", b [k++]);
printf ("\n");
ind += 1; // Next index
}
printf ("___\n");
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
// This procedure is prepared to do a simple AND operator after some
// extensions, so it can be localy a little strange ;-)
int Postings (BaseCtrl *bc, TableQuery tabq [], int ifind, long int *npg
, RecCtrl *rc, long int recvol)
{ int j, state;
long int ipg, rn;
ifpPosting ip;
TableQuery *tq;
if (ifind < 1) goto finish; // nothing to do
ipg = 1L; // Index for page break
{ tq = &tabq [0]; // Let's think only about one term to handle (example)
if (PostSeg (bc, tq) != 0) return (-1);
tq->iall = tq->ihd.ifptotp; // Only 1st Segment has tot. post. number
tq->iitot = 0;
if (PostNext (bc, tq) != 0) return (-1);
PostInv (&ip, tq->ipst);
if (sComp (tq->st, tq->term, 20 * tq->it - 10) != 0)
ipg = 0L; // the term not found
}
if (ipg != 1L) goto finish;
for ( ; ; ) // Global loop...
{
// Is there a field
{ tq = &tabq [0];
if (tq->v != 0)
while (ip.ptag != tq->v)
{
if (PostNext (bc, tq) > 0) goto finish; // No more postings
PostInv (&ip, tq->ipst);
}
}
rn = ip.pmfn;
state = 1;
if (ip.pmfn != rn)
state = 0;
if (state) // We have a record to show
{
if ((j = GetRec (bc, rn, rc, recvol)) == 0)
ShowRec (rc);
else
printf ("Record state = %li\n", j);
do // Shift all postings
{ tq = &tabq [0];
if (PostNext (bc, tq) > 0) goto finish; // No more postings
PostInv (&ip, tq->ipst);
}
while (ip.pmfn <= rn); // MFN equals
}
else // state == 0, Shift all postings
while (ip.pmfn < rn) // The last is to examin
{ tq = &tabq [0];
if (PostNext (bc, tq) > 0) goto finish; // No more postings
PostInv (&ip, tq->ipst);
}
}
finish:
return (0); // The output is finished
}
// ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
idm.jsp 0100644 0001130 0001130 00000051412 07576450563 011046 0 ustar krip krip <%--
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH
--%><%@page
import="org.openisis.*,java.io.IOException,java.util.*,java.net.URLEncoder"
contentType="text/html; charset=iso-8859-1"
%><%!
static final String ID = "$Id: idm.jsp,v 1.4 2002/12/13 21:30:27 kripke Exp $";
static final String EMPTY = "";
static Db db;
static String dberr = EMPTY;
static String me = "idm";
static {
try {
String[] args = new String[] { "-v", "info", "-encoding", "Cp850" };
db = Db.open( "/var/db/idm/doku", args );
} catch (IOException e) {
e.printStackTrace();
dberr = e.toString();
}
};
static String NV ( String x ) { return null==x ? EMPTY : x; }
static int ROWS = 10; // number of rows to show at once
static int TERMS = 50; // number of terms to show at once
static int[] tag = {
100, 101, 200, 201,
301, 302, 430, 435,
28, 450, 800, 18
};
static String[] t = {
"Autor(en)", "author", // 100
"weitere Personen", "other contributors", // 101
"Körperschaft als Hrsg.", "corporate body as editor", // 200
"weitere Körperschaften", "other corporate bodies", // 201
"Titel ohne Autorennennung", "title (without indiv. author)", // 301
"Titel mit Autor", "title (with author)", // 302
"Erscheinungsvermerk", "publisher: name and place", // 430
"Erscheinungsjahr", "year of publication", // 435
"Erscheinungsland", "country of publication", // 28
"Ges.Titel/Ser.Titel/Bandnr.", "series title", // 450
"Schlagworte", "keywords", // 800
"Dokument Nr.", "no of document", // 18
// unused fields:
"Universitätseinrichtung (10):", EMPTY,
"Projekt (12):", EMPTY,
"Erfassungsdatum (16):", EMPTY,
"Dokumententyp (22):", EMPTY,
"Physikalische Form (24):", EMPTY,
"Sprache des Dokumentes (26):", EMPTY,
"Physische Form (440):", EMPTY,
"Quelle (Zeitschrift) (480):", EMPTY,
"Quelle (Monographie) (490):", EMPTY,
"ISBN (600):", EMPTY,
"Abstract (900):", EMPTY,
EMPTY, EMPTY,
EMPTY, EMPTY,
// 50: other texts
"Dokumentations-Datenbank des IDM der Universität Bielefeld",
"documentation database of the IDM at the university of Bielefeld",
"Suchen", "search",
"English", "Deutsch", //54
"?", "?",
"weiter", "more",
"allen Feldern", "all fields", //60
// the ops
"und","and",
"oder","or",
"ohne","without",
"zurücksetzen","reset",
"Name","Surname",//70
"Vorname","Given name",
"Institut","Institution",
"Strasse","Street",
"Ort","City",
"Land","Country", // 80
"Postleitzahl","Zip code",
"Bestellen", "order",
"Abschicken", "Submit",
"Einträge", "rows",
"bis", "to", // 90
"von", "of",
"für", "for",
"Indexeinträge beginnend mit", "terms starting with ",
"neue Suche", "new Query",
// 100: hilfe
"Willkommen bei der Dokumentations-Datenbank des\n"
+" IDM der Universität Bielefeld \n"
+" Sie können bis zu drei Suchbegriffe verknüpfen.\n"
+" Jeder Suchbegriff kann auf ein Feld eingeschränkt werden.Suchtipps \n"
+" Für Prefixsuche beenden Sie den Begriff mit * oder $.Cookies \n"
+" Die Funktion ist stark beeinträchtigt,\n"
+" falls Sie das Session-Cookie ablehnen.welcome to the documentation database of the"
+" IDM at the university of Bielefeld "
+" Up to three search terms may be combined.\n"
+" Each term can be restricted to a specific database field.search tips \n"
+" Use a trailing * or $ to search by prefix.Cookies \n"
+" This page will not work properly,\n"
+" if you do not accept the session cookie.",
Mail.SUBJECT, "online Bestellung",
};
static final int LINES = 3;
static class Ses {
int l; // lang, 0 or 1
int off = 0;
int rows = ROWS;
int view; // last seen view
int idx; // field to save index into
int msec; //
String qry;
int[] set;
int len;
// form content:
String[] t = new String[LINES];
String[] o = new String[ORDER.length]; // order data
int[] f = new int[LINES]; // 0 or 1+index of selected tag
int[] op = new int[LINES];
Iterator i;
String term = EMPTY;
ArrayList orders;
public Ses () {
for ( int i=0; i<%
Ses s;
try {
s = (Ses) session.getAttribute( me );
} catch ( Exception e ) { s = null; }
if ( null == s )
session.setAttribute( me, s = new Ses() );
int task = DO_HELP;
String order = null;
// figure out, what to do
String wants = request.getParameter( "do" );
if ( chkbutton( 52, wants ) )
task = DO_QUERY;
else if ( chkbutton( 54, wants ) )
task = DO_LANG;
else if ( "order".equals( wants ) )
task = DO_ORDER;
else if ( chkbutton( 86, wants ) )
task = DO_ORDERS;
else if ( "more".equals( wants ) )
task = DO_INDEXN;
else if ( "new".equals( wants ) )
task = DO_BLANK;
else if ( null != request.getParameter( "off" ) )
task = DO_QUERYN;
else // check "index" buttons i0, i1 ...
for ( int i=0; i" )
.append(t[112+s.l])
.append( "\n\n" )
.toString();
task = DO_ORDER;
}
}
s.view = VW_ORDER;
}
int maxrow = s.off + s.rows;
if ( maxrow >= s.len )
maxrow = s.len;
%>
<%=t[50+s.l]%>
<%=t[88+s.l]%> <%=s.off+1%>
<%=t[90+s.l]%> <%=maxrow%>
<%=t[92+s.l]%> <%=s.len%>
<%=t[94+s.l]%> '<%=s.qry%>'
<%-- start output --%>
<%
for ( int i=s.off; i
"+t[2*j+s.l]
+" ",
null, // cond,repeated prefix
tag[j], Db.MDL|Db.PS|Db.HTI, 0, 0, // tag and mode, len, off
"; ")
);
}
String ti = r.v("",301,Db.MDL, 0, 0,"");
if ( null == ti || 0 == ti.length() )
ti = r.v("",302,Db.MDL, 0, 0,"");
String link = me+"?do=order"
+ "&id=" + URLEncoder.encode( r.v("",18,Db.MDL, 0, 0,"") )
+ "&ti=" + URLEncoder.encode( ti );
%>
>> <%=t[84+s.l]%>
<%
}
break; // case VW_QUERY
case VW_INDEX:
%>
<%=t[96+s.l]%> '<%=Field.html(s.term)%>'
<%
if ( null == s.i || ! s.i.hasNext() ) {
} else {
int i = TERMS;
while ( 0 < i-- && s.i.hasNext() ) {
String ix = (String)s.i.next();
if ( 0 == ix.length() ) continue;
String u = URLEncoder.encode( ix );
%>
<%=Field.html(ix)%>
>> <%=t[58+s.l]%>
<%
} else
s.i = null;
}
break; // case VW_INDEX
case VW_ORDER:
%>
<%= order %>
<%
} // switch
%>
img/Makefile 0100644 0001130 0001130 00000002516 07615312070 011754 0 ustar krip krip #
#/*
# openisis - an open implementation of the ISIS database
# Version 0.8.x (microversion see file Version)
# Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# This library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
# This software is dedicated to the memory of Eckart Dietrich.
#
# This software is inspired by (but contains no code of) the iAPI
# Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
# See iAPI.txt for what it contains.
#EOH */
#
#
# $Id: Makefile,v 1.1 2003/01/27 20:19:04 kripke Exp $
# img Makefile
#
all: logo.gif isi.gif favicon.ico
favicon.ico: isi.pgm
ppmtowinicon $< >$@
%.gif: %.pgm
ppmtogif $< >$@
%.ico: %.pgm
ppmtowinicon $< >$@
img/isi.cursor 0100664 0001130 0001130 00000000501 07615301370 012332 0 ustar krip krip #define isi_width 16
#define isi_height 16
#define isi_x_hot 1
#define isi_y_hot 5
static unsigned char isi_bits[] = {
0x00, 0x00, 0x00, 0x18, 0x80, 0x25, 0xc0, 0x27, 0xe0, 0x47, 0xfe, 0x27,
0xe4, 0x27, 0xc4, 0x23, 0x08, 0x10, 0x10, 0x08, 0xe0, 0x07, 0xc0, 0x03,
0xc0, 0x03, 0xc0, 0x03, 0xe0, 0x07, 0x00, 0x00};
img/isi.pgm 0100664 0001130 0001130 00000001674 07615275612 011625 0 ustar krip krip P2
16 16
255
255 255 255 255 255 255 255 255 255 255 255 237 151 255 255 255
255 255 255 255 255 255 255 255 255 255 221 110 76 159 255 255
255 255 255 255 255 255 169 58 69 204 103 203 245 84 242 255
255 255 255 255 255 178 7 0 0 30 111 203 255 104 200 255
255 255 255 138 255 75 0 0 0 0 88 154 255 134 181 255
255 255 132 183 106 73 0 0 0 11 109 170 255 115 191 255
255 133 133 255 213 105 68 14 19 106 109 230 252 86 229 255
255 240 84 221 255 178 101 90 91 106 195 255 154 118 255 255
255 255 181 100 240 255 237 184 187 248 255 203 96 233 255 255
255 255 255 176 84 185 255 255 255 252 146 85 220 255 255 255
255 255 255 255 210 55 43 79 72 30 98 240 255 255 255 255
255 255 255 255 255 187 0 0 0 0 188 255 255 255 255 255
255 255 255 255 255 187 0 0 0 2 178 255 255 255 255 255
255 255 255 255 255 187 0 0 0 2 187 255 255 255 255 255
255 255 255 255 236 0 0 0 0 0 0 252 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
img/logo.pgm 0100664 0001130 0001130 00000107651 07615303427 011777 0 ustar krip krip P2
100 100
255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 112 158 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 96 0 0 158 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 96 0 0 0 0 158 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 96 0 0 0 0 0 15 207
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 96 0 0 15 191 46 0 0 46
239 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 96 0 0 15 207 255 223 15 0 0
112 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 96 0 0 15 207 255 255 255 191 0 0
0 207 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 96 0 0 15 207 255 255 255 255 255 112 0
0 64 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 96 0 0 15 207 255 255 255 255 255 255 239 15
0 0 191 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 96 0 0 15 207 255 255 255 255 255 255 255 255 158
0 0 64 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 96 0 0 15 207 255 255 255 255 255 255 255 255 255 255
32 0 0 207 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 239 158 112 64 64 64 64 112 174 255 255 255 255
255 112 0 0 0 207 255 255 255 255 255 255 255 255 255 255 255
158 0 0 112 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 239 127 15 0 0 0 0 0 0 0 0 32 127 255 255
255 158 0 0 0 146 255 255 255 255 255 255 255 255 255 255 255
239 15 0 15 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 207 32 0 0 0 0 0 0 0 0 0 0 0 0 46 207
255 255 158 0 0 0 223 255 255 255 255 255 255 255 255 255 255
255 96 0 0 191 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
158 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15
207 255 255 127 0 0 64 255 255 255 255 255 255 255 255 255 255
255 174 0 0 112 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 207
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
15 223 255 255 32 0 0 191 255 255 255 255 255 255 255 255 255
255 255 0 0 46 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 239 32
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 64 255 255 174 0 0 96 255 255 255 255 255 255 255 255 255
255 255 64 0 0 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 127 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 191 255 255 32 0 0 239 255 255 255 255 255 255 255 255
255 255 127 0 0 191 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 32 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 64 255 255 127 0 0 174 255 255 255 255 255 255 255 255
255 255 158 0 0 158 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 191 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 239 255 191 0 0 127 255 255 255 255 255 255 255 255
255 255 191 0 0 127 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 127 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 174 255 255 0 0 64 255 255 255 255 255 255 255 255
255 255 223 0 0 127 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 96 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 127 255 255 32 0 64 255 255 255 255 255 255 255 255
255 255 255 0 0 84 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 158 127 127 127 127
127 127 127 127 127 127 127 127 127 127 127 158 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 127 255 255 64 0 64 255 255 255 255 255 255 255 255
255 255 255 0 0 64 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 64 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 64 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 127 255 255 64 0 64 255 255 255 255 255 255 255 255
255 255 255 0 0 64 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 64 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 64 255 255 127 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 146 255 255 0 0 64 255 255 255 255 255 255 255 255
255 255 255 0 0 112 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 64 0 0 255 255
255 255 255 255 255 255 255 255 255 64 0 15 255 255 158 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 207 255 223 0 0 96 255 255 255 255 255 255 255 255
255 255 191 0 0 127 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 112 0 0 223 255
255 255 255 255 255 255 255 255 255 112 0 0 207 255 223 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 15 255 255 158 0 0 146 255 255 255 255 255 255 255 255
255 255 191 0 0 127 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 127 0 0 191 255
255 255 255 255 255 255 255 255 255 174 0 0 112 255 255 84 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 127 255 255 84 0 0 207 255 255 255 255 255 255 255 255
255 255 127 0 0 191 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 191 0 0 127 255
255 255 255 255 255 255 255 255 255 239 15 0 32 255 255 191 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 15 239 255 223 0 0 32 255 255 255 255 255 255 255 255 255
255 255 84 0 0 223 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 239 0 0 84 255
255 255 255 255 255 255 255 255 255 255 96 0 0 146 255 255 112
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 158 255 255 96 0 0 146 255 255 255 255 255 255 255 255 255
255 255 32 0 15 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 32 0 0 255
255 255 255 255 255 255 255 255 255 255 223 0 0 15 239 255 255
64 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
96 255 255 207 0 0 15 239 255 255 255 255 255 255 255 255 255
255 223 0 0 84 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 112 0 0 174
255 255 255 255 255 255 255 255 255 255 255 112 0 0 64 255 255
239 84 0 0 0 0 0 0 0 0 0 0 0 0 0 0 112
255 255 239 32 0 0 146 255 255 255 255 255 255 255 255 255 255
255 146 0 0 158 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 174 0 0 96
255 255 255 255 255 255 255 255 255 255 255 239 32 0 0 96 255
255 255 146 15 0 0 0 0 0 0 0 0 0 0 32 174 255
255 239 46 0 0 64 255 255 255 255 255 255 255 255 255 255 255
255 32 0 0 223 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 32 0 0
239 255 255 255 255 255 255 255 255 255 255 255 207 15 0 0 96
239 255 255 239 146 46 0 0 0 0 0 0 46 158 255 255 255
239 46 0 0 15 239 255 255 255 255 255 255 255 255 255 255 255
191 0 0 64 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 127 0 0
127 255 255 255 255 255 255 255 255 255 255 255 255 174 0 0 0
46 207 255 255 255 255 239 191 191 191 191 255 255 255 255 255 191
32 0 0 15 207 255 255 255 255 255 255 255 255 255 255 255 255
96 0 0 158 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 223 0 0
15 239 255 255 255 255 255 255 255 255 255 255 255 255 207 15 0
0 0 96 223 255 255 255 255 255 255 255 255 255 255 191 64 0
0 0 32 207 255 255 255 255 255 255 255 255 255 255 255 255 223
0 0 32 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 96 0
0 127 255 255 255 255 255 255 255 255 255 255 255 255 255 223 64
0 0 0 0 46 127 191 223 255 255 207 191 112 32 0 0 0
0 84 239 255 255 255 255 255 255 255 255 255 255 255 255 255 64
0 0 158 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 223 15
0 0 223 255 255 255 255 255 255 255 255 255 255 255 255 255 255
158 32 0 0 0 0 0 0 0 0 0 0 0 0 0 0 32
174 255 255 255 255 255 255 255 255 255 255 255 255 255 255 174 0
0 32 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 127
0 0 64 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 158 84 0 0 0 0 0 0 0 0 0 0 84 174 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 239 15 0
0 174 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
46 0 0 127 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 239 191 146 127 127 127 127 146 191 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 84 0 0
64 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
207 0 0 0 174 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 127 0 0 15
239 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 146 0 0 15 207 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 158 0 0 0 174
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 96 0 0 15 207 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 158 0 0 0 146 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 64 0 0 15 207 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 158 0 0 0 96 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 239 46 0 0 0 158 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 112 0 0 0 96 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 239 96 0 0 0 96 239 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 223 64 0 0 0 96 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 112 0 0 0 32 174 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 146 15 0 0 0 158 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 174 15 0 0 0 64 191 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 158 32 0 0 0 32 207 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 223 84 0 0 0 0 64 158 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 239 158 32 0 0 0 0 112 239 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 191 32 0 0 0 0 15
96 174 239 255 255 255 255 255 255 255 255 255 255 255 255 223 158
84 0 0 0 0 0 64 191 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 158 46 0 0 0
0 0 0 46 84 127 127 191 191 191 174 127 127 64 32 0 0
0 0 0 0 64 191 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 223 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 64 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 191 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 64 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 191 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 64 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 191 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 64 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 223 119 84 46 68 101 195 255 255
255 255 255 135 101 101 101 101 101 101 178 195 255 255 255 255 96
101 101 101 101 101 101 101 101 101 101 255 211 96 101 101 223 255
255 255 255 146 101 101 158 255 255 223 135 135 195 255 255 255 255
223 146 75 75 75 75 146 223 255 255 255 255 255 135 135 165 255
255 255 255 239 146 84 75 75 75 135 223 255 255 255 255
255 255 255 255 255 255 255 84 68 174 207 255 223 191 68 46 239
255 255 255 46 211 207 207 207 207 191 137 84 46 239 255 255 55
211 207 207 207 207 207 207 207 211 55 255 174 112 211 174 46 255
255 255 255 46 211 211 101 255 255 195 15 15 135 255 255 255 165
32 15 15 15 15 15 15 32 165 255 255 255 255 15 15 75 255
255 255 211 32 15 15 15 15 15 15 15 165 255 255 255
255 255 255 255 255 255 55 158 255 255 223 207 207 255 255 191 46
239 255 255 46 255 255 255 255 255 255 255 255 174 46 255 255 68
255 255 255 255 255 255 255 255 255 75 255 178 137 255 255 101 158
255 255 255 46 255 207 101 255 255 195 15 15 135 255 255 223 32
15 15 105 135 135 75 15 15 15 223 255 255 255 15 15 75 255
255 239 32 15 15 84 135 135 84 15 15 15 195 255 255
255 255 255 255 255 119 119 255 255 68 55 101 84 15 207 255 191
55 255 255 46 255 255 15 46 46 32 64 255 255 105 202 255 64
255 202 32 46 46 46 46 46 46 46 255 178 135 255 255 255 32
239 255 255 46 255 207 101 255 255 195 15 15 135 255 255 135 15
15 178 255 255 255 255 135 15 15 119 255 255 255 15 15 75 255
255 165 15 15 146 255 255 255 255 146 15 15 105 255 255
255 255 255 255 255 32 255 255 46 178 255 255 255 239 32 239 255
68 223 255 46 255 255 46 255 255 255 119 146 255 174 119 255 64
255 202 105 255 255 255 255 255 255 255 255 178 135 255 255 255 158
84 255 255 46 255 207 101 255 255 195 15 15 135 255 255 135 15
15 223 255 255 255 255 239 75 84 165 255 255 255 15 15 75 255
255 135 15 15 178 255 255 255 255 255 75 84 165 255 255
255 255 255 255 195 105 255 185 75 255 255 255 255 255 158 119 255
174 119 255 46 255 255 46 255 255 255 185 135 255 207 101 255 64
255 202 105 255 255 255 255 255 255 255 255 178 135 255 195 211 255
68 202 255 46 255 207 101 255 255 195 15 15 135 255 255 146 15
15 46 165 239 255 255 255 255 255 255 255 255 255 15 15 75 255
255 165 15 15 32 165 223 255 255 255 255 255 255 255 255
255 255 255 255 135 158 255 135 158 255 255 255 255 255 255 64 255
223 75 255 46 255 255 32 185 178 158 32 202 255 119 158 255 64
255 211 64 68 68 68 68 68 46 255 255 178 135 255 127 55 255
223 32 255 46 255 207 101 255 255 195 15 15 135 255 255 239 32
15 15 15 15 64 119 195 255 255 255 255 255 255 15 15 75 255
255 239 46 15 15 15 15 64 119 178 255 255 255 255 255
255 255 255 255 101 207 255 135 178 255 255 255 255 255 255 68 255
255 46 255 46 255 255 127 135 137 135 223 255 255 32 239 255 64
255 255 255 255 255 255 255 255 46 255 255 178 135 255 135 32 158
255 119 127 46 255 207 101 255 255 195 15 15 135 255 255 255 211
46 15 15 15 15 15 15 46 135 255 255 255 255 15 15 75 255
255 255 211 75 15 15 15 15 15 15 32 135 255 255 255
255 255 255 255 101 207 255 135 178 255 255 255 255 255 255 68 255
255 46 255 46 255 255 255 255 255 255 255 191 46 158 255 255 64
255 211 64 64 68 68 68 68 46 255 255 178 135 255 135 165 15
255 255 46 32 255 207 101 255 255 195 15 15 135 255 255 255 255
255 211 119 64 15 15 15 15 15 119 255 255 255 15 15 75 255
255 255 255 255 211 119 64 15 15 15 15 15 84 255 255
255 255 255 255 158 137 255 158 137 255 255 255 255 255 223 84 255
207 84 255 46 255 255 64 64 68 32 46 68 223 255 255 255 64
255 202 105 255 255 255 255 255 255 255 255 178 135 255 135 185 158
84 255 185 0 255 207 101 255 255 195 15 15 135 255 255 255 255
255 255 255 255 223 165 75 15 15 15 223 255 255 15 15 75 255
255 255 255 255 255 255 255 239 165 84 15 15 15 211 255
255 255 255 255 223 84 255 239 46 255 255 255 255 255 101 165 255
158 137 255 46 255 255 46 255 255 255 255 255 255 255 255 255 64
255 202 101 255 255 255 255 255 255 255 255 178 135 255 137 178 255
46 207 255 158 255 202 101 255 255 195 15 15 135 255 255 135 105
119 255 255 255 255 255 255 84 15 15 195 255 255 15 15 75 255
255 135 105 105 255 255 255 255 255 255 119 15 15 195 255
255 255 255 255 255 46 239 255 119 75 239 255 255 119 46 255 255
46 239 255 46 255 255 46 255 255 255 255 255 255 255 255 255 64
255 202 105 255 255 255 255 255 255 255 255 185 135 255 137 178 255
223 46 255 255 255 202 101 255 255 195 15 15 135 255 255 46 15
15 223 255 255 255 255 255 105 15 15 195 255 255 15 15 75 255
255 75 15 15 211 255 255 255 255 255 119 15 15 195 255
255 255 255 255 255 174 55 255 255 174 46 46 46 135 255 255 105
101 255 255 46 255 255 46 255 255 255 255 255 255 255 255 255 64
255 223 64 68 68 68 68 68 68 32 255 185 135 255 137 178 255
255 101 135 255 255 202 101 255 255 195 15 15 135 255 255 135 15
15 64 223 255 255 255 146 15 15 46 255 255 255 15 15 75 255
255 165 15 15 64 195 255 255 255 195 32 15 32 239 255
255 255 255 255 255 255 135 55 239 255 255 255 255 255 255 105 68
255 255 255 46 255 255 46 255 255 255 255 255 255 255 255 255 68
255 255 255 255 255 255 255 255 255 135 255 185 137 255 146 174 255
255 255 15 239 255 207 101 255 255 195 15 15 135 255 255 239 64
15 15 15 46 75 32 15 15 15 178 255 255 255 15 15 75 255
255 255 64 15 15 15 46 75 32 15 15 15 165 255 255
255 255 255 255 255 255 255 174 46 68 137 137 137 84 32 135 255
255 255 255 46 75 75 46 255 255 255 255 255 255 255 255 255 15
68 68 64 68 68 68 68 68 68 32 255 178 32 75 46 174 255
255 255 174 32 68 55 96 255 255 195 15 15 135 255 255 255 239
105 15 15 15 15 15 15 46 195 255 255 255 255 15 15 75 255
255 255 239 105 15 15 15 15 15 15 46 178 255 255 255
255 255 255 255 255 255 255 255 255 239 174 174 174 223 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 239 195 195 223 255 255 255 255
255 223 178 135 135 165 211 255 255 255 255 255 255 195 195 211 255
255 255 255 255 239 195 135 135 146 211 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
255 255 255 255 255 255 255 255 255 255 255 255 255 255 255
index-es.html 0100644 0001130 0001130 00000057503 07534721352 012157 0 ustar krip krip
OpenIsis 0.8.x
versión actual
isis
openisis
manual práctico
acerca de
enlaces
in english
bienvenido a OpenIsis.org
DEMOS:normal
o trate
de buscar caracteres unicode
(
índice unicode )
win32 (0.8.5) ( mingw32 cross-build) debería trabajar en todas las plataformas win32 desde win5b.solaris (0.8.4) ( compilado en solaris 5.8 )
Características:
Julio 2002¿Qué de ISIS hace a ISIS?
0.8.6 Junio 2002notas de formateo para más detalles.
La extensión Perl ahora implementa formateo (véase test.pl),
y versiones mejoradas para los otros lenguajes seguirán pronto.
PHP Junio 2002Braulio José Solano Rojas
de Costa Rica creó una extensión PHP, que puede ser vista en acción en
galileo .
Disponible como descarga
o desde sourceforge modulo php-openisis.
Además, el Institut Teknologi Bandung de Indonesia
cambió su índice web a una versión basada en PHP/OpenIsis.
0.8.5.2 Marzo 2002
0.8.5.1 March 2002método Vn-field-selector-style
y varias utilerías interesantes.
La indentación no está propiamente manejada, dado que no hay una
solución fácil y común en HTML.
Se harán herramientas para una serie de estrategias estándares luego...
0.8.5 Marzo 2002demostración ).
Todos los operadores están ahí (incluyendo /(etiqueta) excepto /(t1,t2...)).
Se hace lo posible para limitar el costo potencial de consultas extremadamente
estúpidas como "$"^"$", de tal manera que resultados no históricos (#n)
o resultados intermedios (para precedencia) son guardados.
Las consultas son procesadas estrictamente de izquierda a derecha en un almacenamiento de 1000
coincidencias.
0.8.4 Enero 2002búsqueda en la bd unesb
con más de 70.000 filas (noten que el
servidor anfitrión es un Celeron de 500MHZ).
0.8.3 Octubre 2001demostración JSP muestra
como una consulta es refinada (más angosta) iterativamente haciendo un
"y" lógico con otra consulta.
Gracias a Verónica Lencinas y colegas,
tenemos una primer versión en español de este documento originalmente
en inglés .
Septiembre 2001sourceforge .
La versión para Windows openisis.exe corre.
0.8.2 Agosto 2001
0.8.1 Junio 2001
0.8.0 Mayo 2001
0.7.9 Mayo 2001
Isis es un sistema de bases de datos sencillo pero potente, que cuenta con una
importante base de usuarios desde los 80. Debido a su buena adaptabilidad a datos
bibliográficos, se usa generalmente en bibliotecas y por su bajo costo, especialmente
en aquellas con escaso presupuesto.
Una base de datos isis consiste en filas de estructura no específica, identificada
con un número único, la clave de fila (mfn). Cada fila contiene una lista de
campos, con un número de campo (tag) y un valor de cadena. Dentro de una fila
puede haber cero, uno o más campos con un número de campo determinado. El valor
del campo generalmente es la representación textual de datos en una u otra codificación
de caracteres (generalmente una de las páginas de código IBM/DOS), pero también
puede contener bytes arbitrarios.
subcampos
Hay una convención que consiste en codificar campos múltiples dentro de un solo
campo, separándolos con un ' ^ ' seguido por un caracter identificador del subcampo.
De esta manera el valor del campo ' ^afoo^bbar^bbaz' representa un campo con un
subcampo 'a' con valor 'foo' y dos subcampos 'b': 'bar' y 'baz' respectivamente.
formateo
Hay un lenguaje de formateo con variables de texto, campos y subcampos, estructuras
de control if-else (según existencia de campos) y búcles for (sobre repetición
de campos) (dicho de manera sencilla).
indización
Se construye un índice, convirtiendo una fila en una lista de palabras (aplicando
formatos opcionales) y ubicándolas según su posición de su ocurrencia en la
fila, en un árbol B (que se extiendo sobre seis archivos). Se puebe buscar una
palabra o un prefijo indicando - o no - la posición (campo). Debido a que todos
los campos pueden ser combinados en un índice, no es necesario (pero posible)
generar múltiples índices.
consultas
Un lenguaje de consultas permite la combinación de palabras usando los operadores
"y" (and), "o" (or) y "no" (not).
uso
Mientras que isis carece de la mayoría de las características de un sistema
de gestión de bases de datos relacionales (RDBMS), como relaciones complejas
entre diferentes entidades, su flexibilidad resulta práctico para catálogos
y directorios con registros altamente variables y un solo nivel de subestructura,
los que actualmente se vierten en documentos XML más que en filas. El mecanismo
flexible de indización combina lo mejor de búsqueda en texto completo con recuperación
estructurada.
La madre de todo el software isis es una versión DOS de "MicroISIS", un sistema
integrado con interfaz de texto. Hay una versión BSD de "CDS/ISIS" que también
corre bajo linux hasta kernels 2.2.x (los kernels 2.4 actuales no permiten el
módulo iBCS para binarios COFF). Además hay varias versiones de "WinISIS" (sólo
para M$-Windows).
Existe una librería compartida de funciones de acceso a bases de datos isis
"isis.dll" que, a pesar de su nombre, también se distribuye en una versión para
linux "isilux". Sin embargo se necesitan algunas librerías muy especiales para
usarla. Un conjunto de utilitarios de línea de comando ("cisis") realizan tareas
tales como la importación de archivos de intercambio de bases de datos bibliográficas
- ISO2709, construcción de archivos invertido, etc. Lo más cercano a un servidor
de bases de datos isis es "wwwisis", que corre como un CGI o bien desde la línea
de comandos y realiza la mayoría de tareas de isis (versiones win y lin). Sin
embargo, corre sólo por encargo, no como un servidor propiamente dicho y no
puede proveer control de procesos simultáneos.
Este software "oficial" de isis, es mantenido por Unesco y/o Bireme , y ha sido complementado por
aportes independientes, algunos de dominio público. Javaisis es una interfaz gráfica
basada en AWT con su correspondiente servidor, que usa wwwisis. Es un software
bueno y abierto que se concentra sobre un frontend multiplataforma, pero sin
acceso directo a la base de datos. Robert
Janusz escribió una librería C de cero (iAPI), que fué el punto de partida
para el software openisis.
¿Y entonces, porqué estamos haciendo openisis? Porque Isis no es un software de
fuente abierta, ni siquiera es gratuito, y esto conduce a una serie de problemas.
Disponibilidad (en teoría) Disponibilidad (en la práctica) Disponibilidad (en términos legales) Disponibilidad (de documentación) Control de fallas (bugs) Extensión Mejoramiento Habilitación
Para encarar estos problemas, percibimos la necesidad de una implementación
abierta de isis. Sin duda lo mejor sería de contar con el código existente para
isis bajo una u otra forma de licencia abierta (GPL, LGPL, artística o similar,
según sea apropiado).
Por otro lado, una implementación secundaria e independiente, tiene sus ventajas.
Permitiría un enfoque diferente, desarrollando determinadas potencialidades,
mientras que en otras situaciones el abordaje sería distinto. Por ejemplo, openisis
tendrá algún soporte para procesos múltiples y unicode, pagando un cierto costo.
Realizar una recodificación por un desarrollador con un bagaje de conocimientos
diferentes, puede aportar nuevas ideas que, luego de haber sido probadas, pueden
ayudar a mejorar el estándar.
OpenIsis como software para acceder bases de datos isis está y estará disponible
gratuitamente para cualquiera, con código fuente completo, sin aranceles ni
restricciones.
En términos generales, no está planeado reimplementar cada línea de código
escrita para isis. Para ser útil, OpenIsis debe mantener la compatibilidad con
el formato de los archivos de la base de datos. De esta manera, se puede usar
winisis o cualquier otro script existente para crear y mantener la base de datos,
empleando la interfaz perl de OpenIsis para un motor de consultas potentes y
la interfaz nativa de Java para aplicaciones web basada en servlets.
OpenIsis privilegiará la provisión de herramientas más que aplicaciones. Por
ejemplo, no intentará recrear la funcionalidad de winisis hasta que el conjunto
de herramientas GUI haya sido completado. Para alcanzar este objetivo, OpenIsis
provee acceso a los lenguajes de programación más importantes: Java para web (HECHO),
perl para scripts (HECHO) y, algún día, Tcl/Tk para interfaces gráficas independientes
de la plataforma (parcialmente hecho). Todos los demás (lenguajes) pueden por cierto, utilizar enlaces
a la librería.
Proximos pasos:
hacer un diseño de archivos configurable para permitir bases de datos
más grandes y de implementación exclusiva en java (HECHO)
implementar la búsqueda (búsqueda en texto libre HECHO)
implementar búsqueda en base a archivo invertido
mayor rendimiento: probar std (HECHO, rinde mal) y homegrown io buffering, además de
acelerar bucles en la función de conversión de ldb's
empezar a trabajar en una implementación pura de java "thread-save"
(voluntarios ?)
preparar lanzamientos binarios para windows (.exe y .dll para java) (HECHO)
implementar el lenguaje de consulta (mayormente HECHO)
implementar formateo (va en camino)
implementar la escritura (mayormente planeada)
... los voluntarios son bienvenidos !
Se comienza bajando el software. Hasta ahora es todo fuente, sin binarios.
Crearemos distribuciones binarias cuando se logre mayor estabilidad. Se desempaca
todo en algún directorio cualquiera. Para las pruebas, se necesitará también
alguna base de datos isis que deberá ser ubicada como archivo db/cds/cds.*.
Pruebe ésta . Controlar que
los nombres de archivo estén en minúsculas.
Si se está en Windows, deberá contar con un entorno cygwin con herramientas
como gmake y gcc o portar el código y escribir el archivo make para su make
y compilador. Erik armará una versión Windows empleando Linux
gcc como compilador cruzado. Si se está en Linux, está todo bien. Con un MAC
... ni idea.
Tipee "make" y disfrute de los mensajes del compilador. Tipee "make demo" y
disfrute de su primer registro openisis. (Ya instaló una base db/cds/cds.*,
no es así? Tiene 42 registros?) Tipee "make run" y observe el contenido de su
base de datos. Ttipee "make test", no debería haber diferencia entre
testout.txt y testres.txt como se entregan (usando esta base de datos cds).
Tipee "make time" para medir el rendimiento, pruebas subsiguientes generalmente
son mucho más rápidas. Mi 800 MHz P3 logra en lectura aleatoria
más de 80.000 registros por segundo, una vez que los archivos estén
en el cache del sistema.
$ make time
time ./openisis -v 3 -perf 100000 -db db/cds/cds
0.63user 0.61system 0:06.20elapsed 19%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+11607minor)pagefaults 0swaps
$ make time
time ./openisis -v 3 -perf 100000 -db db/cds/cds
0.71user 0.53system 0:01.23elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (95major+11607minor)pagefaults 0swaps
Tipee "make perl" para armar el material para perl. Se sobreentiende que previamente
deberá haberse instalado algún tipo de perl 5.*. Tipee "make java" o, si no
le alcanza, tipee "make jdump", para verlo aparecer en su nuevo y brillante
JDK 1.3 Java VM. Algún 1.2.* JDK debería hacerlo también, pero indíquele al
Makefile que no busque en /usr/java/jdk1.3, seteándolo a JAVAHOME.
libopenisis puede enlazarse con tu código; no es necesario instalarlo. Si desea
instalar los binarios 'openisis' en alguna ubicación de su path, simplemente
cópielo, no hay registros de configuración mágicos.
Para instalar openisis para su disponibilidad general en su /usr/lib/perl5
o donde esté instalado, cambie al subdirectorio perl (después de "make perl")
e ingrese "make install" (como root o legitimado de otra manera). Después pruebe
"perldoc OpenIsis" y el script de demostración demo.pl.
Java, like perl, needs to dynamically slurp both some stuff in the own language
and a native shared object. The former is openisis.jar, set your CLASSPATH to
include it, or specify when invoking java like in the Makefile. The latter is
libopenjsis.so on linux (yes, it's j sis). The system dynamic linker
must be able to find it; see NativeDb.java for details.
OpenIsis.org es patrocinado por
,
a service of merconic , Berlín, Alemania. Como un sitio
destinado a estudiantes, allmaxx apoya software de fuente abierto dirigido a
la educación y el manejo de conocimientos. Véase también la open
community for science .
Actualmente el sitio es mantenido por
Erik y Paul .
Los voluntarios son bienvenidos.
El código fuente de Openisis está disponible en
lado a lado con el proyecto PHP isis de Franck Martin's PHP. Gracias Franck!
openisis y PHP isis en sourceforge Unesco Bireme
EL LIBRO manual de referencia CDS/ISIS incl. formato de datos (en español)
Robert Janusz' iAPI
Kafkas Caprazli's TODO acerda de CDS/ISIS
software de código abierto para bibliotecas javaisis
Institut Teknologi Bandung
IsisOnline en Indonesia
Netherlands / international
UK (ISIS PLUS)
Germany (isisnetz) Erik Grziwotz Klaus "Paul" Ripke Braulio José Solano Rojas
ISIS, conjuntos de caracteres y unicode OpenIsis 0.8.x
current version
isis
openisis
howto
about
links
en espanol
welcome to OpenIsis.org
DEMOS:standard
or try
searching for unicode characters
(
unicode index )
win32 (0.8.6)
( mingw32 cross-build) should work on all win32 platforms from win95b on.solaris (0.8.4)
( build on solaris 5.8 )
Please mail us
if you need a more up-to-date solaris binary.
News:
October 19th 2002
0.8.7 October 2002writing.txt for details.
Current TODOs:
interlock multiprocess writing (PHP, Perl CGI)
and fix the windows and solaris versions.
Besides writing support, there is a new streaming record reader,
which groks various formats like the SYSPAR.PAR, email headers and
property files, so you can fill your db from such textfiles.
Next step: new indexing engine.
August 2002the universal ISIS record
July 2002
What is it about ISIS that makes it ISIS?
0.8.6 June 2002formatting notes for details.
The perl binding supports formatting (see the test.pl),
enhanced versions for all those languages are to follow soon.
PHP June 2002Braulio José Solano Rojas
from Costa Rica created a PHP binding, which can be seen in action at
galileo .
Available as download
or from sourceforge module php-openisis.
Also, the Institut Teknologi Bandung of Indonesia
switched it's web index to a PHP/OpenIsis based version.
0.8.5.2 March 2002
0.8.5.1 March 2002Vn-field-selector-style method
and several nice utils.
Indentation is not properly handled, since there is no easy
common solution in HTML. Will build tools for a choice of
standard strategies later ...
0.8.5 March 2002demo ).
All the operators are there (including /(tag), but not /(t1,t2...)).
Every attempt is made to limit the potential costs of even extremely
stupid queries like "$"^"$", so no historical (#n) or intermediate
results (for precedence) are stored.
Queries are processed strictly left to right on a buffer of 1000 hits.
0.8.4 January 2002searching the unesb db
with over 58.000 rows (note the hosting server is a 500MHz Celeron).
0.8.3 October 2001demo
shows how a query is refined (narrowed) iteratively
by ANDing it with a second query.
And thanks to Veronica Lencinas and colleagues,
we have a version of this document en espanol .
September 2001sourceforge .
Windows version openisis.exe running.
0.8.2 August 2001LGPL , no legacy code.
Conversion from file structures governed by abstract dynamic description
rather than C-structs, so we can support different file layouts,
much larger databases, big endian processors and more.
Simple full-scan search available in C-Lib and Java.
Given a random read throughput of about 30.000 Records/sec on a
lame 300 MHz Notebook this seems to be of practic use.
jsp demo under development ;).
0.8.1 June 2001
0.8.0 May 2001
0.7.9 May 2001
Isis is a simple, yet powerful database system with a large installed
base since the 80s. Since it's well suited for bibliographic data,
it's commonly used in libraries, and since it's very low cost,
especially in those running on a low budget.
An isis DB is a list of rows of unspecified structure,
each identified by a unique number, the rowid (a.k.a. mfn).
Each row is a list of fields, and each field has number (tag)
and a string value. Within a row there may be zero, one or more
fields with a given tag. While the field's value usually is a
textual representation of data in one or the other character
encoding (commonly one of the IBM/DOS code pages), it may
actually contain arbitrary bytes.
This is closely modelled after ISO2709 "Information Interchange Format"
(IIF, a.k.a. ANSI/NISO
Z39.2 )
subfields
There is a convention to encode multiple fields in one by separating
them with a '^' followed by one character tagging the subfield.
So the field value '^afoo^bbar^bbaz' represents a field having
one 'a' subfield with value 'foo' and two 'b' subfields 'bar' and 'baz'.
An other separator char may be used,
e.g. ASCII character 31 ("Unit Separator") is used in the
MARC
standard.
formatting
There is a formatting language, with literal text, field and subfield
variables, if-else branches (on field existance)
and for loops (over field repetitions) (roughly speaking).
indexing
An index is build by converting a row into a list of words
(optionally applying formats) and stuffing every word,
qualified by the position of it's occurence in the row,
into a B+-Tree (which is actually spread to six files).
Searching for a word or
word prefix is possible with or without qualifying the position (field).
Since all fields can be combined into one index, it is usually not
necessary (but possible) to set up multiple indexes.
queries
A query language allows for combination of word lookups using
and, or and not(without) operators.
This is very similar to the "Type-1" query of
Z39.50 .
usage
While isis lacks most features of RDBMS like complex relations
between different entities, it's flexibility comes in handy for
many catalogues and directories with highly varying records and
one single level of substructure, which today are usually
modelled in XML documents rather than table rows.
In other words, isis is an ideal storage for many XML applications.
The flexible indexing mechanism combines the best of full text
searching and structured retrieval.
The mother of all isis software is a DOS version of "MicroISIS"
as an integrated system with textual user interface.
There is a BSD version of "CDS/ISIS" which
also runs under linux up to some 2.2.x kernels
(current 2.4 kernels do not support the iBCS module for COFF binaries).
Then there are several versions of "WinISIS" (M$-Windows only,
but runs under linux/wine).
A shared library version "isis.dll" of functions
to access an isis db from your code exists, despite it's name,
also in a linux version ("isilux");
however, you need some pretty special libs to make it run.
A set of command line tools ("cisis") performs tasks like importing
ISO2709 bibliographic databases, inverting (index building) etc.
The thing next to an isis database server is "wwwisis",
which runs as CGI or from the command line and performs
most isis tasks (win and lin versions). However, it actually
runs per request, not as a server itself, and thus cannot
provide concurrency control.
This "official" isis software, which is maintained by
Unesco and/or
Bireme ,
is accompanied by a couple of independent developments,
some of which are in the public domain.
Javaisis
is an AWT-based GUI (3.5 uses SWING) and a corresponding server,
which in turn uses wwwisis.
Robert Janusz
wrote a C-lib (iAPI) from scratch,
which was the starting point for the openisis software.
So why are we writing the openisis software?
Because Isis is not open source software, it's not even free software,
and that leads to a whole bunch of problems.
Availability (in theory)
Availability (in practice)Isisnetz remedied the situation.
Availability (in legal terms)
Availability (of documentation)
Bugfixing
Extending
Improving
Enabling
To address these problems we feel the need for an open source
implementation of isis.
Of course it would be best to have all of the existing isis code
under one or the other form of open license (GPL, LGPL, artistic
or similar as appropriate).
On the other hand, an independent secondary implementation has
advantages in it's own right. It may have a different focus
and develop strengths in one aspect while another aproach
performs better in other situations. For example,
openisis will have some support for multithreading and unicode,
which is paid for by a certain overhead.
A rewrite by developers with a different background might
introduce new ideas which finally, after having had their
indepent test bed, help improve the standard.
OpenIsis as a software to access isis databases is and will be freely
available for everybody with full sourcecode, no fee, no restrictions.
In general, there are no plans to reimplement every piece
of code ever written for isis. To be of practical value,
OpenIsis has to maintain compatibility in the format of
the database files anyway. So, one may use winisis or
whatever existing import scripts to create and maintain
the database, yet deploy OpenIsis' perl interface to run
powerful reports and the Java Native Interface to allow
queries from a Servlet based web application.
OpenIsis will focus on providing tools rather than applications.
For example, there will be no attempt to mirror functionality
of winisis unless the GUI toolkit is done. To achive this,
OpenIsis provides access from the most important programming
languages: Java and PHP for the web (DONE), perl for the scripts (DONE) and
Tcl/Tk for platform independent GUIs (partly DONE).
All others can, of course, link the lib.
Next steps:
make file layout configurable to allow for larger db (DONE)
implement searching (full-scan searching DONE)
implement index-based searching (DONE)
more performance: try std (DONE, performs badly)
and homegrown io buffering,
further accelerate loop in ldb's convert function (DONE)
start work on thread-save pure-java implementation
(cancelled due to lack of demand)
prepare binary releases for windows (.exe and .dll for java) (DONE)
implement query language (simple version DONE)
implement formatting (simple version DONE)
implement writing data (masterfile DONE, index underway)
finish Tcl binding, create GUI version using TK (-> Erik)
implement server version
... volunteers are welcome !
Start by downloading the Software.
Unpack everything in some arbitrary directory.
For the tests, you will also need some isis database,
which must be located as files db/cds/cds.*.
Try this one .
Make sure filenames are lowercase.
If you are on Windows, you should either get yourself the cygwin
environment with tools like gmake and gcc or volunteer as a porter
and start writing the Makefile for your make and compiler.
Erik has build a Windows version using mingw and Linux gcc as crosscompiler.
If you are on Linux, everything is fine.
Ports to MAC OS X and other UNIXes should be no problem.
Type "make" and enjoy the compiler messages.
(If your make complains, e.g. on BSD, try "gmake").
Type "make demo" and enjoy your first open isis record.
(You installed a db/cds/cds.*, didn't you? It has 42 rows?)
Type "make run" and watch the guts of your db passing by.
Type "make test", there should be no difference between the testout.txt
and the testres.txt as provided
(using this cds database from winisis
and this 15 MB 58.000+ row unesb.zip db).
Type "make time" to measure performance,
subsequent tries are usually much faster.
My 800 MHz P3 random-reads more than 179.000 records a second,
once the files are in the system cache.
Typical values:
time ./openisis -perf 1000000 -db db/cds/cds >/dev/null
real 0m5.655s
user 0m3.650s
sys 0m2.000s
time ./openisis -perf 100000 -db db/unesb/unesb >/dev/null
real 0m0.991s
user 0m0.670s
sys 0m0.320s
time ./openisis -fmt mfnf -search 'k$' -db db/unesb/unesb >/dev/null
860 rows for k
real 0m0.044s
user 0m0.040s
sys 0m0.000s
Type "make perl" to build the perl stuff;
some perl 5.* must be installed beforehand.
Type "make java" or, if you just can't get enough, "make jdump",
to see it all happen in your shiny new JDK1.3 Java VM.
Some 1.2.* JDKs should do, but tell the Makefile to not
look in /usr/java/jdk1.3 by setting JAVAHOME.
libopenisis.a can be linked with your code; no installation necessary.
You may wan't to install the 'openisis' binary somewhere in your path
for the fun of it; go ahead, just copy, no magic registry entries.
To install the perl stuff for general availability in your
/usr/lib/perl5 or whatever, cd to the perl subdir (after "make perl")
and issue "make install" (as root or otherwise legitimated).
After that, try "perldoc OpenIsis" and the demo.pl script.
Java, like perl, needs to dynamically slurp both some stuff in
the own language and a native shared object.
The former is openisis.jar, set your CLASSPATH to include it,
or specify when invoking java like in the Makefile.
The latter is libopenjsis.so on linux (yes, it's j sis).
The system dynamic linker must be able to find it;
see NativeDb.java for details.
OpenIsis.org is sponsored by
,
a service of merconic , Berlin, Germany.
As a student's site, allmaxx supports open software with a focus on
education and knowledge management.
See also the open community for science .
Currently the site is maintained by
Erik and Paul .
Volunteers are very welcome.
Openisis sources are available at
side by side with Franck Martin's PHP isis project. Thanks, Franck!
openisis and PHP isis at sourceforge Unesco Bireme
THE BOOK CDS/ISIS reference manual incl. data formats (en espanol)Z39.2 MARC
21, overview Z39.50 ,
overview at
OCLC|Pica ,indexdata ,
makers of excellent free Z39.50 software.
Robert Janusz' iAPI
Kafkas Caprazli's EVERYTHING about CDS/ISIS
open source software for libraries javaisis
Institut Teknologi Bandung
IsisOnline in Indonesia
Netherlands / international
UK (ISIS PLUS)
Germany (isisnetz) Erik Grziwotz Klaus "Paul" Ripke Braulio José Solano Rojas ISIS, charsets and unicode What is it about ISIS that makes it ISIS? the universal ISIS record record writing implementation multi-threading performance
#include
#include
#include /* PATH_MAX */
#include /* or try unixio */
#include
#include
#include
#include
#include /* threaded crashtest */
#define __USE_GNU
#include
#define P 16
// #define S 1024
#define S 8192
char b[P][S];
struct aiocb a[P];
const struct aiocb *pa[P];
struct aioinit ai = { 16, 16, 0,0,0,0, 60 };
int mode = 0;
int trep = 100;
int fd, max;
struct t {
pthread_t t;
int pid;
int id;
int d,f;
} p[P];
/* (p)thread routine */
static void * run ( void *arg ) {
struct t *my = (struct t *)arg;
int fid = dup( fd );
int i;
for ( i=0; iid) % max;
int off = blk*S;
if ( S == pread( fid, b[0], S, off ) )
my->d++;
else
my->f++;
}
return NULL;
}
int main ( int argc, char **argv ) {
struct stat s;
struct timeval t0, t1, t2;
int u, i, did = 0, fail = 0, susp = 0;
if ( 2 < argc )
mode = atoi( argv[2] );
if ( 3 < argc )
trep = atoi( argv[3] );
trep *= 1000;
fd = open( argv[1], O_RDONLY );
fstat( fd, &s );
max = s.st_size / S;
memset( a, 0, sizeof(a) );
for ( i=P; i--; ) {
a[i].aio_fildes = fd;
a[i].aio_buf = b[i];
a[i].aio_nbytes = S;
a[i].aio_sigevent.sigev_notify = SIGEV_NONE;
/*
a[i].aio_sigevent.sigev_notify = SIGEV_SIGNAL;
a[i].aio_sigevent.sigev_signo = SIGIO;
a[i].aio_sigevent.sigev_value.sival_ptr = &a[i];
*/
pa[i] = &a[i];
}
if ( 1 == (mode & ~1) )
aio_init( &ai );
gettimeofday( &t0, NULL );
t1 = t0;
i = 0;
if ( 4 == (mode & ~1) ) {
for ( i=0; i= P ) for ( ;; j++ ) {
if ( i+P == j ) { /* checked all -- wait */
if ( aio_suspend( pa, P, NULL ) ) {
perror( "aio_suspend" );
exit(1);
}
susp++;
j = i;
}
switch ( aio_error( &a[j%P] ) ) {
case EINPROGRESS: continue;
case ENOSYS:
perror( "aio_error" );
exit(1);
case 0:
if ( S == aio_return(&a[j%P]) )
did++;
else
fail++;
nop = 0;
}
break;
}
j %= P;
a[j].aio_offset = off;
if ( aio_read( &a[j] ) ) {
perror( "aio_read" );
exit(1);
}
i = j+1;
if ( nop )
continue;
}
break;
default:
if ( S == pread( fd, b[0], S, off ) )
did++;
else
fail++;
}
if ( ! (did % 1000) && did ) {
gettimeofday( &t2, NULL );
u = t2.tv_usec - t1.tv_usec + (t2.tv_sec - t1.tv_sec)*1000000;
printf( "%d.%06d %d %d %d\n", u / 1000000, u % 1000000, did, fail, susp );
t1 = t2;
}
}
gettimeofday( &t2, NULL );
u = t2.tv_usec - t0.tv_usec + (t2.tv_sec - t0.tv_sec)*1000000;
printf( "\n%d.%06d mode %d did %d fail %d susp %d throughput %d/sec\n",
u / 1000000, u % 1000000, mode, did, fail, susp,
1000 * did / (u / 1000) );
return 0;
}
isocvt 0100775 0001130 0001130 00000000362 07427563367 011016 0 ustar krip krip #!/usr/bin/perl
# tools for ISO 2709 format
#
# field sep is ^^ (char val 30)
# record sep is ^]^J (char val 29 and newline)
$/ = "\035\n";
while () {
print;
last if $. > 20;
# chomp;
# y/\n\t\036/ \t/;
# print $_, "\n";
}
jdocstyle.css 0100664 0001130 0001130 00000002331 07445455333 012262 0 ustar krip krip /* Javadoc style sheet */
/* Define colors, fonts and other style attributes here to override the defaults */
/* Page background color */
body { background-color: #FFFFFF }
/* Table colors */
.TableHeadingColor { background: #00CC00 } /* Dark Green */
.TableSubHeadingColor { background: #AAEEAA } /* Light Green */
.TableRowColor { background: #FFFFFF } /* White */
/* Font used in left-hand frame lists */
.FrameTitleFont { font-size: normal; font-family: normal }
.FrameHeadingFont { font-size: normal; font-family: normal }
.FrameItemFont { font-size: normal; font-family: normal }
/* Example of smaller, sans-serif font in frames */
/* .FrameItemFont { font-size: 10pt; font-family: Helvetica, Arial, sans-serif } */
/* Navigation bar fonts and colors */
.NavBarCell1 { background-color:#AAEEAA;}/* Light Green */
.NavBarCell1Rev { background-color:#00CC00;}/* Dark Green */
.NavBarFont1 { font-family: Arial, Helvetica, sans-serif; color:#000000;}
.NavBarFont1Rev { font-family: Arial, Helvetica, sans-serif; color:#FFFFFF;}
.NavBarCell2 { font-family: Arial, Helvetica, sans-serif; background-color:#FFFFFF;}
.NavBarCell3 { font-family: Arial, Helvetica, sans-serif; background-color:#FFFFFF;}
jtest.bat 0100664 0001130 0001130 00000003675 07533161161 011374 0 ustar krip krip @echo off
rem openisis - an open implementation of the ISIS database
rem Version 0.8.x (microversion see file Version)
rem Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
rem
rem This library is free software; you can redistribute it and/or
rem modify it under the terms of the GNU Lesser General Public
rem License as published by the Free Software Foundation; either
rem version 2.1 of the License, or (at your option) any later version.
rem
rem This library is distributed in the hope that it will be useful,
rem but WITHOUT ANY WARRANTY; without even the implied warranty of
rem MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
rem Lesser General Public License for more details.
rem
rem You should have received a copy of the GNU Lesser General Public
rem License along with this library; if not, write to the Free Software
rem Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
rem
rem This software is dedicated to the memory of Eckart Dietrich.
rem
rem This software is inspired by (but contains no code of) the iAPI
rem Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
rem See iAPI.txt for what it contains.
rem
rem
rem $Id: jtest.bat,v 1.1 2002/08/28 14:47:13 erikisis Exp $
rem Testscript for openisis java classes on win32 platforms
rem
set CLASSPATH=.
if not exist testout.txt goto bug
del testout.txt
:bug
java org.openisis.Test -db db/cds/cds -encoding Cp850 -row 42 >> testout.txt
java org.openisis.Test -db db/cds/cds -encoding Cp850 -search development >> testout.txt
java org.openisis.Test -db db/cds/cds -encoding Cp850 -terms "" >> testout.txt
java org.openisis.Test -db db/cds/cds -encoding Cp850 -mpu -row 130 >> testout.txt
java org.openisis.Test -db db/cds/cds -encoding Cp850 -mhl -hta -row 130 >> testout.txt
java org.openisis.Test -db db/cds/cds -encoding Cp850 -m1 -row 130 >> testout.txt
fc testout.txt jtest.txt
jtest.txt 0100644 0001130 0001130 00000055706 07445370772 011461 0 ustar krip krip got row 42
<24><The> Meteorological problems of the deltaic flood-plains of East Pakistan
<26>^c1966
<26c>1966
<30>^ap. 123-133^billus., maps
<30a>p. 123-133
<30b>illus., maps
<44>Scientific problems of the humid tropical zone deltas and their implications: proceedings of the Dacca Symposium
<50>Incl. bibl.
<69>Paper on: <meteorology><deltas><floods><sea level><climatic data><rain><Bangladesh>
<70>Naqvi, Sibte Nabi
got row 100
<24>Micropolitan development: theory and practice of greater-rural economic development
<26>^aAmes^bIowa State University Press^c1976
<26a>Ames
<26b>Iowa State University Press
<26c>1976
<30>^a456 p.^bmap
<30a>456 p.
<30b>map
<50>Incl. bibl.
<69><regional planning><rural development><USA> - <human resources development><employment><economics of education><poverty><community development><industrialization><land use><economic policy><statistical data>
<70>Tweeten, Luther G.
<70>Brinkman, George Lorris
got row 114
<24>Educational development; a new discipline for self-renewal
<26>^aEugene, Or.^bCenter for the Advanced Study of Educational Administration^c1973
<26a>Eugene, Or.
<26b>Center for the Advanced Study of Educational Administration
<26c>1973
<30>^a343 p.^billus.
<30a>343 p.
<30b>illus.
<50>(CASEA monograph, no.24). Incl. bibl.
<69><educational development><public education><educational planning><educational research><USA> - <educational programmes><curriculum development><teacher training><evaluation of education><information systems><educational management><educational statistics>
<70>Hemphill, John Knox
<70>Rosenau, Fred S.
<71>Far West Laboratory for Educational Research and Development (USA)
<71>University of Oregon (USA). Center for the Advanced Study of Educational Administration
got row 130
<12>Joint FAO/WHO Expert Committee on Nutrition^n9th^pRome^d1974
<24>Food and nutrition strategies in national development
<26>^aRome^bFAO ^c1976
<26a>Rome
<26b>FAO
<26c>1976
<30>^a64 p.
<30a>64 p.
<50>(FAO nutrition meetings report series, no.56). WHO technical report series, no.584
<69><agricultural development><food production><nutrition><national development> - <economic and social development><food resources><research priorities><developing countries>
<71>FAO
<71>WHO
<76>Stratégies de l'alimentation et de la nutrition dans le développement national^zfre
got row 131
<24><The> Measurement of scientific and technical activities - Proposed standard practice for surveys of research and experimental development
<26>^aParis^bOECD^c1976
<26a>Paris
<26b>OECD
<26c>1976
<30>^a138 p.
<30a>138 p.
<69><technology assessment><science and technology><research and development><science statistics><statistical analysis><survey analysis><international standards><guides>
<71>OECD
<74>Proposed standard practice for surveys of research and experimental development
<76>La Mesure des activités scientifiques et techniques^zfre
got row 141
<24>Energy for rural development: renewable resources and alternative technologies for developing countries
<26>^aWashington^bNational Academy of Sciences^c1976
<26a>Washington
<26b>National Academy of Sciences
<26c>1976
<30>^a306 p.^billus.
<30a>306 p.
<30b>illus.
<50>(PB-260-606). Incl. bibl.
<69><renewable resources><solar energy><energy resources><rural development><technology transfer><appropriate technology><developing countries> - <energy conversion><geothermal energy><electrical engineering><wind power><biomass><manufacturing industries><research and development><research centres><directories>
<71>National Academy of Sciences (USA). Ad Hoc Panel on Renewable Energy Resources
<71>National Research Council (USA)
<71>National Academy of Sciences (USA). Board on Science and Technology for International Development
got row 149
<12>Symposium on the Psychological Bases of Programmed Learning ^pTbilisi, USSR^d1976
<24>For a training project based on programmed learning in the developing countries: objectives, means and resources, organization and management, development
<26>^c14 June 1976
<26c>14 June 1976
<30>^a18 p.
<30a>18 p.
<69><educational assistance><programmed instruction><developing countries> - <educational innovations><educational technology><educational needs><case studies><teaching methods><educational experiments><technical and vocational education><adult education><working population><production><programmed materials>
<70>Pocztar, Jerry
got row 150
<24>Micropolitan development: theory and practice of greater-rural economic development
<26>^aAmes^bIowa State University Press^c1976
<26a>Ames
<26b>Iowa State University Press
<26c>1976
<30>^a456 p.^bmap
<30a>456 p.
<30b>map
<50>Incl. bibl.
<69><regional planning><rural development><USA> - <human resources development><employment><economics of education><poverty><community development><industrialization><land use><economic policy><statistical data>
<70>Tweeten, Luther G.
<70>Brinkman, George Lorris
ABBAS, B.M.
ABEYWICKRAMA, B.A.
ABOUT
ABSENCE
ABSORPTION
ACCOMPANYING
ACCOUNT
ACCOUNTING
ACHIEVEMENTS
ACID
ACTION
ACTIVITIES
ACTUAL
ADDRESSES
ADMINISTRATIVE REFORM
ADMISSION
ADMISSION CRITERIA
ADULT
ADULT EDUCATION
AERIAL
AERIAL PHOTOGRAPHY
AESTHETICS
AFFECTIVE
AFGHANISTAN
AFRICA
AFRICAN LANGUAGES
AGE
AGE GROUPS
AGENT
AGENTS
AGRICULTURAL DEVELOPMENT
AGRICULTURAL TRAINING
AGRICULTURE
AGRONOMY
AHMAD, MAJEED
AHMAD, NAZIR
ALLUVIAUX
ALLUVIONS
ALTERNATIVE
AMAZON
AMERICA
ANALYSIS
ANALYTICAL
ANAYA, RICARDO
ANCIENT CIVILIZATIONS
ANIMAL
ANNA
ANNOTATED
ANNUAL
ANTI
ANTIQUITIES
APERCU
APPARATUS
APPLICATION
APPROACH
APPROPRIATE TECHNOLOGY
APRIL
APTER, DAVID ERNEST
AQUIFERS
ARAB COUNTRIES
ARABES
ARCHITECTURE
AREA
AREAS
ARID
ARMAMENTS
ARMS CONTROL
ARMS SALES
ART BOOKS
ART HISTORY
ARTISTIC CREATION
ARTS
ASIA
ASPECTS
ASPIRATION
ASSIMILATION
ASSURANCES
AT
ATANGANA, ENGELBERT
ATTITUDE
AUDIT
AUSTRALIA
AUSTRALIAN
AUTOMATIC
AVAILABLE
AWAY
AYALA, FRANCISCO JOSE
BAEZ, ALBERT V.
BAHOKEN, J.C.
BALANCE
BANGLADESH
BARRS, H.D.
BARS
BASED
BASEL
BASIC
BASIC EDUCATION
BASIS
BAUMGARTNER, ALBERT
BE
BEHAVIOUR
BELOW
BENGAL
BETWEEN
BIBLIOGRAPHIE
BIBLIOGRAPHIES
BIBLIOGRAPHY
BIOLOGICAL EVOLUTION
BIOLOGICAL RESEARCH
BIOLOGY
BIOMASS
BITSIOS, DIMITRI S.
BJELICA, MIHALO
BLACKS
BLOOM, JOSEPH
BOARDING SCHOOLS
BOESCH, H.
BOLIVIA
BONDURANT, JOAN V.
BOOK
BORDIA, ANIL
BORICHANSKY, L.S.
BORING
BOROBUDUR
BOSIAN, G.
BOURDEAU, PHILIPPE F.
BOWRA, SIR CECIL MAURICE
BRACKISH
BRACKISH WATER
BRAIN DRAIN
BRASIL
BRAZIL
BRINKMAN, GEORGE LORRIS
BRIX, H.
BROADCAST
BROADCASTING
BROTHERS
BUDDHISM
BUDEL, J.
BUDGET
BUILDING
BURMA
BUSINESS MANAGEMENT
C
CALCULATION
CAMEROON
CAMEROON UR
CANADA
CARACTERISTIQUES
CARBON
CARBON DIOXIDE
CARBONIQUE
CARIBBEAN AREA
CARLIER, G.
CARMEL
CARTE
CARTOGRAPHY
CASE STUDIES
CASE, JOHN H.
CASPERS, H.
CAT
CATALOGUES
CATSKY, J.
CENTRE
CEREALS
CEREMONY
CEYLON
CHAMBER
CHANCELLOR
CHANDI
CHANGE
CHANGES
CHANGING
CHANGING SOCIETY
CHAO
CHAPMAN, VALENTINE J.
CHEMICAL ANALYSIS
CHEMICAL PROPERTIES
CHEMICAL/BIOLOGICAL WARFARE
CHI
CHILDREN
CHILDREN'S BOOKS
CHIMIQUE
CHO
CHOWDHURY, M.I.
CHURCH
CHURCHILL, STACY
CIVILIZATION
CLASSICAL
CLASSIFICATION
CLAYS
CLIMATE
CLIMATIC
CLIMATIC DATA
CLUBS
CO
COASTAL
COASTAL ZONES
COLOMBIA
COLORIMETRIC
COMMUNICATION
COMMUNICATION HISTORY
COMMUNICATION IMPACT
COMMUNICATION LEGISLATION
COMMUNICATION PERSONNEL TRAINI
COMMUNICATION PLANNING
COMMUNICATION POLICY
COMMUNICATION PROCESS
COMMUNICATION RESEARCH
COMMUNICATION STATISTICS
COMMUNITIES
COMMUNITY
COMMUNITY DEVELOPMENT
COMMUNITY EDUCATION
COMPARATIVE
COMPARATIVE ANALYSIS
COMPARISON
COMPULSORY EDUCATION
COMPUTER APPLICATIONS
COMPUTER HARDWARE
COMPUTER SOFTWARE
COMUNICACAO
CONCENTRATION
CONCLUSIONS
CONDITIONS
CONFERENCE
CONFERENCES
CONFLICT
CONFLICT RESOLUTION
CONFORM
CONQUEST
CONSULTANCY
CONTEMPORARY SOCIETY
CONTRIBUTES
CONTROL
CONTROLLED
CORPORATION
COUGHLAN, MARGARET N.
COUNTRIES
COUNTRY REPORTS
COURSES
COVER
COVERS
CRAIN, N.
CRITICISM
CROISSANCE
CROP
CROPPED
CROSS
CUBA
CULTIVATION
CULTURAL
CULTURAL ACTIVITIES
CULTURAL DEVELOPMENT
CULTURAL FACILITIES
CULTURAL FACTORS
CULTURAL HISTORY
CULTURAL IDENTITY
CULTURAL INSTITUTIONS
CULTURAL PLANNING AND ADMINIST
CULTURAL POLICY
CULTURAL PROGRAMMES
CULTURE
CURRENT
CURRENTS
CURRICULUM DEVELOPMENT
CURRICULUM PLANNING
CYPRUS
D
DACCA
DANCE, F. ROBERT
DANS
DATA COLLECTION
DATA PROCESSING
DAVE
DAVE, RAVINDRAKUMAR H.
DE
DECENTRALIZATION
DECISION
DECISION MAKING
DEFICIT
DEFINIES
DEFINITIONS
DEGRE
DEGREES
DEGUCHI, K.
DELIVERED
DELTA
DELTAIC
DELTAS
DEMAND
DEMOCRATIC KAMPUCHEA
DEMOCRATIZATION OF EDUCATION
DES
DESARROLLO
DESCRIPTION
DESIGN
DETERMINATION
DETERMINING
DEVELOPED COUNTRIES
DEVELOPING
DEVELOPING COUNTRIES
DEVELOPMENT
DEVELOPMENT PLANNING
DEVELOPMENT STRATEGIES
DEVELOPPEMENT
DEW
DIFFERENT
DIFFUSION
DIOXIDE
DIPLOMAS
DIRECTOR
DIRECTORIES
DISARMAMENT
DISC
DISCHARGE OF RIVERS
DISCIPLINE
DISCUSSED
DISPOSITION
DISSEMINATION OF INFORMATION
DOBZHANSKY, THEODOSIUS GRIGORI
DOCUMENTATION
DOCUMENTATION CENTRES
DOCUMENTOS
DOTY, M.S.
DOW, PHILIP S.
DRAINAGE
DRAPER, JAMES A.
DROUGHT
DRY
DU
DUE
DUKE, CHRISTOPHER
E
EAST
EASTERN EUROPE
EAUX
ECOLOGY
ECONOMIC
ECONOMIC AND SOCIAL DEVELOPMEN
ECONOMIC BEHAVIOUR
ECONOMIC CONCENTRATION
ECONOMIC CONDITIONS
ECONOMIC GROWTH
ECONOMIC POLICY
ECONOMICS OF EDUCATION
ECONOMY
ECOSYSTEMS
EDUCATION
EDUCATIVES
EFFECT
EFFECTIFS
EFFECTS
EGOROV, V.V.
ELECTRIC
ELECTRICAL ENGINEERING
ELEMENT
ELKANA, YEHUDA
EMISSION
EMPLOYERS
EMPLOYMENT
EN
ENERGY
ENERGY BALANCE
ENERGY CONVERSION
ENERGY RESOURCES
ENROLMENT
ENROLMENT PROJECTIONS
ENROLMENT RATIOS
ENSEIGNEMENT
ENVIRONMENTAL
ENVIRONMENTAL CHANGES
ENVIRONMENTAL CONTROL
ENVIRONMENTAL EDUCATION
ENVIRONMENTAL QUALITY
EQUATION
ESCAP
ESTABLISHMENT
ESTUARIES
ESTUARINE
ESTUARY
ET
ETHICS
ETUDE
EUROPE
EVALUATING
EVALUATION METHOD
EVALUATION OF EDUCATION
EVAPORATION
EVAPOTRANSPIRATION
EVERYDAY LIFE
EXAMINATIONS
EXCHANGE
EXISTENCE
EXPECTATIONS
EXPERIENCE
EXPERIMENT
EXPERIMENTAL
EXPERIMENTS
EXPERTS
EXTRACTS
FACTORS
FAJARDO, JOSE CARLOS
FAMILY LIFE
FAUNA
FAURE, EDGAR
FEATURES
FELLOWSHIPS
FERTILIZERS
FIELD
FIELDS
FILMS
FINAL
FIRST
FISH
FISHERIES
FLOOD
FLOOD CONTROL
FLOODING
FLOODS
FLORA
FLORAS
FLOW MEASUREMENT AND INSTRUMEN
FOLKLORE
FOOD
FOOD PRODUCTION
FOOD RESOURCES
FORCE
FOREIGN INVESTMENT
FOREIGN TRADE
FOREST
FORESTS
FORMATION
FORMERLY
FOSBERG, F. RAYMOND
FRADIER, GEORGES
FRANCE
FRANCO, C.M.
FRANEKER
FREEDOM
FRESH
FRESHWATER
FULLY
FUTURE
GALE, J.
GANDHIAN
GANGES
GASEOUS
GAUGES
GAZ
GENERAL
GENESIS
GEODYNAMICS
GEOLOGICAL
GEOLOGY
GEOMORPHOLOGIE
GEOMORPHOLOGY
GEOTHERMAL ENERGY
GERMANY FR
GHANI, Q.
GOODMAN, LOUIS WOLF
GOOR, G.A.W. VAN DE
GRADUAL
GRADUATES
GRASS
GREAT
GREATER
GREECE
GRIEVE, B.J.
GROOT, A.J. DE
GROUND
GROUNDWATER
GROUP DISCUSSIONS
GROWTH
GUIDANCE SERVICE
GUIDE
GUIDES
GUTTMANN, LUDWIG
HALLAK, JACQUES
HANDBOOK
HAVELOCK, MARY C.
HAVELOCK, RONALD G.
HEALTH
HEAT
HEAT TRANSFER
HEEMST, H.D.J. VAN
HEMPHILL, JOHN KNOX
HERVIEU, J.
HIGH
HIGHER
HIGHER EDUCATION
HIGHER TECHNICAL PERSONNEL
HISTOIRE
HISTORIC MONUMENTS
HISTORICAL ANALYSIS
HISTORY
HOLDINGS
HOLLANDER, JACK M.
HOLLERWOGER, F.
HOLT, MICHAEL
HOME
HOOK, SIDNEY
HORLAMUS, SEPP
HOUSE
HOUSING
HUDSON, J.P.
HUISMAN, L.
HUMAN
HUMAN ACTIVITIES EFFECTS
HUMAN ECOLOGY
HUMAN RESOURCES DEVELOPMENT
HUMAN RIGHTS
HUMAN SETTLEMENT
HUMID
HUMID ZONES
HUMIDITE
HUMIDITY
HUNGARY
HYDRATURE
HYDRAULIC
HYDRAULIC ENGINEERING
HYDRODYNAMICS
HYDROGEOCHEMISTRY
HYDROGEOLOGY
HYDROGRAPHY
HYDROLOGICAL
HYDROLOGY
HYDROMETEOROLOGICAL
HYDROMETEOROLOGY
HYGROMETER
HYGROMETERS
IBE
IDEAL
IMPLICATIONS
IMPORTANT
IMPROVEMENT
IN-SERVICE TEACHER TRAINING
INCLUDING
INCOME
INCOME DISTRIBUTION
INCOMUNICACAO
INDEX
INDIA
INDIAN
INDIRECT
INDIVIDUAL
INDONESIA
INDUSTRIAL PRODUCTS
INDUSTRIALIZATION
INDUSTRIES
INFLUENCE
INFORMATION PROCESSING AUTOMAT
INFORMATION SERVICES
INFORMATION SOURCES
INFORMATION SYSTEMS
INFORMATION USERS
INFORMATION/LIBRARY FINANCING
INFORMATIQUE
INFRA
INNOVATION
INNOVATION BEHAVIOUR
INSTRUCTIONAL
INTEGRATED
INTEGRATED CURRICULUM
INTERACTION
INTERDISCIPLINARY APPROACH
INTERGOVERNMENTAL ORGANIZATION
INTERNAL
INTERNATIONAL
INTERNATIONAL CONFLICT
INTERNATIONAL COOPERATION
INTERNATIONAL LAW
INTERNATIONAL ORGANIZATIONS
INTERNATIONAL RELATIONS
INTERNATIONAL STANDARDS
INTERNATIONAL TRADE
INTERNATIONALE
INVERTEBRATA
INVOLVED
IRAN (ISLAMIC REPUBLIC)
IRRAWADDY
IRRIGATION
ISLAM, M. AMIRUL
ISRAEL
IT
ITALY
JACKMAN, MARY ELIZABETH
JAVA
JOBORU, MAGDA
JOURNALISM
JOURNALISTS
JUSQU
KARAMAZOV
KARENINA
KARIM, A.Q.M.B.
KENYA
KHAN, F. KARIM
KHUDA, M. QUDRAT
KIDD, JAMES ROBBINS
KLAUSING, O.
KLEINFELD, JUDITH S.
KRAMER, P.J.
KREEB, K.
L
LA
LABORATORY
LABOUR
LABOUR MARKET
LABOUR MIGRATION
LABOUR POLICY
LABOUR RELATIONS
LAND
LAND RECLAMATION
LAND USE
LANGE, O.L.
LANGUAGE BEHAVIOUR
LANGUAGE OF INSTRUCTION
LATIN
LATIN AMERICA
LAW
LE
LEADERSHIP
LEAF
LEARNING
LEARNING PROCESSES
LEAST DEVELOPED COUNTRIES
LEAVES
LEBLOND, C.
LECTURES
LEGISLATION
LEIDEN
LEKOVIC, ZDRAVKO
LEROY, GEORGES
LES
LEVITT, J.
LIBERATION MOVEMENTS
LIETH, H.
LIFE
LIFE-LONG EDUCATION
LIFELONG
LIKELY
LIST
LITERACY PROGRAMMES
LITERARY CRITICISM
LITERATURE
LIVING CONDITIONS
LONG
LOSS
LUMIERE
LYSIMETER
M
MADAGASCAR
MAGALHAES, A.C.
MAHEU
MAIN
MAKING
MAKKINK, G.F.
MAN
MANAGEMENT
MANAGEMENT INFORMATION SYSTEMS
MANAGEMENT OPERATIONS
MANGANESE
MANGOKY
MANGROVE
MANGROVES
MANKIND
MANPOWER NEEDS
MANUEL
MANUFACTURING INDUSTRIES
MAQUIS
MARJORAM, D.T.E.
MARKET
MARRAKECH
MARSHES
MASS
MASS CULTURE
MASS MEDIA
MASTER
MATHEMATICAL
MATHEMATICAL LOGIC
MATHEMATICAL MODELS
MATHEMATICS
MATTER
MAY, WILLIAM H.
MCQUAIL, DENIS
MEANING
MEANS
MEASUREMENT
MEASUREMENT AND INSTRUMENTS
MEASUREMENTS
MEASURING
MEDIA
MEDICINAL PLANTS
MEETING
MEETINGS
MELO, JOSE MARQUES DE
MEMORIAL
MESURE
METABOLISM
METEOROLOGICAL
METEOROLOGY
METHOD
METHODOLOGIE
METHODOLOGY
METHODS
MICHAELI, A.
MICHIGAN
MICROBIOLOGY
MICROCLIMATE
MICROCLIMATIC
MICROPOLITAN
MIDDLEMARCH
MIKHAILOV, V.N.
MILITARY ENGINEERING
MINERAL DEPOSITS
MINERAL RESOURCES
MINERALOGIQUES
MIRON, MURRAY S.
MISE
MODEL
MODELS
MODIFICATIONS
MOISTURE
MOLECULAR BIOLOGY
MOLLUSCA
MONGOLIA
MONOLITHS
MONUMENT
MOORMANN, F.R.
MORPHOLOGY
MORPHOLOGY (LINGUISTICS)
MOUNTAINS
MOUTH
MR
MUCHEKE
MUD
MUELLER-DOMBOIS, D.
MULTINATIONAL
MULTIRACIAL SOCIETIES
NAGARAJA, V.N.
NAMIBIA
NAQVI, SIBTE NABI
NATIONAL
NATIONAL DEVELOPMENT
NATIONAL PLANNING
NATIONAL POLICY
NATIONALISM
NATIONS
NATURAL
NEED
NEEDS
NETHERLANDS
NEW
NEWLY-INDEPENDENT STATES
NEWS AGENCIES
NEWSPAPERS
NIGER
NO
NON-VIOLENCE
NORTH AMERICA
NOTES
NOVEL
NOVELS
NUCLEAR
NUCLEAR WEAPONS
NUTRIENT
NUTRIENTS
NUTRITION
OBJECTIVES
OBLIGATOIRE
OCCUPANT
OCCUPATIONAL QUALIFICATIONS
OCCURENCE
OCEAN CURRENTS
OCEANIA
OHYA, MASAHIKO
OIAPOQUE
OKATCHA, F.M.M.O.
ONE
ONTARIO
OPEN
OPENING
OPERATIONS
ORGANES
ORGANIC MATTER
ORGANIZATION
ORGANIZATIONAL
OSGOOD, CHARLES EGERTON
OTHER
OTRAS
OUEST
PACIFIC ISLANDS
PAKISTAN
PAR
PARASITES
PART
PART-TIME COURSES
PATEL, A.M.
PAYS
PECS
PERDU
PERFORMANCE
PERIODICALS
PERRY, WALTER
PERSONAL
PERSONNEL MANAGEMENT
PERSPECTIVES
PERU
PERUVIAN
PESTS
PETTIGREW, ANDREW M.
PHENOMENOLOGY
PHILIPPINES
PHILOSOPHICAL
PHILOSOPHY
PHOTOGRAPHS
PHOTOSYNTHESIS
PHYA
PHYSICAL EDUCATION
PHYSICAL PROPERTIES
PHYSIOLOGY
PIED
PILOT
PILOT PROJECTS
PINE
PIRES, J. MURCA
PLACE
PLAINS
PLAN
PLANKTON
PLANNING
PLANT
PLANT ECOLOGY
PLANT EVAPOTRANSPIRATION
PLANT PHOTOSYNTHESIS
PLANT PHYSIOLOGY
PLANT TRANSPIRATION
PLANTS
POCZTAR, JERRY
POINT
POLICIES
POLICY
POLITICAL BEHAVIOUR
POLITICAL CONFLICT
POLITICAL ECONOMY
POLITICAL PARTICIPATION
POLITICAL PARTIES
POLITICAL SYSTEMS
POLITICAL THOUGHT
POLITICS
POLITIQUES
POLJAKOFF-MAYBER, A.
POLLUTION
POLSTER, H.
PONENCIAS
POPULATION
POPULATION DENSITY
POPULATION DYNAMICS
POROMETER
PORTUGAL
POST-GRADUATE COURSES
POTENTIAL
POUR
POVERTY
POWER INDUSTRY
PRACTICE
PRATIQUE
PRESENTED
PRESERVATION OF CULTURAL PROPE
PRESERVATION OF MONUMENTS
PRESS
PREVAILING
PRIMAIRE
PRIMARY EDUCATION
PRIMARY SCHOOLS
PROBLEM
PROBLEMES
PROBLEMS
PROCEEDINGS
PRODUCT
PRODUCTION
PRODUCTIVITY
PROFESSIONAL ASSOCIATIONS
PROFESSIONAL TRAINING
PROGRAMMED
PROGRAMMED INSTRUCTION
PROGRAMMED MATERIALS
PROGRAMMES
PROGRAMS
PROGRESS
PROJECT
PROJECTIONS
PROJECTS
PROPOSED
PROWSE, G.A.
PSYCHOANALYSIS
PSYCHOLINGUISTICS
PUBLIC
PUBLIC EDUCATION
PUBLIC HEALTH
PUBLIC OPINION
QUALITY
QUALITY OF LIFE
QUANTITATIF
QUANTITATIFS
QUESTIONNAIRES
QURESHI, M. RAHIMULLAH
R
RACE
RACE RELATIONS
RACIAL SEGREGATION
RADIATION MEASUREMENT AND INST
RADIO
RAIN
RAINFALL
RATES
READINGS
RECENT
RECENTES
RECENTS
RECHERCHE
RECOMMENDATIONS
RED
REDUCTION
REFERENCE
REFORM
REFRACTIVE
REGARD
REGIME
REGIME OF WATERS
REGIONAL COOPERATION
REGIONAL PLANNING
REGIONS
REGULATION
RELATED
RELATING
RELATION
RELATIVE
RELEVANCE
RENE
RENEWABLE
RENEWABLE RESOURCES
RENEWAL
REPORT
REPUBLIC
RESEARCH
RESEARCH AND DEVELOPMENT
RESEARCH CENTRES
RESEARCH PRIORITIES
RESEARCH WORKERS
RESISTANCE
RESOLUTION
RESOLUTIONS
RESOURCE CONSERVATION
RESOURCES
RESPONSE
RESTORATION
REVIEW
RHODESIA
RHODESIAN
RICE
RIEPE, DALE MAURICE
RIGHT TO INFORMATION
RIGHTS
RIVER
RIVER BASINS
RIVER CONTROL
RIVERS
ROLE
ROLE OF EDUCATION
ROONWAL, M.L.
ROSENAU, FRED S.
RUN-OFF
RURAL
RURAL AREAS
RURAL DEVELOPMENT
RURAL POPULATION
RURAL YOUTH
S
SA, ARTUR MOREIRA DE
SAFETY
SAINT ANTONIN, CLAUDE
SALINE
SALINE WATER
SALINITY
SAMMAK, A.
SANCTIONS
SANTEMA, P.
SANYAL, BIKAS C.
SATELLITE
SATELLITE BROADCASTING
SATURATION
SCALE
SCHMID, MAURICE
SCHOELLER, MARC
SCHOLARSHIPS
SCHOOL LABORATORIES
SCHOOL MAPPING
SCHOOL-AGE POPULATION
SCHOOLS
SCIENCE
SCIENCE AND DEVELOPMENT
SCIENCE AND TECHNOLOGY
SCIENCE EDUCATION
SCIENCE FINANCING
SCIENCE POLICY
SCIENCE STATISTICS
SCIENTIFIC
SCIENTIFIC ACTIVITIES
SCIENTIFIC DEVELOPMENT
SCIENTIFIC INSTITUTIONS
SCIENTIFIC PERSONNEL
SCIENTIFIC PROGRAMMES
SCIENTIFIC PROGRESS
SCIENTIFIC RESEARCH
SCOLAIRE
SCOLAIRES
SCOLARISATION
SCULPTURE
SEA
SEA LEVEL
SEA WATER
SECOND
SECONDARY EDUCATION
SECONDARY SCHOOL CURRICULUM
SECONDARY SCHOOLS
SEDIMENT
SEDIMENT TRANSPORT
SEDIMENTATION
SEDIMENTS
SELECTED
SELF
SELF-TEACHING METHODS
SEMANTICS
SEMI
SENEGAL
SEPTEMBER
SERVICES
SEX
SHACHORI, A.Y.
SHIFTING
SHIPWORMS
SHUI
SIGNIFICANCE
SIGNIFICANT
SIMMONS, MELVIN K.
SINGH, B.N.
SIOLI, HAROLD
SKILLED WORKERS
SLATYER, R.O.
SLAVIK, B.
SOCIAL
SOCIAL CHANGE
SOCIAL ENVIRONMENT
SOCIAL FACTORS
SOCIAL INTEGRATION
SOCIAL LIFE
SOCIAL PARTICIPATION
SOCIAL PLANNING
SOCIAL REFORM
SOCIAL SCIENCE RESEARCH
SOCIAL SECURITY
SOCIAL STATUS
SOCIAL STRATIFICATION
SOCIAL SYSTEMS
SOCIALES
SOCIALISM
SOCIO-ECONOMIC FACTORS
SOCIOLOGY
SOCIOLOGY OF COMMUNICATION
SOCIOLOGY OF SCIENCE
SOEKMONO, R.
SOIL
SOIL EROSION
SOIL FERTILITY
SOIL GENESIS
SOIL MECHANICS
SOIL MOISTURE
SOIL SCIENCE
SOIL SURVEYS
SOIL TYPES
SOILS
SOLAR ENERGY
SOLAR RADIATION
SOLS
SOME
SOUTERRAINES
SOUTH EAST ASIA
SOUTHERN AFRICA
SPAIN
SPECIAL
SPORT
SRI LANKA
STAKE, ROBERT E.
STANDARD
STANDARDIZATION
STANHILL, G.
STATEMENT
STATEMENTS
STATES
STATISTICAL ANALYSIS
STATISTICAL DATA
STATISTICAL METHODOLOGY
STATISTICS
STATUS
STELLING-MICHAUD, SVEN
STOMATAL
STONE, E.C.
STORM
STORMS
STRATEGIES
STRATEGY
STRESS
STRUCTURAL
STUDENT ATTITUDE
STUDENT BEHAVIOUR
STUDENT SELECTION
STUDENT SOCIOLOGY
STUDIES
STUDY
SUCTION
SUD
SULPHATE
SUNDERBANS
SUR
SURFACE
SURGES
SURVEY ANALYSIS
SURVEYS
SURVIVAL
SWITZERLAND
SYMPOSIUM
SYSTEMS ANALYSIS
TAIWAN
TEACHER BEHAVIOUR
TEACHER EDUCATORS
TEACHER GUIDES
TEACHER ROLE
TEACHER TRAINING
TEACHING METHODS
TEACHING SKILLS
TECHNICAL
TECHNICAL AND VOCATIONAL EDUCA
TECHNICAL ASSISTANCE
TECHNICAL PERSONNEL
TECHNICAL TRAINING
TECHNIQUE
TECHNIQUES
TECHNOLOGIES
TECHNOLOGY
TECHNOLOGY ASSESSMENT
TECHNOLOGY TRANSFER
TELEVISION
TELEVISION PROGRAMMES
TEMPERATE
TEMPERATURE
TEMPLES
TEMPS
TENDANCES
TENTATIVE
TEREDINID
TERMS
TERRITORY
TEXTURALES
THAILAND
THEBAUD, SCHILLER
THEIR
THEORY
THERMAL PROPERTIES OF SUBSTANC
THREE
TIDAL
TIDES
TIME
TISON, LEON J.
TOOL
TOPICS
TOPOGRAPHY
TOPOLOGY
TOWNS
TRACTS
TRADITIONAL
TRAINING
TRAINING ABROAD
TRAINING CENTRES
TRAINING COURSES
TRANSNATIONAL CORPORATIONS
TRANSPIRANTS
TRANSPIRATION
TRANSPORT
TRICART, J.
TROPICAL
TROPICAL ZONES
TROPICAUX
TROPICS
TURGIDITY
TURKEY
TWEETEN, LUTHER G.
TYPE
TYPES
TYPOLOGY
U
UNDER
UNDERDEVELOPMENT
UNESCO
UNITED
UNIVERSALS
UNIVERSIDAD
UNIVERSITES
UNIVERSITIES
UNIVERSITY
UNIVERSITY COOPERATION
UNIVERSITY COURSES
UNIVERSITY CURRICULUM
UNIVERSITY EXTENSION
URBAN AREAS
URBAN DEVELOPMENT
URBAN LIFE
URBAN PLANNING
URBAN POPULATION
URBAN RENEWAL
URUGUAY
USA
USE
USING
USSR
V
VALEUR
VAPOUR
VARIABILITY
VASAK, KAREL
VEGETATION
VEGETAUX
VELOCITY
VERSLUYS, J.D.N.
VERSTAPPEN, H.TH.
VICE
VIET NAM SR
VILLAGE
VILLAGES
VIOLENCE
VISSER, W.C.
VITALITY
VOLKER, ADRIAAN
VULNERABLE
WANIEWICZ, IGNACY
WAR
WATER
WATER BALANCE
WATER YIELD
WAVES
WAY
WEINRICH, A.K.H.
WENT, F.W.
WEST
WESTERN
WESTERN EUROPE
WETLANDS
WIDE
WILTING
WIND
WIND POWER
WOMEN
WOOD
WOODWELL, GEORGE M.
WORKERS EDUCATION
WORKING POPULATION
WORKS
WORLD
WORLDWIDE
WYNTER, HECTOR
Y
YIELDS
YOSUFZAI, H.K.
YOUTH
YOUTH ORGANIZATIONS
YUGOSLAVIA
ZACKLIN, RALPH
ZAMBIA
ZIMBABWE
ZONE
got row 130
<12>JOINT FAO/WHO EXPERT COMMITTEE ON NUTRITION^N9TH^PROME^D1974
<24>FOOD AND NUTRITION STRATEGIES IN NATIONAL DEVELOPMENT
<26>^AROME^BFAO ^C1976
<26a>ROME
<26b>FAO
<26c>1976
<30>^A64 P.
<30a>64 P.
<50>(FAO NUTRITION MEETINGS REPORT SERIES, NO.56). WHO TECHNICAL REPORT SERIES, NO.584
<69> -
<71>FAO
<71>WHO
<76>STRATÉGIES DE L'ALIMENTATION ET DE LA NUTRITION DANS LE DÉVELOPPEMENT NATIONAL^ZFRE
got row 130
<12>Joint FAO/WHO Expert Committee on Nutrition. 9th. Rome, 1974
<24>Food and nutrition strategies in national development
<26>Rome, FAO , 1976
<30>64 p.
<50>(FAO nutrition meetings report series, no.56). WHO technical report series, no.584
<69>agricultural development; food production; nutrition; national development - economic and social development; food resources; research priorities; developing countries
<71>FAO
<71>WHO
<76>Stratégies de l'alimentation et de la nutrition dans le développement national. fre
MFN : 0130
CONFERENCE : Joint FAO/WHO Expert Committee on Nutrition. 9th. Rome, 1974.
TITLE : Food and nutrition strategies in national development.
IMPRINT : Rome, FAO , 1976.
COLLATION : 64 p.
NOTES : (FAO nutrition meetings report series, no.56). WHO technical report series, no.584.
CORP. BODY : FAO.
WHO.
OTHER LANG : Stratégies de l'alimentation et de la nutrition dans le développement national.
KEYWORDS : agricultural development; food production; nutrition; national development - economic and social development; food resources; research priorities; developing countries.
lbt.c 0100644 0001130 0001130 00000173142 07637615604 010506 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lbt.c,v 1.28 2003/03/24 14:51:48 kripke Exp $
the btree.
*/
#include
#include
/* special
#if defined( __GNUC__ ) && defined ( alloca )
#include
#endif
*/
#if defined( sparc ) || defined( __ppc__ )
# define LDB_BIG_ENDIAN
#endif
#include "lio.h"
#include "luti.h"
#include "lbt.h"
/*
probably all relevant cases can be handled by the more
efficient approach of storing recoded key a la strxfrm
*/
#define LBT_USECMP 0
/* ************************************************************
private types
*/
enum {
/* little endian magic text */
LBT_OPEN = 'O' | 'P'<<8 | 'E'<<16 | 'N'<<24,
LBT_ISIS = 'I' | 'S'<<8 | 'I'<<16 | 'S'<<24
};
enum {
LBT_LEVELS = 16, /* max depth */
LBT_POSBITS = 13, /* bits for position within block */
LBT_POSMASK = 0x1fff,
LBT_POSMAX = LBT_POSMASK /* obviously :) */
};
typedef struct Dict { /* dictionary of block entries */
unsigned short pos; /* 13 bits position of entry, 2 bits val for leaves */
unsigned char vln; /* length of entry's value */
/* for inner nodes, this is the length of qualifying
second key segment (either 0 or bt->vsz <= 23).
for leaves, this, together with higher bits of pos,
is the number of values of bt->vsz each.
*/
unsigned char kln; /* length of entry's key */
/*
the following access macros are needed for leaves only
*/
#define LBT_POS( d ) (LBT_POSMASK & (d)->pos)
#define LBT_VLN( d ) ((d)->vln | (~LBT_POSMASK & (d)->pos)>>(LBT_POSBITS-8))
#define LBT_POSADD( d, a ) do { \
(d)->pos = (~LBT_POSMASK & (d)->pos) | (LBT_POSMASK & ((d)->pos + (a))); \
} while (0)
#define LBT_VLNADD( d, a ) do { \
unsigned _vln = LBT_VLN( d ) + (a); \
(d)->pos = (LBT_POSMASK & (d)->pos) \
| (~LBT_POSMASK & ((_vln)<<(LBT_POSBITS-8))); \
(d)->vln = (unsigned char)(_vln); \
} while (0)
#define LBT_PV( d, p, v ) do { \
(d)->pos = (LBT_POSMASK & (p)) | (~LBT_POSMASK & ((v)<<(LBT_POSBITS-8))); \
(d)->vln = (unsigned char)(v); \
} while (0)
#define LBT_PVK( d, p, v, k ) do { \
LBT_PV( d, p, v ); \
(d)->kln = (unsigned char)(k); \
} while (0)
} Dict;
/* guard space around block */
#ifndef NDEBUG
# define LBT_GUARD 64
/* template for guard space -- blank or pattern */
static unsigned char guard[LBT_GUARD];
#else
# define LBT_GUARD 0
#endif
/*
TODO: check the possible performance gain in findkey
from separating disk block and administrative data
and aligning disk blocks on PAGE boundaries in memory
*/
typedef struct Block {
#if LBT_GUARD
unsigned char grd[LBT_GUARD];
#endif
unsigned char ird; /* is reading */
unsigned char wrd; /* #threads waiting for read */
unsigned char lck; /* is write locked */
unsigned char wlk; /* #threads waiting for write lock */
#define LBT_BUSY( b ) (*(int*)&(b)->ird) /* any of 1st 4 bytes != 0 */
struct Block *nhs; /* next in hash chain */
struct Block *nru; /* next (more) recently used */
/* TODO: possibly double-link lru list, so we can unlink on every access ? */
/* the disk block -- from here on bsz bytes are on disk */
#define LBT_BOFF ((char *)&((Block*)0)->num - (char*)0)
#define LBT_BASE( b ) ((unsigned char *)&(b)->num)
unsigned num; /* number of this block, 0 is root */
unsigned nxt; /* right sibling, "OPEN" or "ISIS" for root */
/* typ,key,col are redundant in non-root blocks, maybe abused ... */
unsigned char typ; /* type: bsz, vsz, flags */
unsigned char key; /* max key length */
unsigned char col; /* collation */
unsigned char lev; /* level over bottom, 0 for leaves */
unsigned short ent; /* 10 bits # entries */
#define LBT_ENT( b ) (b)->ent /* currently other bits arent abused */
unsigned short stk; /* 13 bits offset of keyspace stack */
/* offset of dict RELATIVE to base */
#define LBT_DOFF ((char *)&((Block*)0)->dct - (char*)&((Block*)0)->num)
Dict dct[1];
} Block;
/*
LAYOUT:
dct is an array of dictionary entries in increasing key order
stk is growing downwards from block end towards dict
position of entry i is b->stk <= LBT_POS(b->dct+i) < bt->bsz
decreasing on i (i.e. stack space is in reverse ordering)
at LBT_POS(b->dct+i), there are b->dct[i].kln bytes key
for a leaf entry, the key is followed by
LBT_VLN(b->dct+i) (between 0..1023 incl.) values
of bt->vsz (between 8..23 incl) each
for an inner entry, the key is followed by
4 bytes block number and optionally bt->vsz qualifying value
SIZES:
block size >= 1024
block base overhead LBT_DOFF is 16 bytes
avail size >= 1008
Dict entry is 4 bytes
max. used stack space per entry is:
inner node: kln+4+bt->vsz <= 255+4+23 = 282
new leaf entry (one value): kln+bt->vsz <= 278
max total space for a new entry incl. dict is 286 < avail size/3
*/
typedef struct Chunk {
struct Chunk *nxt;
} Chunk;
enum {
/* number of levels for lru list */
LBT_LRULEV = sizeof(((Idx*)0)->lru)/sizeof(((Idx*)0)->lru[0]),
LBT_ENTBITS = 10, /* bits for # enries */
LBT_ENTMASK = 0x03ff,
LBT_ENTMAX = LBT_ENTMASK,
LBT_MATCH = 0x8000, /* entry matched */
LBT_ENTSIZE = 8 /* minimum cost of entry: 4 byte dict, 4 byte val */
};
typedef struct Batch {
Block *cur;
unsigned got; /* current for this key */
unsigned tot; /* total for this key */
unsigned let;
/* stats */
unsigned keys;
unsigned keyl;
unsigned maxl;
unsigned vals;
unsigned maxv;
unsigned span;
unsigned free;
/* buffer */
Key key;
Val val[LBT_ENTMAX];
} Batch;
typedef enum {
LBT_RD, /* read -- no lock */
LBT_WR, /* write */
LBT_EX /* exclusive */
} lbt_get;
typedef struct { /* structure filled by findkey */
Idx *bt; /* the tree */
Key *key; /* the key searched */
Block *b[LBT_LEVELS]; /* path to leaf, b[0] is leaf */
unsigned n[LBT_LEVELS]; /* block numbers of path */
unsigned short e[LBT_LEVELS]; /* entries in path */
} FindKey;
typedef struct {
unsigned get;
unsigned miss;
unsigned cmp;
unsigned key;
} Stat;
/* buffer to print a value to.
lowest 11 bytes are printed as 4 shorts + one int < 4*5+10 = 30 digits+5sep
higher 12 bytes are printed hex = 24 bytes.
*/
typedef char PrVal[64];
/* ************************************************************
private data
*/
/*
stat access is not guarded by memory barrier.
may be wrong when running on multiple processors
*/
static Stat stat;
/* ************************************************************
private functions
*/
#if defined( sparc )
static unsigned GETINT ( void *m ) /* bus error safe version of *(int*) */
{
unsigned l;
memcpy( &l, m, 4 );
return l;
}
static void SETINT ( void *m, unsigned i )
{
memcpy( m, &i, 4 );
}
#else
#define GETINT( m ) (*(unsigned*)(m))
#define SETINT( m, i ) (*(unsigned*)(m) = (i))
#endif
static void mkval ( Val *v, Hit *ifp )
{
unsigned char *c = v->byt;
switch ( v->len ) {
default: memset( c, 0, v->len-11 ); c += v->len-11;
case 11: *c++ = (unsigned char) (ifp->dbn >> 8);
case 10: *c++ = (unsigned char) ifp->dbn;
case 9: *c++ = (unsigned char) (ifp->mfn >> 24);
case 8: *c++ = (unsigned char) (ifp->mfn >> 16);
case 7: *c++ = (unsigned char) (ifp->mfn >> 8);
case 6: *c++ = (unsigned char) ifp->mfn;
case 5: *c++ = (unsigned char) (ifp->tag >> 8);
case 4: *c++ = (unsigned char) ifp->tag;
case 3: *c++ = (unsigned char) ifp->occ;
case 2: *c++ = (unsigned char) (ifp->pos >> 8);
case 1: *c++ = (unsigned char) ifp->pos;
case 0: ;
}
} /* mkval */
static Hit *rdval ( Hit *ifp, unsigned char *c, unsigned vsz )
{
ifp->mfn = ifp->tag = ifp->occ = ifp->pos = ifp->dbn = 0;
switch ( vsz ) {
default: c += vsz-11;
case 11: ifp->dbn = (unsigned short)*c++ << 8;
case 10: ifp->dbn |= *c++;
case 9: ifp->mfn = (unsigned int)*c++ << 24;
case 8: ifp->mfn |= (unsigned int)*c++ << 16;
case 7: ifp->mfn |= (unsigned int)*c++ << 8;
case 6: ifp->mfn |= *c++;
case 5: ifp->tag = (unsigned short)*c++ << 8;
case 4: ifp->tag |= *c++;
case 3: ifp->occ = *c++;
case 2: ifp->pos = (unsigned short)*c++ << 8;
case 1: ifp->pos |= *c++;
case 0: ;
}
return ifp;
} /* rdval */
static char *prval ( PrVal buf, unsigned char *c, unsigned vsz )
{
char *d = buf;
Hit h;
if ( ! vsz ) {
buf[0] = '#';
buf[1] = 0;
return buf;
}
if ( 23 < vsz )
vsz = 23;
if ( 11 < vsz ) {
while ( 11 < vsz-- ) {
*d++ = luti_hex[ *c >> 4 ];
*d++ = luti_hex[ *c++ & 0xf ];
}
*d++ = '.';
}
rdval( &h, c, vsz );
d += u2a( d, h.dbn ); *d++ = '.';
d += u2a( d, h.mfn ); *d++ = '.';
d += u2a( d, h.tag ); *d++ = '.';
d += u2a( d, h.occ ); *d++ = '.';
d += u2a( d, h.pos ); *d++ = '.';
*d = 0;
return buf;
} /* prval */
/* for PRVAL f->key->val */
#define PRVAL( buf, v ) prval( buf, v.byt, v.len )
/* ************************************************************
block I/O & cache mgmt
*/
static void iniblk ( Block *b, Idx *bt, unsigned num, unsigned char lev )
{
memset( b, 0, LBT_BOFF + bt->bsz + LBT_GUARD );
b->num = num;
b->typ = bt->typ;
b->key = bt->key;
b->col = bt->col;
b->lev = lev;
b->stk = bt->bsz;
} /* iniblk */
static int chkblk ( Block *b, Idx *bt )
{
#if 0
(void)b; (void)bt;
#else
int i;
unsigned short eod = LBT_DOFF + b->ent*sizeof(Dict), siz, pos, vln, key;
unsigned end = bt->bsz, tot = 0;
unsigned char *base = LBT_BASE(b);
#if LBT_GUARD
if ( memcmp( guard, b->grd, LBT_GUARD ) )
return log_msg( ERR_TRASH, "OOPS! initial guard frobbed on blk %d", b->num );
if ( memcmp( guard, LBT_BASE(b)+bt->bsz, LBT_GUARD ) )
return log_msg( ERR_TRASH, "OOPS! initial guard frobbed on blk %d", b->num );
#endif
if ( b->num > bt->len || (b->num && b->nxt > bt->len) )
return log_msg( ERR_TRASH,
"bad num %d / nxt %d on lev %d", b->num, b->nxt, b->lev );
if ( b->typ != bt->typ || b->key != bt->key || b->col != bt->col )
return log_msg( ERR_TRASH,
"bad sig 0x%02x,%d,%d in block %d want 0x%02x,%d,%d",
b->typ, b->key, b->col, b->num, bt->typ, bt->key, bt->col );
if ( b->ent > bt->bsz / LBT_ENTSIZE || b->stk > bt->bsz || b->stk < eod )
return log_msg( ERR_TRASH,
"bad ent/stk %d,%d in block %d eod %d sz %d",
b->ent, b->stk, b->num, eod, bt->bsz );
for ( i=0; ient; i++ ) {
Dict *d = &b->dct[i];
pos = LBT_POS( d );
vln = LBT_VLN( d );
key = d->kln;
siz = key + (b->lev ? vln+4u : vln * bt->vsz);
if ( pos < b->stk || end != (unsigned)pos + siz )
return log_msg( ERR_TRASH,
"bad pos/vln/key %d/%d/%d in entry %d of block %d"
" stack %d sz %d end %d expected %d",
pos, vln, key, i, b->num, b->stk, siz, pos+siz, end );
if ( b->lev ) {
unsigned lnk;
if ( vln && vln != +bt->vsz )
return log_msg( ERR_TRASH,
"bad vln %d in entry %d of block %d vsz %d",
vln, i, b->num, bt->vsz );
lnk = GETINT( base + pos + key + vln );
if ( lnk >= bt->len )
return log_msg( ERR_TRASH,
"bad link %u in entry %d of block %d",
lnk, i, b->num );
}
#if 0 && ! defined( NDEBUG )
if ( i ) {
unsigned short op = LBT_POS( d-1 ), okl = d[-1].kln;
int mc = memcmp( base+pos, base+op, keylev && vln ) { /* compare values */
unsigned ovln = d[-1].vln;
if ( ! ovln || 0 < (j = memcmp( base+pos+key, base+op+okl, vln )) )
goto ok;
}
return log_msg( ERR_TRASH,
"bad key order entry %d of block %d lev %d"
" mc %d j %d key %d okl %d vln %d",
i, b->num, b->lev, mc, j, key, okl, vln
);
}
ok:
#endif
tot += siz;
end = pos;
}
if ( tot > bt->bsz - b->stk )
return log_msg( ERR_TRASH,
"bad total entry size %d in block %d stack %d",
tot, b->num, b->stk );
#ifndef NDEBUG
if ( LOG_DO( LOG_TRACE ) ) {
log_msg( LOG_TRACE,
"block %d lev %d sig 0x%02x,%d,%d ent %d eod %d stk %d",
b->num, b->lev, b->typ, b->key, b->col, b->ent, eod, b->stk
);
for ( i=0; i<2; i++ ) {
int j = i ? b->ent-1 : 0;
Dict *d = &b->dct[j];
if ( i >= b->ent ) /* if b->ent is 0 or 1, log no or one key */
break;
pos = LBT_POS( d );
vln = LBT_VLN( d );
key = d->kln;
end = pos + b->lev ? vln+4u : vln * bt->vsz;
/* start of value */
if ( b->lev )
log_msg( LOG_TRACE, "key %d '%.*s'/%d -> %d",
j, key, base+pos, vln, GETINT(base+pos+key+vln) );
else {
Hit h;
rdval( &h, base+pos+key, bt->vsz );
log_msg( LOG_TRACE, "key %d '%.*s'[%d] = %d[%d,%d,%d]",
j, key, base+pos, vln, h.mfn, h.tag, h.occ, h.pos );
}
}
}
#endif
#endif
return 0;
} /* chkblk */
static Block *addchunk ( Idx *bt, unsigned cnt )
{
unsigned ibsz = LBT_BOFF + bt->bsz + LBT_GUARD; /* internal block size */
/* increase cnt so it's > 0 and add one extra to return */
char *mem = (char *)lmalloc( sizeof(Chunk) + (1 + ++cnt)*ibsz );
Chunk *chk = (Chunk *)mem;
if ( ! mem )
return 0;
/* chain the chunk */
chk->nxt = bt->mem;
bt->mem = chk;
mem += sizeof(Chunk);
bt->clen += cnt;
/* add cnt blocks to level 0 lru list */
if ( ! bt->mru[0] ) /* level 0 cache is empty */
bt->mru[0] = (Block *)mem;
for ( ; cnt--; mem += ibsz ) {
Block *b = (Block *)mem;
iniblk( b, bt, 0 , 0 ); /* initialise empty block */
b->nru = bt->lru[0];
bt->lru[0] = b;
}
iniblk( (Block *)mem, bt, 0 , 0 );
return (Block *)mem;
} /* addchunk */
static int readblk ( Block *b, Idx *bt, unsigned num )
{
int got = lio_pread( &bt->fd, LBT_BASE(b), bt->bsz, num*bt->bsz );
log_msg( LOG_VERBOSE, "reading block %u got %u", num, got );
if ( bt->bsz != (unsigned)got )
return log_msg( LOG_SYSERR, "\twhen reading bt block %d", num );
if ( b->num != num )
return log_msg( ERR_TRASH, "bad num %d in block %d", b->num, num );
return chkblk( b, bt );
} /* readblk */
static void unlock ( Block *b )
{
b->lck = 0;
if ( b->wlk )
(void)LIO_WAKE( b->num );
}
#define ULK( b ) do { \
if ( (b)->lck ) { (b)->lck = 0; if ( b->wlk ) unlock( b ); } \
} while (0)
static int writeblk ( Block *b, Idx *bt, int ulk )
{
int got;
if ( chkblk( b, bt ) ) /* don't write no shize */
return log_msg( ERR_TRASH, "\tbefore writing bt block %d", b->num );
got = lio_pwrite( &bt->fd, LBT_BASE(b), bt->bsz, b->num*bt->bsz );
log_msg( LOG_VERBOSE, "writing block %u got %u", b->num, got );
if ( ulk )
ULK( b );
if ( bt->bsz != (unsigned)got )
return log_msg( LOG_SYSERR, "\twhen writing bt block %d", b->num );
return 0;
} /* writeblk */
#define WRULK( b, bt ) writeblk( b, bt, 1 )
#define NEWBLK( bt, lev, lock ) getblk( bt, bt->len, lev, lock )
/**
get block #num.
1) locate block #num in cache.
2) if it is found:
if it is READING, or we want to lock and it is locked,
incr # waiters, wait for it.
3) if it is not found:
locate a free cache entry, extending the cache if necessary,
set the num, set the READING flag, go read it.
after reading, if there are waiters, wake them up.
since we are the first to get a hold on the block,
we may write lock it, if we want.
4) return the block.
*/
static Block *getblk ( Idx *bt, unsigned num,
unsigned char lev, lbt_get lock )
{
Block *b;
stat.get++; /* dirty */
if ( !(stat.get & 0x3fff) ) { /* every 16K */
static unsigned lmiss;
log_msg( LOG_INFO,
"searched %i keys in %u blks cache %d %d%%"
" depth %u cmp/get/miss %u/%u/%u %d%% (%d %d%%)",
stat.key, bt->len, bt->clen, bt->clen*100 / bt->len,
bt->root->lev, stat.cmp, stat.get,
stat.miss, stat.miss*100 / stat.get,
(stat.miss - lmiss), (stat.miss - lmiss)*100 / 0x4000
);
lmiss = stat.miss;
}
startover: /* only used on absurd congestion */
b = 0;
LOG_DBG( LOG_DEBUG, "get bt block %u %u %d", num, lev, lock );
if ( ! num )
b = bt->root;
else if ( num < bt->len ) {
b = bt->hash[ num % bt->hlen ];
while ( b && num != b->num )
b = b->nhs;
}
if ( b )
while ( b->ird || (lock && b->lck) ) {
int wrd = b->ird;
int wlk = lock && b->lck;
LOG_DBG( LOG_VERBOSE, "lck on block %u %u r%d l%d",
num, lev, b->ird, b->lck );
if ( LBT_EX <= lock ) {
log_msg( LOG_ERROR, "concurrent mod on EX lock" );
return 0;
}
if ( ! lio_lock ) {
log_msg( LOG_FATAL, "bad MT environment: concurrent mod, no lock" );
return 0;
}
if ( wrd && 0xff != b->wrd )
b->wrd++;
if ( wlk && 0xff != b->wlk )
b->wlk++;
(void)LIO_WAIT( num ); /* zzzzz .... sleep tight */
/* some hours later */
if ( num != b->num ) {
log_msg( LOG_WARN, "heavy congestion: block %d has gone", num );
goto startover;
}
if ( wrd && b->wrd )
b->wrd--;
if ( wlk && b->wlk )
b->wlk--;
}
else { /* not found or new block */
/* find free block */
int i = 0, llev, err = 0;
stat.miss++; /* dirty */
LOG_DBG( LOG_DEBUG, "bt block %u %u not found", num, lev );
for ( ; i < (int)LBT_LRULEV; i++ ) {
/* check lru list for level i */
Block **bp = &bt->lru[i];
while ( (b = *bp) && LBT_BUSY(b) )
bp = &b->nru;
if ( ! b ) /* try next level */
continue;
*bp = b->nru; /* dequeue */
if ( bt->mru[i] == b ) /* was the tail */
bt->mru[i] = 0;
break;
}
if ( ! b ) {
log_msg( LOG_WARN, "no free blocks in cache" );
if ( !(b = addchunk( bt, 64 )) )
return 0;
/* TODO: should we also extend the hash ? */
} else if ( b->num ) { /* got block -- should be hashed somewhere */
Block **bp = &bt->hash[ b->num % bt->hlen ];
while ( *bp && *bp != b )
bp = &(*bp)->nhs;
if ( *bp == b )
*bp = b->nhs; /* dehash */
/* else ... hmmm should have found it */
}
assert( ! LBT_BUSY( b ) );
iniblk( b, bt, num, lev );
/* hash it, so waiters can find it */
b->nhs = bt->hash[ b->num % bt->hlen ];
bt->hash[ b->num % bt->hlen ] = b;
if ( num == bt->len ) { /* extend */
b->lev = lev;
bt->len++;
/* TODO: should be configurable */
if ( bt->len > bt->clen << 5 && bt->clen < 0x4000 )
addchunk( bt, bt->clen / 4 );
} else {
b->ird = 1;
(void)LIO_RELE();
err = readblk( b, bt, num );
(void)LIO_LOCK();
b->ird = 0;
if ( ! err && lev != b->lev )
err = log_msg( ERR_TRASH, "bad lev %d in block %d want %d",
b->lev, b->num, lev );
if ( err ) {
Block **bp = &bt->hash[ num % bt->hlen ];
while ( *bp && *bp != b )
bp = &(*bp)->nhs;
if ( *bp == b )
*bp = b->nhs; /* dehash */
b->lev = 0; /* add to level 0 lru list */
b->num = 0; /* tell others that it isn't the expected block */
}
}
/* add to it's levels lru list */
llev = LBT_LRULEV > b->lev ? b->lev : LBT_LRULEV-1;
if ( bt->mru[ llev ] )
bt->mru[ llev ]->nru = b;
else
bt->lru[ llev ] = b;
(bt->mru[ llev ] = b)->nru = 0;
if ( b->wrd )
(void)LIO_WAKE( num );
if ( err ) {
b->wrd = b->wlk = 0; /* mark block unused */
return 0;
}
} /* got free block */
if ( LBT_WR == lock ) {
if ( b->lck ) {
log_msg( LOG_FATAL, "bt consi: unexpected lock on %u", num );
return 0;
}
b->lck = 1;
}
return b;
} /* getblk */
/** binsearch position for key.
position is either the index of the matching entry (0..b->ent-1)
or the index (0..b->ent) where key should be inserted.
@return position for key (0..b->ent), |ifyed with LBT_MATCH, if found
*/
static unsigned short keypos (
#if LBT_USECMP
Idx *bt,
#define KEYPOS( bt, b, key ) keypos( bt, b, key )
#else
#define KEYPOS( bt, b, key ) keypos( b, key )
#endif
Block *b, Key *key )
{
unsigned char *base = LBT_BASE( b );
unsigned char *kbt = key->byt;
unsigned char kln = key->len;
unsigned short l = 0, r = b->ent;
Dict *d = b->dct;
/* binsearch invariant:
key is < all entries i >= r (we do know relation at r)
key is > all entries i < l (don't know relation at l)
*/
while ( l < r ) { /* invariant: l <= keypos < r */
unsigned short i = (l + r) >> 1; /* l <= i < r */
unsigned short pos = LBT_POS(&d[i]);
unsigned char cln = d[i].kln;
int cmp =
#if LBT_USECMP
bt->cmp ? bt->cmp( kbt, base+pos, klnlev /* ignore on leaves -- no duplicate keys here */
|| (!d[i].vln && !key->val.len) /* no value */
)
return LBT_MATCH | i;
/* so we're on an inner node */
if ( ! d[i].vln ) /* common case */
cmp = 1;
else if ( ! key->val.len )
cmp = -1;
else {
unsigned char vln = LBT_VLN(&d[i]);
int c = key->val.len < d[i].vln ? key->val.len : vln;
if ( !(cmp = memcmp( key->val.byt, base+pos+cln, c ))
&& !(cmp = (int) key->val.len - vln)
)
return LBT_MATCH | i;
}
}
if ( cmp > 0 )
l = i+1;
else
r = i;
}
return l; /* == r */
} /* keypos */
/** find position for (leaf) val.
search v in array b of n values of size sz
@return position for value (0..n), |ifyed with LBT_MATCH, if found
*/
static unsigned short valpos ( unsigned char *b,
unsigned short n, unsigned char sz, unsigned char *v )
{
unsigned short l = 0, r = n;
/* binsearch invariant:
v is < all values i >= r (we do know relation at r)
v is > all values i < l (don't know relation at l)
*/
while ( l < r ) { /* invariant: l <= valpos < r */
unsigned short i = (l + r) >> 1; /* l <= i < r */
int cmp = memcmp( v, b+i*sz, sz );
if ( ! cmp )
return LBT_MATCH | i;
if ( cmp > 0 )
l = i+1;
else
r = i;
}
return l; /* == r */
} /* valpos */
#ifndef NDEBUG
static void TRACEKEY ( unsigned lev, FindKey *f )
{
int i;
unsigned n = 0;
Hit h;
PrVal pv;
if ( LOG_DONT( lev ) )
return;
log_msg( lev, "key '%.*s' vln %d val %s",
f->key->len, f->key->byt, f->key->val.len,
PRVAL( pv, f->key->val ) );
if ( f->key->val.len ) {
rdval( &h, f->key->val.byt, f->key->val.len );
log_msg( lev, "\tdb %d mfn %d tag %d occ %d pos %d",
h.dbn, h.mfn, h.tag, h.occ, h.pos );
}
for ( i=LBT_LEVELS; i--; ) {
Block *b = f->b[i];
unsigned on = n, e;
Dict *d;
if ( ! b )
continue;
d = b->dct + (e = (LBT_ENTMASK & f->e[i]));
if ( i )
n = GETINT( LBT_BASE(b)+d->pos+d->kln+d->vln );
log_msg( lev, "lev %d n %d e 0x%x = %d %c '%.*s' val %d %s -> %d%s",
i, f->n[i], f->e[i], e, (LBT_MATCH & f->e[i]) ? 'M' : '-',
d->kln, LBT_BASE(b)+d->pos, d->vln,
prval( pv, LBT_BASE(b)+d->pos+d->kln, d->vln ),
n, on == b->num ? "" : " LY!"
);
if ( d->vln ) {
rdval( &h, LBT_BASE(b)+d->pos+d->kln, d->vln );
log_msg( lev, "\tdb %d mfn %d tag %d occ %d pos %d",
h.dbn, h.mfn, h.tag, h.occ, h.pos );
}
if ( e ) {
d--;
log_msg( lev, "\tleft '%.*s' val %d %s",
d->kln, LBT_BASE(b)+d->pos, d->vln,
prval( pv, LBT_BASE(b)+d->pos+d->kln, d->vln )
);
if ( d->vln ) {
rdval( &h, LBT_BASE(b)+d->pos+d->kln, d->vln );
log_msg( lev, "\tdb %d mfn %d tag %d occ %d pos %d",
h.dbn, h.mfn, h.tag, h.occ, h.pos );
}
}
}
} /* TRACEKEY */
#else
#define TRACEKEY( l, f ) (void)(f)
#endif
static int findkey ( FindKey *f, lbt_get lock )
{
int i;
Block *b = f->bt->root;
unsigned char *pkey = 0, pkln = 0, pvln = 0;
if ( b->lev >= LBT_LEVELS )
return log_msg( ERR_TRASH, "tree is too deep: %u", b->lev );
stat.key++;
for ( i=LBT_LEVELS; i--; ) { /* paranoia */
f->b[i] = 0;
f->n[i] = f->e[i] = 0;
}
for ( i=b->lev;; ) {
unsigned short e;
unsigned n;
Dict *d;
#ifndef NDEBUG
chkblk( b, f->bt );
#endif
f->n[i] = (f->b[i] = b)->num;
f->e[i] = e = KEYPOS( f->bt, b, f->key );
LOG_DBG( LOG_DEBUG, "keypos %x", e );
#ifndef NDEBUG
if ( pkey && b->ent ) {
/* compare parent key against our first key
pkey must be equal for inner node and not greater for leave
*/
int cmp; /* this - pkey >= 0 */
unsigned char *ent, ln;
d = b->dct;
ent = LBT_BASE(b)+LBT_POS(d);
ln = pkln < d->kln ? pkln : d->kln;
cmp = ln ? memcmp( ent, pkey, ln ) : 0;
if ( ! cmp )
cmp = (int)d->kln - (int)pkln;
if ( ! cmp ) {
if ( ! d->vln || ! pvln )
cmp = (int)d->vln - (int)pvln;
else
cmp = memcmp( ent+d->kln, pkey+pkln, f->bt->vsz );
}
if ( cmp && (i || 0 > cmp) ) {
Hit h;
PrVal pv;
TRACEKEY( ERR_TRASH, f );
if ( d->vln ) {
rdval( &h, ent+d->kln, d->vln );
log_msg( ERR_TRASH, "\tfirst val db %d mfn %d tag %d occ %d pos %d",
h.dbn, h.mfn, h.tag, h.occ, h.pos );
}
return log_msg( ERR_TRASH, "\tbad first '%.*s' val %d %s",
d->kln, ent, d->vln, prval( pv, ent+d->kln, d->vln ) );
}
}
#endif
/*
here goes the Lehmann/Yao race detection:
If we ran on right boundary, follow right link
*/
while ( !(LBT_MATCH & e) && b->ent == e && b->nxt && b->num ) {
Block *s;
unsigned se;
b->lck = LBT_WR; /* hold b while checking s */
s = getblk( f->bt, b->nxt, i, i ? LBT_RD : lock );
if ( i || ! lock )
ULK( b );
else
b->lck = lock;
se = KEYPOS( f->bt, s, f->key );
if ( !se ) { /* lower than first */
ULK( s );
break;
}
LOG_DBG( LOG_DEBUG, "LYao %d -> %d lev %d for '%.*s' got 0x%x",
b->num, s->num, i, f->key->len, f->key->byt, se );
/* trade b for s */
ULK( b );
f->n[i] = (f->b[i] = b = s)->num;
e = se;
}
f->e[i] = e;
if ( ! i )
return 0;
if ( !(LBT_MATCH & e) ) {
if ( e )
f->e[i] = --e;
else {
TRACEKEY( ERR_TRASH, f );
return log_msg( ERR_IDIOT,
"ran on left bound of block %u ent %u",
b->num, b->ent );
}
}
d = b->dct + (LBT_ENTMASK & e);
/* direct access ok in inner node */
pkey = LBT_BASE(b)+d->pos;
pkln = d->kln;
pvln = d->vln;
n = GETINT( pkey+pkln+pvln );
i--;
b = getblk( f->bt, n, i, i ? LBT_RD : lock );
}
return -1; /* go away */
} /* findkey */
static int mktree ( Idx *bt )
{
Batch *ba = bt->bat;
unsigned char lev = 0; /* the level to be indexed */
unsigned left = 1; /* block on child level */
/* stats */
unsigned size = bt->len * bt->bsz;
unsigned sval = ba->vals * bt->vsz;
unsigned sove = bt->len * LBT_DOFF + sizeof(Dict)*(ba->keys+ba->span);
unsigned iblk = 0;
unsigned ikey = 0;
unsigned ikeyl = 0;
unsigned ipfx = 0;
unsigned ispn = 0;
log_msg( LOG_INFO,
"end batch:\n"
"\t#blks %u total len %u free %u (%.1f%%) overhead %u (%.1f%%)\n"
"\t#keys %u total len %u (%.1f%%) avg %.1f max %u\n"
"\t#vals %u total len %u (%.1f%%) avg %.1f max %u spans %u",
bt->len, size, ba->free, ba->free*100./size, sove, sove*100./size,
ba->keys, ba->keyl, ba->keyl*100./size, (float)ba->keyl/ba->keys, ba->maxl,
ba->vals, sval, sval*100./size, (float)ba->vals/ba->keys, ba->maxv, ba->span
);
/* build tree */
for (;;) { /* levels */
unsigned len = 0; /* current is n'th block on level */
Block *b = 0;
unsigned have = 0; /* room in current block b */
unsigned char*base = 0; /* of current block b */
Dict *d = 0; /* of current block b */
unsigned next = left; /* child */
left = bt->len;
/* get child c and create child pointer in block b */
while ( next ) {
unsigned char vln = 0;
unsigned char *key;
unsigned char kln;
unsigned need;
Block *c = getblk( bt, next, lev, LBT_EX );
if ( ! c )
return log_msg( ERR_NOMEM, "no child block" );
next = c->nxt;
if ( ! c->ent ) /* OOPS empty node */
continue;
key = LBT_BASE(c)+LBT_POS(c->dct);
kln = c->dct[0].kln;
if ( ! b ) /* the leftmost child on this level */
kln = 0; /* always 0 key on left */
else if ( lev )
vln = LBT_VLN(c->dct);
else if ( ba->key.len == kln /* compare key to last of prev. */
&& !memcmp( ba->key.byt, key, kln ) /* which was saved in ba->key */
) { /* key spans blocks */
LOG_DBG( LOG_DEBUG, "found span key '%.*s' child %u",
kln, key, c->num );
vln = bt->vsz; /* qualify with value */
} else {
#if 1
unsigned char x = 0;
/* find prefix key */
if ( kln > ba->key.len )
kln = ba->key.len;
while ( x < kln && key[x] == ba->key.byt[x] )
x++;
if ( x < kln ) /* they differ at [x] */
kln = x+1;
else /* equal on the shorter length */
kln++;
if ( kln > c->dct[0].kln )
return log_msg( ERR_TRASH,
"bad key '%.*s' on level %u matching previous of length %u",
c->dct[0].kln, key, lev, ba->key.len );
LOG_DBG( LOG_DEBUG, "prefix on key '%.*s' child %u save %u",
kln, key, c->num, c->dct[0].kln - kln );
ipfx += c->dct[0].kln - kln;
#endif
}
if ( vln )
ispn++;
ikeyl += kln;
need = sizeof(Dict) + kln + vln + 4;
if ( need > have ) {
if ( b ) {
log_msg( LOG_VERBOSE,
"level %u block %u is full %u ent stack %u",
lev, b->num, b->ent, b->stk );
b->nxt = bt->len; /* link */
b->lck = 0;
if ( WRULK( b, bt ) ) goto nowrite;
len++;
}
c->lck = LBT_EX; /* paranoia */
b = NEWBLK( bt, lev+1, LBT_EX );
c->lck = 0;
if ( ! b )
return log_msg( ERR_NOMEM, "no free block" );
if ( b->ent || b->stk != bt->bsz )
return log_msg( ERR_TRASH,
"new block %u is not empty: ent %u stk %u",
b->num, b->ent, b->stk );
base = LBT_BASE( b );
b->lck = LBT_EX; /* lock */
have = bt->bsz - LBT_DOFF; /* - ba->let; TODO: should we ??? */
d = b->dct;
len++;
iblk++;
}
SETINT( base+(b->stk -= 4), c->num );
d->kln = kln;
d->vln = vln;
if ( kln += vln ) /* key + consecutive qualifying value */
memcpy( base+(b->stk -= kln), key, kln );
d->pos = b->stk;
d++;
b->ent++;
ikey++;
have -= need;
/* save the last dance for me */
memcpy( ba->key.byt, LBT_BASE(c)+LBT_POS(&c->dct[c->ent-1]),
ba->key.len = c->dct[c->ent-1].kln );
}
if ( !len )
return log_msg( ERR_TRASH, "level %u was empty", lev );
/* close level */
log_msg( LOG_INFO, "closing level %u with %u blocks %u..%u",
lev, len, left, b->num );
b->lck = 0;
if ( 1 == len ) {
memcpy( bt->root, b, LBT_BOFF + bt->bsz );
b = bt->root;
b->num = 0;
}
if ( WRULK( b, bt ) ) goto nowrite;
lev++;
if ( 2 > len )
break;
}
log_msg( LOG_INFO,
"\tused %u inner blocks %u keys %u bytes avg %.1f\n"
"\t%u #spans (%.1f%%) %u pfx (- %.1f%%)",
iblk, ikey, ikeyl, (float)ikeyl/ikey,
ispn, ispn*100.*bt->vsz/ikeyl, ipfx, ipfx*100./(ipfx+ikeyl) );
free( bt->bat );
bt->bat = 0;
return 0;
nowrite:
return log_msg( LOG_SYSERR, "\twhen writing batch block" );
} /* mktree */
/* ************************************************************
package data
*/
/* ************************************************************
package functions
*/
int lbt_batch ( Idx *bt, unsigned char pctfree )
{
bt->bat = (Batch*)lmalloc( sizeof(Batch) );
if ( ! bt->bat )
return log_msg( ERR_NOMEM, "\twhen getting batch" );
memset( bt->bat, 0, sizeof(Batch) );
if ( pctfree > 50 )
pctfree = 50;
bt->bat->let = pctfree * bt->bsz / 100;
lio_trunc( &bt->fd, 0 );
iniblk( bt->root, bt, 0, 0 );
/* TODO: kill hash */
bt->len = 1;
bt->bat->cur = NEWBLK( bt, 0, LBT_EX );
return 0;
} /* lbt_batch */
int lbt_batchval ( Idx *bt, Key *key )
{
int cmp = 0;
Batch *ba = bt->bat;
if ( key != &ba->key ) {
int l = key->len < ba->key.len ? key->len : ba->key.len; /* min */
if ( ! (cmp = memcmp(key->byt, ba->key.byt, l)) )
cmp = (int)key->len - (int)ba->key.len;
if ( 0 > cmp )
return log_msg( LOG_ERROR, "batch key '%.*s' < last '%.*s'",
key->len, key->byt, ba->key.len, ba->key.byt );
}
/* flush ? */
while ( ! key->val.len /* end */
|| LBT_ENTMAX == ba->got /* full */
|| cmp /* next key */
) {
Block *b = ba->cur;
unsigned have = b->stk - LBT_DOFF - b->ent*sizeof(Dict) - ba->let;
unsigned need = sizeof(Dict) + ba->key.len;
unsigned want = need + ba->got * bt->vsz;
unsigned take = ba->got;
if ( have < want ) { /* you can't always get ... */
if ( have <= need )
take = 0;
else {
take = (have - need) / bt->vsz;
if ( 256 > take * bt->vsz )
take = 0;
}
}
#if 0 /* prefix key turbo ??? -- doesn't seem to work all too well */
if ( have < (bt->bsz >> 1) ) { /* ready to waste half a block ? */
Dict *d = &b->dct[ b->ent-1 ];
unsigned char *last = LBT_BASE(b)+LBT_POS(d);
if ( ba->key.byt[0] != last[0]
|| ba->key.byt[1] != last[1]
|| ba->key.byt[2] != last[2]
)
take = 0;
}
#endif
LOG_DBG( LOG_DEBUG, "batch '%.*s' got %u need %u want %u have %u take %u",
ba->key.len, ba->key.byt, ba->got, need, want, have, take );
/* we have enough room in b for take entries */
if ( take ) {
unsigned char *base = LBT_BASE( b );
unsigned char *s = base + b->stk;
int i = take;
Dict *d = &b->dct[ b->ent++ ];
while ( i-- )
memcpy( s -= bt->vsz, ba->val[i].byt, bt->vsz );
memcpy( s -= ba->key.len, ba->key.byt, ba->key.len );
b->stk = (unsigned short) (s - base);
LBT_PV( d, b->stk, take );
d->kln = ba->key.len;
if ( ba->got > take )
memcpy( ba->val, ba->val+take, sizeof(ba->val[0])*(ba->got - take) );
ba->got -= take;
}
if ( ! ba->got )
break;
if ( take ) /* key spans blocks */
ba->span++;
ba->free += b->stk - LBT_DOFF - b->ent*sizeof(Dict); /* wasted */
b->nxt = bt->len; /* link */
LOG_DBG( LOG_DEBUG, "batch: %u entries left, getting block %u",
ba->got, bt->len );
if ( WRULK( b, bt ) )
return log_msg( LOG_SYSERR, "\twhen writing batch block" );
/* so couldn't flush all, thus we need another block */
b = ba->cur = NEWBLK( bt, 0, LBT_EX );
/* will continue on end or keychange */
if ( b->ent || b->stk < bt->bsz )
return log_msg( ERR_TRASH, "new block %u is not empty: ent %u stk %u",
b->num, b->ent, b->stk );
LOG_DBG( LOG_VERBOSE, "new block %u has %u ent %u stk, will keep %u",
b->num, b->ent, b->stk, ba->let );
}
if ( cmp || ! key->val.len ) /* done with key: stats */
if ( ba->maxv < ba->tot )
ba->maxv = ba->tot;
if ( key->val.len ) {
if ( cmp ) { /* set new key */
if ( ba->got )
return log_msg( ERR_IDIOT, "batch not empty on new key" );
ba->tot = 0;
memcpy( ba->key.byt, key->byt, ba->key.len = key->len );
ba->keys++;
ba->keyl += key->len;
if ( ba->maxl < key->len )
ba->maxl = key->len;
}
/* add new val */
/* LOG_HEX( key->val.byt, bt->vsz ); */
memcpy( ba->val[ ba->got++ ].byt, key->val.byt, bt->vsz );
ba->vals++;
ba->tot++;
} else {
WRULK( ba->cur, bt );
mktree( bt );
}
return 0;
} /* lbt_batchval */
int lbt_add ( Idx *bt, Key *key )
{
int ret = 0;
unsigned short e, i = 0;
Block *b, *s = 0; /* mainblock, sibling */
Dict *d;
unsigned char *base, *sbase, *src, *v, *stk;
FindKey f;
int hadkey, split = 0, insibling = 0;
unsigned need;
PrVal pv;
f.bt = bt;
f.key = key;
if ( bt->key && key->len > bt->key )
key->len = bt->key;
if ( findkey( &f, LBT_WR ) ) {
if ( f.b[0] )
ULK( f.b[0] );
b = 0;
LOG_OTO( done, ( ERR_TRASH, "could not find key" ) );
}
e = (LBT_ENTMASK & f.e[0]);
b = f.b[0];
base = LBT_BASE(b);
d = b->dct + e;
if ( ! (hadkey = LBT_MATCH & f.e[0]) ) /* new key */
need = key->len + sizeof(Dict) + bt->vsz;
else {
unsigned vln = LBT_VLN(d);
v = base+LBT_POS(d)+d->kln;
i = valpos( v, vln, bt->vsz, key->val.byt );
if ( LBT_MATCH & i )
goto done; /* value already there */
need = bt->vsz;
}
LOG_DBG( LOG_DEBUG,
"adding key '%.*s'/%s in block %d had %d ent %d stk %d need %d",
key->len, key->byt, PRVAL( pv, key->val ),
b->num, hadkey, b->ent, b->stk, need );
if ( need > b->stk - LBT_DOFF - b->ent*sizeof(Dict) ) {
/* now that's bad: not enough space
create new block s, move last entries there
starting from the end, entries (including the new one) are collected,
until their combined size reaches more than half a block.
If the combined size then is more than 2/3 a block,
and the last entry alone takes more than 1/3 a block, it is split.
TODO: check existing sibling for free space
*/
unsigned half = (bt->bsz - LBT_DOFF) >> 1;
unsigned tot = 0;
unsigned mv = b->ent;
LOG_DBG( LOG_DEBUG, "splitting block %d need %d have %d (stk %d ent %d)",
b->num, need, b->stk - LBT_DOFF - b->ent*sizeof(Dict), b->stk, b->ent );
if ( need > (bt->bsz - LBT_DOFF) / 3 )
LOG_OTO( done, (ERR_IDIOT, "OOPS! excessive need for %d bytes", need) );
/* figure out split position */
if ( !hadkey && b->ent == e ) {
tot = need;
insibling = !0;
}
d = b->dct+mv;
while ( d--, mv-- ) {
unsigned sz = sizeof(Dict)+d->kln+LBT_VLN(d)*bt->vsz;
if ( half > (tot += sz) ) { /* sz fits */
if ( e == mv && !hadkey ) {
/*
check size of new entry.
In case of hadkey, we don't bother for the bt->vsz bytes.
*/
if ( half < (tot + need) ) {
if ( (half - tot) > need/2 ) { /* more balanced if we move */
tot += need;
insibling = !0;
}
break;
}
tot += need;
insibling = !0;
}
if ( mv )
continue; /* else croak on ! mv below */
}
/* exceeded half -- maybe we move entry mv anyway */
if ( tot > (bt->bsz - LBT_DOFF)*2/3 ) {
if ( tot-sz < (bt->bsz - LBT_DOFF)/3 )
split = !0;
else { /* undo -- don't move this entry */
tot -= sz;
d++; mv++;
}
} else if ( ! mv )
LOG_OTO( done, (ERR_IDIOT, "OOPS! didn't find end tot %d bytes", tot) );
break;
} /* while mv-- */
if ( mv < e || (mv == e && hadkey && !split) )
insibling = !0;
/* now mv is the first entry to move */
s = NEWBLK( bt, 0, LBT_WR );
LOG_DBG( LOG_DEBUG,
"split: move %d of %d to new blk %d e %d tot %d insibling %d split %d",
mv, b->ent, s->num, e, tot, insibling, split );
s->nxt = b->nxt;
b->nxt = s->num;
sbase = LBT_BASE(s);
if ( split ) {
/* now total includes full size of entry mv.
if we included a new entry, it might even exceed bsz.
calculate sizes for both blocks w/o mv's values.
*/
unsigned esz = sizeof(Dict)+d->kln; /* entry's base cost */
unsigned vln = LBT_VLN(d);
unsigned keep, diff, mvval, kpval, mvsiz, pos = LBT_POS(d);
/*
from currently used space, substract tot (including all
values and possibly new key "need"), but add entry base cost.
that should be the size remaining if we where moving
all of the values but keep an entry and key.
maybe slightly biased if need is bt->vsz (the hadkey case).
*/
keep = bt->bsz - b->stk + b->ent*sizeof(Dict) + need - tot + esz;
tot -= vln * bt->vsz; /* undo entries */
diff = (keep > tot ? keep - tot : tot - keep) / bt->vsz;
if ( diff > vln )
LOG_OTO( done, (ERR_IDIOT,
"OOPS! got diff %d vln %d entry %d tot %d keep %d",
diff, vln, mv, tot, keep) );
mvval = (vln - diff)/2;
if ( keep > tot )
mvval += diff;
kpval = vln - mvval;
if ( ! mvval || ! kpval ) {
LOG_DBG( LOG_WARN, "bad mv/kpval %d/%d keep %d tot %d vln %d diff %d",
mvval, kpval, keep, tot, vln, diff );
if ( ! mvval )
kpval = vln - (mvval = 1);
else
mvval = vln - (kpval = 1);
}
tot += mvsiz = mvval*bt->vsz;
keep += kpval*bt->vsz;
src = base+pos;
LOG_DBG( LOG_DEBUG,
"split: entry %d '%.*s' mv %d of %d values, i %d keep %d tot %d",
mv, d->kln, src, mvval, vln, i, keep, tot );
if ( keep > (bt->bsz-LBT_DOFF) || tot > (bt->bsz-LBT_DOFF) )
LOG_OTO( done, (ERR_IDIOT,
"OOPS! got diff %d vln %d mvval %d entry %d tot %d keep %d",
diff, vln, mvval, mv, tot, keep) );
/* SIGH -- do it */
memcpy( sbase + (s->stk -= mvsiz),
src+d->kln+kpval*bt->vsz, mvsiz );
if ( e == mv && hadkey && i > kpval ) {
insibling = !0;
s->stk -= bt->vsz;
i -= kpval;
memmove( sbase+s->stk, sbase+s->stk+bt->vsz, i*bt->vsz );
memcpy( sbase+s->stk+i*bt->vsz, key->val.byt, bt->vsz );
mvval++;
}
memcpy( sbase + (s->stk -= d->kln), src, d->kln );
LBT_PVK( s->dct, s->stk, mvval, d->kln );
s->ent = 1;
memmove( src + mvsiz, src, d->kln+kpval*bt->vsz );
pos += mvsiz;
LBT_PV( d, pos, kpval );
d++; mv++;
}
/* now entries mv..b->ent are moved, mv > 0 */
b->stk = LBT_POS(b->dct+mv-1); /* pre-adjust stack */
if ( mv < e ) { /* move entries mv..e-1 */
/* used stack space is from pos of entry e-1 to end of entry mv
(might differ from pos of mv-1 due to split)
*/
unsigned from = LBT_POS(b->dct+e-1), nent = e-mv,
sz = LBT_POS(b->dct+mv)+b->dct[mv].kln
+LBT_VLN(b->dct+mv)*bt->vsz - from;
int adj = (s->stk -= sz) - from; /* entry position adjustment */
memcpy( sbase + s->stk, base+from, sz );
memcpy( s->dct+s->ent, b->dct+mv, nent*sizeof(Dict) );
d = s->dct + s->ent;
s->ent += nent;
for ( ; nent--; d++ )
LBT_POSADD( d, adj );
}
if ( insibling && mv <= e ) { /* now mv or insert e, unless done on split */
d = b->dct + e;
if ( hadkey ) { /* mv existing entry e >= mv, add new val */
/* i is valpos */
unsigned vln = LBT_VLN(d);
src = base+LBT_POS(d)+d->kln;
if ( i < vln ) {
unsigned sz = (vln - i)*bt->vsz;
memcpy( sbase + (s->stk -= sz), src + i*bt->vsz, sz );
}
memcpy( sbase + (s->stk -= bt->vsz), key->val.byt, bt->vsz );
if ( i ) {
unsigned sz = i*bt->vsz;
memcpy( sbase + (s->stk -= sz), src, sz );
}
memcpy( sbase + (s->stk -= d->kln), base+LBT_POS(d), d->kln );
LBT_PVK( s->dct + s->ent, s->stk, vln+1, d->kln );
e++; /* e is moved */
} else {
memcpy( sbase + (s->stk -= bt->vsz), key->val.byt, bt->vsz );
memcpy( sbase + (s->stk -= key->len), key->byt, key->len );
LBT_PVK( s->dct + s->ent, s->stk, 1, key->len );
}
s->ent++;
} /* mv or insert e */
if ( e < b->ent && mv < b->ent ) { /* move entries max(e,mv) to end */
unsigned fst = mv < e ? e : mv;
/* used stack space is from pos of entry b->ent-1 (the original b->stk)
to end of entry fst (might differ from pos of fst-1 due to split)
*/
unsigned from = LBT_POS(b->dct+b->ent-1), nent = b->ent-fst,
sz = LBT_POS(b->dct+fst)+b->dct[fst].kln
+LBT_VLN(b->dct+fst)*bt->vsz - from;
int adj = (s->stk -= sz) - from; /* entry position adjustment */
LOG_DBG( LOG_DEBUG, "split: entries %d..%d sz %d from %d to %d",
fst, b->ent-1, sz, from, s->stk );
memcpy( sbase + s->stk, base+from, sz );
memcpy( s->dct+s->ent, b->dct+fst, nent*sizeof(Dict) );
d = s->dct + s->ent;
s->ent += nent;
for ( ; nent--; d++ ) {
#if !defined(NDEBUG)
unsigned p = LBT_POS(d);
#endif
LBT_POSADD( d, adj );
#if !defined(NDEBUG)
LOG_DBG( LOG_DEBUG, "new ent %d p/v/k %d/%d/%d o %d",
d - s->dct, LBT_POS(d), LBT_VLN(d), d->kln, p );
#endif
}
}
b->ent = mv; /* kill entries */
LOG_DBG( LOG_DEBUG,
"split: blks %d/%d with %d/%d entries, s starts '%.*s'",
b->num, s->num, b->ent, s->ent, s->dct[0].kln, sbase+LBT_POS(s->dct) );
} /* new sibling */
if ( ! insibling ) {
/* now we do have enough space */
d = b->dct + e;
stk = base + b->stk; /* bottom of data stack */
if ( hadkey ) { /* insert value at i */
unsigned char *val = base + LBT_POS(d)+d->kln+i*bt->vsz;
need = bt->vsz;
#if 0
LOG_DBG( LOG_DEBUG,
"in entry %d '%.*s' val %d: memmove( %p %p %d) val %p need %d",
e, d->kln, base+LBT_POS(d), i,
stk-need, stk, (val-stk)+need, val, need );
#endif
memmove( stk - need, stk, (val - stk)+need );
memcpy( val - need, key->val.byt, bt->vsz );
LBT_VLNADD( d, 1 );
} else { /* insert key, value and dict entry */
/* take dict and data space from entry e */
unsigned char *nu;
need = key->len + bt->vsz;
if ( e == b->ent )
nu = base + b->stk - need;
else {
/* at end of next entry, i.e. end of block or beg of previous */
nu = base + LBT_POS(d) + d->kln + LBT_VLN(d)*bt->vsz;
memmove( stk - need, stk, nu - stk );
nu -= need;
memmove( d+1, d, (b->ent - e)*sizeof(Dict) );
}
memcpy( nu, key->byt, key->len );
memcpy( nu + key->len, key->val.byt, bt->vsz );
LBT_PV( d, nu-base, 1 );
d->kln = key->len;
d++;
e++;
b->ent++;
}
/*
need is set to the needed (i.e. additionally used) data space,
e is the first dict entry to redirect need bytes lower
*/
b->stk -= need;
for ( ; e < b->ent; e++, d++ )
LBT_POSADD( d, -need );
} /* ! insibling */
while ( s ) { /* created sibling */
unsigned num = s->num;
unsigned char *e0 = LBT_BASE(s) + LBT_POS(s->dct);
unsigned l = s->lev + 1;
Block *ins;
Key k;
memcpy( k.byt, e0, k.len = s->dct[0].kln );
k.val.len = 0;
if ( s->lev ? s->dct[0].vln : split )
memcpy( k.val.byt, e0+k.len, k.val.len = bt->vsz );
LOG_DBG( LOG_DEBUG, "split: had %d/%d on lev %d key '%.*s' vln %d %s",
b->num, num, s->lev, k.len, k.byt, k.val.len, PRVAL( pv, k.val ) );
/* get parent */
if ( f.b[l]->num == f.n[l] && ! LBT_BUSY(f.b[l]) )
f.b[l]->lck = LBT_WR;
else
f.b[l] = getblk( bt, f.n[l], l, LBT_WR );
LOG_DBG( LOG_DEBUG,
"split: got parent %d lev %d", f.b[l]->num, f.b[l]->lev );
/* MT TODO: again L/Y link hunt */
/* ok, locked the parent -- release s and b */
if ( (ret = WRULK( s, bt )) || (ret = WRULK( b, bt )) )
break;
s = 0;
ins = b = f.b[l];
base = LBT_BASE(b);
e = KEYPOS( bt, b, &k );
LOG_DBG( LOG_DEBUG, "split: got parent pos %d", e );
if ( LBT_MATCH & e ) /* !??? */
LOG_OTO( done, (ERR_TRASH, "OOPS! found key in parent" ) );
if ( ! e ) {
TRACEKEY( ERR_TRASH, &f );
LOG_OTO( done, (ERR_TRASH, "OOPS! parent insert at 0 lev %d", l ) );
}
need = sizeof(Dict) + k.len + 4 + k.val.len;
if ( need > b->stk - LBT_DOFF - b->ent*sizeof(Dict) ) {
unsigned half = ((bt->bsz - b->stk)+b->ent*sizeof(Dict)+need) / 2;
unsigned mv = b->ent, tot = 0, nent, sz;
int adj;
if ( e == b->ent ) {
tot = need;
ins = 0;
}
for ( d = b->dct+mv; d--, mv--; ) {
sz = sizeof(Dict) + d->kln + 4 + d->vln;
if ( half > (tot += sz) ) {
if ( e == mv ) {
if ( half > tot+need || tot+need-half < half-tot ) {
tot += need;
ins = 0; /* new link goes to sibling */
} else
break;
if ( half <= tot )
break;
}
if ( mv )
continue;
}
if ( tot - half > sz/2 ) { /* don't move this */
tot -= sz;
d++; mv++;
} else if ( ! mv ) /* !??? */
mv = 1;
break;
}
/* split here a little bit simpler */
s = NEWBLK( bt, l, LBT_WR );
s->nxt = b->nxt;
b->nxt = s->num;
/* straight move entries mv..b->ent-1 to sibling */
sbase = LBT_BASE(s);
d = b->dct + mv;
sz = b->dct[mv-1].pos - b->stk;
nent = b->ent - mv;
LOG_DBG( LOG_DEBUG,
"split: parent lev %d split %d to new %d mv %d"
" '%.*s'/%s nent %d sz %d",
l, b->num, s->num, mv, d->kln, base+d->pos,
prval( pv, base+d->pos+d->kln, d->vln ), nent, sz );
memcpy( sbase + (s->stk -= sz), base + b->stk, sz );
adj = s->stk - b->stk;
b->stk += sz;
memcpy( s->dct, d, nent*sizeof(Dict) );
b->ent = mv;
s->ent = nent;
for ( d = s->dct; nent--; d++ )
d->pos += adj;
if ( ! ins ) {
ins = s;
e -= mv;
}
if ( (ret = chkblk( s, bt )) )
break;
}
/* now insert link in block ins at pos e */ {
unsigned isz = k.len + 4 + k.val.len, nent = ins->ent - e;
unsigned char *ibase = LBT_BASE(ins);
unsigned end = e ? ins->dct[e-1].pos : bt->bsz;
LOG_DBG( LOG_DEBUG,
"split: ins parent %d lev %d pos %d key '%.*s' -> %d vln %d %s",
ins->num, ins->lev, e, k.len, k.byt, num, k.val.len,
PRVAL( pv, k.val ) );
memmove( ibase + ins->stk - isz, ibase + ins->stk, end - ins->stk );
ins->stk -= isz;
memcpy( ibase + (end -= 4), &num, 4 );
if ( k.val.len )
memcpy( ibase + (end -= k.val.len), k.val.byt, k.val.len );
memcpy( ibase + (end -= k.len), k.byt, k.len );
for ( d = ins->dct+ins->ent; nent--; d-- ) {
*d = d[-1];
d->pos -= isz;
}
d->pos = end;
d->vln = k.val.len;
d->kln = k.len;
ins->ent++;
if ( (ret = chkblk( ins, bt )) )
break;
}
if ( ! s ) /* write b below or in next turn */
break;
if ( ! b->num ) { /* root split */
Block *bb = NEWBLK( bt, l, LBT_WR );
LOG_DBG( LOG_DEBUG, "root split!" );
/* cp all but the num */
memcpy( LBT_BASE(bb)+4, LBT_BASE(b)+4, bt->bsz-4 );
b->nxt = s->nxt; /* restore root's special nxt */
s->nxt = 0;
b->stk = bt->bsz; /* empty the root */
/* add two childs bb and s */
SETINT( base + (b->stk -= 4), bb->num );
LBT_PVK( b->dct, b->stk, 0, 0 );
SETINT( base + (b->stk -= 4), s->num );
e0 = LBT_BASE(s) + LBT_POS(s->dct);
if ( s->dct[0].vln )
memcpy( base + (b->stk -= bt->vsz), e0+s->dct[0].kln, bt->vsz );
memcpy( base + (b->stk -= s->dct[0].kln), e0, s->dct[0].kln );
LBT_PVK( b->dct+1, b->stk, s->dct[0].vln, s->dct[0].kln );
b->ent = 2;
b->lev++; /* increase tree depth */
if ( (ret = WRULK( s, bt )) || (ret = WRULK( bb, bt )) )
break;
s = 0;
break;
}
}
if ( b && !ret )
ret = WRULK( b, bt );
done:
if ( s )
ULK( s );
if ( b )
ULK( b );
if ( !ret && b->lev && findkey( &f, LBT_RD ) )
ret = log_msg( ERR_TRASH, "could not find key after add" );
return ret;
} /* lbt_add */
int lbt_del ( Idx *bt, Key *key )
{
int ret = 0;
unsigned short e, n, i;
Block *b;
Dict *d;
unsigned char *base, *v;
FindKey f;
f.bt = bt;
f.key = key;
if ( findkey( &f, LBT_WR ) )
LOG_OTO( done, ( ERR_TRASH, "could not find key" ) );
if ( !(LBT_MATCH & f.e[0]) ) /* key not found - ok */
goto done;
e = (LBT_ENTMASK & f.e[0]);
b = f.b[0];
d = b->dct + e;
base = LBT_BASE(b);
v = base+LBT_POS(d)+d->kln;
n = LBT_VLN(d);
if ( ! n )
goto done;
i = valpos( v, n, bt->vsz, key->val.byt );
if ( !(LBT_MATCH & i) ) /* val not found - ok */
goto done;
i &= ~LBT_MATCH;
v += i*bt->vsz;
LOG_DBG( LOG_DEBUG,
"del blk %d e %d i/n %d/%d", b->num, e, i, n );
memmove( base+b->stk+bt->vsz, base+b->stk, v-(base+b->stk) );
LBT_VLNADD( d, -1 );
for ( ; e++ < b->ent; d++ )
LBT_POSADD( d, bt->vsz );
WRULK( b, bt );
done:
if ( f.b[0] )
ULK( f.b[0] );
return ret;
} /* lbt_del */
int lbt_loop ( Idx *bt, DXLoop *l )
{
int ret = 0;
unsigned i = 0;
int mode = IDXMODE & l->flg;
Block *b = 0;
Key *cmp = &l->key;
if ( ! cmp->len ) /* start at 1st leaf */
b = getblk( bt, 1, 0, LBT_RD );
else { /* locate key */
FindKey f;
f.bt = bt;
f.key = cmp;
if ( findkey( &f, LBT_RD ) )
LOG_OTO( done, ( ERR_TRASH, "could not find key" ) );
b = f.b[0];
i = (unsigned)(LBT_ENTMASK & f.e[0]);
if ( !(LBT_MATCH & f.e[0]) && IDXEQ == mode )
goto done;
}
if ( IDXUPTO <= mode )
cmp = &l->to;
for ( ;; i=0 ) { /* blocks */
unsigned char *base = LBT_BASE( b );
Dict *d;
if ( ! b || b->lev )
return -ERR_TRASH;
b->lck = LBT_EX; /* lock */
d = b->dct + i;
for ( ; ient; i++, d++ ) {
unsigned short pos = LBT_POS( d );
unsigned short vln = LBT_VLN( d );
unsigned short kln = d->kln;
unsigned short j;
unsigned char *v = base+pos+kln;
Key key;
Hit hit;
int diff = memcmp( cmp->byt, base+pos,
kln <= cmp->len ? kln : cmp->len );
switch ( mode ) {
case IDXPF:
if ( diff || kln < cmp->len )
goto moan;
break;
case IDXEQ:
if ( diff || kln != cmp->len )
goto done;
break;
case IDXUPTO:
if ( 0 >= diff )
goto done;
break;
case IDXINCL:
if ( 0 > diff || (!diff && kln > cmp->len) )
goto done;
break;
default: /* unused */
moan:
log_msg( LOG_INFO,
"loop ends on key '%.*s'(%d) %d %d",
kln, base+pos, kln, diff, cmp->len );
goto done;
}
memcpy( key.byt, base+pos, key.len = kln );
if ( l->cb )
for ( j=0; jvsz ) {
rdval( &hit, v, bt->vsz );
ret = l->cb( l->me, &key, &hit );
if ( ret )
goto done;
}
else { /* internal check */
FindKey f;
f.bt = bt;
f.key = &key;
memcpy( key.val.byt, v, key.val.len = bt->vsz );
if ( findkey( &f, LBT_RD ) )
LOG_OTO( done, ( ERR_TRASH, "could not find key" ) );
if ( !(LBT_MATCH & f.e[0])
|| b != f.b[0]
|| i != (unsigned)(LBT_ENTMASK & f.e[0])
) {
log_msg( ERR_TRASH,
"OOPS! want %u(%p).%u got %u(%p).%u (match %x)",
b->num, b, i, f.b[0]->num, f.b[0], LBT_ENTMASK & f.e[0], f.e[0] );
rdval( &hit, v, bt->vsz );
log_msg( ERR_TRASH,
"\twhen looking for '%.*s'/%u.%u.%u.%u",
key.len, key.byt, hit.mfn, hit.tag, hit.occ, hit.pos );
if ( f.b[0]->ent <= (LBT_ENTMASK & f.e[0]) )
log_msg( ERR_TRASH, "\tI found nothing" );
else {
Dict *dd = f.b[0]->dct + (LBT_ENTMASK & f.e[0]);
unsigned char *bb = LBT_BASE(f.b[0])+LBT_POS(dd);
rdval( &hit, bb+dd->kln, bt->vsz );
log_msg( ERR_TRASH,
"\tI found '%.*s'/%u.%u.%u.%u",
dd->kln, bb, hit.mfn, hit.tag, hit.occ, hit.pos );
}
} else
log_msg( LOG_TRACE, "key ok" );
}
}
ULK( b );
if ( ! b->nxt || !(b = getblk(bt, b->nxt, 0, LBT_RD)) )
break;
}
done:
if ( b )
ULK( b );
if ( l->cb )
l->cb( l->me, 0, 0 );
else
log_msg( LOG_INFO,
"checked %i keys in %u blks of %dK depth %u cmp/get/miss %u/%u/%u",
stat.key, bt->len, bt->bsz>>10, bt->root->lev, stat.cmp, stat.get, stat.miss );
return ret;
} /* lbt_loop */
/* mimik the plain old ldb_search */
int lbt_search ( Idx *bt, Key *key, LdbPost *post, Rec *rec )
{
int ret = 0;
unsigned i = 0;
int cont = rec && rec->len; /* continuing previous terms search */
char *stk = rec ? ((char*)rec + rec->bytes) : 0; /* term stack */
int prefix;
Block *b = 0;
Key start;
LdbP * const p = post->p; /* really just an alias */
/* check for prefix match */
if ( post ) {
if ( 8 != bt->vsz )
return log_msg( ERR_INVAL, "bad legacy search on vsz %d", bt->vsz );
prefix = LDB_PFX & post->mode;
} else {
if ( ! rec )
return log_msg( ERR_INVAL, "lbt_search needs rec or post" );
if ( (prefix = key->len && '$' == key->byt[key->len - 1]) )
key->len--;
}
if ( ! cont )
start = *key;
else {
memset( &start, 0, sizeof(start) );
start.len = 255 < rec->field[rec->len-1].len
? 255 : rec->field[rec->len-1].len;
memcpy( start.byt, rec->field[rec->len-1].val, start.len );
rec->len = 0;
}
if ( ! start.len ) /* start at 1st leaf */
b = getblk( bt, 1, 0, LBT_RD );
else { /* locate key */
FindKey f;
f.bt = bt;
f.key = &start;
if ( findkey( &f, LBT_RD ) )
LOG_OTO( ouch, ( ERR_TRASH, "could not find key" ) );
b = f.b[0];
i = (unsigned)(LBT_ENTMASK & f.e[0]);
if ( !(LBT_MATCH & f.e[0]) && ! prefix )
goto done;
}
for ( ;; i=0 ) { /* blocks */
unsigned char *base = LBT_BASE( b );
Dict *d;
if ( ! b || b->lev )
return -ERR_TRASH;
b->lck = LBT_EX; /* lock */
d = b->dct + i;
for ( ; ient; i++, d++ ) { /* entries */
unsigned char *ent = base + LBT_POS( d ); /* start of key */
unsigned short kln = d->kln;
if ( memcmp( key->byt, ent, kln <= key->len ? kln : key->len )
|| (prefix ? kln < key->len : kln != key->len )
)
goto done;
if ( rec ) { /* record term */
Field *f = rec->field + rec->len; /* field to assign */
/** append term to record, unless it's the same as previous */
if ( rec->len ) {
if ( kln == f[-1].len && !memcmp( f[-1].val, ent, kln ) )
continue;
} else if ( cont ) {
if ( kln == start.len && !memcmp( start.byt, ent, kln ) )
continue;
}
stk -= kln;
if ( stk < (char*)(f+1) ) /* stack reached field dict */
goto done;
memcpy( stk, ent, kln );
f->tag = 0;
f->val = stk;
f->len = kln;
rec->len++;
continue;
}
{ /* more locals */
LdbP merge[1024];
unsigned short vln, k, m=0;
unsigned char *c;
int f = post->fil-1; /* highest pos to consider in given postings */
if ( !(vln = LBT_VLN( d )) )
continue;
ent += kln; /* set to start of values */
c = ent + 8*(vln-1); /* last value */
/* collect postings */
for ( k=vln; k--; c-=8 ) {
/* loop backwards (for the fun of it) postings in segment */
int prow, ptag, ppos;
LdbP e; /* the entry */
LdbP samerow; /* highest possible entry w/ same row as e */
#if defined( LDB_BIG_ENDIAN )
/* the 8 bytes of a posting are BIG ENDIAN ! */
memcpy(e.bytes,c,8);
#else
e.bytes[0] = c[7]; e.bytes[1] = c[6];
e.bytes[2] = c[5]; e.bytes[3] = c[4];
e.bytes[4] = c[3]; e.bytes[5] = c[2];
e.bytes[6] = c[1]; e.bytes[7] = c[0];
#endif
prow = LDBP_ROW( &e );
ptag = LDBP_TAG( &e );
ppos = LDBP_POS( &e );
LOG_DBG( LOG_VERBOSE, "post %d.%hd pos %06x", prow, ptag, ppos );
if ( 0 >= ptag /* bad tag */
|| !prow
)
continue;
if ( (post->cut && prow >= post->cut)
|| (post->tag && post->tag != ptag)
)
continue;
if ( prow < post->skp ) /* quickly bail out on skip condition */
break;
LDBP_SETROWTOP( &samerow, &e ); /* for mfn comparison */
/* sweep down to postings for the same row as e ... */
while ( f >= 0 && LDBP_GT( p+f, &samerow ) )
f--;
if ( LDB_AND & post->mode ) {
int l;
/* loop postings for same row, mark all (that are near enough) */
LDBP_SETROWBOT( &samerow, &e ); /* for mfn comparison */
/* NOTE: postings for row are GT than bottom even if marked */
for ( l = f; l>=0 && LDBP_GT( p+l, &samerow ); l-- ) {
if ( post->near ) {
int dist;
if ( ptag != LDBP_TAG( p+l ) ) continue;
if ( LDB_NEAR_G != post->near ) {
dist = LDBP_POS( p+l ) - LDBP_POS( &e );
if ( dist < 0 ) dist = -dist;
if ( 0 < post->near
? post->near < dist
: -post->near != dist /* exact $$$$ */
) continue;
}
}
LDBP_SETMARK( p+l );
}
} else { /* OR mode */
int add;
if ( ! post->near ) /* add if row not found: ignore details */
add = 0 > f || prow > LDBP_ROW( p+f );
else { /* add if no exact match */
int l;
/* NOTE: we don't use mark bit in OR mode, do we ? */
for ( l = f; l>=0 && LDBP_GT( p+l, &e ); l-- )
;
add = 0 > l || LDBP_GT( &e, p+l );
}
if ( add )
merge[ m++ ] = e;
}
} /* for postings in segment */
if ( m ) { /* merge in the merge buffer */
LdbP *mm = merge;
for ( k = post->fil += m; m && k--; ) {
LdbP src;
if ( k < m || LDBP_GT( mm, &p[k-m] ) ) {
src = *mm++;
m--;
LOG_DBG( LOG_DEBUG, "merging %d at %d", LDBP_ROW(&src), k );
} else
src = p[k-m];
if ( k < post->len )
p[k] = src;
else { /* set cut */
int row = LDBP_ROW( &src );
if ( row < post->cut || !post->cut )
post->cut = row;
}
}
if ( post->fil > post->len )
post->fil = post->len;
if ( post->cut ) /* postings for cut row are unreliable */
while ( post->fil && post->cut <= LDBP_ROW(p+post->fil-1) )
post->fil--;
} /* if ( m ) */
} /* more locals */
} /* for entries in block */
ULK( b );
if ( ! b->nxt || !(b = getblk(bt, b->nxt, 0, LBT_RD)) )
break;
}
done:
if ( rec )
ret = rec->len;
ouch:
if ( b )
ULK( b );
return ret;
} /* lbt_search */
int lbt_init ( Idx *bt )
{
int ret = 0, blkbits, i;
Block *l;
int size = lio_size( bt->fd );
bt->root = 0;
bt->hash = 0;
bt->mem = 0;
for ( i=LBT_LRULEV; i--; )
bt->lru[i] = bt->mru[i] = 0;
if ( 0x3ff & size ) /* not on 1K boundary */
return log_msg( LOG_ERROR, "bt has bad size %dK + %d",
size >> 10, (int)size & 0x3ff );
if ( size ) { /* read root header */
Block b;
unsigned want = sizeof(b) - LBT_BOFF;
int got = lio_pread( &bt->fd, LBT_BASE(&b), want, 0 );
if ( want != (unsigned)got )
return log_msg( LOG_SYSERR, "could not get root head" );
if ( b.num )
return log_msg( ERR_TRASH, "root has num %d", b.num );
/*if ( LBT_ISIS != b.nxt ) log_msg( LOG_WARN, "magic is 0x%08x", b.nxt );*/
if ( bt->typ && bt->typ != b.typ )
log_msg( LOG_WARN, "typ want 0x02%x got 0x02%x", bt->typ, b.typ );
bt->typ = b.typ;
if ( bt->key && bt->key != b.key )
log_msg( LOG_WARN, "key want %d got %d", bt->key, b.key );
bt->key = b.key;
if ( bt->col && bt->col != b.col )
log_msg( LOG_WARN, "col want %d got %d", bt->col, b.col );
bt->col = b.col;
bt->dpt = b.lev;
/*
b.ent
b.stk
*/
} else { /* new index */
bt->dpt = 1;
if ( ! bt->key )
bt->key = 30;
}
bt->vsz = 8+(0xf & bt->typ);
blkbits = 10 + (3 & (bt->typ >> 4));
bt->bsz = 1 << blkbits;
if ( size % bt->bsz )
return log_msg( ERR_TRASH, "size 0x%08x % blksize", size, bt->bsz );
bt->len = size >> blkbits;
if ( 2 > bt->len && !(LBT_WRITE & bt->flg) )
return log_msg( LOG_SYSERR, "attempt to read empty index" );
bt->hlen = 16 + bt->len / 32; /* initial cache size */
if ( bt->hlen < 1000 ) bt->hlen = 1000;
if ( !(bt->root = addchunk( bt, bt->hlen )) )
return log_msg( LOG_SYSERR, "\twhen getting bt cache" );
if ( bt->hlen < 256 ) /* the hash */
bt->hlen = 256;
if ( !(bt->hash = (Block**)lmalloc( bt->hlen * sizeof(Block*) )) )
return log_msg( LOG_SYSERR, "\twhen getting bt hash" );
memset( bt->hash, 0, bt->hlen * sizeof(Block*) );
if ( size && readblk( bt->root, bt, 0 ) )
return log_msg( ERR_TRASH, "could not read root" );
if ( ! bt->len ) {
/* init: link root to leaf */
SETINT( LBT_BASE(bt->root)+(bt->root->stk -= 4), 1 );
LBT_PVK( bt->root->dct, bt->root->stk, 0, 0 );
bt->root->ent = 1;
bt->len = 1;
}
if ( !(l = getblk( bt, 1, 0, LBT_EX )) )
return log_msg( ERR_TRASH, "could not read 1st leaf" );
if ( ! size ) {
log_msg( LOG_INFO, "creating index bsz %u ksz %u vsz %u",
bt->bsz, bt->key, bt->vsz );
bt->root->lev = 1;
if ( WRULK( bt->root, bt ) || WRULK( l, bt ) )
return log_msg( LOG_SYSERR, "\twhen writing base" );
}
return ret;
} /* lbt_init */
/* ************************************************************
public functions
*/
void cXMkVal ( Idx *bt, Val *val, Hit *hit )
{
if ( hit ) {
val->len = bt->vsz;
mkval( val, hit );
} else
val->len = 0;
}
int cXAdd ( Idx *bt, Key *key, Hit *hit )
{
if ( bt->bat ) {
if ( ! key )
key = &bt->bat->key;
cXMkVal( bt, &key->val, hit );
return lbt_batchval( bt, key );
} else if ( !key )
return 0;
else if ( hit ) {
cXMkVal( bt, &key->val, hit );
if ( hit->dbn == 0xffff )
return lbt_del( bt, key );
}
return lbt_add( bt, key );
} /* cXAdd */
lbt.h 0100644 0001130 0001130 00000007135 07637615605 010512 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef LBT_H
/*
$Id: lbt.h,v 1.10 2003/03/24 14:51:49 kripke Exp $
package interface of the btree.
see Lehmann/Yao or the GiST for how it works.
*/
#include "ldb.h" /* LdbPost */
/**
key comparision function.
however, we still assume that two keys have to be identical
in length and bytes to be equal.
*/
typedef int lbt_comp ( const unsigned char *a, const unsigned char *b,
unsigned int l );
/**
variables:
4 bits: #bytes for value 8+(0..15)
2 bits: blocksize 1024<<(0..3)
max key length (4..255)
collation (comparision function) 0..255
*/
typedef struct Idx { /* actually it's a B-L-Tree ;) */
int fd; /* the file */
int flg; /* flags: writeable, batch */
unsigned char typ; /* type: bsz, ifp, flags */
unsigned char key; /* max key length */
unsigned char col; /* collation */
unsigned char dpt; /* depth (level of root over bottom > 0) */
lbt_comp *cmp; /* comparision function */
/* following members (and the depth above) are set automatically.
they are going to stay and you may check them, if you're interested.
*/
unsigned vsz; /* ifp size computed from type */
unsigned bsz; /* block size computed from type */
unsigned len; /* # blocks in index */
/* following members are considered internal.
if you import lbt and rely on them, don't blame me if they change.
*/
unsigned hlen; /* hash length */
unsigned clen; /* cache length */
struct Block *root; /* the root */
struct Block **hash; /* hash array */
struct Block *lru[4]; /* least recently used list for lowest levels */
struct Block *mru[4]; /* tail of lru list (most recently used) */
struct Chunk *mem;
struct Batch *bat;
} Idx;
enum { /* btree flags */
LBT_WRITE = 0x01 /* open for writing */
};
enum { /* btree type */
LBT_BLK1K = 0x00, /* 1K blocks */
LBT_BLK2K = 0x10, /* 2K blocks */
LBT_BLK4K = 0x20, /* 4K blocks */
LBT_BLK8K = 0x30, /* 8K blocks */
LBT_CMPRS = 0x80 /* compressed keys */
};
/**
initialise from an already open fd.
*/
extern int lbt_init ( Idx *bt );
/**
flush and release any cache, close fd.
*/
extern int lbt_close ( Idx *bt );
extern int lbt_batch ( Idx *bt, unsigned char pctfree );
extern int lbt_batchval ( Idx *bt, Key *key );
extern int lbt_add ( Idx *bt, Key *key );
extern int lbt_del ( Idx *bt, Key *key );
extern int lbt_loop ( Idx *bt, DXLoop *l );
extern int lbt_search ( Idx *bt, Key *key, LdbPost *post, Rec *rec );
/* half public ... ??? */
extern void cXMkVal ( Idx *bt, Val *val, Hit *hit );
#define LBT_H
#endif /* LBT_H */
lcom.h 0100644 0001130 0001130 00000007533 07632173711 010655 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2003 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef LCOM_H
#define LCOM_H
/*
$Id: lcom.h,v 1.3 2003/03/05 23:47:56 mawag Exp $
ipc parameters.
*/
/** communication parameters: client request */
#define LCOM_RQS_SID 900 /* session id */
#define LCOM_RQS_SER 901 /* serial no of request */
#define LCOM_RQS_TYPE 902 /* request type */
#define LCOM_RQS_DBN 903 /* db name */
#define LCOM_RQS_TMS 904 /* server timestamp */
#define LCOM_RQS_ROWID 910 /* rowid (read, delete, update) */
#define LCOM_RQS_QMOD 911 /* mode (query) */
#define LCOM_RQS_SKIP 912 /* skip (query) */
#define LCOM_RQS_SIZE 913 /* output size (query) */
#define LCOM_RQS_QFLG 914 /* flags (query) */
#define LCOM_RQS_KEY 915 /* key value (query) */
#define LCOM_RQS_REC 920 /* data structure (insert, update) */
#define LCOM_RQS_IDX 921 /* index record (insert, update) */
/** communication parameters: server response */
#define LCOM_RSP_SID 950 /* session id */
#define LCOM_RSP_SER 951 /* serial no of request */
#define LCOM_RSP_DBID 952 /* db id in case of local db */
#define LCOM_RSP_DBN 953 /* db name */
#define LCOM_RSP_TMS 954 /* server timestamp */
#define LCOM_RSP_ERR 955 /* error code */
#define LCOM_RSP_MSG 956 /* error message */
#define LCOM_RSP_NUMT 957 /* total number of records (query, read) */
#define LCOM_RSP_NUMR 958 /* number in this response (query, read) */
#define LCOM_RSP_CERR 959 /* client side error */
#define LCOM_RSP_ROWID 960 /* rowid (query, maxrowid, insert) */
#define LCOM_RSP_DATA 970 /* structure (query, read, open/fdt) */
/** request types. */
#define LCOM_RQST_OPEN 1 /* open db */
#define LCOM_RQST_CREA 2 /* open or create db */
#define LCOM_RQST_CLOS 3 /* close db */
#define LCOM_RQST_MROW 10 /* get maxrowid of db */
#define LCOM_RQST_QRY 11 /* exec query on db */
#define LCOM_RQST_READ 12 /* get rec for rowid */
#define LCOM_RQST_INS 20 /* insert rec */
#define LCOM_RQST_UPD 21 /* update rec */
#define LCOM_RQST_DEL 22 /* delete rec */
/** fdt of request record. */
extern const OpenIsisFdt *openIsisComRqsPar;
/** fdt of response record. */
extern const OpenIsisFdt *openIsisComRspPar;
/** max length of error messages. */
#define LCOM_ERRMSGLEN 1024
/** additional flags in query request. */
#define LCOM_QRYF_READREC 0x01 /* fetch records */
/** max length of key in query request. */
#define LCOM_QRY_KEYLEN 256
/** callback for delivering response record. */
typedef int DlvCb (OpenIsisSession ses, OpenIsisRec **rsp, void *cld);
/** ldsp.c: server side request processing */
extern int ldsp_process (OpenIsisSession ses,
OpenIsisRec *rqs, DlvCb *cb, void *cld);
#endif /* LCOM_H */
lcs.c 0100644 0001130 0001130 00000015143 07630630466 010476 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lcs.c,v 1.7 2003/03/03 10:40:20 kripke Exp $
charset support
*/
#include /* memset */
#include "luti.h"
#include "lio.h"
#include "lcs.h"
/* ************************************************************
private types
*/
/* ************************************************************
private data
*/
/* ************************************************************
private functions
*/
/* ************************************************************
package functions
*/
unsigned char lcs_latin1_uc[256] = {
0,' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',
' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',
' ','!','"','#','$','%','&','\'','(',')','*','+',',','-','.','/',
'0','1','2','3','4','5','6','7','8','9',':',';','<','=','>','?',
'@','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O',
'P','Q','R','S','T','U','V','W','X','Y','Z','[','\\',']','^','_',
'`','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O',
'P','Q','R','S','T','U','V','W','X','Y','Z','{','|','}','~',' ',
' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',
' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',
' ',161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,
176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,
'A','A','A','A','A','A','A','C','E','E','E','E','I','I','I','I',
'T','N','O','O','O','O','O',215,'O','U','U','U','U','Y','T','S',
'A','A','A','A','A','A','A','C','E','E','E','E','I','I','I','I',
'T','N','O','O','O','O','O',247,'O','U','U','U','U','Y','T','Y'
};
unsigned char lcs_latin1_ct[256] = {
/* NUL STX SOT ETX EOT ENQ ACK BEL */
LCS_F,LCS_C,LCS_C,LCS_C,LCS_C,LCS_C,LCS_C,LCS_C,
/* BS HT LF VT FF CR SO SI */
LCS_C,LCS_C,LCS_F,LCS_C,LCS_R,LCS_F,LCS_C,LCS_C,
/* DLE DC1 DC2 DC3 DC4 NAK SYN ETB */
LCS_C,LCS_C,LCS_C,LCS_C,LCS_C,LCS_C,LCS_C,LCS_C,
/* CAN EM SUB ESC FS GS RS US */
LCS_C,LCS_C,LCS_C,LCS_C,LCS_C,LCS_R,LCS_F,LCS_C,
/*blank ! " # $ % & ' */
LCS_S,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,
/* ( ) * + , - . / */
LCS_P,LCS_P,LCS_P,LCS_P,LCS_T,LCS_P,LCS_P,LCS_P,
/* 0 1 2 3 4 5 6 7 */
LCS_D,LCS_D,LCS_D,LCS_D,LCS_D,LCS_D,LCS_D,LCS_D,
/* 8 9 : ; < = > ? */
LCS_D,LCS_D,LCS_T,LCS_T,LCS_P,LCS_T,LCS_P,LCS_P,
/* @ A B C D E F G */
LCS_P,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,
LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A, /* H-O */
LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A, /* P-W */
/* X Y Z [ \ ] ^ _ */
LCS_A,LCS_A,LCS_A,LCS_P,LCS_P,LCS_P,LCS_P,LCS_I,
/* ` a b c d e f g */
LCS_P,LCS_L,LCS_A,LCS_L,LCS_A,LCS_A,LCS_A,LCS_A,
LCS_A,LCS_L,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,
LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,
/* x y z { | } ~ DEL */
LCS_L,LCS_A,LCS_A,LCS_P,LCS_P,LCS_P,LCS_P,LCS_C,
/* 32 C0 controls */
LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,
LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,
LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,
LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,LCS_P,
/* 32 mostly symbols */
LCS_S,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,
LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,
LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,
LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,LCS_Y,
/* 64 Latin alphas including 2 symbols */
LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,
LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,
LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_Y,
LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,
LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,
LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,
LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_Y,
LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A,LCS_A
};
/* ************************************************************
package functions
*/
int lcs_mktab ( LcsTab *dst, char *n, int l, int bits )
{
int ret = 0;
unsigned char *c = dst->c;
unsigned short *u = dst->u;
int line = 1;
int i = 0;
memset( dst, 0, sizeof(*dst) );
for (;; i++ ) {
unsigned short in = 0;
for ( ; l && (*n < '0' || '9' < *n); n++, l-- ) /* skip nondigs */
if ( '\n' == *n )
line++;
if ( ! l )
break;
for ( ; l && '0' <= *n && *n <= '9'; n++, l--)
in = 10*in + *n - '0';
if ( bits < 0 ) {
if ( i > 127 ) {
log_msg( LOG_WARN, "too many unicode values in input" );
return 0;
}
if ( in <= 0xffff )
u[i] = in;
else
log_msg( ERR_INVAL, "unicode %d too big %dth value line %d",
in, i, line );
continue;
}
if ( in > 0xff )
log_msg( ERR_INVAL, "byte %d too big %dth value line %d",
in, i, line );
else if ( bits )
c[in] |= bits;
else if ( i > 255 ) {
log_msg( LOG_WARN, "too many values in input" );
return 0;
} else
c[i] = (unsigned char)in;
}
return ret;
} /* lcs_mktab */
int lcs_mkrecod ( unsigned char *dst, unsigned short *from,
unsigned short *to )
{
int i = 0, j;
for ( i=0; i<128; i++ )
dst[i] = (unsigned char)i; /* identity */
for ( ; i<256; i++ ) {
unsigned short s = from[ i - 128 ];
dst[i] = ' ';
if ( to )
for ( j=128; j--; ) {
if ( s == to[j] ) {
dst[i] = (unsigned char)128+j;
break;
}
}
else if ( 256 > s ) /* map to latin 1 */
dst[i] = (unsigned char)s;
}
return 0;
} /* lcs_mkrecod */
/* ************************************************************
public functions
*/
lcs.h 0100644 0001130 0001130 00000010312 07630630466 010474 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef LCS_H
/*
$Id: lcs.h,v 1.2 2003/03/03 10:40:20 kripke Exp $
charset support
*/
typedef enum {
LCS_SINGLE, /* some ASCII-compliant single byte charset */
LCS_UTF8, /* unicode in 8bit transfer encoding */
LCS_UCS2 /* the unicode base multilingual plane (BMP, first 64K chars)
in 2byte encoding, native (platform) byte order */
} lcs_type;
typedef enum {
LCS_LATIN1, /* the ISO-8859-1 (Latin 1) charset */
LCS_CP850, /* MS-DOS Latin 1, known as the "OEM" charset */
LCS__SBCS /* # of single byte charsets */
} lcs_single;
typedef enum {
LCS_CTYPE, /* flags see below */
LCS_UNICO, /* 128 unicodes */
LCS_UCASE, /* each characters uppercase code */
LCS_INDEX, /* single byte recoding for index ("strxfrm") */
LCS_XEDNI, /* inverse */
LCS_RECOD, /* single byte recoding to extern (typically Latin1) */
LCS_DOCER, /* inverse */
LCS__TABS
} lcs_tabid;
/* some conversion table on a single byte charset
c may hold an actab (!0 for any character deemed "alpha"),
uctab ("uppercase" replacement code, typically removing diacriticals),
code-to-code conversion or other.
u may hold unicode values for the upper half of an
ASCII compatible single byte charset.
*/
typedef union {
unsigned char c[256];
unsigned short u[128];
} LcsTab;
enum {
/* controls (0..31,127) */
LCS_R = 0x00, /* record separator (FF,GS), */
LCS_F = 0x10, /* field separator (NUL,CR,LF,RS) */
LCS_C = 0x28, /* other control */
/* other separators */
LCS_S = 0x48, /* space(blank,nbsp), */
LCS_T = 0x68, /* other token separator (,:;=), */
/* other punctuation assumed "word" characters */
LCS_P = 0x80,
LCS_Y = 0xb0, /* symbol */
/* identifiers */
LCS_I = 0xc0, /* C-ident (_). */
LCS_D = 0xd0, /* digits ('0'..'9') */
LCS_A = 0xe0, /* ASCII alpha */
LCS_L = 0xe4, /* ... among those formatting literals a,c,i,x */
LCS_N = 0xf0 /* other alpha ("national"/non-ASCII) */
};
#define LCS_ISCONTR( t ) (!(0xc0 & (t)))
#define LCS_ISSPACE( t ) (LCS_S >= (t))
#define LCS_ISPRINT( t ) (0xc0 & (t)) /* (LCS_S <= (t)) */
#define LCS_ISIDENT( t ) (0xc0 == (0xc0 & (t)))
#define LCS_ISALPHA( t ) (0xe0 == (0xe0 & (t)))
/* for record parsing */
#define LCS_ISSEP( t ) (!(0x80 & (t)))
#define LCS_ISWORD( t ) (0x80 & (t))
#define LCS_ISFR( t ) (!(0xe0 & (t))) /* field or record separator */
#define LCS_ISCST( t ) (0x08 & (t)) /* other separator */
enum {
LCS_MKUNI = -1 /* as "bits" value, have mktab create unicode table */
};
/** create the table dst from string containing
a free style sequence of decimal numbers (ignoring any non-digit).
if bits is 0, numbers are assigned sequentially to dst->c,
else if bits < 0, they are assigned to dst->u,
else for every number, bits are set in the corresponding array element
else something strange may happen.
*/
extern int lcs_mktab ( LcsTab *dst, char *numbers, int len, int bits );
/** create single byte conversion table from two unicode tables.
if to is NULL, the trivial (Latin1) table is used.
*/
extern int lcs_mkrecod ( unsigned char *dst, unsigned short *from,
unsigned short *to );
extern unsigned char lcs_latin1_uc[256];
extern unsigned char lcs_latin1_ct[256];
#define LCS_H
#endif /* LCS_H */
ldb.c 0100644 0001130 0001130 00000155613 07637613573 010474 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: ldb.c,v 1.66 2003/03/24 14:34:35 kripke Exp $
implementation of general db access functions.
*/
#include
#include
#include /* PATH_MAX */
#include
/* gettimeofday */
#ifdef WIN32
# include
# include
# define timeval _timeb
#else
# include
#endif
#include
/* special */
#if defined( __GNUC__ ) && defined ( alloca )
#include
#endif
/* gcc defines always a cpu type - this we use for byteorder checking */
#if defined( sparc ) || defined( __ppc__ )
# define LDB_BIG_ENDIAN
/* TODO: figure out fastest "htonl" on those boxes that usually don't swap */
static int rvi ( int i ) {
int r;
((char*)&r)[0] = ((char*)&i)[3];
((char*)&r)[1] = ((char*)&i)[2];
((char*)&r)[2] = ((char*)&i)[1];
((char*)&r)[3] = ((char*)&i)[0];
return r;
}
static short rvs ( short i ) {
short r;
((char*)&r)[0] = ((char*)&i)[1];
((char*)&r)[1] = ((char*)&i)[0];
return r;
}
#define SWI( i ) i = rvi( i )
#define SWS( s ) s = rvs( s )
#else
# define rvi(i) i
# define rvs(s) s
#define SWI( i )
#define SWS( s )
#endif
#if defined( sparc )
# define LDB_NEEDALIGN
#endif
#include "lstr.h"
#include "lio.h"
#include "lbt.h"
#include "lcs.h"
#include "ldb.h"
#include "lfdt.h"
#include "luti.h"
/* ************************************************************
private types
*/
/** extension of master file proper. */
static const char EXT_MST_MST[] = ".mst";
/** extension of master file xref. */
static const char EXT_MST_XRF[] = ".xrf";
typedef enum {
MST_MST,
MST_XRF,
MST_FILES
} mst_file;
static const char * const EXT_MST[MST_FILES] = {
EXT_MST_MST,
EXT_MST_XRF
};
/** extension of inverted file short term nodes. */
static const char EXT_INV_N01[] = ".n01";
/** extension of inverted file short term leaves. */
static const char EXT_INV_L01[] = ".l01";
/** extension of inverted file long term nodes. */
static const char EXT_INV_N02[] = ".n02";
/** extension of inverted file long term leaves. */
static const char EXT_INV_L02[] = ".l02";
/** extension of inverted file control records. */
static const char EXT_INV_CNT[] = ".cnt";
/** extension of inverted file postings. */
static const char EXT_INV_IFP[] = ".ifp";
typedef enum {
INV_N01,
INV_L01,
INV_N02,
INV_L02,
INV_CNT,
INV_IFP,
INV_FILES
} inv_file;
static const char * const EXT_INV[INV_FILES] = {
EXT_INV_N01,
EXT_INV_L01,
EXT_INV_N02,
EXT_INV_L02,
EXT_INV_CNT,
EXT_INV_IFP
};
/** extension of lbt B-Link-Tree.
It's named oxi because that is nicer than oix for OpenIsis indeX.
however, see http://www.oxicenter.com.br/
*/
static const char EXT_LBT_OXI[] = ".oxi";
static const char * const EXT_LBT[] = {
EXT_LBT_OXI
};
/** extension of supporting file alpha character table. */
static const char EXT_SUP_ACT[] = ".act";
/** extension of supporting file uppercase table. */
static const char EXT_SUP_UCT[] = ".uct";
typedef enum {
SUP_ACT,
SUP_UCT,
SUP_FILES
} sup_file;
static const char * const EXT_SUP[SUP_FILES] = {
EXT_SUP_ACT,
EXT_SUP_UCT
};
typedef int lblk[128];
typedef struct {
Db head;
int flags;
const char *path;
int mst[MST_FILES]; /* master file */
int inv[INV_FILES]; /* primary inverted file */
int mfc[LMFC__FL]; /* master file control record */
unsigned mflen; /* master file length */
int xrf[129]; /* last used xrf block : THREAD THREAT */
int xrlen; /* length of xrf (in blocks) */
lblk *mmap; /* memory map of xrf */
int mmlen; /* length of map (in blocks) */
int cnt[LDB_INDEXES][LCNT__FL]; /* two cnt records */
short tlen[LDB_INDEXES]; /* max term length for each index */
int set[2]; /* record sets, should be { LSET_MST, LSET_INV } */
LcsTab ctab[LCS__TABS];
Idx oxi;
} LDb;
/* db flags */
#define DB_OPEN 0x010000
#define DB_INVOPEN 0x020000
#define DB_LBTOPEN 0x040000
#define DB_WRITABLE 0x080000
#define DB_MODIFIED 0x100000
#define DB_MMAP 0x10
#define DB_VARI 0xf /* mask for variant */
/* get master or inverted record set */
#ifdef HAVE_NOSTANDARD_RSETS /* far future */
# define DB_RSET( db, s ) lstrlib[ (db)->set[s] ]
#else /* simply LSET_MST and LSET_INV */
# define DB_RSET( db, s ) lstrlib[ s ]
#endif
/* get xstr for record rec in set */
#define DB_XSTR( db, set, rec ) \
DB_RSET( db, set ).desc[ DB_VARI & (db)->flags ][ rec ]
/* get record names for record rec in set */
#define DB_RNAM( db, set, rec ) \
DB_RSET( db, set ).name[ rec ]
/** packed little endian masterfile control structure.
*/
typedef struct Mfc {
int ctlm;
int nmfn;
int nmfb;
short nmfp;
short type;
int rcnt;
int mfx1;
int mfx2;
int mfx3;
} Mfc;
/** packed little endian masterfile record.
*/
typedef struct Dict {
short tag;
short pos;
short len;
} Dict;
/** packed little endian masterfile record.
*/
typedef struct Mfr {
int mfn;
short recl; /* a.k.a. mfrl */
short bwbl; /* low part of int */
short bwbh; /* high part of int */
short bwp;
/* it is believed, that this first five fields up to here (12 bytes packed)
are to be in one 512-byte block; the manual mentiones even 14 bytes ... ???
*/
short base;
short nvf;
short stat;
Dict dict[1];
} Mfr;
/* ************************************************************
private data
*/
static LDb defdbspace[32];
/* array of open dbs. should expand dynamically. */
static LDb *dbs = defdbspace;
static int dbs_len = sizeof(defdbspace)/sizeof(defdbspace[0]);
static int init;
/* ************************************************************
private functions
*/
static LDb *getDb ( int id )
{
if ( 0 <= id && id < dbs_len && dbs[id].flags ) {
return &dbs[id];
}
log_msg( LOG_ERROR, "attempt to access bad db id %d", id );
return 0;
} /* getDb */
/* ************************************************************
start of io section
*/
enum {
OPEN_WR,
OPEN_RD,
OPEN_OPT
};
/* set extension. fname MUST already end with .xxx.
*/
static char *setext ( char *fname, const char *ext )
{
int l = strlen( fname ) - 4;
memcpy( fname+l, ext, 4 );
if ( 'A'<=fname[l-1] && fname[l-1]<= 'Z' ) {
char *p = fname+l;
for ( ;*p; p++ ) /* use uppercase extensions */
if ( 'a' <= *p && *p <= 'z' )
*p -= 'a'-'A';
}
return fname;
}
static int openfiles ( int *fid, char *path,
const char * const *ext, int nfiles, int readonly )
{
int mode = LIO_SEEK | (readonly ? LIO_RD : LIO_RDWR|LIO_CREAT|LIO_FLOCK);
int i,ret;
for ( i=0; i got )
return got;
#ifndef NDEBUG
if ( LOG_DO( LOG_ALL ) )
LOG_HEX( dst, got );
#endif
if ( siz == (int)got )
return 0;
log_msg( LOG_WARN, "got %u bytes wanted %d at %d in 0x%x",
got, siz, where, fid );
return 1+(int)got;
} /* readblk */
/* ************************************************************
end of io section
*/
static int *nrec ( int *xstr )
{
int *dst = (int*)lmalloc( xstr[LSTR_ILEN] );
if ( dst )
*dst = *xstr;
return dst;
} /* nrec */
typedef struct b8 { char x[8]; } b8;
typedef struct b4 { char x[4]; } b4;
typedef struct b2 { char x[2]; } b2;
static int convert ( int *dst, char *src, int *xstr )
{
static int pow2[] = { 1, 2, 4, 8 };
int occ = -1;
int *xmbrs = xstr+LSTR_XMBR;
int nmbrs = LSTRFIX(*xstr);
int *mbr;
char *buf, *part = src, *srcend = src + xstr[LSTR_XLEN];
if ( ! dst )
return log_msg( ERR_NOMEM, "out of memory (no dst) in convert" );
if ( LSTRLEN(*xstr) > xstr[LSTR_ILEN] )
return log_msg( ERR_INVAL, "bad ilen %d need %d",
xstr[LSTR_ILEN], (int)LSTRLEN(*xstr) );
if ( *dst != *xstr )
return log_msg( ERR_INVAL, "bad *dst 0x%08x need 0x%08x", *dst, *xstr );
/* clean and re-init */
memset( dst, 0, xstr[LSTR_ILEN] );
*dst = *xstr;
mbr = dst+1;
buf = ((char*)dst) + LSTRLEN(*dst);
/* cvt the fixed part (occ==-1) and each occurrence of repeated part. */
for ( ;/* occ < LSTROCC(*dst) */; ) { /* cvt one part */
int i;
/* TODO: this loop is crucial for performance */
for ( i=0; i sbytes )
return log_msg( ERR_INVAL,
"srcbuf too short %d have %d need %d occ %d mbr %d",
xstr[LSTR_XLEN], sbytes, len, occ, i );
if ( !NEEDSWAP( xmbr ) )
/* for ( j = len; j--; ) num.buf[j] = s[j]; */
switch ( ld ) {
case 3: num.x8 = *(b8*)s; break;
case 2: num.x4 = *(b4*)s; break;
case 1: num.x2 = *(b2*)s; break;
case 0: num.buf[0] = *s; break;
}
else /* swap bytes */
for ( j = len; j--; )
num.buf[j] = s[len - 1 - j];
switch ( len ) {
case 8: *mbr = num.ll; break;
/* TODO: defines for 16 and 64 bit compilers */
case 4: *mbr = num.i; break;
case 2: *mbr = num.s; break;
case 1: *mbr = num.buf[0]; break;
}
if ( LMBRISBITS( xmbr ) ) { /* apply bit shift and mask */
*mbr >>= LMBRBITOFF( xmbr );
*mbr &= ~(-1L << LMBRBITLEN( xmbr ));
}
continue;
}
/* else raw data -- that's easy :) */
{
int offset = buf - (char*)dst;
int need;
len = LONG2LEN(xmbr);
need = offset + len;
if ( need > xstr[LSTR_ILEN] )
return log_msg( ERR_INVAL,
"bad buflen %d need %d+%d in occ %d mbr %d",
xstr[LSTR_ILEN], offset, len, occ, i );
if ( len > sbytes )
return log_msg( ERR_INVAL,
"srcbuf too short %d have %d need %d occ %d mbr %d",
xstr[LSTR_XLEN], sbytes, len, occ, i );
memcpy( buf, s, len );
*mbr = buf - (char*)dst;
buf += len;
}
} /* for mbrs */
if ( ++occ >= LSTROCC(*dst) )
break;
if ( occ )
part += (short)xstr[LSTR_XRLO]; /* adv. rep. part len */
else { /* was the fixed part, setup for repeated */
nmbrs = LSTRREP(*xstr);
part += (short)(xstr[LSTR_XRLO]>>16); /* adv. rep. part off */
xmbrs += i;
}
}
return 0;
} /* convert */
static int readrec ( int *dst, int fid, int where, int *xstr )
{
char *buf = (char *)
#ifdef alloca
alloca( xstr[LSTR_XLEN] )
#else
lmalloc( xstr[LSTR_XLEN] )
#endif
;
int got = 0;
int ret = 0;
if ( ! buf )
return -ERR_NOMEM;
if ( 0 > where )
where = -where * xstr[LSTR_XLEN];
got = readblk( buf, xstr[LSTR_XLEN], fid, where );
ret = got ? got : convert( dst, buf, xstr );
#ifndef alloca
free( buf );
#endif
return ret;
} /* readrec */
/* read and log */
static int readlog ( int *dst, int fid, int where, LDb *db, int set, int rec )
{
int ret = readrec( dst, fid, where, DB_XSTR( db, set, rec ) );
if ( !ret && LOG_DO( LOG_TRACE ) )
LOG_STR( dst, lstrlib[ db->set[set] ].name[ rec ] );
return ret;
} /* readlog */
/* ************************************************************
access functions for the record types
*/
static int getOff ( LDb *db, int rowid, int xr )
{
int rowix = rowid - 1; /* mfns count from 1 */
int xrf_block = rowix / 127;
int blkix = xrf_block + 1; /* ... so do xrf blocks */
int off;
if ( xrf_block < db->mmlen ) {
if ( xr )
db->mmap[xrf_block][1+(int)(rowix % 127)] = rvi( xr );
else
xr = rvi( db->mmap[xrf_block][1+(int)(rowix % 127)] );
} else {
int err = 0;
/* if ( LIO_LOCK() ) return -3; */
if ( xr ) { /* write */
if ( blkix <= db->xrlen ) {
SWI( xr );
if ( 4 != lio_pwrite( &db->mst[MST_XRF], &xr, 4,
xrf_block*512 + 4*(1 + (rowix % 127))) )
return 0;
} else { /* extent */
lblk extend;
memset( extend, 0, sizeof(extend) );
while ( db->xrlen < blkix ) { /* extend */
extend[0] = db->xrlen + 1; /* set blk id */
if ( blkix == extend[0] )
extend[1+(int)(rowix % 127)] = rvi( xr );
SWI( extend[0] );
if ( 512 != lio_pwrite( &db->mst[MST_XRF], extend, 512,
db->xrlen*512) )
return 0;
db->xrlen++;
}
}
/* and go on read it back, just to check ... */
}
if ( blkix != db->xrf[LXRF_XPOS] ) {
int ret;
LOG_DBG( LOG_VERBOSE, "fetching xrf block %d had %d",
blkix, ! db->xrf ? -1 : db->xrf[LXRF_XPOS] );
ret = readlog( db->xrf, db->mst[MST_XRF],
-xrf_block, db, LSET_MST, LSTR_XRF );
if ( ret ) {
log_msg( LOG_ERROR, "\twhen fetching xrf block %d", blkix );
err = -1;
} else if ( blkix == -db->xrf[LXRF_XPOS] ) {
LOG_DBG( LOG_DEBUG, "hmmm ... negative" );
db->xrf[LXRF_XPOS] = blkix;
} else if ( blkix != db->xrf[LXRF_XPOS] ) {
log_msg( LOG_WARN, "bad xrf %d wanted %d",
db->xrf[LXRF_XPOS], blkix );
err = -2;
}
}
xr = db->xrf[ LXRF_XREC + (int)(rowix % 127) ];
/* LIO_RELE(); */
if ( err )
return err;
}
/*
21 bits (<<11) signed for the (512 byte) block ("xrmfb")
1 for the first block (offset 0)
0 means, never had such a record
-1 and xrmfp=0: record removed from MST
(there is no record at pos 0 in 1st block,
since there resides the MST header)
other negative value -x or pos!=0:
record logically deleted, was at +x
1 bit (1<<10): this record is new and not yet inverted
1 bit (1<<9): this record is changed and not yet re-inverted
9 bits for the block-relative position ("xrmfp")
*/
off = (((xr & 0xfffff800) >> 2) - 0x200) | (0x1ff & xr);
if ( 0 < (xr & ~0x600) ) {
LOG_DBG( LOG_DEBUG,
"offset for rowid %d is %d (blk %d pos %d) flg 0x%08x at %d[%d]",
rowid, off, (xr>>11)&0xfffff, xr&0x1ff, xr&0x80000600, blkix, rowix%127 );
return off;
}
/* deleted */
log_msg( LOG_INFO,
"offset for rowid %d is %d (blk %d pos %d) flg 0x%08x at %d[%d]",
rowid, off, (xr>>11)&0xfffff, xr&0x1ff, xr&0x80000600, blkix, rowix%127 );
return 0;
} /* getOff */
static int* getMfr ( LDb *db, int off, int *nxtoff )
{
struct mfcxstr {
int xstr[LSTR_LONGS(7+3)];
} my = *(struct mfcxstr *)DB_XSTR( db, LSET_MST, LSTR_MFR );
int head[1+7+3];
int len = 0, base;
int *rec = 0;
char *buf = 0;
#ifdef alloca
int notalloca = 0;
#endif
LOG_DBG( LOG_VERBOSE, "getting MFR at off %d", off );
if ( 498 < off % 512 )
log_msg( LOG_WARN, "blk pos > 498 in offset 0x%08x", off );
*head = *my.xstr;
if ( readrec( head, db->mst[MST_MST], off, my.xstr ) ) {
log_msg( LOG_ERROR, "\twhen reading MFR head at %d", off );
return 0;
}
/* log_str( LOG_VERBOSE, head, lstrlib[LSET_MST].name[LSTR_MFR] ); */
len = head[LMFR_RECL];
LOG_DBG( LOG_VERBOSE, "got MFR %d reclen %d", head[LMFR_MFN], len );
if ( nxtoff ) {
*nxtoff = off + (0 < len ? len : -len); /* an odditiy */
if ( 1 & *nxtoff ) /* an odditiy */
(*nxtoff)++; /* round up to even */
if ( 498 < *nxtoff % 512 ) {
*nxtoff += 512;
*nxtoff &= ~0x1ff;
}
if ( *nxtoff > db->mfc[LMFC_NMFB]*512 + db->mfc[LMFC_NMFP] ) {
LOG_DBG( LOG_VERBOSE, "at end of db: %d > %d*512+%hd",
*nxtoff, db->mfc[LMFC_NMFB], db->mfc[LMFC_NMFP] );
*nxtoff = -1;
}
}
if ( len < 0 ) {
log_msg( LOG_INFO, "found deleted rec len %hd at offset %d", len, off );
return 0;
}
/* check external base length */
base = LONG2OFF(my.xstr[LSTR_XRLO])
+ head[LMFR_NVF]*LONG2LEN(my.xstr[LSTR_XRLO]);
if ( 0 > head[LMFR_BASE] || 0 > head[LMFR_NVF]
|| 0x8fff < head[LMFR_NVF]
|| len < head[LMFR_BASE] || len < base
|| (head[LMFR_NVF] && head[LMFR_BASE] < base)
) {
log_msg( LOG_ERROR,
"bad len %d base %d nvf %d need base %d at offset %d",
len, head[LMFR_BASE], head[LMFR_NVF], base, off );
/* check alignment problem */
base = LONG2OFF(my.xstr[LSTR_XRLO])
+ head[LMFR_STAT]*LONG2LEN(my.xstr[LSTR_XRLO]);
if ( 0 > head[LMFR_NVF] || 0 > head[LMFR_STAT]
|| 0x8fff < head[LMFR_STAT]
|| len < head[LMFR_NVF] || len < base
|| (head[LMFR_STAT] && head[LMFR_NVF] < base)
)
;
else
log_msg( LOG_ERROR, "probably alignment problem, try -format aligned" );
goto cleanup;
}
/* set up external structure for this rec */
my.xstr[LSTR_SIZE] |= head[LMFR_NVF] << 16; /* occ of rep. part */
my.xstr[LSTR_XLEN] = len;
/* internal base length */
base = LSTRLEN( *my.xstr );
/* internal len adjusted for the slightly longer base */
my.xstr[LSTR_ILEN] = len + base - head[LMFR_BASE]; /* the buffer */
rec = nrec( my.xstr );
if ( ! rec ) {
log_msg( LOG_SYSERR, "could not alloc MFR of len %hd", my.xstr[LSTR_ILEN] );
goto cleanup;
}
buf = (char *)
#ifdef alloca
alloca( len );
/* first try faster alloca, but stack may be too limited for large records */
notalloca = ! buf;
if ( notalloca )
buf =
#endif
lmalloc( len );
if ( ! buf ) {
log_msg( LOG_SYSERR, "could not alloc MFR of len %hd", len );
goto cleanup;
}
if ( readblk( buf, len, db->mst[MST_MST], off ) ) {
log_msg( LOG_ERROR, "\twhen reading MFR" );
goto cleanup;
}
#ifndef LDB_BIG_ENDIAN
if ( LVAR_PAC != (DB_VARI & db->flags) ) {
#endif
if ( convert( rec, buf, my.xstr ) ) {
log_msg( LOG_ERROR, "\twhen converting MFR" );
goto cleanup;
}
#ifndef LDB_BIG_ENDIAN
} else { /* 10% faster */
Mfr *mfr = (Mfr*)buf;
short *s = &mfr->dict->tag;
int *f = rec + LMFR__FL;
int *e = f + 3*head[LMFR_NVF];
rec[LMFR_MFN] = mfr->mfn;
rec[LMFR_RECL] = mfr->recl;
/*
rec[LMFR_BWB] = mfr->bwbh<<16 | mfr->bwbl;
rec[LMFR_BWP] = mfr->bwp;
*/
rec[LMFR_BASE] = mfr->base;
rec[LMFR_NVF] = mfr->nvf;
rec[LMFR_STAT] = mfr->stat;
while ( f < e ) {
*f++ = *s++;
*f++ = *s++;
*f++ = *s++;
}
}
#endif
if ( rec[LMFR_STAT] ) {
log_msg( LOG_WARN, "found status %hd", rec[LMFR_STAT] );
goto cleanok;
}
/* do a consistency check */
if ( rec[LMFR_NVF] < 0 || rec[LMFR_BASE] < 0 ) {
log_msg( LOG_ERROR, "found neg. field nvf %hd base %hd",
rec[LMFR_NVF], rec[LMFR_BASE] );
goto cleanup;
}
/* now care for the field values */
{
char *valsrc = buf+rec[LMFR_BASE];
char *recsta = ((char*)rec);
char *valdst = recsta + base;
int xbufl = rec[LMFR_RECL] - rec[LMFR_BASE];
int sumlens = 0;
int i;
for ( i=0; i < rec[LMFR_NVF]; i++ ) {
int *d = &rec[LMFR__FL + i*LMFR__RL];
if ( d[LMFR_POS] < 0 || d[LMFR_LEN] < 0 ) {
log_msg( LOG_ERROR,
"bad field %d at off %d: negativ pos %hd or len %hd",
i, off, d[LMFR_POS], d[LMFR_LEN] );
goto cleanup;
}
if ( d[LMFR_POS] + d[LMFR_LEN] > xbufl ) {
log_msg( LOG_ERROR,
"bad field %d at off %d: pos %hd + len %hd > buf %d",
i, off, d[LMFR_POS], d[LMFR_LEN], xbufl );
goto cleanup;
}
sumlens += d[LMFR_LEN];
if ( sumlens > xbufl ) {
log_msg( LOG_ERROR,
"bad fields at off %d: sum of lengths %d > buf %d",
off, sumlens, xbufl );
goto cleanup;
}
memcpy( valdst, valsrc+d[LMFR_POS], d[LMFR_LEN] );
d[LMFR_POS] = valdst - recsta;
valdst += d[LMFR_LEN];
}
} /* consistency check */
rec[LMFR_BWB] = /* "used" bytes */
rec[LMFR_RECL] = my.xstr[LSTR_ILEN];
rec[LMFR_BWP] = rec[LMFR_NVF]; /* avail fields = used fields */
rec[LMFR_BASE] = base;
if ( LOG_TRACE <= log_lev )
LOG_STR( rec, lstrlib[LSET_MST].name[LSTR_MFR] );
goto done;
cleanup:
if ( nxtoff )
*nxtoff = -1;
cleanok:
if ( rec ) {
free( rec );
rec = 0;
}
done:
if ( buf
#ifdef alloca
&& notalloca
#endif
)
free( buf );
if ( rec )
*rec = db->head.dbid;
return rec;
} /* getMfr */
/** write the record.
If it doesn't yet have a mfn, assign one.
NOTE: on a BIG_ENDIAN, anything but the mfn and recl will be frobbed
after this call
*/
static int putMfr ( LDb *db, Mfr *mfr )
{
int oldpos, newpos;
int ret;
if ( !(db->flags & DB_WRITABLE) ) {
log_msg( LOG_ERROR, "db is not writable" );
return -1;
}
/* minimalist sanity check */
if ( mfr->nvf < 0
|| mfr->base != 18 + 6*mfr->nvf
|| mfr->base > mfr->recl
) {
log_msg( LOG_ERROR, "bad nvf/base/recl %d/%d/%d ",
mfr->nvf, mfr->base, mfr->recl );
return -2;
}
db->flags |= DB_MODIFIED;
if ( mfr->mfn ) {
int block;
if ( db->mfc[LMFC_NMFN] > mfr->mfn )
oldpos = getOff( db, mfr->mfn, 0 );
else {
db->mfc[LMFC_NMFN] = mfr->mfn+1;
oldpos = 0;
}
mfr->bwp = 511 & oldpos;
block = 1 + (oldpos >> 9); /* blockno counting from 1 */
mfr->bwbl = 0xffff & block;
mfr->bwbh = block >> 16;
if ( db->mfc[LMFC_NMFN] <= mfr->mfn )
db->mfc[LMFC_NMFN] = mfr->mfn+1;
} else {
mfr->mfn = db->mfc[LMFC_NMFN]++; /* assign new mfn */
oldpos = 0;
mfr->bwbl = mfr->bwbh = mfr->bwp = 0;
}
mfr->stat = 0;
newpos = db->mflen;
if ( 498 < (newpos & 511) ) /* round up to next block boundary */
newpos = ~511 & (newpos + 14);
if ( 1 & newpos )
newpos++;
db->mflen = newpos + mfr->recl;
#ifdef LDB_BIG_ENDIAN
{ /* swap swap swap */
Dict *d = mfr->dict;
short nvf = mfr->nvf;
SWI( mfr->mfn ); SWS( mfr->recl ); SWS( mfr->bwbl ); SWS( mfr->bwbh );
SWS( mfr->bwp ); SWS( mfr->base ); SWS( mfr->nvf ); SWS( mfr->stat );
for ( ; nvf--; d++ ) {
SWS( d->tag );
SWS( d->pos );
SWS( d->len );
}
}
#endif
ret = lio_pwrite( &db->mst[MST_MST], (char*)mfr, rvs(mfr->recl), newpos );
#ifdef LDB_BIG_ENDIAN
/* restore mnf, recl */
SWI( mfr->mfn );
SWS( mfr->recl );
#endif
if ( ret != mfr->recl )
return log_msg( ERR_TRASH, "could not write Mfr %d bytes got %d",
mfr->recl, ret );
getOff( db, mfr->mfn, (1 << (oldpos ? 9 : 10))
| (((newpos & 0xfffffe00) + 0x200) << 2) | (0x1ff & newpos) );
return 0;
} /* putMfr */
static int putRec ( LDb *db, Rec *rec )
{
int ret = 0, i;
Mfr *mfr = 0;
int buflen = 0;
int reclen = 0;
int contig = 1;
#ifdef alloca
int notalloca = 0;
#endif
const char *rbase = ((char *)rec) + rec->base;
Dict *d;
Field *f = rec->field;
/* TODO: if not rec->len, delete ? */
for ( i = rec->len; i--; f++ ) {
if ( ! f->len )
continue;
if ( ! f->val )
LOG_OTO( cleanup, ( ERR_FAULT, "bad rec NULL val" ) );
contig = contig && (f->val == rbase + buflen);
buflen += f->len;
}
reclen = 18 + 6*rec->len + buflen;
if ( 1 & reclen )
reclen++;
mfr = (Mfr*)
#ifdef alloca
alloca( reclen );
notalloca = ! mfr;
if ( notalloca )
mfr = (Mfr*)
#endif
lmalloc( reclen );
if ( ! mfr )
LOG_OTO( cleanup,
( ERR_NOMEM, "could not alloc MFR of len %hd", reclen ) );
mfr->mfn = rec->rowid;
mfr->recl = reclen;
mfr->bwbl = mfr->bwbh = mfr->bwp = 0;
mfr->base = 18 + 6*rec->len;
mfr->nvf = rec->len;
mfr->stat = 0;
d = mfr->dict;
f = rec->field;
buflen = 0;
for ( i = rec->len; i--; d++, f++ ) {
d->tag = f->tag;
d->pos = buflen;
buflen += (d->len = f->len);
}
if ( contig )
memcpy( ((char*)mfr)+mfr->base, rbase, buflen );
else {
char *mbase = ((char*)mfr)+mfr->base;
d = mfr->dict;
f = rec->field;
for ( i = rec->len; i--; d++, f++ )
if ( d->len )
memcpy( mbase + d->pos, f->val, d->len );
}
ret = putMfr( db, mfr );
if ( !ret && !rec->rowid )
rec->rowid = mfr->mfn;
cleanup:
if ( mfr
#ifdef alloca
&& notalloca
#endif
)
free( mfr );
return ret;
} /* putRec */
/* ************************************************************
utilities
*/
static int search ( LDb *db, const char *key, LdbPost *post,
Rec *rec, DXLoop *lp )
{
int i, j, prefix, idx, ret, ock;
int pos;
int *leaf, *entry;
char *term;
int *xstr;
struct { /* terms cursor */
char key[LDB_MAX_KEYLEN+1]; /* key or key prefix */
short klen; /* key length to compare */
char imin; /* minimum index to search */
char imax; /* maximum index to search */
int leaf[LDB_INDEXES][LDB_TERMBUF]; /* one leaf buffer per index */
short lpos[LDB_INDEXES]; /* next position in leaf, -1 if done */
} crs;
short klen; /* length for initial locate */
int block[128]; /* buffer to read one block */
int blockpos = 0;
LdbP *p = 0;
if ( ! key )
key = "$";
/* prepare cursor struct */
memset( &crs, 0, sizeof(crs) ); /* tabula rasa */
crs.klen = strlen( key );
/* check for prefix match */
if ( post )
prefix = LDB_PFX & post->mode;
else if ( (prefix = crs.klen && '$' == key[crs.klen - 1]) )
crs.klen--;
/* check out minimum index to search */
for ( crs.imin=0; crs.klen > db->tlen[(int)crs.imin]; )
if ( LDB_INDEXES == ++(crs.imin) )
return log_msg( ERR_INVAL, "bad keylen %d key '%.64s'", crs.klen, key );
/* prepare key */
memset( crs.key, ' ', sizeof(crs.key)-1 );
{
unsigned char *uc = (unsigned char*)crs.key;
unsigned char *uk = (unsigned char*)key;
for ( i=crs.klen; i--; )
uc[i] = db->ctab[LCS_UCASE].c[ uk[i] ];
}
if ( prefix )
crs.imax = LDB_INDEXES-1;
else {
crs.imax = crs.imin;
crs.klen = db->tlen[(int)crs.imin];
}
log_msg( LOG_INFO, "search for '%.*s'%c", crs.klen, crs.key, prefix?'$':' ' );
key = crs.key;
klen = crs.klen;
if ( rec && rec->len ) {
/* use last key from record to locate starting position */
key = rec->field[rec->len-1].val;
klen = rec->field[rec->len-1].len;
rec->len = 0;
}
for ( i=crs.imin; i<=crs.imax; i++ ) { /* find leaf positions */
int nFile = INV_N01 + 2*i; /* node file index */
int nStr = LSTR_N01 + 2*i; /* node struct index */
int *nstr = DB_XSTR( db, LSET_INV, nStr );
int lvl;
short cmplen = klen <= db->tlen[i] ? klen : db->tlen[i];
pos = db->cnt[i][LCNT_POSR]; /* pos of root record */
j = 0;
for ( lvl = 0; 0= nstr[LSTR_ILEN] );
*node = *nstr;
ret = readlog( node, db->inv[nFile], 1-pos, db, LSET_INV, nStr );
if ( pos != node[LN0X_POS] /* wrong address */
|| i+1 != node[LN0X_TYPE] /* wrong type */
|| node[LN0X_OCK] < 1 /* no keys */
|| 2*db->cnt[i][LCNT_ORDN] < node[LN0X_OCK] /* too many keys */
)
return log_msg( ERR_TRASH, "bad node pos %d type %d keys %d",
node[LN0X_POS], node[LN0X_TYPE], node[LN0X_OCK]
);
ock = node[LN0X_OCK];
for ( j=1;
jp;
xstr = DB_XSTR( db, LSET_INV, LSTR_IFP );
for (;;) { /* loop terms in prefix mode */
/* vars for postings: */
int infb, infp; /* block and pos where to read postings */
int added; /* postings added or marked per term */
int blkno; /* postings block number */
int remain = 0; /* postings to fetch from next block of segment */
int ifp[LIFP__FL]; /* postings header */
idx = -1; /* index to use */
/* compare index terms, load leafes if needed */
for ( i = crs.imin; i <= crs.imax; i++ ) {
short cmplen = klen <= db->tlen[i] ? klen : db->tlen[i];
leaf = crs.leaf[i];
if ( leaf[LL0X_OCK] <= crs.lpos[i] ) { /* load */
int lFile = INV_L01 + 2*i; /* leaf file index */
int lStr = LSTR_L01 + 2*i; /* leaf struct index */
int *lstr = DB_XSTR( db, LSET_INV, lStr );
crs.lpos[i] = -1;
reread:
if ( ! (pos = leaf[LL0X_PS]) )
continue;
LOG_DBG( LOG_DEBUG, "leaf %d", pos );
assert( (int)sizeof(crs.leaf[i]) >= lstr[LSTR_ILEN] );
*leaf = *lstr;
ret = readlog( leaf, db->inv[lFile], 1-pos, db, LSET_INV, lStr );
if ( pos != leaf[LL0X_POS] /* wrong address */
|| i+1 != leaf[LL0X_TYPE] /* wrong type */
|| leaf[LL0X_OCK] < 1 /* no keys */
|| 2*db->cnt[i][LCNT_ORDN] < leaf[LL0X_OCK] /* too many keys */
)
return log_msg( ERR_TRASH, "bad leaf pos %d type %d keys %d",
leaf[LL0X_POS], leaf[LL0X_TYPE], leaf[LL0X_OCK] );
ock = leaf[LL0X_OCK];
/* advance to first term which is not too small
(should be needed only for first leaf of an index)
*/
for ( j=0;
j crs.lpos[i] )
continue;
if ( 0 > idx ) {
idx = i;
continue;
}
/* compare this index next term to that of index idx */
/* assume that index w/ lower number has shorter keys */
ret = memcmp(
((char*)leaf)+leaf[LL0X__FL + crs.lpos[i]*LL0X__RL + LL0X_KEY],
((char*)crs.leaf[idx])+
crs.leaf[idx][LL0X__FL + crs.lpos[idx]*LL0X__RL + LL0X_KEY],
db->tlen[idx] );
if ( 0 > ret )
idx = i;
}
if ( 0 > idx )
goto done;
j = crs.lpos[idx];
leaf = crs.leaf[idx];
entry = leaf + LL0X__FL + j*LL0X__RL;
term = ((char*)leaf) + entry[LL0X_KEY];
if ( memcmp( crs.key, term, crs.klen ) )
goto done;
crs.lpos[idx]++;
if ( rec ) { /* record the term */
/* field to assign */
Field *f = rec->field + rec->len;
short tlen = db->tlen[idx];
/* end of available buffer */
char *b = rec->len
? (char*)f[-1].val /* before previously assigned field */
: ((char*)rec + rec->bytes); /* end of record */
while ( tlen && ' ' == term[tlen-1] )
tlen--;
b -= tlen;
if ( b < (char*)(f+1) ) /* no space left on device */
goto done;
/* probably we're nuking the locator now: */
memcpy( b, term, tlen );
f->tag = 0;
f->val = b;
f->len = tlen;
rec->len++;
/* reset key from locator to prefix */
key = crs.key;
klen = crs.klen;
}
if ( ! post && ! lp )
continue;
/* collect postings */
infb = entry[LL0X_INFB];
infp = entry[LL0X_INFP];
/* the IFP file is organized in blocks of 128 longs.
1st int is block number followed by 127 data.
postings are organized in chained segments so that each segment
fits within one such block. a segment has five longs header,
giving number of postings and pointer to next segment.
*/
added = 0;
for ( blkno=0; infb; blkno++ ) { /* segments */
LdbP merge[127/2]; /* buffer to collect new postings */
int *base; /* start of data */
int *b; /* start of postings */
int n; /* max postings in this seg's 1st block */
int xlen; /* external length to read */
int f = post ? post->fil - 1 : 0; /* highest pos to consider in given postings */
int m = 0; /* fill merge buffer */
int k; /* loop segment */
if ( infp > 127-2-5 ) {
return log_msg( ERR_TRASH, "found bad IFP pos %d blk %d for %.*s",
infp, blkno, klen, key );
}
if ( remain ) { /* consecutive block of same segment */
n = remain;
if ( n > 127/2 )
n = 127/2;
xlen = 8*n;
} else {
n = (127 - 5 - infp)/2;
xlen = 20 + 8*n;
}
pos = (infb - 1) * 512 + (infp + 1) * 4;
if ( blockpos
&& !((pos-blockpos) >> 9) /* 0 <= (pos-blockpos) < 512 */
&& pos+xlen <= blockpos+ 1 + (0x1ff & ~blockpos)
)
base = block + (pos - blockpos)/sizeof(int);
else {
int blklen = 1 + (0x1ff & ~pos);
assert( xlen <= blklen );
assert( blklen <= (int)sizeof(block) );
assert( 0 == (0x1ff & (pos + blklen)) );
ret = readblk( block, blklen, db->inv[INV_IFP], pos );
if ( ret )
return log_msg( ERR_IO, "\twhen reading IFP" );
blockpos = pos;
base = block;
}
if ( remain ) { /* no header to convert */
remain -= n;
b = base; /* no header */
} else {
assert( (int)sizeof(ifp) >= xstr[LSTR_ILEN] );
*ifp = *xstr;
ret = convert( ifp, (char *)base, xstr );
if ( ret )
return log_msg( ERR_TRASH, "\twhen converting IFP header" );
if ( n > ifp[LIFP_SEGP] )
n = ifp[LIFP_SEGP];
remain = ifp[LIFP_SEGP] - n;
b = base+5; /* after header */
}
LOG_DBG( LOG_VERBOSE,
"key %d.%d '%.*s' blk %d post %d/%d r %d xlen %d at b/p %d.%d=%d",
leaf[LL0X_PS], j, db->tlen[idx], term, blkno,
n, ifp[LIFP_TOTP], remain, xlen, infb, infp, pos );
if ( LOG_DO( LOG_TRACE ) )
LOG_STR( ifp, lstrlib[ db->set[LSET_INV] ].name[ LSTR_IFP ] );
assert( (size_t)n <= sizeof(merge)/sizeof(merge[0]) );
if ( lp ) {
Key kbf;
Hit hit;
unsigned char tlen = (unsigned char) db->tlen[idx];
while ( tlen && ' ' == term[tlen-1] )
tlen--;
memcpy( kbf.byt, term, kbf.len = tlen );
for ( k=0; k> 16);
hit.pos = (unsigned short)ppos;
if ( lp->cb( lp->me, &kbf, &hit ) )
goto done;
}
}
if ( post ) for ( k=n; k--; ) {
/* loop backwards (for the fun of it) postings in segment */
int prow, ptag, ppos;
unsigned char *c = (unsigned char *)&b[k*2];
LdbP e; /* the entry */
LdbP samerow; /* highest possible entry w/ same row as e */
#if defined(LDB_BIG_ENDIAN )
/* the 8 bytes of a posting are BIG ENDIAN ! */
memcpy(e.bytes,c,8);
#else
e.bytes[0] = c[7]; e.bytes[1] = c[6];
e.bytes[2] = c[5]; e.bytes[3] = c[4];
e.bytes[4] = c[3]; e.bytes[5] = c[2];
e.bytes[6] = c[1]; e.bytes[7] = c[0];
#endif
prow = LDBP_ROW( &e );
ptag = LDBP_TAG( &e );
ppos = LDBP_POS( &e );
LOG_DBG( LOG_VERBOSE, "post %d.%hd pos %06x key '%.*s'",
prow, ptag, ppos, db->tlen[idx], term );
if ( 0 >= ptag /* bad tag */
|| !prow || prow >= db->mfc[LMFC_NMFN] /* bad mfn */
)
continue;
if ( ! post
|| (post->cut && prow >= post->cut)
|| (post->tag && post->tag != ptag)
)
continue;
if ( prow < post->skp ) /* quickly bail out on skip condition */
break;
LDBP_SETROWTOP( &samerow, &e ); /* for mfn comparison */
/* sweep down to postings for the same row as e ... */
while ( f >= 0 && LDBP_GT( p+f, &samerow ) )
f--;
if ( LDB_AND & post->mode ) {
int l;
/* loop postings for same row, mark all (that are near enough) */
LDBP_SETROWBOT( &samerow, &e ); /* for mfn comparison */
/* NOTE: postings for row are GT than bottom even if marked */
for ( l = f; l>=0 && LDBP_GT( p+l, &samerow ); l-- ) {
if ( post->near ) {
int dist;
if ( ptag != LDBP_TAG( p+l ) ) continue;
if ( LDB_NEAR_G != post->near ) {
dist = LDBP_POS( p+l ) - LDBP_POS( &e );
if ( dist < 0 ) dist = -dist;
if ( 0 < post->near
? post->near < dist
: -post->near != dist /* exact $$$$ */
) continue;
}
}
LDBP_SETMARK( p+l );
added++;
}
} else { /* OR mode */
int add;
if ( ! post->near ) /* add if row not found: ignore details */
add = 0 > f || prow > LDBP_ROW( p+f );
else { /* add if no exact match */
int l;
/* NOTE: we don't use mark bit in OR mode, do we ? */
for ( l = f; l>=0 && LDBP_GT( p+l, &e ); l-- )
;
add = 0 > l || LDBP_GT( &e, p+l );
}
if ( add )
merge[ m++ ] = e;
}
} /* for postings in segment */
if ( m ) { /* merge in the merge buffer */
LdbP *mm = merge;
added += m;
for ( k = post->fil += m; m && k--; ) {
LdbP src;
if ( k < m || LDBP_GT( mm, &p[k-m] ) ) {
src = *mm++;
m--;
LOG_DBG( LOG_DEBUG, "merging %d at %d", LDBP_ROW(&src), k );
} else
src = p[k-m];
if ( k < post->len )
p[k] = src;
else { /* set cut */
int row = LDBP_ROW( &src );
if ( row < post->cut || !post->cut )
post->cut = row;
}
}
if ( post->fil > post->len )
post->fil = post->len;
if ( post->cut ) /* postings for cut row are unreliable */
while ( post->fil && post->cut <= LDBP_ROW(p+post->fil-1) )
post->fil--;
}
if ( remain ) { /* advance to start of next block */
infb++;
infp = 0;
} else {
infb = ifp[LIFP_NXTB];
infp = ifp[LIFP_NXTP];
}
} /* for segments */
LOG_DBG( LOG_VERBOSE, "added %d postings for key '%.*s'",
added, db->tlen[idx], term );
} /* for terms in prefix/postings mode */
done:
if ( post /* fixup */
&& LDB_AND & post->mode && !(LDB_KEEPMARKS & post->mode)
) {
int mark = LDB_NOT & post->mode ? 0 : 0x8000;
j=0;
for ( i=0; ifil; i++ )
if ( mark == LDBP_MARK(p+i) ) {
LDBP_CLRMARK(p+i);
p[j++] = p[i];
}
post->fil = j;
}
return ! rec ? 0 : rec->len;
} /* search */
static int is_abs_path (const char *path) {
if (! path || ! *path) {
return 0;
}
if ('/' == *path) {
return 1;
}
#ifdef WIN32
if ('\\' == *path) {
return 1;
}
if (':' == path[1] && ('\\' == path[2] || '/' == path[2])) {
return 1;
}
#endif
return 0;
}
static int ldb_open (const char *dbname, Rec *dbpar, Rec *syspar)
{
LDb ndb, *db;
int plen, sz, dbid;
int ret = 0, invret = 0, lbtret = 0, autoformat = 1, writable = 1;
char buf[4096], *p = buf;
char path[ PATH_MAX ];
memset( &ndb, 0, sizeof(ndb) );
/* these should be 0 by memsetting to 0 anyway ... */
ndb.path = 0; ndb.mmap = 0;
ndb.flags |= DB_MMAP; /* it mean's: we'll try */
/* loglevel */ {
int llev = rInt2(dbpar, syspar, OPENISIS_SLOGV, -1);
if ( 0 <= llev )
cLog( llev, 0 );
}
if ( dbname ) {
if (DBNLEN <= strlen (dbname)) {
strncpy (ndb.head.name, dbname, DBNLEN - 1) [DBNLEN - 1] = 0;
log_msg( LOG_WARN, "truncating dbname '%s' to '%s'",
dbname, ndb.head.name );
} else
strcpy (ndb.head.name, dbname);
} else {
if (! dbpar && ! syspar)
return -ERR_FAULT;
if (! rString2 ( dbpar, syspar, OPENISIS_DNAME, path, sizeof(path)))
return -ERR_FAULT;
if (DBNLEN <= strlen (path)) {
strncpy (ndb.head.name, path, DBNLEN - 1) [DBNLEN - 1] = 0;
log_msg( LOG_WARN, "truncating dbname '%s' to '%s'",
path, ndb.head.name );
} else
strcpy (ndb.head.name, path);
}
/* more init AFTER honoring verbosity */
if ( ! init ) {
lstr_auto(0);
init = !0;
}
for ( dbid=0; dbidhead.dbid = dbid;
db->set[ LSET_MST ] = LSET_MST;
db->set[ LSET_INV ] = LSET_INV;
if (dbpar || syspar) {
char fmtstr[32];
if (rString2 (dbpar, syspar, OPENISIS_DTYPE, fmtstr, sizeof(fmtstr))) {
if (! strcmp ("aligned", fmtstr)) {
db->flags |= LVAR_ALI;
autoformat = 0;
}
else if (! strcmp ("naligned", fmtstr)) {
autoformat = 0;
}
}
}
/* preset record sizes */
db->mfc[0] = *DB_XSTR( db, LSET_MST, LSTR_MFC );
db->xrf[0] = *DB_XSTR( db, LSET_MST, LSTR_XRF );
db->cnt[0][0] =
db->cnt[1][0] = *DB_XSTR( db, LSET_INV, LSTR_CNT );
/* isis-1 index term lengths */
db->tlen[0] = 10;
db->tlen[1] = 30;
/* only the packed little endian ("DOS") format is writable
test later ...
if ( LVAR_PAC != (DB_VARI & db->flags) )
writable = 0;
*/
/* prepare path */
plen = strlen (db->head.name);
*path = 0;
if (dbpar || syspar) {
char dbpath[PATH_MAX];
int slen = 0, dlen = 0;
if ( rString2 (dbpar, syspar, OPENISIS_DPATH, dbpath, sizeof(path)) ) {
dlen = strlen (dbpath);
if (! dlen || ! is_abs_path (dbpath)) {
if (rString2 (dbpar, syspar, OPENISIS_SPATH,
path, sizeof(path))) {
slen = strlen (path);
}
}
}
if (sizeof(path) < (unsigned int)
(slen + 1 + dlen + 1 + plen + 4 + 1)) {
return log_msg( ERR_FAULT, "path(%d+%d) or filename(%d) too long",
slen, dlen, plen);
}
if (slen) {
path[slen++] = '/';
}
if (dlen) {
memcpy (path + slen, dbpath, dlen);
path[slen += dlen] = '/';
++slen;
}
memcpy (path + slen, db->head.name, plen);
path[slen += plen] = 0;
plen = slen;
}
else {
memcpy (path, db->head.name, plen);
path[plen] = 0;
}
/* save params */
db->path = strdup( path );
memcpy( path+plen, ".???", 5 );
if (dbpar || syspar) {
int ro = rInt2 (dbpar, syspar, OPENISIS_DRO, -1);
if (0 < ro) {
writable = 0;
}
}
/* open files */
if ( writable
&& !openfiles( db->mst, path, EXT_MST, MST_FILES, OPEN_WR )
)
ret = 0;
else {
writable = 0;
ret = openfiles( db->mst, path, EXT_MST, MST_FILES, OPEN_RD );
if ( ret )
goto cleanup;
}
/* XXX: MMMOOO */
if ( writable
&& !openfiles( &db->oxi.fd, path, EXT_LBT, 1, OPEN_WR )
)
lbtret = 0;
else {
writable = 0;
lbtret = openfiles( &db->oxi.fd, path, EXT_LBT, 1, OPEN_RD );
}
/* trad. index is never openend writable. */
invret = openfiles( db->inv, path, EXT_INV, INV_FILES, OPEN_RD );
if ( autoformat ) {
if ( invret )
log_msg( LOG_WARN, "cannot guess format -- no inverted file" );
else {
unsigned len = lio_size( db->inv[INV_CNT] );
if ( 56L == len ) {
db->flags |= LVAR_ALI;
/* writable = 0; we do not write aligned format */
} else if ( 52L != len )
log_msg( LOG_WARN, "cannot guess format -- bad .cnt len %d", len );
log_msg( LOG_INFO, "using autoformat %saligned for .cnt len %d",
(db->flags & LVAR_ALI) ? "":"un", len );
}
}
if ( (ret = readlog( db->mfc, db->mst[MST_MST], 0, db, LSET_MST, LSTR_MFC )) )
{
/* NEW goto cleanup; */
memset( db->mfc, 0, sizeof(db->mfc) );
db->mfc[LMFC_NMFN] = 1;
db->mfc[LMFC_NMFB] = 1;
db->mfc[LMFC_NMFP] = 64;
db->mflen = 64;
} else {
int lastblock = (db->mflen = lio_size( db->mst[MST_MST] ))/512;
if ( 511 & db->mflen ) lastblock++; /* counting from 1 */
/*
the next record's block should be either the last one we have
or the next one to follow
*/
if ( db->mfc[LMFC_NMFB] != lastblock && db->mfc[LMFC_NMFB] != lastblock+1 )
log_msg( LOG_VERBOSE, "NMFB mismatch: NMFB %d ~ %d",
db->mfc[LMFC_NMFB], lastblock );
/* set LOGICAL mf length */
db->mflen = (db->mfc[LMFC_NMFB]-1)*512 + db->mfc[LMFC_NMFP];
}
db->xrlen = lio_size( db->mst[MST_XRF] ) / 512;
if ( (DB_MMAP & db->flags)
&& db->xrlen
&& db->xrlen*512
== lio_mmap( &db->mst[MST_XRF], (void**)&db->mmap, db->xrlen*512 )
)
db->mmlen = db->xrlen;
else
db->flags &= ~DB_MMAP;
/* supporting files, ctables */
if ( 0 >= (sz = lio_slurp( &p, sizeof(buf), setext(path,EXT_SUP_ACT), 1 ))
|| lcs_mktab( db->ctab+LCS_CTYPE, p, sz, LCS_A )
)
memcpy( db->ctab+LCS_CTYPE, lcs_latin1_ct, sizeof(db->ctab[0]) );
if ( 0 >= (sz = lio_slurp( &p, sizeof(buf), setext(path,EXT_SUP_UCT), 1 ))
|| lcs_mktab( db->ctab+LCS_UCASE, p, sz, 0 )
)
memcpy( db->ctab+LCS_UCASE, lcs_latin1_uc, sizeof(db->ctab[0]) );
/* fill header */
{
struct timeval tv;
Fd *fds;
Rec *rec;
int num = fdtFromFile (path, &fds, &rec);
if (0 < num) {
db->head.fdt = cFdt (ses0, num, fds, rec);
}
num = gettimeofday (&tv, 0);
if (num) {
log_msg (LOG_WARN, "dbopen(%s): gettimeofday returns %d %d\n",
db->head.name, num, errno);
db->head.tms = 0;
}
else {
db->head.tms = (int) tv.tv_sec;
}
if (dbpar) {
db->head.cfg = rDup (ses0, dbpar, 0, 0);
}
}
/* done */
db->flags |= DB_OPEN;
if ( writable )
db->flags |= DB_WRITABLE;
if (dbpar || syspar) {
int dmp = rInt2 (dbpar, syspar, OPENISIS_DDUMP, -1);
if (0 < dmp) {
int off = 0;
int *r;
do {
if ( (r = ldb_readRecAtOff(dbid,off,&off)) )
free( r );
} while ( 0 < off );
exit(0);
}
}
if ( writable )
db->oxi.flg |= LBT_WRITE;
{
char *oxityp = getenv("OXITYP");
int ot;
if ( oxityp && 0 < (ot = atoi(oxityp)) && 4 > ot )
db->oxi.typ = ot << 4;
}
if ( ! lbtret
&& ! lbt_init( &db->oxi )
)
db->flags |= DB_LBTOPEN;
if ( LVAR_PAC != (DB_VARI & db->flags) )
writable = 0;
if ( ! invret
&& ! (ret = readlog( db->cnt[0], db->inv[INV_CNT],
0, db, LSET_INV, LSTR_CNT ))
&& ! (ret = readlog( db->cnt[1], db->inv[INV_CNT],
-1, db, LSET_INV, LSTR_CNT ))
)
db->flags |= DB_INVOPEN;
return dbid;
cleanup:
/* cleanup ... */
db->flags = 0; /* MW: creation mode? */
return 0 > ret ? ret : ret ? -ret : -1;
} /* ldb_open */
/* ************************************************************
package data
*/
/* ************************************************************
package functions
*/
int *ldb_readRecAtOff ( int dbid, lxref off, int *nxtoff )
{
int *rec;
LDb *db = getDb( dbid );
if ( ! db ) {
log_msg( LOG_ERROR, "\tat ldb_readRecAtOff" );
return 0;
}
if ( 0 == off )
off = 64;
rec = getMfr( db, off, nxtoff );
if ( ! rec )
return 0;
LOG_DBG( LOG_VERBOSE, "db %d off %d: got %hd bytes",
dbid, off, !rec ? -1 : rec[LMFR_RECL] );
return rec;
} /* ldb_readRecAtOff */
int ldb_search ( int dbid, const char *key, LdbPost *post, Rec *rec )
{
LDb *db = getDb( dbid );
Key k;
if ( ! db )
return -ERR_BADF;
if ( post ) { /* prepare for postings */
if ( ! post->len )
post->len = sizeof(post->p)/sizeof(post->p[0]); /* standard length */
if ( LDB_NOT & post->mode )
post->mode |= LDB_AND;
}
if ( DB_INVOPEN & db->flags )
return search( db, key, post, rec, 0 );
if ( !(DB_LBTOPEN & db->flags) )
return -ERR_BADF;
if ( db->oxi.bat )
return -ERR_BUSY;
memset( &k, 0, sizeof(k) );
if ( ! key ) {
k.byt[0] = '$';
k.len = 1;
} else {
unsigned char *uk = (unsigned char*)key;
int l = strlen( key );
if ( l > 255 )
l = 255;
k.len = (unsigned char)l;
while ( l-- )
k.byt[l] = db->ctab[LCS_UCASE].c[ uk[l] ];
}
return lbt_search( &db->oxi, &k, post, rec );
} /* ldb_search */
int ldb_p2s ( Set *set, LdbPost *post )
{
int *s = set->id;
int last=0, max = set->len;
int i;
set->len = 0;
if ( ! max )
max = OPENISIS_SETLEN;
max--;
if ( !post->fil )
return 0L;
s[0] = LDBP_ROW(post->p);
for ( i=1; ifil && last < max; i++ ) {
int row = LDBP_ROW(post->p+i);
if ( s[last] != row )
s[++last] = row;
}
return set->len = last+1;
} /* ldb_p2s */
#if 0
LcsTab *ldb_tabs( int dbid )
{
LDb *db = getDb( dbid );
return ! db ? 0 : db->ctab;
} /* ldb_tabs */
#endif
Db *ldb_getdb (int dbid) {
LDb *db = getDb (dbid);
return db ? &db->head : 0;
}
/* ************************************************************
public functions
*/
int dMaxId ( Session ses, int dbid )
{
LDb *db = getDb( dbid );
(void)ses;
if ( ! db )
return -ERR_BADF;
return db->mfc[LMFC_NMFN] - 1;
} /* dMaxId */
Raw *dRaw ( Session ses, int dbid, int rowid )
{
int off;
int *rec = 0;
LDb *db;
(void)ses;
if ( LIO_LOCK() ) return 0;
db = getDb( dbid );
if ( ! db ) {
log_msg( LOG_ERROR, "\tat openIsisReadRaw %d", rowid );
goto done;
}
off = getOff( db, rowid, 0 );
log_msg( LOG_INFO, "found xref 0x%08x for %d", off, rowid );
if ( 0 >= off ) {
log_msg( LOG_INFO, "found deleted xref 0x%08x for %d", off, rowid );
goto done;
}
rec = getMfr( db, off, 0 );
if ( ! rec ) {
log_msg( LOG_WARN, "\tno record at %d rowid %d", off, rowid );
goto done;
}
LOG_DBG( LOG_VERBOSE, "db %d row %d: got %hd bytes",
dbid, rowid, !rec ? -1 : rec[LMFR_RECL] );
if ( rec[LMFR_MFN] != rowid ) {
log_msg( LOG_ERROR, "got mfn %d expected %d", rec[LMFR_MFN], rowid );
free( rec );
goto done;
}
done:
(void)LIO_RELE();
return (Raw*)rec;
} /* dRaw */
int dWritex ( Session ses, int dbid, Rec *rec, Rec *idx )
{
LDb *db = getDb( dbid );
int ret = 0;
(void)ses;
if ( ! db )
return -ERR_BADF;
if ( !(DB_WRITABLE & db->flags) )
return log_msg( ERR_INVAL, "db %d not writable", dbid );
if ( rec && (ret = putRec( db, rec )) )
return ret;
if ( idx ) {
const unsigned char *const uc = db->ctab[LCS_UCASE].c;
Hit h;
Key k;
Field *f = idx->field, *last = f + idx->len - 1;
for ( ; f <= last; f++ ) {
int tag = f->tag;
int del = 0 > tag;
if ( del )
tag = -tag;
else if ( ! tag )
continue; /* ??? */
memset( &k, 0, sizeof(k) ); /* we're so paranoid ! */
switch ( tag ) {
case XVAL: /* binary val */
/* if ( f->len < db->oxi.vsz )
memset( k.val.byt, 0, db->oxi.vsz - f->len );
*/
memcpy( k.val.byt
+ (f->len < (int)db->oxi.vsz ? (int)db->oxi.vsz - f->len : 0),
f->val, f->len > (int)db->oxi.vsz ? (int)db->oxi.vsz : f->len );
k.val.len = db->oxi.vsz;
break;
case XHIT:
if ( sizeof(Hit) != f->len )
return log_msg( ERR_INVAL, "bad length %d for XHIT need %d",
f->len, sizeof(Hit) );
memcpy( &h, f->val, sizeof(Hit) );
if ( !h.mfn && rec )
h.mfn = rec->rowid;
break;
case XDOT: {
int i = 0, v[5], *pv = v;
const char *p = f->val, *e = p + f->len;
for ( ; p < e && i<5; i++ ) {
int dig = a2il( p, e-p, v+i );
p += dig;
if ( ! dig || (e > p && '.' != *p) )
return log_msg( ERR_INVAL, "bad XDOT '%.*s'", e-p, p );
p++;
}
memset( &h, 0, sizeof(h) );
switch ( i ) {
case 5: h.dbn = (unsigned short)*pv++;
case 4: h.mfn = (unsigned)*pv++;
case 3: h.pos = (unsigned short)pv[2];
case 2: h.occ = (unsigned short)pv[1];
case 1: h.tag = (unsigned short)pv[0];
/* case 0: ! f->len */
}
if ( 4 > i && rec )
h.mfn = rec->rowid;
} break; /* case XDOT */
case XFST:
return log_msg( ERR_IDIOT, "sorry, XFST not implemented" );
}
if ( ! k.val.len )
cXMkVal( &db->oxi, &k.val, &h );
while ( f < last && !f[1].tag ) { /* collect XKEYs */
int l = (++f)->len;
unsigned char *dst = k.byt;
const unsigned char *src = (const unsigned char *)f->val;
if ( 255 < l )
l = 255;
k.len = (unsigned char)l;
while ( l-- )
*dst++ = uc[ *src++ ];
LOG_DBG( LOG_DEBUG,
"field %d key '%.*s' hit %d.%d.%d.%d.%d",
f - idx->field, k.len, k.byt,
h.dbn, h.mfn, h.tag, h.occ, h.pos );
ret = del ? lbt_del( &db->oxi, &k ) : lbt_add( &db->oxi, &k );
}
}
}
return ret;
} /* dWritex */
int dWrite ( Session ses, int dbid, Rec *rec )
{
/* TODO: use FST lines as idx */
return dWritex( ses, dbid, rec, 0 );
} /* dWrite */
Rec* dTerm ( Session ses, Rec *rec, int dbid, const char *key )
{
(void)ses;
return 0 > ldb_search( dbid, key, 0, rec ) ? 0 : rec;
} /* dTerm */
int dXLoop ( Session ses, int dbid, DXLoop *l )
{
LDb *db = getDb( dbid );
(void)ses;
if ( !db )
return -ERR_BADF;
if ( OPENISIS_IDXTRAD & l->flg ) {
if ( !(db->flags & DB_INVOPEN) )
return -ERR_BUSY;
return search( db, 0, 0, 0, l );
}
if ( !(db->flags & DB_LBTOPEN) || db->oxi.bat )
return -ERR_BUSY;
return lbt_loop( & db->oxi, l );
} /* dXLoop */
int cInit ( int argc, const char **argv, CLockFunc lockfunc )
{
(void)argc; (void)argv;
cOpen( 0 );
if ( lockfunc )
lio_lock = lockfunc;
return 0;
}
Db* cDOpen (const char *dbname, Rec *dbpar, Rec *syspar) {
int dbid;
if ( ! init )
cOpen( 0 );
dbid = ldb_open (dbname, dbpar, syspar);
if (0 <= dbid) {
return &dbs[dbid].head;
}
return 0;
}
int cDOpenv ( const char *dbname, const char **argv, int argc )
{
Rec *parms = 0;
int rt;
if ( ! init )
cOpen( 0 );
if (argc) {
if (argc & ~RARGC) {
log_msg (LOG_WARN, "illegal argc %d for dbopenv %s",
argc, dbname ? dbname : "");
}
else {
parms = rSet (ses0, 0,
RARGV | RFDT | RNOC | RIGN | argc,
openIsisSyspar, argv);
}
}
rt = ldb_open (dbname, parms, 0);
if (parms) {
mFree (ses0, parms);
}
return rt;
}
int cDClose ( int dbid )
{
LDb *db = getDb( dbid );
if ( ! db )
return -ERR_BADF;
if ( LIO_LOCK() ) return -ERR_BUSY;
if ( DB_MODIFIED & db->flags ) {
/* write back the MF control */
Mfc mfc;
/* if ( 498 < (db->mflen & 511) ) db->mflen = ~511 & (db->mflen + 14); */
mfc.ctlm = rvi( db->mfc[LMFC_CTLM] );
mfc.nmfn = rvi( db->mfc[LMFC_NMFN] );
mfc.nmfb = rvi( 1 + (db->mflen >> 9) );
mfc.nmfp = rvs( 511 & db->mflen );
mfc.type = rvs( db->mfc[LMFC_TYPE] );
mfc.rcnt = rvi( db->mfc[LMFC_RCNT] );
mfc.mfx1 = rvi( db->mfc[LMFC_MFX1] );
mfc.mfx2 = rvi( db->mfc[LMFC_MFX2] );
mfc.mfx3 = rvi( db->mfc[LMFC_MFX3] );
if ( sizeof(mfc) != lio_pwrite( &db->mst[MST_MST], &mfc, sizeof(mfc), 0) )
log_msg( ERR_TRASH, "could not write MST header" );
}
if ( db->mmap )
lio_mmap( 0, (void**)&db->mmap, db->mmlen*512 );
db->mmlen = 0;
closefiles( db->mst, MST_FILES );
if ( DB_INVOPEN & db->flags )
closefiles( db->inv, INV_FILES );
db->flags = 0L;
if ( db->path ) free( (char*)db->path );
if (db->head.cfg) mFree (ses0, db->head.cfg);
if (db->head.fdt) cFreeFdt (ses0, db->head.fdt);
memset( db, 0, sizeof(db) );
(void)LIO_RELE();
return 0;
} /* cDClose */
int cDCheck ( int dbid, int flags )
{
static char dot = '.';
int *r;
LDb *db = getDb( dbid );
int nxtoff = 64, off;
if ( ! db )
return -ERR_BADF;
(void)flags;
do {
lio_write( &lio_out, &dot, 1 );
if ( (r = ldb_readRecAtOff(dbid,off=nxtoff,&nxtoff)) ) {
int o = getOff( db, r[LMFR_MFN], 0 );
if ( o != off ) {
log_msg( LOG_WARN, "mfn %d xrf %d != real %d\n",
r[LMFR_MFN], o, off );
}
free( r );
}
} while ( 0 < nxtoff );
return 0;
} /* cDCheck */
OpenIsisIdx *cXOpen ( int dbid, int mode )
{
LDb *db = getDb( dbid );
if ( !db
|| !(db->flags & DB_LBTOPEN)
|| !(db->oxi.flg & LBT_WRITE) /* may be writable if db is not */
|| db->oxi.bat
/*
preliminary undocumented feature:
mode -1 gives direct access in non-batch mode
*/
|| (0 <= mode && lbt_batch( & db->oxi, (unsigned char)mode ))
)
return 0;
return & db->oxi;
} /* cXOpen */
ldb.h 0100644 0001130 0001130 00000013265 07630630466 010466 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef LDB_H
/*
$Id: ldb.h,v 1.18 2003/03/03 10:40:21 kripke Exp $
package interface of the general db access functions.
*/
#include "luti.h"
#include "lll.h"
/* #define LDB_MAGIC 0x0FE91515 read OPENISIS */
enum { /* MFR members */
LMFR_MFN = 1, /* the rowid */
LMFR_RECL, /* total external length (even), negative for locked record */
LMFR_BWB, /* block of rec's previous version */
LMFR_BWP, /* pos of " */
LMFR_BASE, /* offset of contents area */
LMFR_NVF, /* number of fields */
LMFR_STAT, /* state, if != 0, about to be deleted */
LMFR__FL, /* offset of repeated part */
LMFR_TAG = 0,
LMFR_POS,
LMFR_LEN,
LMFR__RL /* length of repeated part */
};
/**
read a raw isis record by rowid.
the memory must be freed after usage.
*/
extern int *ldb_readRec ( int db, int rowid );
/**
numerical type for a records file position.
To support large DBs, this may be lll.
*/
typedef int lxref;
/**
read a raw isis record by xref.
the memory must be freed after usage.
*/
extern int *ldb_readRecAtOff ( int dbid, lxref off, int *nxtoff );
/*
** ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
** detailed search structures and functions
**
** ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
*/
enum {
LDB_MAX_KEYLEN = 31, /* actually, it's 30 for isis-1 */
LDB_TERMBUF = 128, /* ints */
LDB_INDEXES = 2
};
/**
the isis-1 posting is an 64bit number.
In the file it's in big endian order, to allow ordering using memcmp.
In the LdbP we have native order to use native 64bit.
Until we need more bits for postings, we may use the 64bit type
provided by gcc and some other compilers.
the 8 bytes are: highest 3 for the rowid, then 2 tag, then 3 pos.
the 3 pos bytes in turn are 1 byte for the occurence
and 2 bytes for the word count.
(e.g. value 2<<16 | 7 if term was found in the 7th word of
the 2nd occurence of field 'tag').
we usually don't care, since for a near condition we always
want a small distance w/in the same occurence which may be
conveniently checked by the difference of 3 byte pos values.
BUT NOTE: ... unfortunately we DO NEED one more bit for marking
postings during AND operations. we abuse the highest bit of the
word counter, since it's for sure useless for any reasonable
distance check.
*** BE CAREFUL WHEN COMPARING POSTINGS DURING "AND" OP ***
use the LDBP_xxM macros below.
*/
typedef union {
char bytes[8];
lll val;
} LdbP;
/* all macros operate on pointers */
/* access parts */
#define LDBP_ROW( p ) (0xffffffL & (int)((p)->val >> 40))
#define LDBP_TAG( p ) ( 0xffffL & (int)((p)->val >> 24))
#define LDBP_POS( p ) (0xff7fffL & (int)((p)->val))
#define LDBP_MARK( p ) ( 0x8000 & (int)((p)->val))
#define LDBP_SETMARK( p ) ((p)->val |= LLL( 0x8000))
#define LDBP_CLRMARK( p ) ((p)->val &= LLL(0xffffffffffff7fff))
/* value w/o the mark bit */
#define LDBP_IGNMARK( p ) ((p)->val & LLL(0xffffffffffff7fff))
/* compare according to rowid, tag, pos */
#define LDBP_EQ( a, b ) ((a)->val == (b)->val)
#define LDBP_GT( a, b ) ((a)->val > (b)->val)
/* compare IGNORING MARK according to rowid, tag, pos */
#define LDBP_EQM( a, b ) (LDBP_IGNMARK(a) == LDBP_IGNMARK(b))
#define LDBP_GTM( a, b ) (LDBP_IGNMARK(a) > LDBP_IGNMARK(b))
/* manipulation: set bottom / top for row of given posting */
#define LDBP_SETROWBOT( d, s ) ((d)->val = (s)->val & LLL(0xffffff0000000000))
#define LDBP_SETROWTOP( d, s ) ((d)->val = (s)->val | LLL( 0xffffffffff))
/**
postings come in arrays with some header.
The standard structure gives a 8k buffer,
but member len may give other actual length.
*/
typedef struct {
short mode; /* in: merge flags */
short near; /* in: near distance; in OR mode: collect pos info */
int tag; /* in: tag, to which postings are restricted */
int skp; /* in: ignore mfns < skp */
int len; /* in: length (# of postings) of buffer (if 0 : default length) */
int fil; /* io: number of postings actually used */
int cut; /* io: min mfn ignored due to buffer length */
LdbP p[OPENISIS_SETLEN];
/**
8 bytes as in IFP file: mfn[3],tag[2],occ[1],cnt[2]
highest bit of cnt (1LL<<15) is used as mark
*/
} LdbPost;
enum {
LDB_OR,
LDB_AND,
LDB_NOT, /* like AND, but keep unmarked postings */
LDB_PFX = 4, /* prefix match */
LDB_KEEPMARKS = 8 /* do not compact after AND/NOT */
};
enum {
LDB_NEAR_F = 0x7fff, /* the (F): same occurence of field */
LDB_NEAR_G = -0x8000 /* the (G): same field */
};
/**
*/
extern int ldb_search ( int db, const char *key, LdbPost *post,
OpenIsisRec *rec );
extern int ldb_p2s ( OpenIsisSet *set, LdbPost *post );
extern Db* ldb_getdb (int dbid);
#define LDB_H
#endif /* LDB_H */
ldsp.c 0100644 0001130 0001130 00000025771 07635347015 010667 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2003 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: ldsp.c,v 1.13 2003/03/17 09:05:25 mawag Exp $
server side request processing.
*/
#include
#include
#include /* vsnprintf */
#include
#include "openisis.h"
#include "loi.h"
#include "lcom.h"
#include "ldb.h"
#include "lfdt.h"
#include "luti.h"
#ifdef WIN32
#define vsnprintf _vsnprintf
#endif
#define ISID 0
#define ISER 1
#define ITMS 2
#define SSTS 3
#define RSPVAR \
Rec *rsp = 0; \
Db *dbh = 0; \
int sdbid;
#define RSPHEAD \
rsp = rAddI (ses, rsp, LCOM_RSP_SID, sst[ISID], !0); \
rsp = rAddI (ses, rsp, LCOM_RSP_SER, sst[ISER], !0);
#define RSPNOERR \
rsp = rAddI (ses, rsp, LCOM_RSP_ERR, 0, !0);
#define RSPDB \
RADDS (ses, rsp, LCOM_RSP_DBN, dbn, !0);
#define RSPFREE \
if (rsp) { mFree (ses, rsp); }
#define GETDB(rname) \
sdbid = ldspGetDb (&dbh, sst, dbn); \
if (0 > sdbid) { \
int rt = _RqsError (ses, sst, ERR_INVAL, cb, \
cld, rname, sst[ITMS] ? \
"db %s remounted" : "db %s not open", dbn); \
RSPFREE \
return rt; \
}
#define DELIVER \
if (rsp) { \
int rt = (*cb) (ses, &rsp, cld); \
RSPFREE \
return rt; \
} \
sMsg (ses, LOG_ERROR, "ldsp: deliver: rsp == 0"); \
return 0;
/* ************************************************************
*/
static int ldspGetDb (Db **db, int sst[], const char *dbn) {
Db *dbh = nDbByName (stub0, dbn);
if (! dbh) {
sst[ITMS] = 0;
return -1;
}
if (sst[ITMS]) {
if (-1 == sst[ITMS] || /* dont want default of rInt call */
(dbh->tms && dbh->tms != sst[ITMS])
) {
sst[ITMS] = dbh->tms;
return -1;
}
}
if (db) {
*db = dbh;
}
return dbh->dbid;
}
static int _RqsError (
Session ses, int sst[], int err, DlvCb *cb,
void *cld, const char *rname, const char *fmt, ...
) {
RSPVAR
char msg[LCOM_ERRMSGLEN];
va_list ap;
(void)sdbid; (void)dbh;
RSPHEAD
if (sst[ITMS] && -1 != sst[ITMS]) {
rsp = rAddI (ses, rsp, LCOM_RSP_TMS, sst[ITMS], !0);
}
rsp = rAddI (ses, rsp, LCOM_RSP_ERR, err, !0);
if (0 != fmt && 0 != *fmt) {
va_start (ap, fmt);
vsnprintf (msg, LCOM_ERRMSGLEN - 1, fmt, ap);
msg[LCOM_ERRMSGLEN - 1] = 0;
va_end (ap);
RADDS (ses, rsp, LCOM_RSP_MSG, msg, !0);
sMsg (ses, LOG_WARN, "ldsp: err on rqs %s: %d(%x) - %s",
rname, err, err, msg);
}
else {
sMsg (ses, LOG_WARN, "ldsp: err on rqs %s: %d(%x)",
rname, err, err);
}
DELIVER
}
static int _RqsOpen (
Session ses, int sst[], const char *dbn, DlvCb *cb, void *cld
) {
#define STRERRNOLEN 128
RSPVAR
Schema *sc;
char msg[STRERRNOLEN];
sst[ITMS] = 0;
sc = nSchema (stub0);
if (! sc) {
int rt = _RqsError (ses, sst, ERR_BADF, cb, cld,
"open", "%s: local schema not initialized", dbn);
RSPFREE
return rt;
}
errno = 0;
dbh = cDOpen (dbn, 0, sc->cfg);
if (! dbh) {
int rt;
strncpy (msg, strerror (errno), STRERRNOLEN - 1)
[STRERRNOLEN - 1] = 0;
rt = _RqsError (ses, sst, ERR_BADF, cb, cld,
"open", "%s: %s", dbn, msg);
RSPFREE
return rt;
}
sdbid = dbh->dbid;
RSPHEAD
RSPNOERR
RSPDB
rsp = rAddI (ses, rsp, LCOM_RSP_DBID, dbh->dbid, !0);
rsp = rAddI (ses, rsp, LCOM_RSP_TMS, dbh->tms, !0);
if (dbh->fdt) {
rsp = rAddI (ses, rsp, LCOM_RSP_NUMT, dbh->fdt->len, !0);
rsp = rAddI (ses, rsp, LCOM_RSP_NUMR, dbh->fdt->len, !0);
rsp = fdt2rec (ses, rsp, dbh->fdt, LCOM_RSP_DATA);
}
else {
rsp = rAddI (ses, rsp, LCOM_RSP_NUMT, 0, !0);
rsp = rAddI (ses, rsp, LCOM_RSP_NUMR, 0, !0);
}
DELIVER
}
static int _RqsClose (
Session ses, int sst[], const char *dbn, DlvCb *cb, void *cld
) {
RSPVAR
int err;
GETDB("close")
err = cDClose (sdbid);
RSPHEAD
rsp = rAddI (ses, rsp, LCOM_RSP_DBID, sdbid, !0);
rsp = rAddI (ses, rsp, LCOM_RSP_ERR, err, !0);
RSPDB
DELIVER
}
static int _RqsMaxrow (
Session ses, int sst[], const char *dbn, DlvCb *cb, void *cld
) {
RSPVAR
int maxrow;
GETDB("maxrow")
maxrow = dMaxId (ses, sdbid);
if (0 > maxrow) {
int rt = _RqsError (ses, sst, ERR_TRASH, cb, cld, "maxrow",
"no maxrow for %s", dbn);
RSPFREE
return rt;
}
RSPHEAD
RSPNOERR
RSPDB
rsp = rAddI (ses, rsp, LCOM_RSP_ROWID, maxrow, !0);
DELIVER
}
static int _RqsQuery (
Session ses, int sst[], const char *dbn, const char *key,
int mode, int skip, int size, int flags, DlvCb *cb, void *cld
) {
RSPVAR
Set set;
Rec *rec;
int lrt;
int j;
int rdr = LCOM_QRYF_READREC & flags;
GETDB("query")
set.len = 0; /* input */
lrt = dQuery (ses, &set, sdbid, key, mode, skip);
if (0 > lrt) {
int rt = _RqsError (ses, sst, ERR_TRASH, cb, cld, "query",
"%d,%s = %d", sdbid, key, lrt);
RSPFREE
return rt;
}
RSPHEAD
RSPNOERR
RSPDB
rsp = rAddI (ses, rsp, LCOM_RSP_NUMT, lrt, !0);
if (0 < size && lrt > size) {
lrt = size;
}
rsp = rAddI (ses, rsp, LCOM_RSP_NUMR, lrt, !0);
for (j = 0; lrt > j; ++j) {
rsp = rAddI (ses, rsp, LCOM_RSP_ROWID, set.id[j], !0);
}
if (rdr) {
for (j = 0; rsp && lrt > j; ++j) {
rec = dRead (ses, sdbid, set.id[j]);
if (0 == rec) {
int rt = _RqsError (ses, sst, ERR_TRASH, cb, cld,
"query", "non existent rec %d", set.id[j]);
RSPFREE
return rt;
}
rsp = luti_wrap (ses, rsp, rec, LCOM_RSP_DATA);
mFree (ses, rec);
}
}
DELIVER
}
static int _RqsRead (Session ses,
int sst[], const char *dbn, int row, DlvCb *cb, void *cld
) {
RSPVAR
Rec *rec;
GETDB("read")
RSPHEAD
RSPNOERR
RSPDB
rsp = rAddI (ses, rsp, LCOM_RSP_ROWID, row, !0);
rec = dRead (ses, sdbid, row);
if (rec) {
rsp = rAddI (ses, rsp, LCOM_RSP_NUMT, 1, !0);
rsp = rAddI (ses, rsp, LCOM_RSP_NUMR, 1, !0);
rsp = luti_wrap (ses, rsp, rec, LCOM_RSP_DATA);
mFree (ses, rec);
}
else {
rsp = rAddI (ses, rsp, LCOM_RSP_NUMT, 0, !0);
rsp = rAddI (ses, rsp, LCOM_RSP_NUMR, 0, !0);
}
DELIVER
}
static int _RqsUpdate (Session ses, int sst[],
const char *dbn, int row, Rec *rec, Rec *idx, DlvCb *cb, void *cld
) {
RSPVAR
int rtw;
GETDB("update")
rec->rowid = row;
rec->dbid = sdbid;
rtw = dWritex (ses, sdbid, rec, idx);
if (rtw) {
rtw = _RqsError (ses, sst, ERR_TRASH, cb, cld,
"update", "write error %d", rtw);
RSPFREE
if (rec) { mFree (ses, rec); }
if (idx) { mFree (ses, idx); }
return rtw;
}
RSPHEAD
RSPNOERR
RSPDB
rsp = rAddI (ses, rsp, LCOM_RSP_ROWID, rec->rowid, !0);
if (rec) { mFree (ses, rec); }
if (idx) { mFree (ses, idx); }
DELIVER
}
static int _RqsDelete (Session ses,
int sst[], const char *dbn, int row, DlvCb *cb, void *cld
) {
(void)dbn; (void)row;
return _RqsError (ses, sst, ERR_TRASH, cb, cld, "delete",
"request not implemented yet");
}
/* ************************************************************
package functions
*/
int ldsp_process (Session ses, Rec *rqs, DlvCb *cb, void *cld) {
char buf[DBNLEN];
char *dbn;
int sst[SSTS];
int rtyp;
if (0 == ses) {
sMsg (ses0, LOG_ERROR, "ldsp_process: no session");
return -1;
}
if (0 == rqs) {
sMsg (ses, LOG_ERROR, "ldsp_process: no request");
return -1;
}
if (0 == cb) {
sMsg (ses, LOG_ERROR, "ldsp_process: no sender");
return -1;
}
rtyp = rInt (rqs, LCOM_RQS_TYPE, -1, 0);
sst[ISID] = rInt (rqs, LCOM_RQS_SID, -1, 0);
sst[ISER] = rInt (rqs, LCOM_RQS_SER, -1, 0);
sst[ITMS] = rInt (rqs, LCOM_RQS_TMS, -1, 0);
dbn = rString (rqs, LCOM_RQS_DBN, 0, buf, DBNLEN);
if (!dbn) {
i2a (buf, rtyp);
return _RqsError (ses, sst, ERR_INVAL, cb, cld, buf,
"no db name given");
}
switch (rtyp) {
case LCOM_RQST_OPEN:
case LCOM_RQST_CREA:
return _RqsOpen (ses, sst, dbn, cb, cld);
case LCOM_RQST_CLOS:
return _RqsClose (ses, sst, dbn, cb, cld);
case LCOM_RQST_MROW:
return _RqsMaxrow (ses, sst, dbn, cb, cld);
case LCOM_RQST_QRY: {
char kybuf[LCOM_QRY_KEYLEN];
char *key;
int mode, skip, size;
int flg;
key = rString (rqs, LCOM_RQS_KEY, 0, kybuf, LCOM_QRY_KEYLEN);
mode = rInt (rqs, LCOM_RQS_QMOD, -1, 0);
skip = rInt (rqs, LCOM_RQS_SKIP, 0, 0);
size = rInt (rqs, LCOM_RQS_SIZE, -1, 0);
flg = rInt (rqs, LCOM_RQS_QFLG, 0, 0);
if (0 > mode || 0 == key) {
return _RqsError (ses, sst, ERR_INVAL, cb, cld, "query",
"missing mode or key");
}
return _RqsQuery (ses, sst, dbn, key,
mode, skip, size, flg, cb, cld);
}
case LCOM_RQST_READ: {
int rowid;
rowid = rInt (rqs, LCOM_RQS_ROWID, -1, 0);
if (0 > rowid) {
return _RqsError (ses, sst, ERR_INVAL, cb, cld, "read",
"missing rowid");
}
return _RqsRead (ses, sst, dbn, rowid, cb, cld);
}
case LCOM_RQST_INS:
case LCOM_RQST_UPD: {
Rec *rec, *idx;
int rowid = 0;
int ins = rtyp == LCOM_RQST_INS;
rec = luti_unwrap (ses, rqs, 0, LCOM_RQS_REC, -1);
idx = luti_unwrap (ses, rqs, 0, LCOM_RQS_IDX, -1);
if (!(rec || idx)) {
return _RqsError (ses, sst, ERR_INVAL, cb, cld,
"update", "missing record or index");
}
if (! ins) {
rowid = rInt (rqs, LCOM_RQS_ROWID, 0, 0);
if (0 >= rowid) {
return _RqsError (ses, sst, ERR_INVAL, cb, cld,
"update", "missing rowid");
}
}
return _RqsUpdate (ses, sst, dbn, rowid, rec, idx, cb, cld);
}
case LCOM_RQST_DEL: {
int rowid;
rowid = rInt (rqs, LCOM_RQS_ROWID, -1, 0);
if (0 > rowid) {
return _RqsError (ses, sst, ERR_INVAL, cb, cld, "delete",
"missing rowid");
}
return _RqsDelete (ses, sst, dbn, rowid, cb, cld);
}
}
i2a (buf, rtyp);
return _RqsError (ses, sst, ERR_INVAL, cb, cld, buf,
"unrecognized request type");
}
lfdt.c 0100644 0001130 0001130 00000035307 07635347015 010652 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lfdt.c,v 1.8 2003/03/16 18:58:49 mawag Exp $
implementation of FDT calls and static FDTs.
*/
#include
#include
#include "openisis.h"
#include "lfdt.h"
#include "luti.h"
#define EXT_FDT ".fdt"
#define FD_NAMELEN 30
#define FD_PATLEN 20
#define FD_TOOLLEN 6
#define FD_MAXTOOLS 64
/* ************************************************************
private types
*/
/* ************************************************************
private data
*/
static Fd paramf[] = {
/* i, t, r, l, n, s, d */
/* 700 ... OpenIsis system parameters */
{ OPENISIS_SPATH, FTX, 0, 256, "syspath", "", "Gloval Database Path" },
{ OPENISIS_SLOGF, FTX, 0, 256, "logfile", "", "Logfile Name" },
{ OPENISIS_SLOGV, FTE, 0, 256, "v", "", "Verbosity of Logging" },
{ OPENISIS_SLOGV, FTV, 0, 0, "off", "", "don't log anything" },
{ OPENISIS_SLOGV, FTV, 0, 1, "fatal", "", "" },
{ OPENISIS_SLOGV, FTV, 0, 3, "syserr", "", "" },
{ OPENISIS_SLOGV, FTV, 0, 4, "error", "", "" },
{ OPENISIS_SLOGV, FTV, 0, 5, "warn", "", "" },
{ OPENISIS_SLOGV, FTV, 0, 6, "info", "", "" },
{ OPENISIS_SLOGV, FTV, 0, 7, "verbose", "", "" },
{ OPENISIS_SLOGV, FTV, 0, 8, "trace", "", "" },
{ OPENISIS_SLOGV, FTV, 0, 9, "debug", "", "" },
{ OPENISIS_SLOGV, FTV, 0, 10, "all", "", "" },
{ OPENISIS_SSCHEMA, FTX, 1, 256, "schema", "shp", "Schema entry" },
/* db part starts at pos 14, see below */
{ OPENISIS_DNAME, FTX, 0, 256, "db", "", "Database Name" },
{ OPENISIS_DTYPE, FTE, 0, 256, "format", "", "Database Format" },
{ OPENISIS_DTYPE, FTV, 0, 0, "autoformat", "", "Database Format" },
{ OPENISIS_DTYPE, FTV, 0, 1, "naligned", "", "Database Format" },
{ OPENISIS_DTYPE, FTV, 0, 2, "aligned", "", "Database Format" },
{ OPENISIS_DRO , FTB, 0, 1, "ro", "", "Readonly Flag" },
{ OPENISIS_DPATH, FTX, 0, 256, "dbpath", "", "Database Path" },
{ OPENISIS_DDUMP, FTB, 0, 1, "internaldump", "", "Internal Dump Flag" }
};
static Fdt paramS = {
sizeof( paramf )/sizeof( paramf[0] ),
paramf,
0
};
static Fdt paramD = {
sizeof( paramf )/sizeof( paramf[0] ) - 14,
paramf + 14,
0
};
/* ************************************************************
public data
*/
const Fdt *openIsisSyspar = ¶mS;
const Fdt *openIsisDbpar = ¶mD;
/* ************************************************************
private functions
*/
static char *fgetsnl (FILE *fp) {
static char buf[4096];
char *res;
int len;
while (1) {
if (0 == (res = fgets (buf, 4095, fp))) {
break;
}
buf[4095] = 0;
if (0 == (len = strlen (res))) {
continue;
}
if ('\n' == res[--len]) {
if (0 == len) {
continue;
}
res[len] = 0;
}
break;
}
return res;
}
static void strrtrim (char *str, int len) {
char *p = str + len - 1;
while (p >= str && (' ' == *p || '\t' == *p)) {
*p-- = 0;
}
}
static char *desc2name (char *d, char *n) {
char *name = n;
while (1) {
if (!(*n = *d)) {
return name;
}
if ('A' <= *n && 'Z' >= *n) {
*n += 'a' - 'A';
++n;
++d;
continue;
}
if ('_' == *n ||
('a' <= *n && 'z' >= *n) ||
('0' <= *n && '9' >= *n)) {
++n;
++d;
continue;
}
while (1) {
++d;
if (!*d) {
*n = 0;
return name;
}
if (('A' <= *d && 'Z' >= *d) ||
('a' <= *d && 'z' >= *d) ||
('0' <= *d && '9' >= *d)) {
break;
}
}
*n++ = '_';
}
}
static int line2Fd (char *line,
char *name, char *pat, int *tag, int *len, int *type, int *rep
) {
if (FD_NAMELEN + FD_PATLEN >= strlen (line)) {
return 0;
}
strncpy (name, line, FD_NAMELEN) [FD_NAMELEN] = 0;
strrtrim (name, strlen (name));
if (! *name) {
return 0;
}
line += FD_NAMELEN;
strncpy (pat, line, FD_PATLEN) [FD_PATLEN] = 0;
strrtrim (pat, strlen (pat));
line += FD_PATLEN;
if (4 != sscanf (line, "%d %d %d %d", tag, len, type, rep)) {
return 0;
}
switch (*type) {
case 0: *type = OPENISIS_FTX; break;
case 1: *type = OPENISIS_FTA; break;
case 2: *type = OPENISIS_FTN; break;
case 3: *type = OPENISIS_FTP; break;
default: return 0;
}
if (0 != *rep && 1 != *rep) {
return 0;
}
return !0;
}
static int line2Tool (char *line, int *type, char **tools) {
int len;
int num = 0;
switch (*line) {
case 'w':
case 'W':
*type = OPENISIS_DFMT;
break;
case 'f':
case 'F':
*type = OPENISIS_DPFT;
break;
case 's':
case 'S':
*type = OPENISIS_DFST;
break;
default:
return 0;
}
line += 2;
len = strlen (line);
while (0 < len && FD_MAXTOOLS > num) {
tools[num] = line;
if (FD_TOOLLEN > len) {
strrtrim (line, len);
++num;
break;
}
strrtrim (line, FD_TOOLLEN);
line += FD_TOOLLEN;
len -= FD_TOOLLEN;
++num;
}
return num;
}
/* ************************************************************
package functions
*/
#if 0
const char *lfdt_argv (Fdt *fdt, int fdid,
const char *name, const char *val, const char *dflt) {
Fd *D;
if (! fdt || ! name || ! val) {
return dflt;
}
D = fdById (fdt, fdid);
if (! D) {
return dflt;
}
if (strcmp (name, D->name)) {
return dflt;
}
return val;
}
int lfdt_argi (Fdt *fdt, int fdid,
const char *name, const char *val, int dflt) {
val = lfdt_argv (fdt, fdid, name, val, 0);
if (! val) {
return dflt;
}
return a2id (val, -1, dflt);
}
int lfdt_arge (Fdt *fdt, int fdid,
const char *name, const char *val, int dflt) {
Fd *D;
int ev;
if (! fdt || ! name) {
return NOENUM;
}
D = fdById (fdt, fdid);
if (! D) {
return NOENUM;
}
if (strcmp (name, D->name)) {
return NOENUM;
}
if (! val) {
return dflt;
}
ev = fdEnum (fdt, D->id, val);
if (NOENUM == ev) {
return dflt;
}
return ev;
}
#endif /* 0 */
/* ************************************************************
public functions
*/
Fd* fdById ( const Fdt *fdt, int id )
{
Fd *f = fdt->fd, *e = f + fdt->len;
for ( ; f < e; f++ )
if ( id == f->id )
return f;
return 0;
} /* fdById */
Fd* fdByName ( const Fdt *fdt, const char *name )
{
Fd *f = fdt->fd, *e = f + fdt->len;
const char *p;
int l;
if ( '-' == *name ) /* ignore leading dash */
name++;
if ( '0' <= *name && *name <= '9' )
return fdById( fdt, a2i( name, -1 ) );
p = name;
while ( 'a' <= *p ? *p <= 'z' : '9' >= *p ? *p >= '0' : '_' == *p )
p++;
l = p - name;
if ( ! l || l > OPENISIS_FD_NAMELEN - 1 )
return 0;
for ( ; f < e; f++ )
if ( *name == *f->name
&& !(0xf0 & f->type) /* is field */
&& !memcmp( name, f->name, l )
&& !f->name[l]
)
return f;
return 0;
} /* fdByName */
/**
lookup enum:
if name is numeric, the value is returned, if legal
considered are value entries with given id or id 0
- if there is an exact match with same id, this is used
- if there is an exact match with id 0, this is used
- if name is a unique prefix on given id, this is used
- if name is no prefix on given id, but a unique prefix on 0, this is used
*/
int fdEnum ( Fdt *fdt, int id, const char *name )
{
Fd *f = fdt->fd, *e = f + fdt->len,
*x0 = 0, *pi = 0, *p0 = 0;
int ui = 1, u0 = 1; /* unique */
int l = strlen( name );
if ( ! l || l > OPENISIS_FD_NAMELEN - 1 )
return NOENUM;
if ( ('0' <= *name && *name <= '9')
|| ('-' == *name && '0' <= name[1] && name[1] <= '9')
) {
int v = a2i( name, l );
for ( ; f < e; f++ ) {
if ( FTV == f->type && v == f->len && (f->id == id || !f->id) )
return v;
}
return NOENUM;
}
for ( ; f < e; f++ ) {
if ( FTV != f->type
|| *name != *f->name
|| memcmp( name, f->name, l )
|| (f->id && f->id != id)
)
continue;
if ( !f->name[l] ) { /* exact match */
if ( f->id == id )
return f->len;
x0 = f; /* f->id is 0 */
continue;
}
/* prefix match */
if ( f->id == id ) {
if ( pi )
ui = 0;
else
pi = f;
} else
if ( p0 )
u0 = 0;
else
p0 = f;
}
return x0 ? x0->len
: pi ? (ui ? pi->len : NOENUM)
: p0 && u0 ? p0->len : NOENUM;
} /* fdEnum */
Fdt *cFdt ( Session ses, int len, Fd *fd, Rec *rec )
{
Fdt *fdt = (Fdt*) mAlloc (ses, sizeof(Fdt));
if ( ! fdt )
return 0;
fdt->len = len;
fdt->fd = fd;
fdt->rec = rec;
/* TODO: if we have no fd, create one from rec */
return fdt;
} /* cFdt */
void cFreeFdt ( Session ses, Fdt *fdt ) {
if (fdt) {
if (fdt->fd) {
mFree (ses, fdt->fd);
}
if (fdt->rec) {
mFree (ses, fdt->rec);
}
mFree (ses, fdt);
}
}
#define CLN_FR \
if (0 != fds && 0 != *fds) { \
mFree (ses, *fds); \
*fds = 0; \
} \
if (0 != rec && 0 != *rec) { \
mFree (ses, *rec); \
*rec = 0; \
} \
return -1
int fdtFromFile (char *path, Fd **fds, Rec **rec) {
Session ses;
char name[1 + FD_NAMELEN];
char pat[1 + FD_PATLEN];
char tool[1 + FD_TOOLLEN];
char *tools[FD_MAXTOOLS];
char *line;
Fd *nfd;
FILE *fp;
int tag, len, num, type, rep, j;
ses = OPENISIS_SES0 ();
/* ldb::setext */
len = strlen (path) - 4;
memcpy (path + len, EXT_FDT, 4);
if ( 'A'<=path[len-1] && path[len-1]<= 'Z' ) {
char *p = path + len;
for ( ;*p; p++ ) /* use uppercase extensions */
if ( 'a' <= *p && *p <= 'z' )
*p -= 'a'-'A';
}
if (0 != fds) {
*fds = 0;
}
if (0 != rec) {
*rec = 0;
}
fp = fopen (path, "r");
if (0 == fp) {
sMsg (ses, LOG_INFO, "no such fdt: %s", path);
return -1;
}
sMsg (ses, LOG_VERBOSE, "> reading fdt: %s", path);
num = 0;
while (0 != (line = fgetsnl (fp))) {
if ('*' == *line) {
continue;
}
if (':' == line[1]) {
if (0 != rec) {
len = line2Tool (line, &type, tools);
for (j = 0; len > j; ++j) {
strncpy (tool, tools[j], FD_TOOLLEN) [FD_TOOLLEN] = 0;
OPENISIS_RADDS (ses, *rec, type, tool, !0);
if (0 == *rec) {
CLN_FR;
}
}
}
continue;
}
if (0 != fds &&
line2Fd (line, name, pat, &tag, &len, &type, &rep)) {
nfd = mAlloc (ses, (1 + num) * sizeof (Fd));
if (0 == nfd) {
CLN_FR;
}
if (0 != num) {
memcpy (nfd, *fds, num * sizeof (Fd));
mFree (ses, *fds);
}
*fds = nfd;
nfd = *fds + num++;
memset (nfd, 0, sizeof (Fd));
nfd->id = tag;
nfd->type = (char) type;
nfd->rep = (char) rep;
nfd->len = (short) len;
desc2name (name, nfd->name);
strcpy (nfd->subf, pat);
strcpy (nfd->desc, name);
}
}
fclose (fp);
sMsg (ses, LOG_VERBOSE, "< %d entries in fdt", num);
return num;
}
Rec *fd2rec (Session ses, Fd *fd) {
char buf[12];
Rec *rec = 0;
if (0 == fd) {
return 0;
}
i2a (buf, fd->id);
RADDS (ses, rec, OPENISIS_FDID, buf, !0);
i2a (buf, fd->type);
RADDS (ses, rec, OPENISIS_FDTYPE, buf, !0);
i2a (buf, fd->rep);
RADDS (ses, rec, OPENISIS_FDREP, buf, !0);
i2a (buf, fd->len);
RADDS (ses, rec, OPENISIS_FDLEN, buf, !0);
RADDS (ses, rec, OPENISIS_FDNAME, fd->name, !0);
RADDS (ses, rec, OPENISIS_FDSUBF, fd->subf, !0);
RADDS (ses, rec, OPENISIS_FDDESC, fd->desc, !0);
return rec;
}
Rec *fdt2rec (Session ses, Rec *rec, Fdt *fdt, int tag) {
Fd *F;
Rec *R;
if (! fdt) {
return rec;
}
for (F = fdt->fd; fdt->len > F - fdt->fd; ++F) {
R = fd2rec (ses, F);
if (!R) {
return 0;
}
rec = luti_wrap (ses, rec, R, tag);
mFree (ses, R);
if (!rec) {
return 0;
}
}
return rec;
}
#define CLN_F \
if (0 == buf) { \
mFree (ses, fd); \
} \
return 0
Fd *rec2fd (Session ses, Rec *rec, Fd *buf) {
Field *F;
Fd *fd;
int val, len;
int got = 0;
if (0 == rec) {
return 0;
}
if (0 == buf) {
fd = mAlloc (ses, sizeof (Fd));
if (0 == fd) {
return 0;
}
}
else {
fd = buf;
}
memset (fd, 0, sizeof (Fd));
for (F = rec->field; F - rec->field < rec->len; ++F) {
switch (F->tag) {
case OPENISIS_FDID:
if (0 != (0x01 & got)) { CLN_F; } got |= 0x01;
if (0 > (val = a2id (F->val, (int)F->len, -1))) { CLN_F; }
fd->id = val;
break;
case OPENISIS_FDTYPE:
if (0 != (0x02 & got)) { CLN_F; } got |= 0x02;
if (0 > (val = a2id (F->val, (int)F->len, -1))) { CLN_F; }
fd->type = (char) val;
break;
case OPENISIS_FDREP:
if (0 != (0x04 & got)) { CLN_F; } got |= 0x04;
if (0 > (val = a2id (F->val, (int)F->len, -1))) { CLN_F; }
fd->rep = (char)(0 != val);
break;
case OPENISIS_FDLEN:
if (0 != (0x08 & got)) { CLN_F; } got |= 0x08;
if (0 > (val = a2id (F->val, (int)F->len, -1))) { CLN_F; }
fd->len = (short) val;
break;
case OPENISIS_FDNAME:
if (0 != (0x10 & got)) { CLN_F; } got |= 0x10;
if (! F->len) { CLN_F; }
if (30 < (len = (int)F->len)) {
len = 30;
}
strncpy (fd->name, F->val, len) [len] = 0;
break;
case OPENISIS_FDSUBF:
if (0 != (0x20 & got)) { CLN_F; } got |= 0x20;
if (F->len) {
if (20 < (len = (int)F->len)) {
len = 20;
}
strncpy (fd->subf, F->val, len) [len] = 0;
}
break;
case OPENISIS_FDDESC:
if (0 != (0x40 & got)) { CLN_F; } got |= 0x40;
if (F->len) {
if (30 < (len = (int)F->len)) {
len = 30;
}
strncpy (fd->desc, F->val, len) [len] = 0;
}
break;
}
}
if (0x13 != (got & 0x13)) { CLN_F; }
return fd;
}
Fdt *rec2fdt (Session ses, Rec *rec, int idnum, int idfd, int *pos) {
Rec *R;
Fdt *fdt;
Fd fdbuf;
Fd *D, *arr, *E, *A;
int num;
if (!rec) {
return 0;
}
num = rInt (rec, idnum, -1, pos);
if (0 >= num) {
return 0;
}
arr = (Fd*) mAlloc (ses, num * sizeof (Fd));
if (!arr) {
return 0;
}
for (E = (A = arr) + num; E > A; ++A) {
R = luti_unwrap (ses, rec, pos, idfd, -1);
if (!R) {
mFree (ses, arr);
return 0;
}
D = rec2fd (ses, R, &fdbuf);
mFree (ses, R);
if (!D) {
mFree (ses, arr);
return 0;
}
memcpy (A, D, sizeof (Fd));
}
fdt = cFdt (ses, num, arr, 0);
if (!fdt) {
mFree (ses, arr);
return 0;
}
return fdt;
}
lfdt.h 0100644 0001130 0001130 00000003076 07635347015 010655 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef LFDT_H
/*
$Id: lfdt.h,v 1.5 2003/03/16 18:58:49 mawag Exp $
FDT support
*/
#include "loi.h"
extern int fdtFromFile (char *path, Fd **fds, Rec **rec);
extern Rec *fd2rec (Session ses, Fd *fd);
/* append fdt to rec */
extern Rec *fdt2rec (Session ses, Rec *rec, Fdt *fdt, int tag);
extern Fd *rec2fd (Session ses, Rec *rec, Fd *buf);
/* extract fdt from rec */
extern Fdt *rec2fdt (Session ses, Rec *rec, int idnum, int idfd, int *pos);
#define LFDT_H
#endif /* LFDT_H */
lfmt.c 0100644 0001130 0001130 00000106017 07630630466 010660 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lfmt.c,v 1.13 2003/03/03 10:40:22 kripke Exp $
implementation of record cooking.
*/
#include /* memset et al */
#include "ldb.h"
#define LFMT_DBG LOG_WARN /* level of debugging */
/* ************************************************************
private types
*/
typedef struct { /* function definition */
const char *name; /* hmmm ... fixed char[] probably faster */
const char *sign;
int tag;
} LfmtToken;
typedef enum {
LFMT_MP, /* proof */
LFMT_MH, /* heading */
LFMT_MD /* data */
} lfmtMode;
typedef struct LfmtIter_ { /* value iterator */
const unsigned char *def; /* position in format where iter was opened. */
const unsigned char *end; /* position in format where iter was closed. */
int pos; /* next field pos */
int off; /* offset to start search for next subfield */
int occ; /* next occ */
/* filter */
int tag;
char sub;
char submode; /* primary loop (counted by occ) is over subfields. */
int from; /* primary range. */
int to;
int sfrom; /* additional subfield range. */
int sto;
} LfmtIter;
#define LFMT_NITER 8
typedef struct LfmtRec_ { /* record context -- pushed by the r in REF */
struct LfmtRec_ *prev;
Rec *r;
int frame; /* stack frame associated with this record */
const unsigned char *loop; /* position in format where loop was opened. */
LfmtIter iter[LFMT_NITER]; /* iterators in loop */
int occ; /* loop occ */
unsigned char i; /* iterator count */
char more; /* more flag */
char had; /* had flag */
char sub; /* database's subfield separator, typically ^ or $ */
} LfmtRec;
typedef struct LfmtFmt_ { /* format context -- pushed by @ */
struct LfmtFmt_ *prev;
const unsigned char *f;
const unsigned char *e; /* end -- just for paranoia check */
lfmtMode mode;
int upcase; /* uppercase flag */
} LfmtFmt;
typedef struct { /* other type */
const LfmtToken *tok; /* recognized token */
int data; /* start of params in output */
int used; /* used len of output up to our params */
char pos; /* position in signature */
char want; /* currently expected type */
char expl; /* explicit flag */
char rec; /* record flag */
} LfmtFrame;
/*
list of special token tags after the format tags 1 .. 999.
unlike formatting tokens, which just emit a field with their negative tag,
the literal's tag is reset to 0,
and all other tokens emit fields with tags based on their parameters.
tokens are organized in groups of (up to) 16.
the group of an operator determines it's precedence.
*/
#define LFMT_DEFGROUP(i) ((i) << 4) /* * 16 */
#define LFMT_GROUP(i) ((i) >> 4)
/* token groups */
enum {
LFMT_LITERAL = 63, /* literals group starting from 63*16 = 1008 */
/* syntactical groups starting from 64*16 = 0x400 */
LFMT_FUNCTIONS,
LFMT_VALUES = 96, /* starting from 96*16 = 0x600 */
LFMT_ITERATORS,
LFMT_OPITER, /* iterator operators */
LFMT_OPSTR, /* string operators */
LFMT_OPMUL, /* multiplicative operators */
LFMT_OPADD, /* additive operators */
LFMT_OPREL, /* relational operators */
LFMT_OPBOOL, /* boolean operators */
LFMT_OPASSIGN, /* assignment */
LFMT_STOPPER = 128, /* starting from 128*16 = 0x800 */
LFMT_SYNTAX
};
enum {
/* literals */
LFMT_LIT = LFMT_DEFGROUP( LFMT_LITERAL ),
LFMT_COND,
LFMT_REP,
LFMT_COMM,
LFMT_BARE, /* field of bareword C&A literal */
LFMT_NUMBER, /* standalone numeric literal */
/* functions */
LFMT_OPEN = LFMT_DEFGROUP( LFMT_FUNCTIONS ),
LFMT_AMP,
LFMT_AT,
LFMT_XREF,
LFMT_S,
LFMT_Z,
LFMT_F,
/* */
LFMT_MDL,
LFMT_MDU,
LFMT_MHL,
LFMT_MHU,
LFMT_MPL,
LFMT_MPU,
/* values */
LFMT_SI = LFMT_DEFGROUP( LFMT_VALUES ),
LFMT_EI,
/* iterators */
LFMT_V = LFMT_DEFGROUP( LFMT_ITERATORS ),
LFMT_D,
LFMT_N,
/* operator groups by decreasing precedence */
LFMT_INDEX = LFMT_DEFGROUP( LFMT_OPITER ),
LFMT_SUBFIELD,
LFMT_STAR = LFMT_DEFGROUP( LFMT_OPSTR ),
LFMT_DOT, LFMT_INDENT,
LFMT_MUL = LFMT_DEFGROUP( LFMT_OPMUL ),
LFMT_DIV,
LFMT_PLUS = LFMT_DEFGROUP( LFMT_OPADD ),
LFMT_MINUS,
LFMT_EQ = LFMT_DEFGROUP( LFMT_OPREL ),
LFMT_NE, LFMT_LT, LFMT_LE, LFMT_GT, LFMT_GE, LFMT_CT,
LFMT_AND = LFMT_DEFGROUP( LFMT_OPBOOL ),
LFMT_OR, LFMT_NOT,
LFMT_ASSIGN = LFMT_DEFGROUP( LFMT_OPASSIGN ),
/* stoppers */
LFMT_COMMA = LFMT_DEFGROUP( LFMT_STOPPER ),
LFMT_RANGE,
LFMT_CLOSE,
/* syntax */
LFMT_BLANK = LFMT_DEFGROUP( LFMT_SYNTAX ),
LFMT_LOOP,
LFMT_WHILE,
LFMT_REF,
LFMT_IF, LFMT_THEN, LFMT_ELSE, LFMT_FI,
LFMT_SELECT, LFMT_CASE, LFMT_ELSECASE, LFMT_ENDSEL,
LFMT__LAST
};
/*
while fields emitted by formatting and literal tokens
have tags between -1 and -1023 (0xffffffff to 0xfffffc01)
with all high bits set, numerics have lowest negative values
ranging from 0x80000000 to 0xbfffffff (-2147483648 to -1073741825).
That means the highest two bits are 1 and 0.
All numeric fields have a val of NULL and use len to hold a number.
Besides pure int or fixed-point numbers,
there are other numerical types:
next two bits may indicate a range or field selector.
The lower 3 bytes may give fractional part or range end.
*/
#define LFMT_NMASK 0xf0000000L /* highest bits 1 and 0 mark a number */
#define LFMT_NUM 0x80000000L /* basic number */
#define LFMT_ITR 0xa0000000L /* with iterator bit */
#define LFMT_VAL 0x00ffffffL /* mask for 3 bytes additional info */
/* is any numeric */
#define LFMT_ISNUM(t) (LFMT_NUM == (LFMT_NMASK & (t)))
/* is pure int */
#define LFMT_ISINT(t) (LFMT_NUM == (t))
/* is iterator */
#define LFMT_ISITR(t) (LFMT_ITR == (LFMT_NMASK & (t)))
/* ************************************************************
private data
*/
static const LfmtToken number = { "", "_1i", LFMT_NUMBER };
/* while a numeric *field* has large LFMT_NUM tag,
the anonymous number *token* has a literal tag,
so we don't take it as high precedence operator.
*/
static const LfmtToken blank = { "_", "_0o_", LFMT_BLANK };
static const LfmtToken loop = { "loop", "_0o_", LFMT_LOOP };
/*
32 = 33 = ! 34 = " 35 = # 36 = $ 37 = % 38 = & 39 = '
40 = ( 41 = ) 42 = * 43 = + 44 = , 45 = - 46 = . 47 = /
48 = 0 49 = 1 50 = 2 51 = 3 52 = 4 53 = 5 54 = 6 55 = 7
56 = 8 57 = 9 58 = : 59 = ; 60 = < 61 = = 62 = > 63 = ?
64 = @ 65 = A 66 = B 67 = C 68 = D 69 = E 70 = F 71 = G
72 = H 73 = I 74 = J 75 = K 76 = L 77 = M 78 = N 79 = O
80 = P 81 = Q 82 = R 83 = S 84 = T 85 = U 86 = V 87 = W
88 = X 89 = Y 90 = Z 91 = [ 92 = \ 93 = ] 94 = ^ 95 = _
96 = ` 97 = a 98 = b 99 = c 100 = d 101 = e 102 = f 103 = g
104 = h 105 = i 106 = j 107 = k 108 = l 109 = m 110 = n 111 = o
112 = p 113 = q 114 = r 115 = s 116 = t 117 = u 118 = v 119 = w
120 = x 121 = y 122 = z 123 = { 124 = | 125 = } 126 = ~
*/
/* stops dictionary search */
static const char sentinel[] = "\x7f";
#define ENDDICT { sentinel, NULL, 0 }
static const LfmtToken dictSpecial[] = {
{ "", "_0", 0 }, /* eof */
{ "!", "_1x", OPENISIS_FMT_ESC },
{ "\"", "_1x", LFMT_COND },
{ "#", "_0", OPENISIS_FMT_SHARP },
/* { "$", "_0", LFMT_LIT }, */
{ "%", "_0", OPENISIS_FMT_PERCENT },
{ "&", "_1as_", LFMT_AMP },
{ "'", "_1x", LFMT_LIT },
{ "(", "_0", LFMT_OPEN },
{ ")", "_0", LFMT_CLOSE },
{ "*", "n1n", LFMT_MUL },
{ "+", "n1n", LFMT_PLUS },
{ ",", "_0", LFMT_COMMA },
{ "-", "n1n", LFMT_MINUS },
{ ".", "s1i", LFMT_DOT },
{ "..","_0", LFMT_RANGE },
{ "/", "n1n", LFMT_DIV }, /* alias { "/", "_0", OPENISIS_FMT_SLASH }, */
{ "/*","_1x", LFMT_COMM },
/* 0 - 9 */
{ ":", "s1s", LFMT_CT },
{ ":=","v1f", LFMT_ASSIGN },
/* { ";", "b_", LFMT_LIT }, */
{ "<", "n1n", LFMT_LT },
{ "<=","n1n", LFMT_LE },
{ "<>","n1n", LFMT_NE },
{ "=", "n1n", LFMT_EQ },
{ ">", "n1n", LFMT_GT },
{ ">=","n1n", LFMT_GE },
/* { "?", "b_", LFMT_LIT }, */
{ "@", "_1a", LFMT_AT },
/* A - Z */
{ "[", "v1nn", LFMT_INDEX },
/* { "\\", "_0", LFMT_LIT }, */
{ "]", "_0", LFMT_CLOSE },
{ "^", "v1c", LFMT_SUBFIELD },
/* { "_", "_0", LFMT_LIT }, */
/* { "`", "_0", LFMT_LIT }, */
/* a - z */
{ "{", "_0", OPENISIS_FMT_OPEN },
{ "|", "_1x", LFMT_REP },
{ "}", "_0", OPENISIS_FMT_CLOSE },
{ "~", "_1n", LFMT_NOT },
ENDDICT };
static const LfmtToken dictA[] = { ENDDICT };
static const LfmtToken dictB[] = {
{ "B", "_0", OPENISIS_FMT_B },
ENDDICT };
static const LfmtToken dictC[] = {
{ "C", "_1i", OPENISIS_FMT_C },
ENDDICT };
static const LfmtToken dictD[] = {
{ "D", "_1i", LFMT_D },
ENDDICT };
static const LfmtToken dictE[] = {
{ "E", "_1i", LFMT_EI },
{ "ELSE", "_1s_", LFMT_ELSE },
ENDDICT };
static const LfmtToken dictF[] = {
/* { "F", "_1nii", LFMT_F }, */
{ "F", "_1i", OPENISIS_FMT_F },
ENDDICT };
static const LfmtToken dictG[] = { ENDDICT };
static const LfmtToken dictH[] = { ENDDICT };
static const LfmtToken dictI[] = {
{ "I", "_0", OPENISIS_FMT_I },
{ "IF", "_1n", LFMT_IF },
ENDDICT };
static const LfmtToken dictJ[] = { ENDDICT };
static const LfmtToken dictK[] = { ENDDICT };
static const LfmtToken dictL[] = {
{ "LINK", "_1s_", OPENISIS_FMT_LINK },
ENDDICT };
static const LfmtToken dictM[] = {
{ "M", "_1ii", OPENISIS_FMT_M },
{ "MDL", "_0", LFMT_MDL },
{ "MDU", "_0", LFMT_MDU },
{ "MHL", "_0", LFMT_MHL },
{ "MHU", "_0", LFMT_MHU },
{ "MPL", "_0", LFMT_MPL },
{ "MPU", "_0", LFMT_MPU },
ENDDICT };
static const LfmtToken dictN[] = {
{ "N", "_1i", LFMT_N },
{ "NC", "_0i", OPENISIS_FMT_NC },
ENDDICT };
static const LfmtToken dictO[] = { ENDDICT };
static const LfmtToken dictP[] = {
{ "PICT", "_1s", OPENISIS_FMT_PICT },
ENDDICT };
static const LfmtToken dictQ[] = {
{ "QC", "_0", OPENISIS_FMT_QC },
ENDDICT };
static const LfmtToken dictR[] = {
{ "REF", "_2rs_", LFMT_REF },
ENDDICT };
static const LfmtToken dictS[] = {
{ "S", "_1s_", LFMT_S },
ENDDICT };
static const LfmtToken dictT[] = {
{ "TAB", "_0i", OPENISIS_FMT_TAB },
ENDDICT };
static const LfmtToken dictU[] = {
{ "UL", "_0", OPENISIS_FMT_UL },
ENDDICT };
static const LfmtToken dictV[] = {
{ "V", "_1i", LFMT_V },
ENDDICT };
static const LfmtToken dictW[] = { ENDDICT };
static const LfmtToken dictX[] = {
{ "X", "_0i", OPENISIS_FMT_X },
ENDDICT };
static const LfmtToken dictY[] = { ENDDICT };
static const LfmtToken dictZ[] = {
{ "Z", "_1is", LFMT_Z },
ENDDICT };
static const LfmtToken * const dictAZ[26] = {
dictA, dictB, dictC, dictD, dictE, dictF, dictG, dictH, dictI,
dictJ, dictK, dictL, dictM, dictN, dictO, dictP, dictQ, dictR,
dictS, dictT, dictU, dictV, dictW, dictX, dictY, dictZ
};
/* character conversion */
static const unsigned char upcase[256] = {
0,' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',
' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ',
' ','!','"','#','$','%','&','\'', '(',')','*','+',',','-','.','/',
'0','1','2','3','4','5','6','7', '8','9',':',';','<','=','>','?',
'@','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O',
'P','Q','R','S','T','U','V','W','X','Y','Z','[','\\',']','^','_',
'`','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O',
'P','Q','R','S','T','U','V','W','X','Y','Z','{','|','}','~',' ',
'?','?','?','?','?','?','?','?','?','?','?','?','?','?','?','?',
'?','?','?','?','?','?','?','?','?','?','?','?','?','?','?','?',
'?','?','?','?','?','?','?','?','?','?','?','?','?','?','?','?',
'?','?','?','?','?','?','?','?','?','?','?','?','?','?','?','?',
'?','?','?','?','?','?','?','?','?','?','?','?','?','?','?','?',
'?','?','?','?','?','?','?','?','?','?','?','?','?','?','?','?',
'?','?','?','?','?','?','?','?','?','?','?','?','?','?','?','?',
'?','?','?','?','?','?','?','?','?','?','?','?','?','?','?','?'
};
enum {
DIG=0x01, /* digit */
ALP=0x02, /* ascii alphabetic */
IDE=0x04, /* other identifier character */
LIT=0x10, /* as signature type, eat literal (a,c,i,x) */
ALT=ALP|LIT
};
/* character type */
static const unsigned char ctype[256] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
DIG,DIG,DIG,DIG,DIG,DIG,DIG,DIG,DIG,DIG, 0, 0, 0, 0, 0, 0,
/*'@','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O',*/
0,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,
/*'P','Q','R','S','T','U','V','W','X','Y','Z','[','\\',']','^','_',*/
ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP, 0, 0, 0, 0,IDE,
0,ALT,ALP,ALT,ALP,ALP,ALP,ALP,ALP,ALT,ALP,ALP,ALP,ALP,ALP,ALP,
ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALP,ALT,ALP,ALP, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
};
/* ************************************************************
private functions
*/
/* don't rely on braindead ctype.h, it may use locale */
#define ISALPHA(u) ( ALP & ctype[(unsigned char)(u)])
#define ISDIGIT(u) ( DIG & ctype[(unsigned char)(u)])
#define ISALNUM(u) ((ALP|DIG) & ctype[(unsigned char)(u)])
#define ISIDENT(u) ((ALP|DIG|IDE) & ctype[(unsigned char)(u)])
#define ISLITER(u) ( LIT & ctype[(unsigned char)(u)])
/* ************************************************************
package functions
*/
/* ************************************************************
public functions
*/
Rec* rFmt ( Session ses, Rec *buf, const char *fmt, Rec *irec )
{
/* context */
Rec *o = buf;
#define ADD( ntag, s, n ) do { \
RADD( ses, o, ntag, s, n, o!=buf ); \
if ( NULL == o ) goto outofmem; \
} while(0)
#define CAT( s, n ) do { \
RCAT( ses, o, s, n, o!=buf ); \
if ( NULL == o ) goto outofmem; \
} while(0)
/* field add */
#define FADD( f ) ADD( f.tag, f.val, f.len )
/* string add */
#define SADD( ntag, s ) ADD( ntag, s, strlen(s) )
#define SCAT( s ) CAT( s, strlen(s) )
#define SERR( err ) do { \
o = rMsg( ses, o, o!=buf, -LOG_ERROR, \
"@%d: %s", fp - f->f, err ); \
if ( NULL == o ) goto outofmem; \
} while(0)
#define SERR1( fmt, a ) do { \
o = rMsg( ses, o, o!=buf, -LOG_ERROR, \
"@%d: " fmt, fp - f->f, a ); \
if ( NULL == o ) goto outofmem; \
} while(0)
#define SERR2( fmt, a, b ) do { \
o = rMsg( ses, o, o!=buf, -LOG_ERROR, \
"@%d: " fmt, fp - f->f, a, b ); \
if ( NULL == o ) goto outofmem; \
} while(0)
/* number add */
#define NADD( i, d ) do { \
ADD( LFMT_NUM | (d), NULL, 0 ); \
o->field[o->len - 1].val = NULL; \
o->field[o->len - 1].len = i; \
} while(0)
LfmtFrame stack[128], *s = stack; /* current frame (always == stack+sp) */
int sp = -1;
/* push new frame for token t */
#define PUSHFRAME( t ) do { \
if ( ++sp == sizeof(stack)/sizeof(stack[0]) ) goto overflow; \
s = stack+sp; \
memset( s, 0, sizeof(*s) ); \
s->data = o->len; \
s->used = o->used; \
s->tok = (t); \
s->want = (t)->sign[2]; \
} while(0)
LfmtFmt fmt0, *f = NULL; /* default and current format */
const unsigned char *fp = NULL; /* format pointer */
#define PUSHFORMAT( pfmt, fstr ) do { \
memset( pfmt, 0, sizeof(*(pfmt)) ); \
(pfmt)->prev = f; \
f = (pfmt); \
fp = f->f = (const unsigned char *)fstr; \
f->e = fp + strlen(fstr); \
} while(0)
LfmtRec rec0, *r = NULL; /* default and current record */
#define PUSHRECORD( prec, nrec ) do { \
memset( prec, 0, sizeof(*(prec)) ); \
(prec)->prev = r; \
r = (prec); \
r->frame = sp; \
r->r = nrec; \
r->sub = '^'; \
} while(0)
int tlen = 0; /* token length */
PUSHFRAME( &blank );
s->rec = 1;
PUSHFORMAT( &fmt0, fmt );
PUSHRECORD( &rec0, irec );
o->dbid = irec->dbid ;
o->rowid = irec->rowid;
for ( ;; fp += tlen ) {
const LfmtToken *tok = NULL;
Field lit;
unsigned char u;
int start = s->data + s->pos; /* start of current param in output */
int flds = o->len - start; /* # fields for current param */
int vpos = 0; /* pos in dict where we had match of tlen chars */
int dpos = 0; /* pos in dict */
int expl = 0;
unsigned char eatlit = 0;
const LfmtToken *dict;
enum {
CLOSE_OP, /* don't close anything, operator wants field */
CLOSE_VAR, /* close variables only */
CLOSE_FIELD, /* coerce field only */
CLOSE_PARAM, /* compact param only (eat comma) */
CLOSE_EXPL, /* close frame explicitly (eat closing token) */
CLOSE_IMPL /* close frame implicitly, continue on token */
} close;
static const char *closename[] = {
"", "var", "field", "param", "expl", "impl" };
lit.tag = 0;
tlen = 0; /* verified length */
/*
LOG_DBG( LFMT_DBG,
"frame %s%c%s id %d pos %d got %d want %c%s",
s->tok->name, s->expl ? '(' : ' ', s->tok->sign, s->tok->tag,
s->pos, o->len - s->data, s->want, flds ? " (have)" : "" );
*/
/* lookup token */
while ( ' ' == (u = upcase[*fp]) ) /* eat whitespace */
fp++;
if ( ISALPHA(u) ) {
dict = dictAZ[u-'A'];
if ( sentinel != dict->name )
u = upcase[ fp[tlen = 1] ];
} else if ( ISDIGIT( *fp ) ) {
tok = &number; /* read numeric literal */
goto countnumber;
} else
dict = dictSpecial;
/* tok is the last candidate that matched up to IT'S length */
for ( ;; dpos++ ) {
unsigned char test;
LOG_DBG( LOG_TRACE, "trying token '%s'", dict[dpos].name );
if ( tlen && vpos != dpos
&& strncmp( dict[dpos].name, dict[vpos].name, tlen ) )
break; /* ran out of verified length */
if ( u == dict[dpos].name[tlen] )
vpos = dpos;
while ( (test = dict[dpos].name[tlen]) && u == test )
/* test this entry -- same pos, next char */
u = upcase[ fp[++tlen] ];
if ( !test ) /* dict entry ends here -- a hit so far */
tok = &dict[dpos];
/*
LOG_DBG( LFMT_DBG, "\thit on token %s(%s) len %d",
tok->name, tok->sign, tlen );
*/
if ( u < test || !u ) /* too large - bail out*/
break;
}
if ( NULL != tok ) { /* had some match */
if ( tok < &dict[vpos] ) /* had later == longer match */
tok = NULL;
else if ( dict != dictSpecial ) {
/* alpha-token must not be followed by alpha */
assert( tlen == (int)strlen(tok->name) );
if ( ISALPHA( u ) && ISALPHA( tok->name[tlen-1] ) )
tok = NULL;
}
}
if ( NULL == tok /* no hit in alphadict */
|| (dictSpecial == tok && *fp) /* false eof hit in special dict */
) {
SERR2( "unrecognized token '%.*s'", tlen+1, fp );
goto broken;
}
if ( ISLITER( tok->sign[2] ) ) /* eat literals */
switch ( tok->sign[2] ) { /* c, a, i, x */
case 'c':
if ( (lit.len = (eatlit = fp[tlen]) ? 1 : 0 ) ) {
lit.tag = -LFMT_BARE;
lit.val = (char *)fp + tlen;
tlen++;
}
break;
case 'a':
if ( (eatlit = ISIDENT( fp[tlen] )) ) {
lit.tag = -LFMT_BARE;
lit.val = (char *)fp + tlen;
lit.len = 0;
while ( ISIDENT( fp[tlen] ) ) {
lit.len++;
tlen++;
}
}
break;
case 'i':
countnumber:
lit.val = NULL;
lit.len = 0;
if ( (eatlit = ISDIGIT( u = fp[tlen] )) ) {
lit.tag = LFMT_NUM;
while ( u && 10 > (u -= '0') ) {
lit.len = 10*lit.len + u;
u = fp[++tlen];
}
}
/* decimal digits on standalone numeric literal */
if ( &number == tok
&& (unsigned char)('.'-'0') == u
&& '.' != fp[tlen+1] /* avoid eating 1..2 */
) {
int dec = 0;
int v = 0;
while ( (u = fp[++tlen]) && 10 > (u -= '0') )
if ( 6 > dec ) {
v = 10*v + u;
dec++;
}
while ( 6 < dec++ ) v *= 10;
lit.tag = LFMT_NUM | (v & LFMT_VAL);
}
break;
case 'x':
switch ( tok->tag ) {
case LFMT_COMM:
eatlit='*';
break;
case OPENISIS_FMT_ESC:
eatlit=fp[tlen++];
break;
default:
eatlit=tok->name[0];
break;
}
lit.tag = - tok->tag;
lit.val = (char *)fp+1;
while ( (u = fp[tlen++])
&& (eatlit != u || (LFMT_COMM==tok->tag && '/' != fp[tlen]))
) ;
if ( !u ) {
SERR1( "unterminated %s-literal", tok->name );
goto broken;
}
lit.len = tlen - 2;
if ( LFMT_COMM == tok->tag ) {
tlen++;
continue;
}
}
if ( tok->sign[2] ) { /* token wants params */
if ( (!eatlit || tok->sign[3]) && (expl = '(' == fp[tlen]) )
tlen++; /* eat ( */
}
/* got token */
LOG_DBG( LFMT_DBG,
"at %d: '%.*s' %s(%s) id %d (%x) group %d pos %d%c of %s%c%s",
fp - f->f, tlen > 10 ? 10 : tlen, fp,
tok->name, tok->sign, tok->tag, tok->tag, LFMT_GROUP(tok->tag),
s->pos, s->want, s->tok->name, s->expl?'(':' ', s->tok->sign
);
/* close what needs to be closed */
do {
int group = LFMT_GROUP( tok->tag );
int dflt, wantnum, gotnum;
close = CLOSE_FIELD;
if ( '_' != tok->sign[0] /* token is operator ... */
&& (s->expl /* ... within explicit frame or ... */
|| group < LFMT_GROUP( s->tok->tag )) /* ... of higher prec */
) /* we take it -- even as a variable ? */
close = 'v' == tok->sign[0] ? CLOSE_OP : CLOSE_VAR;
else switch ( tok->tag ) {
case LFMT_COMMA:
case LFMT_RANGE:
close = CLOSE_PARAM;
break;
case LFMT_CLOSE:
close = CLOSE_EXPL;
break;
}
if ( CLOSE_VAR > close ) /* leave variable to operator */
break; /* close loop */
if ( flds && LFMT_ISITR( o->field[ o->len-1 ].tag ) ) {
/* dereference iterator variable */
LfmtIter * iter = r->iter + r->i;
if ( ! iter->occ ) {
if ( ! iter->from )
iter->from = 1; /* field defaults to 1..0 (all) */
if ( ! iter->sfrom )
iter->sfrom = iter->sto = 1; /* subfield defaults to 1..1 (1st) */
}
LOG_DBG( LFMT_DBG, "iterator %d/%d V%d[%d..%d]^%c[%d..%d]%s",
o->field[ o->len-1 ].len, r->occ,
iter->tag, iter->from, iter->to,
iter->sub ? iter->sub : ' ', iter->sfrom, iter->sto,
iter->submode ? " subfield mode" : ""
);
o->len--; /* kill the variable. */
/* move to next legal position.
In standard mode, where we advance one field occurence at a time,
this is just the occ >= from.
A legal position may still emit no field,
if a selected subfield is not available in the current field occ.
In subfield mode, we may have to advance 0, 1 or several field
occurences to find next occ of subfield.
*/
r->had = 0;
do {
int socc;
if ( 0 > iter->occ ) /* we were already done */
break;
/* if have legal occurence, ADD it */
if ( iter->occ >= iter->from ) for ( socc=1;; socc++ ) {
/* not initialization pass. */
Field *v = r->r->field + iter->pos;
const char *src = v->val;
int len = v->len;
if ( iter->sub ) { /* find subfield */
const char *p = src + iter->off, *e = src+len;
while ( p < e && (
r->sub != *p++
|| e == p
|| (iter->sub != *p++ && iter->sub != '*')
) )
;
iter->off = p - src;
if ( p >= e )
break;
if ( iter->sfrom && socc < iter->sfrom )
continue;
if ( iter->sto && socc > iter->sto )
break;
src = p;
while ( p < e && r->sub != *p )
p++;
iter->off += len = p - src;
}
/* make sure there's enough room, +3 for DATA mode */
ADD( v->tag, NULL, len+3 );
if ( ! f->mode )
CAT( src, len );
else {
const char *e = src + len;
char *dst = (char *)o->field[ o->len - 1 ].val;
for ( ; src < e; ) {
if ( '<' == *src ) {
for ( src++; src < e; *dst++ = *src++ )
if ( '=' == *src || '>' == *src ) {
while ( '>' != *src++ && src < e )
;
if ( src < e && '<' == *src ) { /* have >< */
*dst++ = ';';
*dst++ = ' ';
}
break;
}
} else if ( r->sub == *src ) {
if ( ++src == e )
break;
if ( dst == o->field[ o->len - 1 ].val )
; /* no spearator */
else if ( 'a' == *src )
*dst++ = ';';
else if ( 'j' > *src )
*dst++ = ',';
else
*dst++ = '.';
src++;
} else
*dst++ = *src++;
}
o->used +=
o->field[ o->len - 1 ].len =
dst - o->field[ o->len - 1 ].val;
if ( LFMT_MD == f->mode )
CAT( ". ", 3 );
} /* f->mode */
if ( f->upcase ) {
char *p = (char *)o->field[ o->len - 1 ].val;
char *e = p + o->field[ o->len - 1 ].len;
for ( ; p < e; p++ )
*p = upcase[(unsigned char)*p];
}
r->had = 1;
if ( iter->submode || ! iter->sub )
break;
}
if ( iter->submode ) {
if ( iter->occ && iter->occ < iter->from ) /* was skipped */
iter->off++;
for ( ; iter->pos < r->r->len; iter->pos++, iter->off = 0 )
if ( iter->tag == r->r->field[ iter->pos ].tag || ! iter->tag ) {
Field *v = r->r->field + iter->pos;
const char *p = v->val + iter->off, *e = v->val + v->len;
while ( p < e && (
r->sub != *p++
|| e == p
|| (iter->sub != *p++ && iter->sub != '*')
) )
;
if ( p < e ) { /* hit */
iter->off = (p - v->val) - 2;
break;
}
}
} else {
if ( iter->occ )
iter->pos++;
while ( iter->pos < r->r->len
&& iter->tag
&& iter->tag != r->r->field[ iter->pos ].tag
)
iter->pos++;
iter->off = 0;
}
iter->occ++;
if ( iter->pos >= r->r->len /* end of record */
|| (iter->to && iter->to < iter->occ) /* end of selected occ */
) {
iter->occ = -1;
break;
}
} while ( iter->occ <= iter->from );
if ( 0 < iter->occ ) /* we have a next occurence */
r->more = 1;
if ( ! r->had && flds && -LFMT_REP == o->field[ o->len - 1 ].tag ) {
flds--;
o->len--;
o->used -= o->field[o->len].len;
}
r->i++; /* advance to next iterator */
} /* dereference iterator variable */
if ( CLOSE_FIELD > close ) /* leave field to operator */
break; /* close loop */
if ( ! flds ) { /* had no field */
if ( CLOSE_PARAM > close ) /* nothing to do */
break; /* close loop */
} else if ( ! s->expl /* frame is implicit ... */
&& ( LFMT_LOOP != s->tok->tag /* ... and not a loop ... */
|| CLOSE_EXPL == close /* ... or we saw a hard closer ... */
|| (r->i && CLOSE_PARAM == close)
/* ... or a loop after 1st iterator on param closer */
)
) /** upgrade the closing mode to IMPL */
close = CLOSE_IMPL;
/* close the field, i.e. coerce it */
dflt = !flds /* param was not given ... */
&& (s->pos /* but was started explicitly by a comma like F( 3, ) */
|| CLOSE_PARAM == close /* or is closed explicitly like F( ,3 ) */
);
wantnum = 'i' == s->want || 'n' == s->want;
gotnum = flds && LFMT_ISNUM( o->field[ o->len-1 ].tag );
LOG_DBG( LFMT_DBG,
"\tclose %s %d%c pos %d%c of %s%c%s id %x",
closename[close],
flds, flds ? gotnum ? 'n' : 's' : dflt ? 'd' : '-', s->pos, s->want,
s->tok->name, s->expl?'(':' ', s->tok->sign, s->tok->tag );
/* close and fix last field */
if ( gotnum && !wantnum ) {
/* coerce number to string, so it may get concatenated */
char b[32];
int ll = lprint( b, o->field[ o->len-1 ].len );
b[ll++] = ' ';
o->len--;
ADD( 0, b, ll );
gotnum = 0;
}
if ( CLOSE_PARAM > close )
break; /* close loop */
/* close and fix param */
if ( wantnum ) {
if ( dflt ) { /* default 0 */
NADD( 0, 0 );
flds = 1;
} else if ( 1 < flds ) {
SERR( "multiple fields for numerical param" );
goto broken;
} else if ( flds && !gotnum ) {
SERR( "expected number" );
goto broken;
}
if ( 'i' == s->want ) /* kill decimal */
o->field[s->data].tag &= ~LFMT_VAL;
} else {
if ( dflt ) {
SADD( 0, "" );
flds = 1;
} else if ( gotnum ) {
SERR( "expected string" ); /* shouldn't happen !? */
goto broken;
} else if ( 1 < flds /* concat strings ... */
&& ! s->rec /* ... unless in record mode frame (blank,loop,ref) */
) {
int i;
for ( i = start+1; i < o->len; i++ )
if ( o->field[i].val )
o->field[ start ].len += o->field[ i ].len;
o->len = start + 1;
}
}
/* done fixing param */
if ( CLOSE_IMPL > close ) {
tok = NULL; /* eat token */
if ( CLOSE_EXPL > close ) { /* i.e. CLOSE_PARAM */
if ( flds ) { /* increment param pos, set next wanted type */
int sl = strlen( s->tok->sign ); /* frame takes sl-2 params */
if ( ++(s->pos) < sl-2 ) {
if ( '_' != s->tok->sign[ 2 + s->pos ] ) /* repeated param */
s->want = s->tok->sign[ 2 + s->pos ]; /* else keep last val */
} else if ( '_' != s->tok->sign[sl-1] ) { /* too much */
SERR2( "expected at most %d params for '%s'",
sl-2, s->tok->name );
goto broken;
}
}
break; /* close loop */
}
}
/* done with token -- close and fix frame */
#define KILLARGS() do { \
o->len = s->data; \
o->used = s->used; \
} while(0)
#define KILLOP() do { \
o->len = s->data - 1; \
o->used = s->used - (o->field[o->len].val ? o->field[o->len].len : 0); \
} while(0)
switch ( s->tok->tag ) {
case LFMT_PLUS: /* arithmetic */
case LFMT_MINUS:
case LFMT_MUL:
case LFMT_DIV: {
lll a = o->field[s->data-1].len;
lll b = o->field[s->data].len;
/* LOG_DBG( LFMT_DBG, "\tarith on %Ld %Ld", a, b ); */
switch ( s->tok->tag ) {
case LFMT_PLUS: a += b; break;
case LFMT_MINUS: a -= b; break;
case LFMT_MUL: a *= b; break;
case LFMT_DIV: if ( 0 != b ) a /= b; break;
}
KILLOP();
NADD( (int)a, 0 );
} break;
case LFMT_F: {
char b[32];
int ll = lprint( b, o->field[ s->data ].len );
KILLARGS();
ADD( 0, b, ll );
} break;
case LFMT_V:
r->iter[ r->i ].tag = o->field[ s->data ].len;
/* set iterator field */
o->field[ s->data ].tag = LFMT_ITR | r->iter[ r->i ].tag;
o->field[ s->data ].len = r->i;
break;
case LFMT_INDEX: {
LfmtIter * iter = r->iter + r->i;
if ( 1 != r->occ ) {
SERR( "OOPS! index not within loop" );
goto internalerr;
}
if ( ! iter->sub || (iter->submode = !iter->from) ) {
/* primary loop */
if ( !(iter->from = (int) o->field[ s->data ].len) )
iter->from = 1;
iter->to = s->pos ? (int) o->field[ s->data+1 ].len : iter->from;
} else {
/* additional subfield loop */
if ( !(iter->sfrom = (int) o->field[ s->data ].len) )
iter->sfrom = 1;
iter->sto = s->pos ? (int) o->field[ s->data+1 ].len : iter->sfrom;
}
iter->end = fp + tlen;
KILLARGS();
} break; /* case LFMT_INDEX: */
case LFMT_SUBFIELD:
if ( 1 != r->occ ) {
SERR( "OOPS! subfield not within loop" );
goto internalerr;
}
r->iter[ r->i ].sub = o->field[ s->data ].val[0];
r->iter[ r->i ].end = fp + tlen;
KILLARGS();
break;
case LFMT_LOOP:
if ( r->more ) {
fp = r->loop; /* start over */
tlen = 0; /* care for += tlen */
tok = NULL;
r->occ++;
r->more = 0;
r->i = 0;
goto doneclosing;
}
r->loop = 0;
break;
}
/* close frame, unless it's the outmost */
if ( ! sp )
break;
/* TODO: close loop ? */
s = &stack[ --sp ];
start = s->data + s->pos; /* start of current param in output */
flds = o->len - start; /* # fields for current param */
} while ( CLOSE_IMPL == close );
doneclosing:
if ( NULL == tok )
continue;
if ( ! tok->tag )
goto done;
/* add new token */
if ( NULL == r->loop ) /* open loop ? */
switch ( LFMT_GROUP(tok->tag) ) {
case LFMT_FUNCTIONS:
if ( LFMT_OPEN != tok->tag )
break;
goto openloop; /* else should do, but gcc doesn't grok it right */
case LFMT_LITERAL:
if ( LFMT_COND != tok->tag && LFMT_REP != tok->tag )
break;
case LFMT_ITERATORS:
openloop:
LOG_DBG( LFMT_DBG, "\topening loop on %s", tok->name );
PUSHFRAME( &loop );
s->rec = stack[sp-1].rec; /* inherit record mode */
r->loop = fp;
r->occ = 1;
r->more = 0;
r->had = 0;
r->i = 0;
if ( (s->expl = (LFMT_OPEN == tok->tag)) ) {
r->loop += tlen;
continue;
}
}
if ( tok->sign[2] && LFMT_LITERAL != LFMT_GROUP(tok->tag) ) {
/* token opens standard frame */
if ( LFMT_ITERATORS == LFMT_GROUP(tok->tag) ) {
if ( 1 != r->occ ) { /* not first run */
if ( r->iter[r->i].def != fp ) {
SERR1( "internal error at %d. iterator", r->i + 1 );
goto internalerr;
}
NADD( r->i, LFMT_ITR | r->iter[r->i].tag ); /* push iterator */
fp = r->iter[r->i].end; /* skip parsing */
tlen = 0; /* care for += tlen */
continue; /* next token */
}
if ( LFMT_NITER-1 == r->i ) {
SERR1( "maximum number of iterators %d exceeded", LFMT_NITER );
goto overflow;
}
memset( r->iter + r->i, 0, sizeof(r->iter[0]) );
r->iter[r->i].def = fp;
r->iter[r->i].end = fp + tlen;
}
PUSHFRAME( tok );
switch ( tok->tag ) { /* special treatment */
case LFMT_INDEX:
s->expl = 1;
break;
default:
if ( expl )
s->expl = 1;
}
LOG_DBG( LFMT_DBG, "\topen %s%c%s id %x",
s->tok->name, s->expl?'(':' ', s->tok->sign, s->tok->tag );
}
if ( lit.tag ) { /* a literal */
if ( NULL != lit.val ) { /* string literal */
switch ( lit.tag ) {
case -LFMT_COND:
if ( r->i ? !r->more : r->occ > 1 )
continue;
break;
case -LFMT_REP:
if ( r->i && !r->had )
continue;
break;
}
ADD( lit.tag, lit.val, lit.len );
LOG_DBG( LFMT_DBG, "\tlit %s%.*s id %x",
tok->name, lit.len, lit.val, tok->tag );
} else { /* numeric literal */
NADD( lit.len, lit.tag );
LOG_DBG( LFMT_DBG, "\tlit %d", lit.len );
}
/* new implicit frame immediatly closed by literal ?
problem are implicit loops
if ( o->len == s->data + 1 && ! s->expl ) {
tok = NULL;
goto closeframe;
}
*/
} else switch ( tok->tag ) { /* other special tokens */
case LFMT_MDL: f->mode = LFMT_MD; f->upcase = 0; break;
case LFMT_MDU: f->mode = LFMT_MD; f->upcase = 1; break;
case LFMT_MHL: f->mode = LFMT_MH; f->upcase = 0; break;
case LFMT_MHU: f->mode = LFMT_MH; f->upcase = 1; break;
case LFMT_MPL: f->mode = LFMT_MP; f->upcase = 0; break;
case LFMT_MPU: f->mode = LFMT_MP; f->upcase = 1; break;
}
} /* for token */
broken:
internalerr:
overflow:
outofmem:
done:
if ( NULL != o ) {
int i = o->len;
while ( i-- )
if ( NULL == o->field[i].val ) {
;
} else if ( -LFMT_LIT >= o->field[i].tag )
o->field[i].tag = 0;
}
return o;
} /* rFmt */
lio.c 0100644 0001130 0001130 00000053471 07644300474 010504 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lio.c,v 1.32 2003/04/07 13:11:48 kripke Exp $
I/O support for the openisis library.
*/
#include
#include /* vsnprintf */
#include
#include
#include
#include /* PATH_MAX */
#include
#ifdef WIN32
#define WIN32_LEAN_AND_MEAN
#define NONAMELESSUNION
# include
# include
# define SHUT_RD SD_RECEIVE
# define SHUT_WR SD_SEND
# define SHUT_RDWR SD_BOTH
# define vsnprintf _vsnprintf
# include
#else /* have operating system */
# define __USE_UNIX98
/*
# define _POSIX_MAPPED_FILES
# define _POSIX_SYNCHRONIZED_IO
*/
# include /* or try unixio */
# ifndef O_SYNC
# if defined( __FreeBSD__ )
# define O_SYNC O_FSYNC
# else
# define O_SYNC 0
# endif
# endif
# include
# include /* flock */
# include
# include
# include
#endif /* WIN32 */
#include /* gettimeofday */
#include /* localtime */
#include "luti.h" /* logging */
#include "lses.h"
#include "lstr.h" /* log_str */
#include "lio.h"
/* ************************************************************
private data
*/
#ifdef WIN32
/* it's not thread-safe, anyway */
static HANDLE hand[128];
#endif
/* ************************************************************
private functions
*/
#ifndef WIN32
static int lerrno ( int errno_ )
{
switch ( errno_ ) {
case EINTR:
case EAGAIN:
return ERR_OK;
case EFAULT: /* structurally bad address */
case ENAMETOOLONG:
case ELOOP:
return ERR_FAULT;
case EBADF: /* file "does not exist" */
case ENOENT:
case ENOTDIR:
return ERR_BADF;
case EIO: /* file "is not accessible" */
case EEXIST:
case EISDIR:
case EACCES:
case ENXIO:
case ENODEV:
case EROFS:
case ETXTBSY:
case ENOSPC:
case EPIPE:
case ESPIPE:
return ERR_IO;
case ENOMEM:
case EMFILE:
case ENFILE:
return ERR_NOMEM;
case EBUSY:
return ERR_BUSY;
}
return ERR_INVAL;
} /* lerrno */
/**
check unixio errno.
if harmless, return != 0 (is ok).
else return 0, possibly closing the file
*/
int unixio_ok ( int *file, int io )
{
switch ( errno ) {
case EINTR:
case EAGAIN:
return !0;
case EFAULT:
case ESPIPE:
break;
default:
log_msg( LOG_IOERR, "closing file %x for %x on %d", *file, io, errno );
lio_close( file, io );
}
return 0;
} /* unixio_err */
#endif
/**
return wether printing should be skipped
*/
static int seterrlev ( int *err, int *level, int code )
{
*level = LOG_MASK & code;
*err = ERR_MASK & code;
if ( ! code )
return 0;
if ( ! *err )
switch ( *level ) {
case LOG_FATAL: *err = ERR_IDIOT; break;
case LOG_ERROR: *err = ERR_INVAL; break;
case LOG_IOERR:
#ifndef WIN32
if ( EAGAIN == errno || EINTR == errno ) /* no error */
return !0;
case LOG_SYSERR:
*err = lerrno( errno );
#else
case LOG_SYSERR:
*err = ERR_INVAL;
#endif
break;
default:
*err = 0;
}
else if ( ! *level ) {
if ( ERR_TRASH <= *err )
*level = LOG_FATAL;
else if ( ERR_IO <= *err )
*level = LOG_SYSERR;
else if ( ERR_FAULT <= *err )
*level = LOG_ERROR;
else
*level = LOG_VERBOSE;
}
return *level > (int)log_lev;
} /* seterrlev */
/* ************************************************************
package data
*/
CLockFunc *lio_lock;
LogLevel log_lev = LOG_ERROR;
#ifndef WIN32
# define LIO_STATICFD( nm, flg ) int nm = flg
#else
# define LIO_STATICFD( nm, flg ) int nm
#endif
LIO_STATICFD( lio_in , LIO_IN | 0 );
LIO_STATICFD( lio_out, LIO_OUT | 1 );
LIO_STATICFD( lio_err, LIO_OUT | 2 );
/* ************************************************************
package functions
*/
int lio_init ()
{
#ifndef WIN32
/* static */
#else
if ( lio_in )
return 0;
lio_in = -1; /* prevent recursion */
lio_in = lio_open( "CONIN$", LIO_RD );
lio_out = lio_open( "CONOUT$", LIO_WR );
lio_err = lio_open( "oisiserr.txt", LIO_WR|LIO_CREAT );
#endif
return 0;
} /* lio_init */
int lio_open ( const char *name, int flags )
{
int fd = -1;
int faillvl = (LIO_OPT == ((LIO_OPT|LIO_WR)&flags))
? (LOG_VERBOSE|ERR_BADF) : LOG_IOERR;
if ( '&' == name[0] ) {
int i = 1;
fd = 0;
while ( '0' <= name[i] && name[i] <= '9' )
fd = 10*fd + name[i++] - '0';
}
#ifndef WIN32
if ( 0 > fd ) {
int f = !(LIO_WR&flags) ? O_RDONLY
: ((LIO_RD&flags) ? O_RDWR : O_WRONLY);
if ( LIO_CREAT & flags && LIO_WR & flags ) f |= O_CREAT;
if ( LIO_NBLK & flags ) f |= O_NONBLOCK;
if ( LIO_SYNC & flags ) f |= O_SYNC;
if ( LIO_TRUNC & flags ) f |= O_TRUNC;
if ( ! (LIO_SEEK & flags) ) f |= O_APPEND;
fd = open( name, f, 00664 ); /* let umask finetune */
if ( LIO_FD < fd ) {
log_msg( LOG_ERROR, "got big fd %d", fd );
fd = -1;
}
if ( 0 > fd )
return log_msg( faillvl, "could not open '%s' %x", name, flags );
/*
we want a non-shareable lock that still allows readers.
under linux, flock is never mandatory, while lockf may be.
/usr/src/linux/Documentation/mandatory.txt
an alternative would be to lock an otherwise useless byterange?
if ( LIO_FLOCK & flags )
flock( fd, LOCK_EX|LOCK_NB );
*/
} else if ( -1 == fcntl( fd, F_GETFL ) ) /* check open */
return -ERR_BADF;
#else
if ( ! lio_in )
lio_init();
if ( 0 > fd ) {
HANDLE h;
int acc = 0;
int shr = FILE_SHARE_READ | (LIO_FLOCK&flags ? 0 : FILE_SHARE_WRITE);
int cre = (LIO_CREAT & flags && LIO_WR & flags)
? (LIO_TRUNC & flags) ? CREATE_ALWAYS : OPEN_ALWAYS
: (LIO_TRUNC & flags) ? TRUNCATE_EXISTING : OPEN_EXISTING;
int f = (LIO_WR&flags) ? FILE_ATTRIBUTE_ARCHIVE : FILE_ATTRIBUTE_NORMAL;
if ( LIO_RD & flags ) acc |= GENERIC_READ; /* FILE_READ_DATA; */
if ( LIO_WR & flags ) acc |= GENERIC_WRITE;
/* some routines like GetFileSize are "documented" to require
GENERIC_READ/GENERIC_WRITE, so it may not be sufficient to use:
(LIO_SEEK & flags) ? FILE_WRITE_DATA : FILE_APPEND_DATA */;
/* if ( LIO_NBLK & flags ) f |= FILE_FLAG_OVERLAPPED; */
if ( LIO_SYNC & flags ) f |= FILE_FLAG_WRITE_THROUGH;
f |= (LIO_SEEK & flags) ? FILE_FLAG_RANDOM_ACCESS
: FILE_FLAG_SEQUENTIAL_SCAN;
for ( fd=0; hand[fd]; )
if ( sizeof(hand)/sizeof(hand[0]) == ++fd )
return -EMFILE;
h = CreateFile( name, acc, shr, 0, cre, f, 0 );
if ( INVALID_HANDLE_VALUE == h )
return log_msg( faillvl, "could not open '%s' %x", name, flags );
hand[fd] = h;
if ( (LIO_WR & flags) && ! (LIO_SEEK & flags) )
SetFilePointer( hand[fd], 0, 0, FILE_END );
} else if ( (int)(sizeof(hand)/sizeof(hand[0])) <= fd || ! hand[fd] )
return -ERR_BADF;
#endif
fd |= flags & LIO_WANT;
/* add status flags */
if ( LIO_RD & flags ) fd |= LIO_IN;
if ( LIO_WR & flags ) fd |= LIO_OUT;
return fd;
} /* lio_open */
int lio_close ( int *file, int flags )
{
int fd = 0xffff & *file;
int op = LIO_INOUT & *file;
int cl = LIO_INOUT & flags;
if ( ! op || ! cl )
return 0;
if ( 0 > *file ) { /* probably failed open */
*file = 0;
return 0;
}
*file &= ~cl;
if ( op & ~cl ) { /* remains partly open */
int how = (LIO_IN & cl) ? SHUT_RD : SHUT_WR;
if ( !(LIO_SOCK & fd) )
return 0;
#ifndef WIN32
if ( ! shutdown( fd, how ) )
return 0;
#else
/* TODO */
#endif
log_msg( LOG_SYSERR, "could not shutdown sock %d %d", fd, how );
return errno ? -errno : -1;
}
#ifndef WIN32
for ( fsync( fd ); close( fd ); errno = 0 )
if ( EINTR != errno ) {
log_msg( LOG_SYSERR, "could not close file %d", fd );
return errno ? -errno : -1;
}
#else
if ( (int)(sizeof(hand)/sizeof(hand[0])) <= fd )
return -ERR_BADF;
CloseHandle( hand[fd] );
hand[fd] = 0;
#endif
return 0;
} /* lio_close */
int lio_size ( int file )
{
if ( !((LIO_IN|LIO_OUT) & file) )
return log_msg( ERR_BADF, "*file 0x%x not open for stat", file );
{
int fd = LIO_FD & file;
#ifndef WIN32
struct stat s;
return fstat( fd, &s ) ? 0 : s.st_size;
#else
return GetFileSize( hand[fd], 0 );
#endif
}
} /* lio_size */
int lio_read ( int *file, void *buf, unsigned count )
{
if ( !(LIO_IN & *file) )
return log_msg( ERR_BADF, "*file 0x%x not open for reading", *file );
{
int fd = LIO_FD & *file;
#ifndef WIN32
int got = read( fd, buf, count );
/* log_msg( LOG_ERROR, "read 0x%x got %d", *file, got ); */
if ( got )
return 0 < got ? got
: unixio_ok( file, LIO_IN ) ? 0
: log_msg( LOG_IOERR, "could not read %d bytes from %d", count, fd );
else if ( LIO_SEEK & *file ) /* special EOF treatment */
return 0;
else {
lio_close( file, LIO_IN );
return -ERR_EOF;
}
#else
unsigned got = 0;
int ok = ReadFile( hand[fd], buf, count, &got, 0 );
return ok ? got
: log_msg( LOG_IOERR, "could not read %d bytes from %d", count, fd );
#endif
}
} /* lio_read */
int lio_write ( int *file, const void *buf, unsigned count )
{
if ( !(LIO_OUT & *file) )
return log_msg( ERR_BADF, "*file 0x%x not open for writing", *file );
{
int fd = LIO_FD & *file;
#ifndef WIN32
int got = write( fd, buf, count );
return 0 <= got ? got
: unixio_ok( file, LIO_OUT ) ? 0
: log_msg( LOG_IOERR, "could not read %d bytes from %d", count, fd );
#else
unsigned got = 0;
int ok = WriteFile( hand[fd], buf, count, &got, 0 );
return ok ? got
: log_msg( LOG_IOERR, "could not read %d bytes from %d", count, fd );
#endif
}
} /* lio_write */
int lio_pread ( int *file, void *buf, unsigned count, int offset )
{
if ( !(LIO_IN & *file) )
return log_msg( ERR_BADF, "*file 0x%x not open for reading", *file );
{
int fd = LIO_FD & *file;
#ifndef WIN32
int got = pread( fd, buf, count, offset );
return 0 <= got ? got
: unixio_ok( file, LIO_IN ) ? 0
: log_msg( LOG_IOERR, "could not read %d bytes from %d at %d",
count, fd, offset );
#else
return 0xffffffff /* "INVALID_SET_FILE_POINTER" */
== SetFilePointer( hand[fd], offset, 0, FILE_BEGIN )
? -1 : lio_read( file, buf, count );
#endif
}
} /* lio_pread */
int lio_pwrite ( int *file, const void *buf, unsigned count, int offset )
{
if ( !(LIO_OUT & *file) )
return log_msg( ERR_BADF, "*file 0x%x not open for writing", *file );
{
int fd = LIO_FD & *file;
#ifndef WIN32
int got = pwrite( fd, buf, count, offset );
return 0 <= got ? got
: unixio_ok( file, LIO_OUT ) ? 0
: log_msg( LOG_IOERR, "could not read %d bytes from %d at %d",
count, fd, offset );
#else
return 0xffffffff /* "INVALID_SET_FILE_POINTER" */
== SetFilePointer( hand[fd], offset, 0, FILE_BEGIN )
? -1 : lio_write( file, buf, count );
#endif
}
} /* lio_pwrite */
int lio_trunc ( int *file, int offset )
{
int fd = LIO_FD & *file;
#ifndef WIN32
ftruncate( fd, offset );
#else
/* SetFileValidData( hand[fd], offset ); XP only */
return 0xffffffff /* "INVALID_SET_FILE_POINTER" */
== SetFilePointer( hand[fd], offset, 0, FILE_BEGIN )
? -1 : SetEndOfFile( hand[fd] )
? 0 : log_msg( LOG_IOERR, "could not truncate %d to %d", fd, offset );
#endif
return 0;
} /* lio_trunc */
int lio_mmap ( int *file, void **map, int length )
{
if ( ! map )
return log_msg( ERR_INVAL, "no map" );
if ( 0 >= length && 0 >= (length = lio_size( *file )) ) {
*map = 0;
return length;
}
#ifndef WIN32
if ( file ) {
if ( *map )
return msync( *map, length, (LIO_SYNC & *file)
? (MS_SYNC|MS_INVALIDATE) : MS_ASYNC )
? log_msg( LOG_IOERR, "msync" ) : 0;
*map = mmap( 0, length, PROT_READ | ((LIO_OUT&*file) ? PROT_WRITE : 0),
MAP_SHARED, LIO_FD&*file, 0 );
if ( MAP_FAILED != *map )
return length;
*map = 0;
return log_msg( ERR_NOMEM, "mmap failed on fd %08x", *file );
}
if ( ! *map )
return log_msg( ERR_INVAL, "no *map" );
munmap( *map, length );
*map = 0;
return 0;
#else
/* awfully complicated here
-- need CreateFileMapping, MapViewOfFile
stupid piece of shrott
true mapping supported in NT family only, 9x copies to swap
*/
*map = 0;
return 0;
#endif
} /* lio_mmap */
int lio_slurp ( char **buf, int sz, const char *name, int opt )
{
int file = lio_open( name, LIO_RD|LIO_SEEK|(opt ? LIO_OPT : 0) );
int size;
int ret = -ERR_NOMEM;
char *p = *buf;
/* log_msg( LOG_IOERR, "open '%s' = %d", name, file ); */
if ( 0 > file )
return file;
size = lio_size( file );
if ( 0 >= size )
return size;
if ( size > sz )
LOG_OTO( done, ( ERR_INVAL, "file '%.30s' too big: %d > %d",
name, size, sz ) );
if ( ! p && !(p = lmalloc(size)) )
goto done;
ret = lio_read( &file, p, size );
if ( size == ret )
*buf = p;
else {
if ( ret >= 0 )
ret = log_msg( ERR_IO, "OOPS! got %d of %d bytes from '%.30s'",
ret, size, name );
if ( ! *buf )
free( p );
}
done:
lio_close( &file, LIO_INOUT );
return ret;
} /* lio_slurp */
/* ************************************************************
package functions
*/
int log_msg ( int code, const char *fmt, ... )
{
int err, level;
va_list ap;
va_start( ap, fmt );
#if 0
ret = sMsg( ses0, MSG_VA|code, fmt, ap );
make log_msg unbuffered, use sMsg for buffered logging
#else
if ( ! seterrlev( &err, &level, code ) && (LIO_OUT & lio_err) ) {
char buf[1024];
int len = vsnprintf( (char*)buf, sizeof(buf), fmt, ap );
if ( 0 < len ) {
if ( len < (int)sizeof(buf) )
buf[len++] = '\n';
lio_write( &lio_err, buf, len );
}
}
#endif
va_end( ap );
return -err;
} /* log_msg */
void log_str ( LogLevel level, int *rec, const char **desc )
{
int occ = -1;
int nmbrs = LSTRFIX(*rec);
int *mbr = rec+1;
char *base = (char*)rec;
if ( level > log_lev || ! desc )
return;
sMsg( ses0, 2, "record %.20s\n", *desc++ );
/* dump the fixed part (occ==-1) and each occurrence of repeated part. */
for ( ;/* occ < LSTROCC(*dst) */; ) { /* dump one part */
int i;
for ( i=0; i= LSTROCC(*rec) )
break;
if ( ! occ ) { /* was the fixed part, setup for repeated */
nmbrs = LSTRREP(*rec);
desc += i;
}
}
} /* log_str */
void log_hex ( LogLevel level, const void *mem, int len )
{
const char *p = (const char *)mem;
char buf[82];
int i = 0;
if ( level > log_lev )
return;
for ( ; i 16 )
left = 16;
for ( ; j < left; j++ ) {
sprintf( buf+pos, "%02x", p[j] );
pos += 2;
if ( 3 == j%4 ) { buf[pos++] = ' '; buf[pos++] = ' '; }
}
for ( ; j < 16; j++ ) {
buf[pos++] = ' '; buf[pos++] = ' ';
if ( 3 == j%4 ) { buf[pos++] = ' '; buf[pos++] = ' '; }
}
/* got 50 = 10 + 16*2 + 4*2 */
for ( j=0; j < left; j++ ) /* add up to 16 */
buf[pos++] = (0x60 & p[j]) ? p[j] : '.';
buf[pos++] = '\n';
buf[pos] = 0;
sMsg( ses0, 2, "%.*s", pos, buf );
}
} /* log_hex */
/* ************************************************************
public functions
*/
int timeUpd ( Tm *tm )
{
Tm o = *tm;
#ifdef WIN32
struct _timeb tb;
_ftime( tb );
tm->millis = tb.time*LLL(1000) + tb.millitm;
#else
struct timeval tv;
gettimeofday( &tv, 0 );
tm->millis = tv.tv_sec*LLL(1000) + tv.tv_usec/1000;
#endif
return (int)(tm->millis - o.millis);
} /* timeUpd */
static int timeLoc ( struct tm *t, Tm *tm )
{
Tm x;
time_t tt;
if ( !tm || !tm->millis )
timeUpd( tm ? tm : (tm = &x) );
tt = (time_t)(tm->millis / 1000);
#ifdef WIN32 /* did I mention it's not threadsafe ? */
*t = localtime( &tt );
#else
localtime_r( &tt, t );
#endif
return (int)(tm->millis % 1000);
} /* timeLoc */
char *timeGtf ( char *buf, Tm *tm )
{
struct tm t;
timeLoc( &t, tm );
snprintf( buf, 15, "%04u%02u%02u%02u%02u%02u",
1900+t.tm_year, 1+t.tm_mon, t.tm_mday,
t.tm_hour, t.tm_min, t.tm_sec );
buf[14] = 0;
return buf;
} /* timeGtf */
char *timeGtfm ( char *buf, Tm *tm )
{
struct tm t;
int millis = timeLoc( &t, tm );
snprintf( buf, 19, "%04u%02u%02u%02u%02u%02u%03u",
1900+t.tm_year, 1+t.tm_mon, t.tm_mday,
t.tm_hour, t.tm_min, t.tm_sec, millis );
buf[18] = 0;
return buf;
} /* timeGtfm */
void timeSleep ( Tm *tm )
{
struct timespec ts;
ts.tv_sec = tm->millis / 1000;
ts.tv_nsec = 1000000 * (int)(tm->millis % 1000);
nanosleep( &ts, 0 );
} /* timeSleep */
int ioStream ( Ios *s, int op )
{
Buf *b;
switch ( op ) {
case LIO_SSIZE:
return sizeof(Ios);
case LIO_SOPEN:
s->pos = s->b.fill = s->b.done = 0;
case LIO_SCLOSE:
s->file &= ~LIO_INOUT;
return 0;
case LIO_SPURGE:
if ( !(LIO_IN & s->file) )
return -ERR_BADF;
/* try to get 4k chunk */
b = /* s->alt ? s->alt : */ &s->b;
if ( LIO_BUFSIZ/2 < b->fill && b->done ) { /* move contents downwards */
if ( b->done < b->fill )
memmove( b->c, b->c + b->done, b->fill - b->done );
s->pos += b->done;
b->fill -= b->done;
b->done = 0;
}
return LIO_BUFSIZ <= b->fill ? 0
: LIO_BUFSIZ/2 < b->fill ? LIO_BUFSIZ - b->fill
: LIO_BUFSIZ/2;
case LIO_SFILL:
case LIO_SFLUSH:
s->file &= ~LIO_INOUT;
}
return -ERR_EOF;
} /* ioStream */
int ioStdio ( Ios *s, int op )
{
Buf *b;
int ret;
switch ( op ) {
case LIO_SOPEN:
ioStream( s, op );
if ( 0 < (ret = lio_open( s->name, s->file )) )
s->file = ret;
return ret;
case LIO_SCLOSE:
if ( LIO_OUT & s->file )
s->func( s, LIO_SFLUSH );
/* don't close borrowed fd */
return '&' == *s->name ? 0 : lio_close( &s->file, LIO_INOUT );
case LIO_SFILL:
if ( !(LIO_IN & s->file) )
return -ERR_BADF;
if ( 0 < (ret = ioStream( s, LIO_SPURGE ))
&& 0 <= (ret = lio_read( &s->file, s->b.c + s->b.fill, ret ))
)
s->b.fill += ret;
log_msg( LOG_ALL, "LIO_FILL: got %d now %d", ret, s->b.fill );
return ret;
case LIO_SFLUSH:
if ( !(LIO_OUT & s->file) )
return -ERR_BADF;
b = /* s->alt ? s->alt : */ &s->b;
if ( b->fill <= b->done )
return 0;
if ( 0 < (ret = lio_write( &s->file, b->c + b->done, b->fill - b->done )) )
b->done += ret;
if ( b->fill == b->done ) {
s->pos += b->done;
b->fill = b->done = 0;
} else if ( b->done && LIO_BUFSIZ/2 < b->fill )
ioStream( s, LIO_SPURGE );
return ret;
}
return ioStream( s, op );
} /* ioStdio */
int sMsg ( Session s, int to, const char *fmt, ... )
{
int code = (LOG_MASK|ERR_MASK) & to, err, level;
int fd = LIO_FD & to; /* stream id uses same mask as system fd */
Ios *o = s->io[ fd ? fd : code ? 2 : 1 ];
va_list ap;
int space;
int ret;
if ( LSES_FILE_MAX <= fd || ! o )
return -1;
if ( seterrlev( &err, &level, code ) ) /* logging */
return -err;
/* core fprintf */
if ( LIO_BUFSIZ/2 < o->b.fill )
LIO_FLUSH( o );
space = LIO_BUFSIZ - o->b.fill;
va_start( ap, fmt );
ret = vsnprintf( (char*)o->b.c + o->b.fill, space, fmt,
(MSG_VA & to) ? va_arg( ap, va_list ) : ap );
va_end( ap );
if ( ret < 0 || space < ret ) /* outta space */
ret = space;
o->b.fill += ret;
if ( ! code )
return ret;
/* logging afterburner */
if ( LOG_SYSERR == level || LOG_IOERR == level ) {
int len;
char *syserr =
#ifndef WIN32
strerror(errno);
#else
0;
char buf[256] = "";
if (
FormatMessage(
FORMAT_MESSAGE_FROM_SYSTEM|FORMAT_MESSAGE_IGNORE_INSERTS,
0, GetLastError(), 0, buf, sizeof(buf)-1, 0 )
)
syserr = buf;
buf[ sizeof(buf)-1 ] = 0;
#endif
if ( syserr ) {
len = strlen( syserr );
if ( o->b.fill+4+len < LIO_BUFSIZ ) {
memcpy( o->b.c + o->b.fill, "\n\t: ", 4 );
memcpy( o->b.c + o->b.fill+4, syserr, len );
o->b.fill += 4+len;
}
}
}
if ( o->b.fill < LIO_BUFSIZ )
o->b.c[ o->b.fill++ ] = '\n';
LIO_FLUSH( o );
return -err;
} /* sMsg */
int sGets ( Session ses, int fd, char **ptr, char delim )
{
Ios *s = ses->io[ LIO_FD & fd ];
if ( s->b.fill > s->b.done || 0 < LIO_FILL( s ) ) for (;;) {
char *f = memchr( s->b.c + s->b.done, delim, s->b.fill - s->b.done );
int len;
log_msg( LOG_ALL, "sGets: done %d fill %d", s->b.done, s->b.fill );
f = memchr( s->b.c + s->b.done, delim, s->b.fill - s->b.done );
if ( ! f ) {
if ( (s->b.fill < LIO_BUFSIZ || s->b.done) /* buffer avail */
&& (LIO_IN & s->file) /* file open */
&& 0 < LIO_FILL( s )
)
continue;
else if ( s->b.done >= s->b.fill )
break;
f = (char*)s->b.c + s->b.fill;
}
len = f - (*ptr = (char*)s->b.c + s->b.done);
s->b.done += len + 1;
if ( len && '\n' == delim && '\r' == (*ptr)[len-1] )
len--;
return len;
}
*ptr = 0;
return -1;
} /* sGets */
void cLog ( int level, const char *filename )
{
if ( 0 > level ) /* no change */
;
else if ( LOG_LEVELS > level ) /* by basic number */
level <<= LOG_SHIFT;
else if ( 'z' >= level ) { /* by ascii value */
switch ( level ) {
case '-': level = LOG_NOCHANGE; break;
case 'o': level = LOG_OFF; break;
case 'f': level = LOG_FATAL; break;
case 's': level = LOG_IOERR /*LOG_SYSERR*/; break;
default:
if ( '0' <= level && level <= '9' ) {
level = (level - '0')<= 0 )
log_lev = (LogLevel)level;
(void)filename;
/* TODO if ( filename )
sOpen( cOpen(0), "2>"filename, 0, ioStdio ) */
} /* cLog */
lio.h 0100644 0001130 0001130 00000021021 07637671272 010504 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef LIO_H
/*
$Id: lio.h,v 1.17 2003/03/24 21:03:54 kripke Exp $
I/O support for the openisis library.
Provides non-blocking, look-ahead network streams and other nice stuff.
*/
#include "luti.h"
#include "lll.h"
/* ************************************************************
time
*/
/* struct wrapper just for struct Tm *Time */
typedef struct Tm {
lll millis;
} Tm;
/**
set Tm to current time, return difference in millis
*/
extern int timeUpd ( Tm * );
/**
print generalized time yyyyMMddHHmmss to buffer
if Tm is 0, current time is used
if Tm.millis is 0, Tm is updated
return buffer, which must have 15 bytes
*/
extern char *timeGtf ( char *buf, Tm * );
/**
like timeAsc, but with additional 4 digits millis
return buffer, which must have 19 bytes
*/
extern char *timeGtfm ( char *buf, Tm * );
/**
nanosl
*/
extern void timeSleep ( Tm * );
/* ************************************************************
system files
*/
/**
system file is a non-negative 32bit int including status bits.
main status bits are LIO_IN and LIO_OUT,
which are set by open and cleared by close.
negative values are reserved for use by non-system based streams.
*/
#define LIO_WANT 0xfff00000 /* what we want: open flags */
#define LIO_FD 0x0000ffff /* what we get: system fd */
#define LIO_STAT 0x000f0000 /* how it's going: it's status */
/* status flags */
#define LIO_IN 0x00010000 /* support input */
#define LIO_OUT 0x00020000 /* support output */
#define LIO_INOUT 0x00030000 /* support both */
#define LIO_ISOPEN( file ) (LIO_INOUT & (file))
/* common requirements that are also used by non-system streams. */
#define LIO_RD 0x00100000 /* shall be opened for input */
#define LIO_WR 0x00200000 /* shall be opened for output */
#define LIO_RDWR 0x00300000 /* shall be opened for both */
#define LIO_SYNC 0x00400000 /* syncing output */
#define LIO_NBLK 0x00800000 /* non blocking */
/* system files only. */
#define LIO_CREAT 0x01000000 /* with WR: shall be created */
#define LIO_OPT 0x01000000 /* with RD only: do not complain if open fails */
#define LIO_TRUNC 0x02000000 /* shall be truncated */
#define LIO_SEEK 0x04000000 /* random access, else append mode */
#define LIO_SOCK 0x08000000 /* is a socket (support shutdown) */
#define LIO_FLOCK 0x10000000 /* shall be locked (EX with WR) */
/** create a new fid based on name and flags.
name may be &i to use fd i, esp. for i=0,1 or 2.
@return non-negative file or some error code
*/
extern int lio_open ( const char *name, int flags );
extern int lio_close ( int *file, int flags );
extern int lio_size ( int file );
extern int lio_in;
extern int lio_out;
extern int lio_err;
extern int lio_init ();
/*
Like the syscalls, this returns the number of bytes on success.
Unlike the syscalls, it returns 0 rather than an error
when no bytes are available after interrupt (EINTR)
or on non-blocking IO (EAGAIN); i.e. you may try later.
On error, a negative value is returned.
On errors that render the file unusable,
it is closed for input or output, resp.
These are most errors but EFAULT (bad buf)
and ESPIPE (which clears the SEEK bit).
For files that do not have the SEEK bit set,
EOF is considered such an error and -LERR_EOF is returned.
If you want to stream a disk file like 'tail -f', set the SEEK bit.
*/
extern int lio_read ( int *file, void *buf, unsigned count );
extern int lio_write ( int *file, const void *buf, unsigned count );
extern int lio_pread ( int *file, void *buf, unsigned count, int offset );
extern int lio_pwrite ( int *file, const void *buf, unsigned count, int offset );
extern int lio_trunc ( int *file, int offset );
/**
open, sync or close a memory mapping
@param file pointer to open file handle to map.
if file is NULL, *map is unmapped (and set to NULL)
else if *map is NULL, length bytes are mapped
else *map is synced (flushed)
@param length number of bytes to map; if 0, size will be used
@return mapped length; if <= 0, *map is set to 0, else to memory region
*/
extern int lio_mmap ( int *file, void **map, int length );
/** slurp in a whole file at once.
@param buf points to buffer of size sz.
lio_slurp will allocate one, if *buf is NULL
@param sz maximum number of bytes to read
@param name of file to slurp
@param opt if != 0, do not complain on failure
@return number of bytes read or negative on error
*/
extern int lio_slurp ( char **buf, int sz, const char *name, int opt );
/* ************************************************************
streams
*/
/** an I/O buffered stream similar to stdio's FILE.
Unlike FILE with it's stupid ungetc,
it supports efficient scanning of the whole input buffer without
actually consuming it.
Also unlike at least Linux' glibc's stdio, access is not interlocked.
The buffer is typically used for input OR output.
However, for internal pipes, the same buffer is used from both ends.
High level access functions like printf operate directly on the buffer
and call the stream function on buffer underflow or overflow
(or explicit flush).
*/
typedef int lio_sfunc ( struct Ios *s, int op );
enum {
LIO_BUFSIZ = 8192
};
typedef struct Buf {
/* struct Buf *nxt; chain -- not used */
short fill; /* total #bytes in the buffer */
short done; /* #bytes done */
unsigned char c[LIO_BUFSIZ]; /* actual buffer may be longer */
} Buf;
/* fill - done is the number of available bytes */
#define LIO_BAVAIL( b ) ((b)->fill - (b)->done)
#define LIO_BINIT( b ) do { \
/*(b)->nxt = 0;*/ (b)->fill = 0; (b)->done = 0; \
} while (0)
typedef struct Ios {
lio_sfunc *func;
const char *name; /* stream's name */
int file; /* file "id" and flags as above */
int pos; /* logical position of buffer start within stream */
Buf b;
} Ios;
#define LIO_SISOPEN( s ) LIO_ISOPEN( (s)->file )
#define LIO_SAVAIL( s ) LIO_BAVAIL( &(s)->b )
#define LIO_SINIT( s, fun, nam, fil ) do { \
(s)->func = fun; (s)->name = nam; (s)->file = fil; \
(s)->pos = 0; \
LIO_BINIT( &(s)->b ); \
} while (0)
#define LIO_STDINIT( s, nam, fil ) LIO_SINIT( s, lio_stdio, nam, fil )
/** stream operations.
*/
enum {
LIO_SSIZE, /* tell the size of the stream structure */
LIO_SOPEN, /* initialize the structure */
LIO_SCLOSE, /* close */
LIO_SPURGE, /* kill the done bytes. return fill buffer size */
LIO_SFILL, /* try to refill the buffer (poll input). return # new bytes */
LIO_SFLUSH, /* try to flush the buffer. return # bytes written */
LIO_SPUSH /* push input stream: munge prefilled input */
};
/**
"abstract" base implementation of stream.
not very useful in itself, but used by derived impl.
*/
extern int ioStream ( Ios *s, int op );
/**
stream func based on system file to mimic stdio.
on LIO_SOPEN, the fid is set by lio_open based on name.
other ops used the correponding lio system call.
*/
extern int ioStdio ( Ios *s, int op );
#define LIO_OPEN( s ) (s)->func( (s), LIO_SOPEN )
#define LIO_CLOSE( s ) (s)->func( (s), LIO_SCLOSE )
#define LIO_FILL( s ) (s)->func( (s), LIO_SFILL )
#define LIO_FLUSH( s ) (s)->func( (s), LIO_SFLUSH )
/* ************************************************************
inter process communication
*/
extern CLockFunc *lio_lock;
#define LIO_LOCK() (NULL != lio_lock && lio_lock(OPENISIS_LOCK))
#define LIO_RELE() (NULL != lio_lock && lio_lock(OPENISIS_RELE))
#define LIO_WAIT( c ) (NULL != lio_lock && lio_lock(OPENISIS_WAIT \
| (OPENISIS_COND & (c))))
#define LIO_WAKE( c ) (NULL != lio_lock && lio_lock(OPENISIS_WAKE \
| (OPENISIS_COND & (c))))
#define LIO_H
#endif /* LIO_H */
lll.h 0100664 0001130 0001130 00000003166 07630630313 010500 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef LLL_H
/*
$Id: lll.h,v 1.1 2003/03/03 10:39:07 kripke Exp $
64 bit integers
*/
#ifdef __GNUC__
# define LLL_BUILTIN long long
# define LLL( v ) v##LL
#elif defined( _MSC_VER ) || defined( __BORLANDC__ )
# define LLL_BUILTIN __int64
# define LLL( v ) v /* is there some special suffix ??? */
#elif 0xFFFFFFFFL != ~0L
# define LLL_BUILTIN long
# define LLL( v ) v##L
#endif
#ifndef LLL_BUILTIN
# error "please use a compiler providing a 64 bit integer type. thanks."
#endif
typedef LLL_BUILTIN lll;
#define LLL_H
#endif /* LLL_H */
loi.h 0100644 0001130 0001130 00000016724 07641110323 010476 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef LOI_H
/*
$Id: loi.h,v 1.22 2003/03/28 18:13:39 kripke Exp $
lib openisis: provides the "openisis namespace",
i.e. short names for most openIsis names,
so we don't get to dislike those letters ...
*/
#include
#define OPENISIS_NOPRE09
#include "openisis.h"
#define Fld OpenIsisFld
#define Field OpenIsisField
#define Raw OpenIsisRaw
#define Rec OpenIsisRec
#define BASESZ OPENISIS_BASESZ
#define INITREC OPENISIS_INITREC
#define INITSTD OPENISIS_INITSTD
#define INITBUF OPENISIS_INITBUF
#define CLRREC OPENISIS_CLRREC
#define RECOK OPENISIS_RECOK
#define rGet openIsisRGet
#define rOccurence openIsisROccurence
#define rInt openIsisRInt
#define rInt2 openIsisRInt2
#define rEnum openIsisREnum
#define rString openIsisRString
#define rString2 openIsisRString2
#define a2i openIsisA2i
#define a2il openIsisA2il
#define a2id openIsisA2id
#define i2a openIsisI2a
#define u2a openIsisU2a
#define rSer openIsisRSerialize
#define rDeser openIsisRDeserialize
#define FTX OPENISIS_FTX
#define FTA OPENISIS_FTA
#define FTN OPENISIS_FTN
#define FTP OPENISIS_FTP
#define FTI OPENISIS_FTI
#define FTE OPENISIS_FTE
#define FTB OPENISIS_FTB
#define FTT OPENISIS_FTT
#define FTS OPENISIS_FTS
#define FTF OPENISIS_FTF
#define FTV OPENISIS_FTV
#define NOENUM OPENISIS_NOENUM
#define Fd OpenIsisFd
#define Fdt OpenIsisFdt
#define fdById openIsisFdById
#define fdByName openIsisFdByName
#define fdEnum openIsisFdEnum
#define Ses OpenIsisSes
#define Session OpenIsisSession /* Ses* */
#define mAlloc openIsisMAlloc
#define mFree openIsisMFree
#define mDup openIsisMDup
#define toHtml openIsisToHtml
#define utf8Chk openIsisUtf8Chk
#define Stb OpenIsisStb
#define Stub OpenIsisStub /* Stb* */
#define Db OpenIsisDb
#define Schema OpenIsisSchema
#define DBNLEN OPENISIS_DB_NAMELEN
#define SCNLEN OPENISIS_SC_NAMELEN
#define DBIMSK OPENISIS_DBIDMSK
#define SCIMSK OPENISIS_SCIDMSK
#define nSchema openIsisNSchema
#define nSend openIsisNSend
#define nRecv openIsisNRecv
#define nClean openIsisNClean
#define nDbByName openIsisNDbByName
#define nDbById openIsisNDbById
#define Ios OpenIsisIos
#define Stream OpenIsisStream /* Ios* */
#define SFunc OpenIsisSFunc
#define ERR_OK OPENISIS_ERR_OK
#define ERR_EOF OPENISIS_ERR_EOF
#define ERR_FAULT OPENISIS_ERR_FAULT
#define ERR_INVAL OPENISIS_ERR_INVAL
#define ERR_BADF OPENISIS_ERR_BADF
#define ERR_IO OPENISIS_ERR_IO
#define ERR_NOMEM OPENISIS_ERR_NOMEM
#define ERR_BUSY OPENISIS_ERR_BUSY
#define ERR_TRASH OPENISIS_ERR_TRASH
#define ERR_IDIOT OPENISIS_ERR_IDIOT
#define ERR_ERRORS OPENISIS_ERR_ERRORS
#define ERR_MASK OPENISIS_ERR_MASK
#define ERR_SHIFT OPENISIS_ERR_SHIFT
#define LOG_OFF OPENISIS_LOG_OFF
#define LOG_FATAL OPENISIS_LOG_FATAL
#define LOG_SYSERR OPENISIS_LOG_SYSERR
#define LOG_IOERR OPENISIS_LOG_IOERR
#define LOG_ERROR OPENISIS_LOG_ERROR
#define LOG_WARN OPENISIS_LOG_WARN
#define LOG_INFO OPENISIS_LOG_INFO
#define LOG_VERBOSE OPENISIS_LOG_VERBOSE
#define LOG_TRACE OPENISIS_LOG_TRACE
#define LOG_DEBUG OPENISIS_LOG_DEBUG
#define LOG_ALL OPENISIS_LOG_ALL
#define LOG_NOCHANGE OPENISIS_LOG_NOCHANGE
#define LOG_LEVELS OPENISIS_LOG_LEVELS
#define LOG_MASK OPENISIS_LOG_MASK
#define LOG_SHIFT OPENISIS_LOG_SHIFT
#define MSG_VA OPENISIS_MSG_VA
#define LogLevel OpenIsisLogLevel
#define sMsg openIsisSMsg
#define sOpen openIsisSOpen
#define sGets openIsisSGets
#define sReadln openIsisSReadln
#define sGetr openIsisSGetr
#define rDup openIsisRDup
#define rMsg openIsisRMsg
#define rFmt openIsisRFmt
#define rSplitf openIsisRSplitf
#define rSet openIsisRSet
#define rAddI openIsisRAddI
#define RSPACE OPENISIS_RSPACE
#define RADD OPENISIS_RADD
#define RCAT OPENISIS_RCAT
#define RADDS OPENISIS_RADDS
#define RADDF OPENISIS_RADDF
#define RCATS OPENISIS_RCATS
#define RARGC OPENISIS_RARGC
#define RARGV OPENISIS_RARGV
#define RFDT OPENISIS_RFDT
#define RDIS OPENISIS_RDIS
#define RNOC OPENISIS_RNOC
#define RIGN OPENISIS_RIGN
#define RCHG OPENISIS_RCHG
#define RDEL OPENISIS_RDEL
#define ROP OPENISIS_ROP
#define Set OpenIsisSet
#define Val OpenIsisVal
#define Key OpenIsisKey
#define Hit OpenIsisHit
#define XKEY OPENISIS_XKEY
#define XVAL OPENISIS_XVAL
#define XHIT OPENISIS_XHIT
#define XHEX OPENISIS_XHEX
#define XDOT OPENISIS_XDOT
#define XFST OPENISIS_XFST
#define dRead openIsisDRead
#define dRaw openIsisDRaw
#define dFmt openIsisDFmt
#define dMaxId openIsisDMaxId
#define dWrite openIsisDWrite
#define dWritex openIsisDWritex
#define dScan openIsisDScan
#define dTerm openIsisDTerm
#define dQuery openIsisDQuery
#define DXCb OpenIsisDXCb
#define DXLoop OpenIsisDXLoop
#define IDXEQ OPENISIS_IDXEQ
#define IDXPF OPENISIS_IDXPF
#define IDXUPTO OPENISIS_IDXUPTO
#define IDXINCL OPENISIS_IDXINCL
#define IDXMODE OPENISIS_IDXMODE
#define dXLoop openIsisDXLoop
#define Idx OpenIsisIdx
#define Index OpenIsisIndex /* Idx* */
#define CLockFunc OpenIsisCLockFunc
#define cOpen openIsisCOpen
#define cInit openIsisCInit
#define cLog openIsisCLog
#define cSession openIsisCSession
#define cDOpen openIsisCDOpen
#define cDOpenv openIsisCDOpenv
#define cDCheck openIsisCDCheck
#define cDClose openIsisCDClose
#define cXOpen openIsisCXOpen
#define cXMkVal openIsisCXMkVal
#define cXAdd openIsisCXAdd
#define cXClose openIsisCXClose
#define cFdt openIsisCFdt
#define cFreeFdt openIsisCFreeFdt
#define stub0 openisis_stub0
#define ses0 openisis_ses0
extern Session ses0;
/*
the following are not yet blessed by an official OpenIsis prefix
*/
/* lio */
#define Buf LioBuf
#define ioStream lio_stream
#define ioStdio lio_stdio
/* lsv */
#define Que LsvQue
#define Pool LsvPool
#define Wrk LsvWrk
#define Srv LsvSrv
#define Con LsvCon
#define svCur lsv_cur
#define svPlain lsv_plain
#define svEcho lsv_echo
#define svRun lsv_run
#define LOI_H
#endif /* LOI_H */
lqry.c 0100644 0001130 0001130 00000013632 07630630466 010705 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lqry.c,v 1.6 2003/03/03 10:40:25 kripke Exp $
implementation of record cooking.
*/
#include /* memset et al */
#include "ldb.h"
/* ************************************************************
private types
*/
/* ************************************************************
private data
*/
static unsigned char op[256];
/* ************************************************************
private functions
*/
static int lqry_scan ( OpenIsisSet *set, int db,
const char *key, int mode, int skip )
{
(void)set;(void)db;(void)key;(void)mode;(void)skip;
log_msg( LOG_ERROR, "scan mode not implemented yet, sorry!" );
return -1;
} /* lqry_scan */
/* ************************************************************
package functions
*/
/* ************************************************************
public functions
*/
int dQuery ( Session ses, Set *set, int db,
const char *key, int mode, int skip )
{
LdbPost p;
int ret = 0;
memset( &p, 0, sizeof(p) );
p.skp = skip;
p.tag = mode >> 16;
mode &= 0xff;
if ( OPENISIS_QRY_SIMPLE > mode ) {
char *tmp = (char*)key;
if ( OPENISIS_QRY_SCANE <= mode )
return lqry_scan( set, db, key, mode, skip );
switch ( (int)mode ) {
case OPENISIS_QRY_KEYEQ: break;
case OPENISIS_QRY_KEYAT:
{
size_t l = strlen(key);
if ( !l || '$' != key[l-1] )
break;
tmp = mAlloc( ses, l );
if ( ! tmp )
return -ERR_NOMEM;
memcpy( tmp, key, l-1 );
tmp[l-1] = 0;
}
case OPENISIS_QRY_KEYPF:
p.mode = LDB_PFX;
}
ret = ldb_search( db, tmp, &p, 0 );
if ( tmp != key )
mFree( ses, tmp );
/* } else if ( OPENISIS_QRY_SIMPLE < mode ) { */
} else { /* handling query expression (simple) */
size_t klen = strlen(key);
unsigned char *buf, *uc, *term, *end, utmp;
if ( ! op['.'] ) /* operators */
op['.'] = op['$'] = op['('] = op['*'] = op['+'] = op['^'] = op['/'] = 1;
buf = (unsigned char*)mAlloc( ses, klen+1 );
if ( ! buf )
return -ERR_NOMEM;
/* startover: */
memcpy( buf, key, klen+1 ); /* reload, might have frobed the buf on \ */
uc = buf;
p.mode = LDB_OR;
nextterm:
while ( *uc && ' ' >= *uc ) uc++; /* skip white */
if ( !*uc ) goto done;
/* take as term, whatever it is ... */
term = uc;
if ( '"' == *term ) { /* anything up to next unescaped " */
int bs = 0; /* count of backslashes seen so far */
term++;
while ( *++uc && '"' != *uc ) {
if ( '\\' == *uc ) {
if ( ! *++uc ) break; /* trailing \ */
bs++;
}
if ( bs ) uc[-bs] = *uc;
}
end = uc++ - bs;
if ( '$' == end[-1] ) {
p.mode |= LDB_PFX;
end--;
}
} else {
while ( *++uc && !op[*uc]
&& (' '<*uc || (uc==term+3 && !memcmp("ANY",term,3)))
); /* find white or op */
end = uc;
if ( '$' == *uc ) {
p.mode |= LDB_PFX;
uc++;
}
}
/* now end is on 1st char after term, possibly on a $ or ".
uc is on next pos to scan. */
/* check for tag */
p.tag = 0;
if ( '/' == *uc ) {
int parenths = '(' == *++uc;
if ( parenths ) uc++; /* accept tag w o w/o () */
/* eat arbitrary large numbers :)) */
while ( '0'<=*uc && *uc<='9' ) p.tag = 10*p.tag + *uc++ - '0';
if ( parenths ) while ( *uc && ')'!=*uc++ ); /* mv behind ) */
}
/* do it */
utmp = *end; *end = 0; /* terminate term */
if ( !(p.mode & (LDB_AND|LDB_NOT)) )
p.near = -1; /* collect all pos */
else if ( !p.fil ) /* nothing to AND */
goto operator;
if ( 0 > (ret = ldb_search( db, (const char*)term, &p, 0 )) )
goto done;
sMsg( ses, LOG_INFO, "src %d '%.30s'@%d %hd(%hd): fill %d cut %d",
p.skp, term, p.tag, p.mode, p.near, p.fil, p.cut );
operator:
*end = utmp; /* restore */
/* look for next operator */
p.mode = LDB_AND;
p.near = 0;
while ( *uc && ' ' >= *uc ) uc++; /* skip white */
if ( !*uc ) goto done;
switch ( *uc ) {
case '*': uc++; break;
case '^': p.mode = LDB_NOT; uc++; break;
case '+': p.mode = LDB_OR; uc++; break;
case '.': while ( '.' == *uc++ ) p.near++; break;
case '$': while ( '$' == *uc++ ) p.near--; break;
case '(':
if ( ! uc[1] ) goto done;
if ( ')' != uc[2] ) break;
if ( 'F' == uc[1] ) p.near = LDB_NEAR_F;
if ( 'G' == uc[1] ) p.near = LDB_NEAR_G;
if ( '0'= uc[2] ) { p.mode = LDB_OR; uc += 3; }
break;
case 'A':
if ( 'N' == uc[1] && 'D' == uc[2] && ' ' >= uc[3] ) uc += 4;
break;
case 'N':
if ( 'O' == uc[1] && 'T' == uc[2] && ' ' >= uc[3] ) {
p.mode = LDB_NOT; uc += 4;
}
break;
}
goto nextterm;
done:
/* somewhat screws the idea of limiting costs ...
if ( !p.fil && p.cut > p.skp ) {
sMsg( ses, LOG_INFO, "OUCH! full cut at %d for '%.50s' >= %d",
p.cut, key, p.skp );
p.skp = p.cut;
p.cut = 0;
goto startover;
}
*/
mFree( ses, buf );
}
return 0 > ret ? ret : ldb_p2s( set, &p );
} /* openIsisQuery */
lrec.c 0100644 0001130 0001130 00000055103 07641115777 010647 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lrec.c,v 1.43 2003/03/28 19:01:19 kripke Exp $
implementation of record cooking.
*/
#include
#include
#include /* vsnprintf */
#include /* free */
#include /* memset et al */
#ifdef WIN32
# define vsnprintf _vsnprintf
#endif /* WIN32 */
#include "ldb.h"
#include "lio.h"
#include "lcs.h"
/* ************************************************************
private types
*/
typedef struct {
int len; /* length of tagf */
int tagf[2]; /* tag and modification flags */
} LrecMF;
#define MF_SET 0x01
#define MF_DEL 0x02
/* ************************************************************
private data
*/
/* ************************************************************
private functions
*/
#define FIXUP( r ) \
{ \
char * base = (char*)r; \
int i; \
for ( i=0; i < (r)->len; i++ ) \
(r)->field[i].val = base + (int)(r)->field[i].val; \
}
static LrecMF* mfCtor (Session ses, Rec *rec) {
int len = (int)(2 * rec->len);
int *M;
Field *F;
LrecMF *lmf = (LrecMF*) mAlloc (ses, (1 + len) * sizeof (int));
if (0 == lmf) {
return 0;
}
memset (lmf->tagf, 0, len * sizeof (int));
lmf->len = len;
for (M = lmf->tagf, F = rec->field; len > M - lmf->tagf; ++F, M += 2) {
*M = (int) F->tag;
}
return lmf;
}
static int rCompact (Session ses, Rec *rec) {
int rt = 0;
if (0 != rec) {
char *V = (char*)rec + rec->base;
Field *F = rec->field;
Field *L = F + rec->len;
int used = rec->base;
while (F < L) {
if (V != F->val) {
rt = !0;
break;
}
V += F->len;
used += F->len;
++F;
}
if (rt) {
char *buf = (char*) mAlloc (ses, rec->used - (V - (char*)rec));
char *B = buf;
int nb = 0;
if (0 == buf) {
return 0;
}
while (F < L) {
memcpy (B, F->val, F->len);
F->val = V;
B += F->len;
V += F->len;
nb += F->len;
++F;
}
memcpy ((char*)rec + used, buf, nb);
rec->used = used + nb;
mFree (ses, buf);
}
}
return rt;
}
/** @param
val == 0 delete mode
@return
-2 - malloc failed,
-1 - tag not present,
0 - compactification not neccessary,
1 - rec data buffer has gaps
*/
static int rReplace (
Session ses, Rec **rec, LrecMF *lmf,
int tag, int occr, const char *val, int disc
) {
Rec *nr;
Field *F;
int *M;
int vlen;
int fnd = -1;
int ocnt = -1;
for (M = lmf->tagf; lmf->len > M - lmf->tagf; M += 2) {
if (tag == M[0]) {
if (0 <= occr) {
if (++ocnt == occr) {
M[1] = 0 == val ? MF_DEL : MF_SET;
fnd = (M - lmf->tagf) / 2;
break;
}
}
else {
if (0 == (MF_SET & M[1])) {
M[1] = 0 == val ? MF_DEL : MF_SET;
fnd = (M - lmf->tagf) / 2;
for (M += 2; lmf->len > M - lmf->tagf; M += 2) {
if (tag == M[0]) {
M[1] = MF_DEL;
}
}
break;
}
}
}
}
if (0 > fnd) {
return -1;
}
if (0 == val) {
return 1;
}
F = (*rec)->field + fnd;
vlen = strlen (val);
if (F->len == vlen) {
memcpy ((char*)F->val, val, vlen);
return 0;
}
if (F->len > vlen) {
memcpy ((char*)F->val, val, vlen);
F->len = vlen;
return 1;
}
if ((*rec)->bytes < (*rec)->used + vlen) {
nr = rDup (ses, *rec, vlen, disc);
if (0 == nr) {
return -2;
}
F = (*rec = nr)->field + fnd;
}
F->val = (char*)(*rec) + (*rec)->used;
F->len = vlen;
memcpy ((char*)F->val, val, vlen);
(*rec)->used += vlen;
return 1;
}
static int rDel (Rec *rec, LrecMF *lmf) {
Field *F;
int *M;
int idx;
int dcnt = 0;
int done = 0;
for (M = lmf->tagf; lmf->len > M - lmf->tagf; M += 2) {
if (0 != (MF_DEL & M[1])) {
idx = (M - lmf->tagf) / 2 - dcnt;
if (idx < rec->len - 1) {
F = rec->field + idx;
memmove (F, F + 1, (rec->len - idx - 1) * sizeof (Field));
}
--(rec->len);
++dcnt;
done = !0;
}
}
return done;
}
/* ************************************************************
package functions
*/
/* ************************************************************
public functions
*/
Field *rGet ( Rec *r, int tag, int *pos )
{
if (r) {
int p = pos ? *pos : 0;
for ( ; p < r->len; p++ )
if ( tag == r->field[p].tag ) {
if ( pos )
*pos = p+1;
return &r->field[p];
}
if ( pos )
*pos = r->len;
}
return 0;
} /* rGet */
Field *rOccurence ( Rec *r, int tag, int occ ) {
Field *F;
int pos = 0;
do {
F = rGet (r, tag, &pos);
} while (F && 0 <= --occ);
return F;
}
char *rString (Rec *rec, int tag, int *pos, char *buf, int len) {
Field *F = rGet (rec, tag, pos);
if (!F) {
return 0;
}
if (len > 1 + F->len) {
len = 1 + (int) F->len;
}
if (0 >= --len) {
*buf = 0;
return buf;
}
strncpy (buf, F->val, len) [len] = 0;
return buf;
}
char *rString2 (Rec *rec, Rec *dflt, int tag, char *buf, int len) {
char *rt = rString (rec, tag, 0, buf, len);
if (rt) {
return rt;
}
return rString (dflt, tag, 0, buf, len);
}
int rInt ( Rec *r, int tag, int def, int *pos )
{
if (r) {
int p = pos ? *pos : 0;
int rt;
for ( ; p < r->len; p++ )
if ( tag == r->field[p].tag ) {
if ( pos )
*pos = p+1;
rt = a2i( r->field[p].val, r->field[p].len );
if (rt) {
return rt;
}
if (0 < luti_true (r->field[p].val, r->field[p].len)) {
return 1;
}
return 0;
}
if ( pos )
*pos = r->len;
}
return def;
} /* rInt */
int rInt2 (Rec *rec, Rec *dflt, int tag, int def) {
int rt1 = rInt (rec, tag, def, 0);
if (def == rt1) {
return rInt (dflt, tag, def, 0);
}
return rt1;
}
int rEnum (Fdt *fdt, Rec *rec, int tag, int def, int *pos) {
char buf[OPENISIS_FD_NAMELEN];
Field *F;
Fd *D;
int len, ev;
if (! fdt || ! rec) {
return def;
}
D = fdById (fdt, tag);
if (! D) {
return def;
}
F = rGet (rec, tag, pos);
if (! F) {
return def;
}
len = (int) F->len;
if (OPENISIS_FD_NAMELEN <= len) {
len = OPENISIS_FD_NAMELEN - 1;
}
strncpy (buf, F->val, len) [len] = 0;
ev = fdEnum (fdt, tag, buf);
if (NOENUM == ev) {
return def;
}
return ev;
}
Rec* rDup ( Session ses, Rec *r, int room, int discard )
{
Rec *nr = 0;
int nfields, nbytes, hadfields, hadcontent;
int compacted;
if ( ! r ) {
nfields = 80; /* for a 1K base */
nbytes = 8*1024;
if ( nbytes < BASESZ(80) + room*3/2 )
nbytes = BASESZ(80) + room*3/2;
hadfields = 0;
hadcontent = 0;
} else {
compacted = rCompact (ses, r);
hadfields = r->len;
hadcontent = r->used - r->base;
if ( 0 > room ) { /* shrink to fit */
nfields = r->len;
nbytes = BASESZ(nfields) + hadcontent;
} else {
if (compacted && room <= hadcontent) {
return r;
}
nfields = 54 > r->len ? 80 : (r->len * 3 / 2); /* add 50% */
if ( nfields < r->fields )
nfields = r->fields;
nbytes = 6*1024 > r->bytes ? 8*1024 : (r->bytes *3 / 2);
if ( nbytes < r->used + room )
nbytes = r->used + room*3/2;
nbytes += (nfields - r->fields)*sizeof(Field);
}
}
LOG_DBG( LOG_DEBUG,
"extending rec size %d (cont %d) to %d bytes, %d -> %d fields",
!r ? -1 : r->bytes, hadcontent, nbytes, hadfields, nfields );
assert( nbytes >= BASESZ( nfields ) + hadcontent + room );
if ( ! (nr = mAlloc( ses, nbytes )) )
return 0;
memset( nr, 0, nbytes ); /* paranoia */
nr->bytes = nbytes;
nr->fields = nfields;
nr->base = BASESZ( nfields );
nr->used = nr->base + hadcontent;
nr->len = hadfields;
if ( ! r ) {
nr->dbid = -1; /* no valid dbid if new rec */
}
else {
char *obuf = ((char*)r) + r->base;
char *nbuf = ((char*)nr) + nr->base;
nr->dbid = r->dbid;
/* for ! r, the following are 0 by memset */
nr->rowid = r->rowid;
nr->state = r->state;
if ( hadcontent )
memcpy( nbuf, obuf, hadcontent );
if ( hadfields ) {
int i;
int displace = nbuf - obuf; /* ptrdiff_t */
Field *f = nr->field;
memcpy( nr->field, r->field, hadfields*sizeof(Field) );
for ( i=hadfields; i--; )
if ( f[i].val )
f[i].val += displace;
#ifndef NDEBUG
{
char *end = ((char*)nr) + nr->used;
for ( i=hadfields; i--; )
if ( f[i].val
&& (f[i].val < nbuf || f[i].val + f[i].len > end)
) {
int wasok = r->field[i].val >= obuf
&& r->field[i].val + r->field[i].len <= obuf + hadcontent;
sMsg( ses,
LOG_ERROR, "OOPS! nuked field %d which previously was %s",
i, wasok ? "ok" : "already broken" );
return 0; /* no cleanup, we're nearly dead anyway */
}
}
#endif
}
if ( discard )
free( r );
}
assert( RECOK(nr) );
return nr;
} /* rDup */
Rec* rMsg ( Session ses, Rec *r, int discard, int tag, const char *fmt, ... )
{
char buf[1024];
int l;
va_list ap;
va_start( ap, fmt );
l = vsnprintf( buf, sizeof(buf), fmt, ap );
if ( 0 > l ) /* older versions return -1 */
l = sizeof(buf);
va_end( ap );
RADD( ses, r, tag, buf, l, discard );
return r;
} /* rMsg */
/*
#define openIsisPrintf( r, d, t, f, a... ) \
openIsisRMsg( OPENISIS_SES0(), r, d, t, f, ## a )
requires gcc ...
*/
Rec* openIsisPrintf ( Rec *r, int discard, int tag, const char *fmt, ... )
{
char buf[1024];
int l;
va_list ap;
va_start( ap, fmt );
l = vsnprintf( buf, sizeof(buf), fmt, ap );
if ( 0 > l ) /* older versions return -1 */
l = sizeof(buf);
va_end( ap );
RADD( ses0, r, tag, buf, l, discard );
return r;
} /* openIsisPrintf */
Rec *rAddI (Session ses, Rec *rec, int tag, int val, int discard) {
char buf[32];
int len = i2a (buf, (int)val);
RADD (ses, rec, tag, buf, len, discard);
return rec;
}
Rec* rSet ( Session ses, Rec *oldr, int mode, ... )
{
const char **argv = 0;
LrecMF *lmf = 0;
Fdt *fdt = 0;
Rec *newr = oldr;
int argc = 0xffff & mode;
int dis = RDIS & mode;
int op = ROP & mode;
int compact = 0;
va_list ap;
va_start( ap, mode );
if (RFDT & mode) {
fdt = va_arg( ap, Fdt* );
}
if (RARGV & mode) {
argv = va_arg( ap, const char** );
}
if (! fdt) {
fdt = luti_fdt_from_rec (oldr);
}
if (op) {
if (oldr) {
if (0 == (lmf = mfCtor (ses, oldr))) {
goto done;
}
}
else if (RDEL == op) {
goto done;
}
}
for ( ;; ) {
char ib[16];
Rec *tmpr;
const char *v = 0;
int id = -1;
int occr = -1;
Fd *fd = 0;
int rpl;
if ( argv ) {
if ( 1 > argc )
break;
v = luti_parse_path (*argv, fdt, &fd, &id, &occr);
if (!v || *v) {
fd = 0;
id = -1;
}
v = 0;
++argv;
--argc;
}
else {
id = va_arg( ap, int );
if ( ! id )
break;
fd = fdt ? fdById( fdt, id ) : 0;
if (RDEL != op) {
v = va_arg( ap, char* );
}
}
if ( fd ) {
if (RDEL != op) {
int e;
if (argv) {
if ( 1 > argc ) {
if (FTB != fd->type) {
break;
}
}
else {
v = *argv++;
--argc;
}
}
switch ( fd->type ) {
case FTE:
e = fdEnum( fdt, fd->id, v );
if ( NOENUM == e ) {
if ( RIGN & mode )
continue;
sMsg( ses, LOG_ERROR,
"bad enum value '%s' for id %d", v, fd->id );
goto error;
}
i2a( ib, e );
v = ib;
break;
case FTB:
e = v ? luti_true (v, -1) : 1;
if (0 > e) {
/* non-given values default to true */
++argc;
--argv;
v = "1";
}
else {
v = e ? "1" : "0";
}
break;
}
} /* RDEL != op */
} /* fd */
else {
if (argv) {
if (RDEL != op) {
if ( 1 > argc ) {
break;
}
v = *argv++;
--argc;
}
}
if (fdt /*|| 0 >= id*/) {
if ( RIGN & mode )
continue;
if ( argv )
sMsg( ses, LOG_ERROR, "unknown field name '%s'",
argv[RDEL == op ? -1 : -2] );
else
sMsg( ses, LOG_ERROR, "unknown field id %d", id );
break;
}
}
if (!v) {
v = "";
}
if (op) {
rpl = rReplace (ses, &newr, lmf, id, occr,
RDEL == op ? 0 : v, dis || newr != oldr);
if (-2 == rpl) {
goto done;
}
if (-1 != rpl) {
if (1 == rpl) {
compact = !0;
}
continue;
}
if (RDEL == op) {
continue;
}
/* -1: fall thru */
}
tmpr = newr;
RADDS( ses, newr, id, v, dis || newr != oldr );
if (0 == newr) {
goto done;
}
if (newr != tmpr) {
compact = 0;
}
sMsg( ses, LOG_VERBOSE,
"added v '%s' id %d as %dth", v, id, newr->len );
}
error:
if (lmf) {
if (rDel (newr, lmf)) {
compact = !0;
}
}
if (compact && 0 == (RNOC & mode)) {
rCompact (ses, newr);
}
done:
if (lmf) {
mFree (ses, lmf);
}
va_end( ap );
return newr;
} /* rSet */
Rec *dRead ( Session ses, int db, int rowid )
{
Rec *r = (Rec *) dRaw( ses, db, rowid );
if ( ! r )
return 0;
FIXUP( r )
assert( RECOK( r ) );
#if 1
return r;
#else
#if 1 /* clone test */
return rDup( ses, r, -1, !0 );
#else /* hurt me plenty test */
{
Rec *nr = 0;
int i;
Field *f = r->field;
for ( i=0; ilen; i++ ) {
#if 0
RADD( ses, nr, f[i].tag, f[i].val, f[i].len, !0 );
#else /* even worse, but works: */
RADD( ses, nr, f[i].tag, 0, 0, !0 );
RCAT( ses, nr, f[i].val, f[i].len, !0 );
#endif
}
if ( ! nr )
RSPACE( ses, nr, 0, 0 );
if ( nr ) {
nr->dbid = r->dbid;
nr->rowid = r->rowid;
nr->state = r->state;
}
return rDup( ses, nr, -1, !0 ); /* and re-shrink */
}
#endif
#endif
} /* dRead */
Rec *dFmt ( Session ses, Rec *buf, const char *fmt, int db, int rowid )
{
Rec *r = dRead( ses, db, rowid );
Rec *q;
if ( ! r )
return 0;
q = rFmt( ses, buf, fmt, r );
free( r );
return q;
} /* dFmt */
Rec *dScan ( Session ses, int db, int rowid, int tag, const char *txt )
{
int max = dMaxId( ses, db );
int tlen = strlen( txt );
const char f = *txt;
for ( ; rowid <= max; rowid++ ) {
int i, found = 0;
Rec *r = (Rec *) dRaw( ses, db, rowid );
char *base = (char*)r;
if ( ! r )
continue;
for ( i=0; i < r->len; i++ ) {
const char *c, *e;
r->field[i].val = base + (int)r->field[i].val;
if ( found
|| (tag > 0 && tag != r->field[i].tag)
|| tlen > r->field[i].len
)
continue;
c = r->field[i].val;
e = c + r->field[i].len - tlen;
for ( ; c<=e; c++ )
if ( f == *c && !memcmp( c, txt, tlen ) ) {
found = !0;
break;
}
}
if ( found )
return r;
}
return 0;
} /* dScan */
Rec *rSplitf ( Session ses, Rec *r, const Field* field )
{
int i;
int nfields = 0; /* number of subfields */
int size; /* byte length of field list */
const char *p, *e;
if ( ! field || 2 > field->len )
return 0;
if ( '^' != field->val[0] ) /* initial anonymous subfield */
nfields = 1;
/* go counting hats ... */
for ( e = (p=field->val) + field->len - 1; plen, field->val );
if ( r ) {
if ( 1 > r->len )
return 0;
size = sizeof(Rec) + (r->len - 1) * sizeof(Field);
if ( nfields > r->len )
nfields = r->len;
} else {
/* first field already counted in sizeof(Rec) */
size = sizeof(Rec) + (nfields - 1) * sizeof(Field);
r = (Rec *)mAlloc( ses, size );
if ( ! r )
return 0;
}
memset( r, 0, size );
r->rowid = 0;
r->len = nfields;
e = (p = field->val) + field->len;
if ( '^' == *p ) /* else initial anonymous subfield */
p++;
for ( i=0; i= e ) {
sMsg( ses, LOG_ERROR,
"confused cats hound counting hats at %d in '%.*s'",
i, field->len, field->val );
r->len = i;
break;
}
/* p is after a hat */
r->field[ i ].tag = p==field->val ? 0 : (unsigned char)*p++;
r->field[ i ].val = p;
/* advance in buf */
while ( pfield[ i ].len = p - r->field[ i ].val;
/* log_msg( LOG_ERROR, "subf %d of %d tag %c off %d",
i, nfields, r->field[ i ].tag, r->field[ i ].val - field->val ); */
}
return r;
} /* rSplitf */
enum { /* stream reading state */
RS_REC, /* not in any record */
RS_TAG, /* in tag: done is at beginning of line */
RS_SEP, /* in sep: done is at beginning of sep */
RS_VAL, /* done is somewhere in the value ... */
RS_EOL /* not really a state, but a flag */
};
/**
read a record from a stream.
This may either
read a record and return 1
not get enough bytes on a non-blocking stream and return 0
have some error and return negative
*/
int sGetr ( Session ses, OpenIsisRecStream *s )
{
Ios *lio = s->in;
int state = 3 & s->flg;
int emptyline = 0;
int ret = 0;
if ( ! s->rec
&& ! (s->rec = s->buf) /* have buffer ? */
&& ! (s->rec = rDup( ses, 0, 0, 0 ))
/* allocate and prepare an 8k standard record */
) {
ret = sMsg( ses, LOG_ERROR, "could not get record" );
goto fatal;
}
if ( RS_REC == state ) { /* initialize */
s->rec->rowid = 0;
s->rec->len = 0; /* nuke all fields */
s->rec->used = s->rec->base; /* nuke value buffer */
/* let base/fields untouched, it hopefully has a reasonable value */
state = RS_TAG;
}
if ( ! s->rec->len )
state = RS_TAG;
for (;;) {
unsigned char *b, *p, *e; /* begin, pointer, end */
if ( lio->b.done >= lio->b.fill ) {
reload:
if ( ! LIO_FILL( lio ) )
goto done; /* EAGAIN -- see ya later */
/* so we have bytes or EOF */
if ( lio->b.done >= lio->b.fill ) {
if ( LIO_IN & lio->file ) {
sMsg( ses, LOG_ERROR, "confused in instream: no bytes and no EOF" );
lio_close( &lio->file, LIO_IN );
}
break;
}
}
/* so we have bytes */
p = b = lio->b.c + lio->b.done;
e = lio->b.c + lio->b.fill;
/* try to eat one line */
switch ( state ) {
case RS_TAG: {
int tag = -1;
int recognized = 0;
if ( LCS_ISFR(lcs_latin1_ct[*p]) ) { /* line starts with line/record sep */
if ( '\n' == *p && (RS_EOL & s->flg) ) { /* TODO */
s->flg &= ~RS_EOL;
continue;
}
emptyline = !0;
break;
}
for (;;p++) {
if ( p < e ) {
if ( LCS_ISWORD(lcs_latin1_ct[*p]) )
continue; /* tight loop */
break;
}
if ( 4096 > p - b )
goto reload;
sMsg( ses, ERR_TRASH, "tag too long" );
goto fatal;
}
/* identify tag between s and p */
if ( p == b )
recognized = !0;/* empty tag */
else if ( '0' <= *b && *b <= '9' ) {
unsigned char *d = b; /* digit ? */
tag = 0;
while ( d < p && '0' <= *d && *d <= '9' )
tag = 10*tag + *d++ - '0';
if ( !(recognized = d == p) ) /* all digits */
tag = -1;
}
if ( ! recognized && s->dict ) { /* dict lookup */
int i=s->dict->len, l = p-b;
Field *f = s->dict->field;
for ( ; i--; f++ )
if ( l == f->len && memcmp( b, f->val, l ) ) {
tag = f->tag;
break;
}
}
/* create new field unless it's a continuation line */
if ( p == b && s->rec->len )
;
else if ( recognized )
RADD( ses, s->rec, tag, 0, 0, s->rec != s->buf );
else
RADD( ses, s->rec, tag, b, p-b, s->rec != s->buf );
if ( ! s->rec )
goto outamem;
/* did that */
lio->b.done = (b = p) - lio->b.c;
if ( LCS_ISFR(lcs_latin1_ct[*p]) )
break;
state = RS_SEP;
} case RS_SEP:
/* p = b is at beginning of separator */
while ( p < e && ' ' == *p ) /* skip leading blanks */
p++;
if ( p < e && LCS_ISCST(lcs_latin1_ct[*p]) && '\t' != *p++ )
while ( p < e && ' ' == *p ) /* skip trailing blanks */
p++;
if ( e == p ) {
if ( 4096 > p - b )
goto reload;
sMsg( ses, ERR_TRASH, "sep too long" );
goto fatal;
}
assert( s->rec->len );
if ( s->rec->field[s->rec->len - 1].len )
RCAT( ses, s->rec, b, p-b, s->rec != s->buf );
if ( ! s->rec )
goto outamem;
lio->b.done = (b = p) - lio->b.c;
if ( LCS_ISFR(lcs_latin1_ct[*p]) )
break;
state = RS_VAL;
case RS_VAL: default: /* ??? */
while ( p < e && !LCS_ISFR(lcs_latin1_ct[*p]) )
p++;
assert( s->rec->len );
if ( p > b )
RCAT( ses, s->rec, b, p-b, s->rec != s->buf );
if ( ! s->rec )
goto outamem;
lio->b.done = (b = p) - lio->b.c;
if ( p == e )
goto reload;
/* else we found line or record separator */
} /* switch state */
if ( '\r' == *p ) { /* do that ole' ugly CR/NL quatsch */
if ( e-p < 2 && 0 < LIO_FILL(lio) ) /* extend end */
e = lio->b.c + lio->b.fill;
if ( e-p < 2 )
s->flg |= RS_EOL; /* possibly check later */
else if ( '\n' == p[1] )
p++;
}
lio->b.done = 1 + p - lio->b.c; /* consider p eaten */
if ( LCS_R == lcs_latin1_ct[*p]
|| (emptyline && (OPENISIS_STOPONEMPTY & s->flg))
)
break;
state = RS_TAG;
}
/* got record */
state = RS_REC;
if ( s->rec && s->rec->len ) {
if ( (OPENISIS_AUTOCLONE & s->flg) && s->rec == s->buf )
s->rec = rDup( ses, s->buf, 0, 0 );
ret = 1;
}
goto done;
outamem:
fatal:
if ( LIO_SISOPEN( s->in ) )
LIO_CLOSE( s->in );
done:
s->flg = state | (~3 & s->flg); /* save state */
/* sMsg( ses, LOG_ERROR, "fill %d done %d", lio->b.fill, lio->b.done ); */
return ret;
} /* openIsisReadStream */
int rSer ( char *buf, OpenIsisRec *rec )
{
Field *f = rec->field;
int i = rec->len;
char *p = buf, *e;
for ( ; i--; f++ ) {
p += i2a( p, f->tag );
*p++ = 9; /* TAB */
memcpy( p, f->val, f->len );
for ( e = p + f->len; p p; p++ ) {
if ( 10 == *p ) { /* blank line */
if ( OPENISIS_STOPONEMPTY & flg )
break;
continue;
}
p += a2il( p, e-p, &tag );
if ( e > p && 9 == *p ) /* TAB */
p++;
RADD( ses, rec, tag, 0, e-p, dis || rec!=*rp );
if ( !rec )
break;
f = rec->field + rec->len-1;
for ( q=(char*)f->val; e>p && 10 != *p; )
if ( 11 == (*q++ = *p++) )
q[-1] = 10;
rec->used += f->len = q - f->val;
}
*rp = rec;
return p-buf;
} /* rDeser */
lrectest.c 0100644 0001130 0001130 00000015764 07635347015 011553 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2003 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lrectest.c,v 1.5 2003/03/16 17:44:13 mawag Exp $
tests for lrec methods.
*/
#include
#include
#include
#include
#include "openisis.h"
#include "loi.h"
#include "lfdt.h"
#include "luti.h"
#ifdef WIN32
#define snprintf _snprintf
#define vsnprintf _vsnprintf
#endif
static const char* _TheSummary = "Meeting on the Implementation of the Recommendations of the Third Regional Conference of Ministers of Education and Ministers Responsible for Economic Planning in the Arab States";
#define SUMLEN (strlen (_TheSummary))
static Fd _TheFds[] = {
/* tag type rep len name subf desc */
{ 20, FTX, 0, 32, "author", "", "" },
{ 21, FTX, 1, 32, "names", "", "" },
{ 22, FTX, 1, 32, "year", "", "" },
{ 42, FTX, 1, 32, "title", "", "" },
{ 49, FTX, 1, 32, "summary", "", "" },
{ 70, FTX, 1, 32, "country", "", "" },
{ 80, FTX, 1, 32, "conference", "", "" },
{ 81, FTE, 0, 32, "format", "", "" },
{ 81, FTV, 0, 1, "autoformat", "", "" },
{ 81, FTV, 0, 2, "naligned", "", "" },
{ 81, FTV, 0, 73, "aligned", "", "" },
{ 82, FTB, 0, 10, "soldout", "", "" }
};
static Fdt _TheFdt = {
sizeof(_TheFds) / sizeof(_TheFds[0]),
_TheFds,
0
};
static void check (
char *msg, Rec *rec, int *tag, const char **val, int len, int used
) {
Field *F;
int idx, vlen, ok;
if (0 == rec) {
fprintf (stderr, "ERR %s: 0 == rec\n", msg);
exit (1);
}
if (! OPENISIS_RECOK (rec)) {
fprintf (stderr, "ERR %s: ! RECOK: b %d u %d/%d f %d/%d\n",
msg, rec->base, rec->used, rec->bytes, rec->len, rec->fields);
exit (1);
}
if (len != rec->len) {
fprintf (stderr, "ERR %s: len %d != %d\n", msg, rec->len, len);
log_rec (LOG_ERROR, rec, 0, 0);
exit (1);
}
if (0 <= used && used != rec->used - rec->base) {
fprintf (stderr, "ERR %s: used %d (%d-%d) != %d\n",
msg, (rec->used - rec->base), rec->used, rec->base, used);
log_rec (LOG_ERROR, rec, 0, 0);
exit (1);
}
ok = !0;
for (idx = 0; len > idx; ++idx) {
vlen = strlen (val[idx]);
F = rec->field + idx;
if (F->tag != tag[idx]) {
fprintf (stderr, "ERR %s: fld[%d] tag %d != %d\n",
msg, idx, F->tag, tag[idx]);
ok = 0;
}
if (F->len != vlen ||
strncmp (F->val, val[idx], vlen)) {
fprintf (stderr, "ERR %s: fld[%d] val <%.*s> != <%s>\n",
msg, idx, (int)F->len, F->val, val[idx]);
ok = 0;
}
}
if (! ok) {
exit (1);
}
}
int main (int argc, char **argv) {
const char *args[32];
const char *vals[32];
int tags[32];
Rec *rec;
Session ses;
ses = OPENISIS_SES0 ();
rec = 0;
(void)argv;
if (1 < argc) {
printf ("entering never ending loop ...\n");
while (!0) {
int j;
rec = 0;
for (j = 1000; 1 <= --j; ) {
rec = rAddI (ses, rec, j, j, !0);
}
mFree (ses, rec);
}
}
args[0] = "20";
args[1] = "Mary";
rec = rSet (ses, rec, RARGV | 2, args);
tags[0] = 20;
vals[0] = args[1];
check ("A", rec, tags, vals, 1, 4);
args[0] = "22";
args[1] = "1953";
args[2] = "70";
args[3] = "Italy";
args[4] = "98";
args[5] = "illegal";
args[6] = "20";
args[7] = "John";
rec = rSet (ses, rec, RARGV | RFDT | 8, &_TheFdt, args);
tags[0] = 20;
vals[0] = "Mary";
tags[1] = 22;
vals[1] = "1953";
tags[2] = 70;
vals[2] = "Italy";
tags[3] = 20;
vals[3] = "John";
check ("B", rec, tags, vals, 3, 13);
args[0] = "98";
args[1] = "illegal";
args[2] = "20";
args[3] = "John";
rec = rSet (ses, rec, RARGV | RFDT | RIGN | 4, &_TheFdt, args);
check ("C", rec, tags, vals, 4, 17);
rec = rSet (ses, rec, RDEL, 20, 0);
tags[0] = 22;
vals[0] = "1953";
tags[1] = 70;
vals[1] = "Italy";
check ("D", rec, tags, vals, 2, 9);
rec = rSet (ses, rec, 0, 20, "Mary", 20, "James", 90, "Alabama", 0);
tags[2] = 20;
vals[2] = "Mary";
tags[3] = 20;
vals[3] = "James";
tags[4] = 90;
vals[4] = "Alabama";
check ("E", rec, tags, vals, 5, 25);
rec = rSet (ses, rec, RFDT | RCHG, &_TheFdt,
20, "Lary", 70, "Greece", 70, "Cuba", 42, "Folklore from Africa",
81, "invalid", 0);
tags[0] = 22;
vals[0] = "1953";
tags[1] = 70;
vals[1] = "Greece";
tags[2] = 20;
vals[2] = "Lary";
tags[3] = 90;
vals[3] = "Alabama";
tags[4] = 70;
vals[4] = "Cuba";
tags[5] = 42;
vals[5] = "Folklore from Africa";
check ("F", rec, tags, vals, 6, 45);
rec = rSet (ses, rec, 0, 20, "Peter", 20, "Gary", 70, "Egypt", 0);
tags[6] = 20;
vals[6] = "Peter";
tags[7] = 20;
vals[7] = "Gary";
tags[8] = 70;
vals[8] = "Egypt";
check ("G", rec, tags, vals, 9, 59);
args[0] = "author[1]";
args[1] = "81";
args[2] = "70";
rec = rSet (ses, rec, RFDT | RARGV | RDEL | 3, &_TheFdt, args);
tags[0] = 22;
vals[0] = "1953";
tags[1] = 20;
vals[1] = "Lary";
tags[2] = 90;
vals[2] = "Alabama";
tags[3] = 42;
vals[3] = "Folklore from Africa";
tags[4] = 20;
vals[4] = "Gary";
check ("H", rec, tags, vals, 5, 39);
args[0] = "author[0]";
args[1] = "May";
args[2] = "summary";
args[3] = _TheSummary;
args[4] = "soldout";
args[5] = "format";
args[6] = "aligned";
rec = rSet (ses, rec, RFDT | RARGV | RCHG | RNOC | 7, &_TheFdt, args);
vals[1] = "May";
tags[5] = 49;
vals[5] = _TheSummary;
tags[6] = 82;
vals[6] = "1";
tags[7] = 81;
vals[7] = "73";
check ("I", rec, tags, vals, 8, 42 + SUMLEN);
args[0] = "20[0]";
args[1] = "Marilyn";
args[2] = "49[1]";
args[3] = _TheSummary;
rec = rSet (ses, rec, RARGV | RCHG | 4, args);
vals[1] = "Marilyn";
tags[8] = 49;
vals[8] = _TheSummary;
check ("J", rec, tags, vals, 9, 45 + 2 * SUMLEN);
rec = rSet (ses, rec, RDEL | 2, 82, 22, 0);
tags[0] = 20;
vals[0] = "Marilyn";
tags[1] = 90;
vals[1] = "Alabama";
tags[2] = 42;
vals[2] = "Folklore from Africa";
tags[3] = 20;
vals[3] = "Gary";
tags[4] = 49;
vals[4] = _TheSummary;
tags[5] = 81;
vals[5] = "73";
tags[6] = 49;
vals[6] = _TheSummary;
check ("K", rec, tags, vals, 7, 40 + 2 * SUMLEN);
rec = rSet (ses, rec, RFDT | RCHG, &_TheFdt,
82, "false", 81, "2", 49, "overview", 0);
vals[4] = "overview";
vals[5] = "2";
tags[6] = 82;
vals[6] = "0";
check ("L", rec, tags, vals, 7, 48);
mFree (ses, rec);
printf ("ok.\n");
return 0;
}
lscvs 0100755 0001130 0001130 00000001467 07576377762 010655 0 ustar krip krip #!/usr/bin/perl
# list files under CVS control
# $Id: lscvs,v 1.2 2002/12/13 15:42:42 kripke Exp $
sub dodir {
my $dir = shift;
print STDERR "$dir\n";
local *FH;
open( FH, $dir.'CVS/Entries' );
while () {
# print STDERR $_;
chomp;
my ($type,$name,$version,$date,$opt,$tagdate) = split( /\// );
if ( '' eq $type ) {
if ( $l ) {
print join("\t",$dir.$name,$version,$date,$x),"\n";
} else {
print $dir.$name,"\n";
}
} elsif ( 'D' eq $type ) {
dodir( $dir.$name.'/' ) if $r && $name;
# "`D' on a line all by itself indicates that the program
# which wrote the `Entries' file does record subdirectories"
} else {
print STDERR "uknown type '$type' in $dir ",
join("\t",$name,$version,$date,$x),"\n";
}
}
close FH;
}
$l = $ARGV[0] =~ /l/;
$r = $ARGV[0] =~ /r/;
dodir;
lses.c 0100644 0001130 0001130 00000015527 07637671272 010700 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lses.c,v 1.12 2003/03/24 21:03:54 kripke Exp $
implementation of general db access functions.
*/
#include
#include /* memset */
#define LSES_C
#include "lses.h"
/* ************************************************************
private types
*/
typedef struct {
Ses h;
int brk;
} SesI;
enum {
LSES_MAX = 0x1000,
LSES_MSK = 0x0fff
};
/* ************************************************************
package data
*/
/* ************************************************************
private data
*/
static SesI lses0; /* the default session */
static SesI *tab[ LSES_MAX ] = {
&lses0
};
static Ses *htab[ LSES_MAX ];
static int maxid;
/* ************************************************************
private functions
*/
void lses_tini ()
{
int i, s = LSES_MAX;
/* lio_write( &lio_out, "tini time\n", 9 ); */
while ( s-- )
if ( tab[s] )
for ( i=0; i<3; i++ )
if ( tab[s]->h.io[i] )
LIO_CLOSE( tab[s]->h.io[i] );
} /* lses_tini */
/* ************************************************************
package functions
*/
/* legacy */
void * lmalloc ( int siz )
{
static const char oops[] = "out of memory\n";
void *mem = malloc( siz );
if ( ! mem )
lio_write( &lio_err, oops, sizeof(oops)-1 );
return mem;
} /* lmalloc */
/* ************************************************************
public functions
*/
Session ses0;
Ses *cOpen ( Rec *args )
{
if ( ! ses0 ) {
ses0 = &lses0.h;
lio_init();
lses0.brk = 1; /* fake */
sOpen ( ses0, "<&0", 0, lio_stdio );
sOpen ( ses0, ">&1", 0, lio_stdio );
sOpen ( ses0, "2>&2", 0, lio_stdio );
atexit( lses_tini );
}
if ( args ) { /* add args */
/* TODO: merge args */
lses0.h.prop = rDup( ses0, args, -1, 0 );
}
return ses0;
} /* cOpen */
Ses *cSession ( Rec *args )
{
SesI *sesi = 0;
Ses *ses = 0;
int id = 1;
/** XXX unless we actually free sessions,
we should start checking with maxid
or check for expiry
*/
while ( tab[id] )
if ( LSES_MAX == ++id )
return 0;
if ( maxid < id )
maxid = id;
sesi = tab[id] = (SesI*)malloc( sizeof(SesI) );
memset( sesi, 0, sizeof(SesI) );
ses = &sesi->h;
ses->id = id;
ses->hash = -1; /* not hashed (hash is non-negative) */
/* prepare session heap */
sesi->brk = 1; /* fake */
ses->prop = args ? rDup( ses, args, -1, 0 ) : 0;
/* check props */
/* open files */
sOpen ( ses, "2>&2", 0, lio_stdio );
return ses;
} /* cSession */
void *mAlloc ( Session s, int size )
{
void *p = malloc( size );
(void)s;
if (p) {
memset (p, 0, size);
}
return p;
} /* mAlloc */
void mFree ( Session s, void *mem )
{
(void)s;
if ( mem )
free( mem );
} /* mFree */
void *mDup ( Session s, const void *str, int sz )
{
void *m = mAlloc( s, 0<=sz ? sz : (sz = strlen( str ) + 1) );
if ( m )
memcpy( m, str, sz );
return m;
} /* mDup */
int sOpen ( Session s, const char *name, int flags,
SFunc *type )
{
int fd = 0;
const char *p = name;
lio_sfunc *func = type ? (lio_sfunc*)type : lio_stdio;
/* check for i<> */
while ( '0' <= *p && *p <= '9' )
fd = 10*fd + *p++ - '0';
if ( '<' != *p && '>' != *p )
p = name; /* forget it */
else {
if ( name == p && '>' == *p ) /* no fd given, > defaults to 1 */
fd = 1;
flags |= '<' == *p ? LIO_RD : LIO_WR;
p++;
}
if ( p == name ) { /* no fd given, find one */
for ( fd=0; s->io[fd]; )
if ( LSES_FILE_MAX == ++fd )
return -ERR_NOMEM;
} else if ( s->io[fd] ) { /* close existing */
func( s->io[fd], LIO_SCLOSE );
mFree( s, s->io[fd] );
}
s->io[fd] = (Ios*)mAlloc( s, func( 0, LIO_SSIZE ) );
if ( ! s->io[fd] )
return -ERR_NOMEM;
s->io[fd]->func = func;
s->io[fd]->name = mDup( s, p, -1 );
s->io[fd]->file = flags;
return func( s->io[fd], LIO_SOPEN );
} /* sOpen */
Ses *openIsisId2Session (int sid) {
if (0 > sid || LSES_MAX <= sid || 0 == tab[sid]) {
return (Ses*)0;
}
return &(tab[sid]->h);
} /* openIsisId2Session */
int openIsisSession2Id (Ses *ses) {
if (0 == ses) {
return -1;
}
return ses->id;
} /* openIsisSession2Id */
Ses *cSesByName ( char *name, int nlen, Tm *now, Tm *expire )
{
char tbuf[20];
int h,i;
Ses *s;
if ( nlen < 0 )
nlen = strlen(name);
if ( nlen > 63 )
nlen = 63;
for ( i=nlen; i--; )
if ( !(0xe0 & name[i]) )
name[i] = '_';
h = lhash( name, nlen );
for ( s = htab[h & LSES_MSK]; s; s = s->nxt )
if ( h == s->hash
&& !memcmp( name, s->name, nlen )
&& !s->name[nlen]
) { /* match */
if ( !expire || s->atime.millis >= expire->millis )
goto gotit; /* valid */
if ( s->cur || s->que ) {
log_msg( LOG_WARN, "expired session %d %s still busy!",
s->id, s->name );
goto gotit; /* valid anyway */
}
goto refresh;
}
/* didn't find by name */
if ( ! expire ) /* bad thing -- we're going to run out of sessions ... */
i = maxid+1;
else /* check other expired */
for ( i=1; i<=maxid; i++ ) {
s = &tab[i]->h; /* sind wir heute defensiv ... */
if ( s && s->atime.millis < expire->millis
&& s->atime.millis && 0 <= s->hash /* else is none of our biz */
&& !s->cur && !s->que /* else is busy !? */
)
break;
}
if ( i > maxid ) {
s = cSession( 0 );
if ( ! s )
return 0;
} else /* got one */
log_msg( LOG_INFO, "reusing session %d '%s' atime %s",
s->id, s->name, s->atime.millis ? timeGtf(tbuf,&s->atime) : "-" );
if ( 0 <= s->hash ) { /* was hashed -- unlink */
Ses **spp = &htab[s->hash & LSES_MSK];
for ( ; *spp; spp = &(*spp)->nxt )
if ( s == *spp ) {
*spp = s->nxt;
break;
}
}
memset( s->name, 0, sizeof(s->name) );
memcpy( s->name, name, nlen );
s->hash = h;
s->nxt = htab[h & LSES_MSK];
htab[h & LSES_MSK] = s;
refresh:
s->ctime.millis = s->mtime.millis = now ? now->millis : 0;
s->accnt = 0;
if ( s->prop )
CLRREC( s->prop );
if ( s->res )
CLRREC( s->res );
gotit:
s->atime.millis = now ? now->millis : 0;
return s;
} /* cSesByName */
lses.h 0100644 0001130 0001130 00000004262 07637671273 010700 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef LSES_H
/*
$Id: lses.h,v 1.6 2003/03/24 21:03:55 kripke Exp $
session
*/
#include "loi.h"
#include "lio.h"
enum {
LSES_FILE_MAX = 16
};
typedef struct Ses {
int id;
struct Ses *nxt; /* chain in hash */
int hash; /* -value of name */
char name[64];
int accnt; /* # accesses */
Tm ctime;
Tm mtime;
Tm atime;
Rec *prop; /* session state */
Rec *res; /* a reusable result */
struct Con *cur; /* current connection */
struct Con *que; /* connection queue */
Ios *io[LSES_FILE_MAX];
/* actually this is followed by internal members ... */
} Ses;
#ifndef LSES_C
extern Ses lses0; /* the default session */
#endif
/* set up the default session */
extern int lses_init ( Rec *args );
/**
get or create session by name (of len nlen).
name will be cleaned of control chars (<32).
touch atime (and ctime on new) with now (if not 0).
clean and reuse sessions with atime < expire (if not 0)
*/
extern Ses *cSesByName ( char *name, int nlen, Tm *now, Tm *expire );
#define LSES_H
#endif /* LSES_H */
lstb.c 0100644 0001130 0001130 00000052160 07637613576 010673 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2003 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lstb.c,v 1.21 2003/03/24 14:34:38 kripke Exp $
implementation of client side session and communication functions.
*/
#include
#include
#include
#include
#include "openisis.h"
#include "loi.h"
#include "lcom.h"
#include "ldb.h"
#include "lfdt.h"
#include "lses.h"
#include "luti.h"
#ifdef WIN32
#define snprintf _snprintf
#define vsnprintf _vsnprintf
#endif
#define LSTB_STHOSTLEN 32
#define LSTB_MAXDBS 255 /* according to OPENISIS_DBIDMSK */
#define LSTB_MAXSTUBS 255 /* according to OPENISIS_SCIDMSK */
#define LSTB_INCRDB 8
#define LSTB_INCRSTUB 8
/* ************************************************************
* meta
*/
static Fd _RqsFields[] = {
/* tag type rep len name subf desc */
{ LCOM_RQS_SID, FTN, 0, 2, "sid", "", "Client Session Id" },
{ LCOM_RQS_SER, FTN, 0, 5, "ser", "", "Request Serial No." },
{ LCOM_RQS_TYPE, FTE, 0, 32, "type", "", "Client Session Id" },
{ LCOM_RQS_TYPE, FTV, 0, LCOM_RQST_OPEN, "open", "", "open db" },
{ LCOM_RQS_TYPE, FTV, 0, LCOM_RQST_CREA, "create", "", "create db" },
{ LCOM_RQS_TYPE, FTV, 0, LCOM_RQST_CLOS, "close", "", "close db" },
{ LCOM_RQS_TYPE, FTV, 0, LCOM_RQST_MROW, "maxrow", "", "get maxrowid" },
{ LCOM_RQS_TYPE, FTV, 0, LCOM_RQST_QRY, "query", "", "exec query" },
{ LCOM_RQS_TYPE, FTV, 0, LCOM_RQST_READ, "read", "", "fetch row" },
{ LCOM_RQS_TYPE, FTV, 0, LCOM_RQST_INS, "insert", "", "insert rec" },
{ LCOM_RQS_TYPE, FTV, 0, LCOM_RQST_UPD, "update", "", "update rec" },
{ LCOM_RQS_TYPE, FTV, 0, LCOM_RQST_DEL, "delete", "", "delete row" },
{ LCOM_RQS_DBN, FTX, 0, DBNLEN, "db", "", "DB Identification" },
{ LCOM_RQS_TMS, FTN, 0, 5, "tms", "", "Server Db Timestamp" },
{ LCOM_RQS_ROWID, FTN, 0, 5, "rowid", "", "RowId" },
{ LCOM_RQS_QMOD, FTN, 0, 5, "mode", "", "Query Mode" },
{ LCOM_RQS_SKIP, FTN, 0, 5, "skip", "", "Query Skip" },
{ LCOM_RQS_SIZE, FTN, 0, 5, "size", "", "Result Length" },
{ LCOM_RQS_QFLG, FTN, 0, 5, "flags", "", "Query Flags" },
{ LCOM_RQS_KEY, FTX, 0, LCOM_QRY_KEYLEN, "key", "", "Query Key" },
{ LCOM_RQS_REC, FTS, 1, 0, "rec", "", "Record to Update" },
{ LCOM_RQS_IDX, FTS, 1, 0, "idx", "", "Index to Update" }
};
static Fdt _RqsFdt = {
sizeof(_RqsFields) / sizeof(_RqsFields[0]),
_RqsFields,
0
};
static Fd _RspFields[] = {
/* tag type rep len name subf desc */
{ LCOM_RSP_SID, FTN, 0, 2, "sid", "", "Client Session Id" },
{ LCOM_RSP_SER, FTN, 0, 5, "ser", "", "Request Serial No." },
{ LCOM_RSP_DBN, FTX, 0, DBNLEN, "db", "", "DB Identification" },
{ LCOM_RSP_TMS, FTN, 0, 5, "tms", "", "Server Db Timestamp" },
{ LCOM_RSP_ERR, FTN, 0, 5, "error", "", "Error Code" },
{ LCOM_RSP_CERR, FTN, 0, 5, "error2", "", "Client Side Error" },
{ LCOM_RSP_MSG, FTX, 0, LCOM_ERRMSGLEN, "msg", "", "Error Message" },
{ LCOM_RSP_NUMT, FTN, 0, 5, "total", "", "Total No. of Records" },
{ LCOM_RSP_NUMR, FTN, 0, 5, "size", "", "Number of Records" },
{ LCOM_RSP_ROWID, FTN, 1, 5, "rowid", "", "RowId" },
{ LCOM_RSP_DATA, FTS, 1, 0, "data", "", "Fetched Structure" }
};
static Fdt _RspFdt = {
sizeof(_RspFields) / sizeof(_RspFields[0]),
_RspFields,
0
};
const Fdt *openIsisComRqsPar = &_RqsFdt;
const Fdt *openIsisComRspPar = &_RspFdt;
/* stub id 0 has idx -1 */
/* stub id 0x0100 has idx 0 */
#define SC_X2I(s) ( ( (1 + (s)) << 8 ) & OPENISIS_SCIDMSK )
#define SC_I2X(s) ( ( ((s) & OPENISIS_SCIDMSK) >> 8 ) - 1 )
#define DB_X2I(d,s) ( ((s) & OPENISIS_SCIDMSK) | ((d) & OPENISIS_DBIDMSK) )
#define DB_I2X(d) ( (d) & OPENISIS_DBIDMSK )
#define DB_IDX_OK(st,idx) \
(0 <= (idx) && (st)->head.ndbs > (idx) && * (st)->head.dbs[idx]->name)
/* ************************************************************
* client view of db
*/
typedef struct {
Db head;
} CDb;
static void cdbCtor (CDb *that,
const char *name, int dbid, int tms, Fdt *fdt
) {
strncpy (that->head.name, name, DBNLEN - 1) [DBNLEN - 1] = 0;
that->head.dbid = dbid;
that->head.tms = tms;
that->head.fdt = fdt;
}
static void cdbDtor (CDb *that) {
if (that->head.fdt) {
mFree (ses0, that->head.fdt);
}
memset (that, 0, sizeof (CDb));
}
/* ************************************************************
* client session
*/
typedef struct OpenIsisStb {
Schema head;
char host[LSTB_STHOSTLEN];
int port;
int srq; /* rqs serial no */
Rec *rqs; /* last request */
Rec *rsp; /* last response */
Db *rdb; /* db of embedded rsp recs */
Session ses; /* session used for actual/last rqs/rsp */
LutiLT hsh;
CDb *dbase; /* db array of non-local schema */
OpenIsisRspCb *actcb;
void *actcld;
OpenIsisStubCbData cbdta;
} OpenIsisStb;
Stub stub0 = 0;
static Stub RmtStubs = 0;
static int NumStubs = 0;
static LutiLT StubLT = 0;
#define STBSIZE (sizeof(Stb))
static void stbCtor (Stub that,
const char *name, const char *host, int port, int scid,
OpenIsisStubCbData *cbd
) {
strncpy (that->head.name, name, SCNLEN - 1) [SCNLEN - 1] = 0;
strncpy (that->host, host, LSTB_STHOSTLEN - 1) [LSTB_STHOSTLEN - 1] = 0;
that->head.scid = scid;
that->port = port;
that->rdb = 0;
that->hsh = luti_ltnew ();
if (cbd) {
memcpy (&that->cbdta, cbd, sizeof (OpenIsisStubCbData));
}
luti_ltadd (StubLT, that->head.name, scid);
}
static int cdbNewIdx (Stub that) {
char **D, **E;
CDb *obase;
int ndb = that->head.ndbs;
int rt, j, diff;
for (j = 0; ndb > j; ++j) {
if (! * that->dbase[j].head.name) {
return j;
}
}
obase = that->dbase;
rt = luti_ptrincr (ses0, &that->dbase, &ndb,
LSTB_INCRDB, sizeof (CDb), LSTB_MAXDBS);
if (0 > rt) {
return -1;
}
rt = luti_ptrincr (ses0, &that->head.dbs, &that->head.ndbs,
LSTB_INCRDB, sizeof (CDb*), LSTB_MAXDBS);
if (0 > rt) {
return -1;
}
diff = (char*) that->dbase - (char*) obase;
if (diff) {
E = (D = (char**) that->head.dbs) + rt;
while (E > D) {
*D += diff;
++D;
}
}
return rt;
}
static int cdbNewIdx0 (Stub that, int dbid, int *idx) {
int rt;
*idx = dbid;
if (dbid < that->head.ndbs) {
if (that->head.dbs[dbid]) {
return 0;
}
return 1;
}
rt = luti_ptrincr (ses0, &that->head.dbs, &that->head.ndbs,
1 + dbid - that->head.ndbs, sizeof (CDb*), -1);
if (0 > rt) {
return -1;
}
return 1;
}
static int stbNewIdx () {
int j;
for (j = 0; NumStubs > j; ++j) {
if (! * RmtStubs[j].head.name) {
return j;
}
}
return luti_ptrincr (ses0, &RmtStubs, &NumStubs,
LSTB_INCRSTUB, STBSIZE, LSTB_MAXSTUBS);
}
static Stub stbById (int sid) {
int idx;
if (0 > sid || ! stub0) {
return 0;
}
if (0 == sid) {
return stub0;
}
idx = SC_I2X (sid);
if (0 > idx) {
log_msg (LOG_ERROR, "stbById: illegal id %x\n", sid);
return 0;
}
if (NumStubs <= idx) {
return 0;
}
if (! * RmtStubs[idx].head.name) {
return 0;
}
return RmtStubs + idx;
}
static void stbDtor (Stub that) {
luti_ltrmv (StubLT, that->head.name);
if (that->cbdta.delcb) {
if (that->actcld) {
(*that->cbdta.delcb) (
that->cbdta.delcld, that, that->actcld);
}
if (that->cbdta.dfltcld) {
(*that->cbdta.delcb) (
that->cbdta.delcld, that, that->cbdta.dfltcld);
}
(*that->cbdta.delcb) (that->cbdta.delcld, that, 0);
}
if (that->dbase) {
int j;
for (j = that->head.ndbs; 0 <= --j; ) {
if (* that->dbase[j].head.name) {
cdbDtor (that->dbase + j);
}
}
mFree (ses0, that->dbase);
}
if (that->head.dbs) {
mFree (ses0, that->head.dbs);
}
if (that->head.cfg) {
mFree (ses0, that->head.cfg);
}
luti_ltdel (that->hsh);
nClean (that);
memset (that, 0, STBSIZE);
}
static Rec* stbMetaFor (const char *scname) {
Rec *ent = 0;
Field *F = 0;
Field *N = 0;
int pos = 0;
if (! stub0->head.cfg) {
return 0;
}
while (1) {
F = rGet (stub0->head.cfg, OPENISIS_SSCHEMA, &pos);
if (ent) {
mFree (ses0, ent);
ent = 0;
}
if (! F) {
break;
}
ent = rSplitf (ses0, 0, F);
if (! ent) {
break;
}
N = rGet (ent, OPENISIS_SSCNAME, 0);
if (! N) {
log_msg (LOG_WARN, "illegal schema entry %d: %.*s\n",
pos, (int)F->len, F->val);
continue;
}
if (! strncmp (scname, N->val, (int)N->len)) {
break;
}
}
return ent;
}
static int stbDbidFromName (Stub that, const char *dbn) {
int id;
if (! dbn) {
return -1;
}
id = luti_ltget (that->hsh, dbn);
return id;
}
static int stbDbIdxFromName (Stub that, const char *dbn) {
int idx = stbDbidFromName (that, dbn);
if (0 > idx) {
return -1;
}
idx = DB_I2X (idx);
if (! DB_IDX_OK (that, idx)) {
return -1;
}
return idx;
}
static int stbDbIdxFromRec (Stub that, Rec *rec, int tag) {
char dbn[DBNLEN];
return stbDbIdxFromName (that,
rString (rec, tag, 0, dbn, DBNLEN));
}
static int stbDbidFromRec (Stub that, Rec *rec, int tag) {
char dbn[DBNLEN];
return stbDbidFromName (that,
rString (rec, tag, 0, dbn, DBNLEN));
}
static int stbOpenDb (Stub that) {
char dbn[DBNLEN];
Fdt *fdt;
int idx, dbid, tms;
int newdb = 0;
if (! rString (that->rsp, LCOM_RSP_DBN, 0, dbn, DBNLEN)) {
log_msg (LOG_ERROR, "stbOpenDb: missing db name in rsp\n");
return -1;
}
idx = stbDbIdxFromName (that, dbn);
if (0 > idx) {
idx = cdbNewIdx (that);
if (0 > idx) {
return -1;
}
newdb = !0;
}
else {
cdbDtor (that->dbase + idx);
}
fdt = rec2fdt (ses0, that->rsp, LCOM_RSP_NUMR, LCOM_RSP_DATA, 0);
tms = rInt (that->rsp, LCOM_RSP_TMS, 0, 0);
dbid = DB_X2I (idx, that->head.scid);
cdbCtor (that->dbase + idx, dbn, dbid, tms, fdt);
if (newdb) {
luti_ltadd (that->hsh, dbn, dbid);
}
return 0;
}
static int stbOpenDb0 (Stub that) {
Db *dbh;
int idx;
int newdb = 0;
int dbid = rInt (that->rsp, LCOM_RSP_DBID, -1, 0);
if (0 > dbid) {
log_msg (LOG_ERROR, "stbOpenDb0: missing dbid in rsp");
return -1;
}
newdb = cdbNewIdx0 (that, dbid, &idx);
if (0 > newdb) {
log_msg (LOG_ERROR, "stbOpenDb0: out of memory");
return -1;
}
dbh = ldb_getdb (idx);
if (! dbh) {
log_msg (LOG_ERROR, "stbOpenDb0: no such db %d %d", dbid, idx);
return -1;
}
log_msg (LOG_INFO, "stbOpenDb0 %d %d %s", dbid, idx, dbh->name);
that->head.dbs[idx] = dbh;
if (newdb) {
luti_ltadd (that->hsh, dbh->name, idx);
}
return 0;
}
static int stbCloseDb (Stub that) {
CDb *db;
int dbid = stbDbIdxFromRec (that, that->rsp, LCOM_RSP_DBN);
if (0 > dbid) {
log_msg (LOG_ERROR, "stbCloseDb: missing or illegal db name in rsp");
return -1;
}
db = that->dbase + dbid;
luti_ltrmv (that->hsh, db->head.name);
cdbDtor (db);
return 0;
}
static int stbCloseDb0 (Stub that) {
char dbn[DBNLEN];
int dbid = rInt (that->rsp, LCOM_RSP_DBID, -1, 0);
if (0 > dbid) {
log_msg (LOG_ERROR, "stbCloseDb0: missing dbid in rsp");
return -1;
}
if (that->head.ndbs <= dbid) {
log_msg (LOG_ERROR, "stbCloseDb0: illegal dbid %d(%d) in rsp",
dbid, that->head.ndbs);
return -1;
}
if (! rString (that->rsp, LCOM_RSP_DBN, 0, dbn, DBNLEN)) {
log_msg (LOG_ERROR, "stbCloseDb0: missing dbn in rsp");
return -1;
}
log_msg (LOG_INFO, "stbCloseDb0 %d %s", dbid, dbn);
luti_ltrmv (that->hsh, dbn);
that->head.dbs[dbid] = 0;
return 0;
}
/* (local) response callback
MMM TBD if called from file event handler,
read stub id, determine ses from stub,
deserialize and allocate rsp and call cb
NOTE: rsp may be discarded in this call (s. err2)
*/
static int stbRspCb (Session ois, Rec **rsp, void *cld) {
Stub that;
Rec *rsp2;
int sid, ser, err, err2, err3;
(void)cld;
sid = rInt (*rsp, LCOM_RSP_SID, -1, 0);
ser = rInt (*rsp, LCOM_RSP_SER, -1, 0);
err = rInt (*rsp, LCOM_RSP_ERR, -1, 0);
that = stbById (sid);
if (! that) {
log_msg (LOG_WARN, "illegal stub id %d in stbRspCb", sid);
return -1;
}
if (ser != that->srq) {
log_msg (LOG_WARN, "serial mismatch %d %d stub %d",
ser, that->srq, sid);
return -1;
}
if (! that->rqs) {
log_msg (LOG_WARN, "response without request %d stub %d",
ser, sid);
return -1;
}
if (ois != that->ses) {
log_msg (LOG_ERROR, "session mismatch: got %d(%d), exp %d(%d)",
ois, ois->id, that->ses, (that->ses ? that->ses->id : 0));
return -1;
}
if (that->rsp) {
log_msg (LOG_WARN, "multiple responses for request %d stub %d",
ser, sid);
mFree (that->ses, that->rsp);
}
that->rdb = 0;
that->rsp = *rsp;
*rsp = 0;
err2 = err3 = 0;
if (0 == err) {
int rtyp = rInt (that->rqs, LCOM_RQS_TYPE, -1, 0);
if (LCOM_RQST_CLOS == rtyp) {
err2 = sid ? stbCloseDb (that) : stbCloseDb0 (that);
}
else {
int rdid, rx;
if (LCOM_RQST_OPEN == rtyp || LCOM_RQST_CREA == rtyp) {
err2 = sid ? stbOpenDb (that) : stbOpenDb0 (that);
}
rdid = stbDbidFromRec (that, that->rsp, LCOM_RSP_DBN);
if (0 > rdid) {
log_msg (LOG_ERROR, "stbRspCb: unknown db id\n");
err2 = -1;
}
/* see nDbById */
rx = DB_I2X (rdid);
if (! DB_IDX_OK (that, rx)) {
log_msg (LOG_ERROR, "stbRspCb: illegal db id %x\n", rdid);
err2 = -1;
}
else {
that->rdb = that->head.dbs[rx];
}
}
}
else {
int tms;
tms = rInt (that->rsp, LCOM_RSP_TMS, 0, 0);
if (tms && sid) { /* remounted */
err2 = stbCloseDb (that);
}
}
rsp2 = rAddI (that->ses, that->rsp, LCOM_RSP_CERR, err2, !0);
if (rsp2) {
that->rsp = rsp2;
}
if (that->actcb) {
err3 = (*that->actcb) (that->actcld,
that, that->rsp, that->rdb);
if (that->actcld && that->cbdta.delcb) {
(*that->cbdta.delcb) (
that->cbdta.delcld, that, that->actcld);
}
that->actcb = 0;
that->actcld = 0;
}
else if (that->cbdta.dfltcb) {
err3 = (*that->cbdta.dfltcb) (that->cbdta.dfltcld,
that, that->rsp, that->rdb);
}
/* if (err) return err;
nSend should not return processing errors
*/
if (err2) return err2;
return err3;
}
/* ************************************************************
package functions
*/
/* ************************************************************
public functions
*/
Stub openIsisNInit (
int argc, const char **argv, OpenIsisStubCbData *dta
) {
Session ses;
if (stub0) {
return stub0;
}
ses = OPENISIS_SES0 ();
StubLT = luti_ltnew ();
stub0 = mAlloc (ses, STBSIZE);
memset (stub0, 0, STBSIZE);
stbCtor (stub0, "", "", 0, 0, dta);
if (argc) {
if (argc & ~RARGC) {
log_msg (LOG_WARN, "illegal argc %d for stub0", argc);
}
else {
stub0->head.cfg = rSet (ses, 0, RFDT | RARGV | RIGN | argc,
openIsisSyspar, argv);
}
}
return stub0;
}
Stub openIsisNOpen (
const char *name, int argc, const char *argv[], OpenIsisStubCbData *dta
) {
char host[LSTB_STHOSTLEN];
Stub that = 0;
Rec *meta = 0;
int idx, port;
if (! stub0 || ! name || ! *name) {
return 0;
}
(void)argc; (void)argv; /* MMM TBD */
idx = luti_ltget (StubLT, name);
if (0 <= idx) {
idx = SC_I2X (idx);
if (0 > idx || NumStubs <= idx || ! * RmtStubs[idx].head.name) {
log_msg (LOG_ERROR, "nOpen: illegal idx %d(%d) for %s\n",
idx, NumStubs, name);
return 0;
}
return RmtStubs + idx;
}
meta = stbMetaFor (name);
if (! meta) {
log_msg (LOG_WARN, "nOpen: unknown schema <%s>\n", name);
goto done;
}
if (! rString (meta, OPENISIS_SSCHOST, 0, host, LSTB_STHOSTLEN)) {
log_msg (LOG_WARN, "nOpen: no host for schema %s\n", name);
goto done;
}
port = rInt (meta, OPENISIS_SSCPORT, 0, 0);
if (! port) {
log_msg (LOG_WARN, "nOpen: no port for schema %s\n", name);
goto done;
}
idx = stbNewIdx ();
if (0 > idx) {
goto done;
}
that = RmtStubs + idx;
idx = SC_X2I (idx);
stbCtor (that, name, host, port, idx, dta);
done:
if (meta) {
mFree (ses0, meta);
}
return that;
}
void openIsisNClose (Stub that) {
if (that && *that->head.name) {
stbDtor (that);
}
}
void openIsisNDeinit () {
if (stub0) {
int j;
if (RmtStubs) {
for (j = NumStubs; 0 <= --j; ) {
if (* RmtStubs[j].head.name) {
stbDtor (RmtStubs + j);
}
}
mFree (ses0, RmtStubs);
RmtStubs = 0;
NumStubs = 0;
}
for (j = stub0->head.ndbs; 0 <= --j; ) {
if (stub0->head.dbs[j]) {
cDClose (stub0->head.dbs[j]->dbid);
}
}
stbDtor (stub0);
mFree (ses0, stub0);
stub0 = 0;
luti_ltdel (StubLT);
StubLT = 0;
}
}
Schema* nSchema (Stub that) {
int idx;
if (that && ( stub0 == that || ( *that->head.name &&
0 <= (idx = SC_I2X (that->head.scid)) && NumStubs > idx
))) {
return &that->head;
}
return 0;
}
void nClean (Stub that) {
if (that->ses) {
if (that->rqs) {
mFree (that->ses, that->rqs);
that->rqs = 0;
}
if (that->rsp) {
mFree (that->ses, that->rsp);
that->rsp = 0;
}
that->rdb = 0;
that->ses = 0;
}
}
int nSend (Stub that,
Session ses, Rec *rqs, OpenIsisRspCb *actcb, void *actcld, int dup
) {
char sid[16];
char ser[16];
char tms[16];
int dbid;
if (!(that && ses && rqs)) {
return -1;
}
if (stub0 != that && ! *that->head.name) {
return -1;
}
if (that->ses) {
if (! that->rsp) {
log_msg (LOG_WARN, "stub %s: attempt to send multiple requests",
that->head.name);
return -1;
}
nClean (that);
}
i2a (sid, that->head.scid);
if (0 >= ++that->srq) {
that->srq = 1;
log_msg (LOG_INFO, "stub %s: srq loop over 0", that->head.name);
}
i2a (ser, that->srq);
dbid = -1;
if (0 > rInt (rqs, LCOM_RQS_TMS, -1, 0)) {
dbid = stbDbIdxFromRec (that, rqs, LCOM_RQS_DBN);
if (0 <= dbid) {
i2a (tms, that->head.dbs[dbid]->tms);
}
}
if (dup) {
rqs = rDup (ses, rqs, 0, 0);
if (! rqs) {
return -1;
}
}
rqs = rSet (ses, rqs, RFDT | RDIS | RCHG, openIsisComRqsPar,
LCOM_RQS_SID, sid, LCOM_RQS_SER, ser,
0 <= dbid ? LCOM_RQS_TMS : 0, tms, 0);
if (! rqs) {
return -1;
}
that->ses = ses;
that->rqs = rqs;
if (that->actcld && that->cbdta.delcb) {
/* ooups */
(*that->cbdta.delcb) (
that->cbdta.delcld, that, that->actcld);
}
if (actcb) {
that->actcb = actcb;
that->actcld = actcld;
}
else {
that->actcb = 0;
that->actcld = 0;
}
return ldsp_process (ses, that->rqs, &stbRspCb, 0);
}
Rec* nRecv (Stub that, Db **db) {
if (that && (
stub0 == that || *that->head.name
)) {
if (db) {
*db = that->rdb;
}
return that->rsp;
}
return 0;
}
Db* nDbByName (Stub that, const char *dbname) {
if (that && (
stub0 == that || *that->head.name
)) {
int dbid = stbDbIdxFromName (that, dbname);
if (0 <= dbid) {
return that->head.dbs[dbid];
}
}
return 0;
}
Db* nDbById (int id) {
Stub that = stbById (id & OPENISIS_SCIDMSK);
if (! that) {
return 0;
}
id = DB_I2X (id);
if (! DB_IDX_OK (that, id)) {
return 0;
}
return that->head.dbs[id];
}
/* ************************************************************
MMM util ? used in lstbtest
*/
#define FREEPTR(p) \
if (0 != p) { \
mFree (ses, p); \
p = 0; \
}
int openIsisNGetResult (
Stub that, Session ses, int **rowids, Rec ***recs, int *tot
) {
Field *F;
Rec **R;
int *I;
int numt, numr, err, pos, j;
if (0 != rowids) {
*rowids = 0;
}
if (0 != recs) {
*recs = 0;
}
if (0 != tot) {
*tot = 0;
}
if (! that || ! that->rsp || ! that->rdb || ! ses) {
return 0;
}
err = rInt (that->rsp, LCOM_RSP_ERR, 0, 0);
if (err) {
return -1;
}
numr = rInt (that->rsp, LCOM_RSP_NUMR, -1, 0);
numt = rInt (that->rsp, LCOM_RSP_NUMT, -1, 0);
if (0 > numr || 0 > numt) {
return -2;
}
if (numr) {
if (rowids) {
*rowids = (int*) mAlloc (ses, numr * sizeof (int));
if (! *rowids) {
return -3;
}
for (I = *rowids, j = pos = 0; numr > j; ++I, ++j) {
*I = rInt (that->rsp, LCOM_RSP_ROWID, -1, &pos);
if (0 > *I) {
sMsg (ses, LOG_ERROR,
"nGetResult: missing rowid %d(%d)", j, numr);
FREEPTR (*rowids);
return -2;
}
}
}
if (recs) {
*recs = (Rec**) mAlloc (ses, numr * sizeof (Rec*));
if (! *recs) {
FREEPTR (*rowids);
return -3;
}
memset (*recs, 0, numr * sizeof (Rec*));
R = *recs;
j = pos = 0;
F = rGet (that->rsp, LCOM_RSP_DATA, &pos);
if (! F) {
sMsg (ses, LOG_WARN,
"nGetResult: no records in response");
FREEPTR (*recs);
}
else {
--pos;
while (numr > j) {
*R = luti_unwrap (ses, that->rsp,
&pos, LCOM_RSP_DATA, that->rdb->dbid);
if (! *R) {
FREEPTR (*rowids);
luti_free (ses, (void**)*recs, j);
*recs = 0;
return -2;
}
++R;
++j;
}
}
} /* recs */
} /* numr */
if (0 != tot) {
*tot = numt;
}
return numr;
}
lstbtest.c 0100644 0001130 0001130 00000040175 07635347015 011564 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2003 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lstbtest.c,v 1.15 2003/03/17 09:05:27 mawag Exp $
tests for lstb methods.
*/
#include
#include
#include
#include
#include
#include /* unlink */
#include "openisis.h"
#include "loi.h"
#include "lcom.h" /* openIsisComRqsPar, LCOM_RSP_XXX */
#include "ldb.h" /* ldb_getdb */
#include "luti.h" /* luti_unwrap */
static int _RspCld[1] = { 0 };
static int _DelCld[1] = { 0 };
static OpenIsisStubCbData cbdta;
static Session ses1;
static Stub st0, st1, st2, st3;
static Schema *sc0, *sc1, *sc2, *sc3;
static Db *db;
static Rec *rqs, *rsp;
static char *args[64];
static int alen;
/* ************************************************************
utils
*/
static void _assert (int cond, const char *where, char *fmt, ...) {
if (! cond) {
va_list ap;
va_start (ap, fmt);
fprintf (stderr, "ERR %s: ", where ? where : "");
vfprintf (stderr, fmt, ap);
fprintf (stderr, "\n");
va_end (ap);
exit (1);
}
}
static int _rsp_cb (void *cld, Stub stb, Rec *rsprec, Db *x) {
(void)stb; (void)rsprec; (void)x;
if (cld) {
++(*((int*)cld));
}
return 0;
}
static void _del_cb (void *cld, Stub stb, void *cbd) {
(void)stb;
++(*((int*)cld));
if (cbd) {
_assert (cbd == (void*)_RspCld, "DELCB",
"%d: %p != %p", *((int*)cld), cbd, _RspCld);
--(*((int*)cbd));
}
}
static void _cmp_rec (const char *msg, Rec *r1, Rec *r2) {
char buf1[32], buf2[32];
Field *f1, *f2;
int j;
_assert (0 != r1, msg, "chkr: r1 == 0");
_assert (0 != r2, msg, "chkr: r2 == 0");
_assert (r1->len == r2->len, msg, "chkr: reclen = %d/%d",
r1->len, r2->len);
buf1[31] = buf2[31] = 0;
for (f1 = r1->field, f2 = r2->field, j = 0;
r1->len > j;
++f1, ++f2, ++j) {
_assert (f1->tag == f2->tag, msg, "chkr: tag[%d] = %d/%d",
j, f1->tag, f2->tag);
_assert (f1->len == f2->len, msg, "chkr: len[%d] = %d/%d",
j, f1->len, f2->len);
strncpy (buf1, f1->val, 31 < f1->len ? 31 : f1->len);
strncpy (buf2, f2->val, 31 < f1->len ? 31 : f1->len);
_assert (0 == memcmp (f1->val, f2->val, f1->len), msg,
"field[%d] = %s/%s", j, buf1, buf2);
}
}
static void _chk_rec (const char *msg, Rec *rec, int dbid, int rowid) {
Rec *r2 = dRead (ses0, dbid, rowid);
_cmp_rec (msg, rec, r2);
mFree (ses0, r2);
}
static Rec* _buildrqs (Session ses, int argc, char **argv) {
return rSet (ses, 0, RARGV | RFDT | argc, openIsisComRqsPar, argv);
}
static int _set_tms (const char *msg, char *dbn, int delta) {
static char tbuf[32];
int j;
_assert (0 != sc0, msg, "(tms) sc0 == 0");
_assert (0 < sc0->ndbs, msg, "(tms) sc0->ndbs = %d", sc0->ndbs);
for (j = 0, db = 0; sc0->ndbs > j; ++j) {
if (0 == strcmp (sc0->dbs[j]->name, dbn)) {
db = sc0->dbs[j];
break;
}
}
_assert (0 != db, msg, "(tms) no such db: %s", dbn);
_assert (0 != db->tms, msg, "(tms) tms == 0");
sprintf (tbuf, "%d", (db->tms + delta));
args[alen++] = "tms";
args[alen++] = tbuf;
return db->tms;
}
/* ************************************************************
tests
*/
static void _open_srv (const char *msg, int argc, const char **argv) {
st0 = openIsisNInit (argc, argv, &cbdta);
_assert (0 != st0, msg, "st0 == 0");
_assert (0 != ses0, msg, "ses0 == 0");
sc0 = openIsisNSchema (st0);
_assert (0 != sc0, msg, "sc0 == 0");
_assert (0 == sc0->scid, msg, "sc0.scid = %x", sc0->scid);
st1 = openIsisNOpen ("tonno", 0, 0, &cbdta);
_assert (0 != st1, msg, "st1 == 0");
_assert (st0 != st1, msg, "st1 == st0");
sc1 = openIsisNSchema (st1);
_assert (0 != sc1, msg, "sc1 == 0");
_assert (0x0100 == sc1->scid, msg, "sc1.scid = %x", sc1->scid);
st2 = openIsisNInit (0, 0, 0);
_assert (st2 == st0, msg, "st2 %p != %p", st2, st0);
sc2 = openIsisNSchema (st2);
_assert (sc2 == sc0, msg, "sc2 %p != %p", sc2, sc0);
st3 = openIsisNOpen ("tonno", 0, 0, 0);
_assert (st3 == st1, msg, "st3 %p != %p", st3, st1);
sc3 = openIsisNSchema (st1);
_assert (sc3 == sc1, msg, "sc3 %p != %p", sc3, sc1);
}
static void _open_db (const char *msg, char *dbn) {
int rt;
alen = 0;
args[alen++] = "type";
args[alen++] = "open";
args[alen++] = "db";
args[alen++] = dbn;
rqs = _buildrqs (ses1, alen, args);
_assert (0 != rqs, msg, "rqs == 0");
rt = openIsisNSend (st0, ses1, rqs, &_rsp_cb, _RspCld, 0);
_assert (0 == rt, msg, "rt = %d", rt);
rsp = openIsisNRecv (st0, 0);
_assert (0 != rsp, msg, "rsp == 0");
rt = rInt (rsp, LCOM_RSP_ERR, -1, 0);
_assert (0 == rt, msg, "err = %d", rt);
db = ldb_getdb (0);
_assert (0 != db, msg, "db == 0");
_assert (0 == strcmp (db->name, "cds"), msg, "db0: %s", db->name);
}
static void _close_db (const char *msg, char *dbn, int dbid) {
int rt;
/*
MMM TBD db header in stub 0 unabh von ldb in kopie pflegen ?
*/
alen = 0;
args[alen++] = "type";
args[alen++] = "close";
args[alen++] = "db";
args[alen++] = dbn;
rqs = _buildrqs (ses1, alen, args);
_assert (0 != rqs, msg, "rqs == 0");
rt = openIsisNSend (st0, ses1, rqs, &_rsp_cb, _RspCld, 0);
_assert (0 == rt, msg, "rt = %d", rt);
rsp = openIsisNRecv (st0, 0);
_assert (0 != rsp, msg, "rsp == 0");
rt = rInt (rsp, LCOM_RSP_ERR, -1, 0);
_assert (0 == rt, msg, "err = %d", rt);
if (0 <= dbid) {
db = ldb_getdb (dbid);
_assert (0 == db, msg, "db != 0");
}
}
static void _maxrow (const char *msg, char *dbn, int exp) {
int rt;
alen = 0;
args[alen++] = "type";
args[alen++] = "maxrow";
args[alen++] = "db";
args[alen++] = dbn;
rqs = _buildrqs (ses1, alen, args);
_assert (0 != rqs, msg, "rqs == 0");
rt = openIsisNSend (st0, ses1, rqs, &_rsp_cb, _RspCld, 0);
_assert (0 == rt, msg, "rt = %d", rt);
rsp = openIsisNRecv (st0, 0);
_assert (0 != rsp, msg, "rsp == 0");
rt = rInt (rsp, LCOM_RSP_ERR, -1, 0);
_assert (0 == rt, msg, "err = %d", rt);
rt = rInt (rsp, LCOM_RSP_NUMT, -1, 0);
_assert (-1 == rt, msg, "numt = %d", rt);
rt = rInt (rsp, LCOM_RSP_ROWID, -1, 0);
if (0 <= exp) {
_assert (exp == rt, msg, "rowid = %d(%d)", rt, exp);
}
else {
_assert (0 <= rt, msg, "rowid = %d", rt);
}
}
static void _maxrowerr (const char *msg, char *dbn) {
int rt, tms;
alen = 0;
args[alen++] = "type";
args[alen++] = "maxrow";
args[alen++] = "db";
args[alen++] = dbn;
tms = _set_tms (msg, dbn, -2);
rqs = _buildrqs (ses1, alen, args);
_assert (0 != rqs, msg, "rqs == 0");
rt = openIsisNSend (st0, ses1, rqs, &_rsp_cb, _RspCld, 0);
_assert (0 == rt, msg, "rt = %d x%x", rt, rt);
rsp = openIsisNRecv (st0, 0);
_assert (0 != rsp, msg, "rsp == 0");
rt = rInt (rsp, LCOM_RSP_ERR, -1, 0);
_assert (ERR_INVAL == rt, msg, "err = %d", rt);
rt = rInt (rsp, LCOM_RSP_TMS, -1, 0);
_assert (tms == rt, msg, "tms = %d(%d)", rt, tms);
}
static void _read (const char *msg, int rid, int rlen) { /* no _rsp_cb */
Rec *rec;
Db *rdb;
char rbuf[32];
int rt;
alen = 0;
args[alen++] = "type";
args[alen++] = "read";
args[alen++] = "db";
args[alen++] = "cds";
_set_tms (msg, "cds", 0);
sprintf (rbuf, "%d", rid);
args[alen++] = "rowid";
args[alen++] = rbuf;
rqs = _buildrqs (ses1, alen, args);
_assert (0 != rqs, msg, "rqs == 0");
rt = openIsisNSend (st0, ses1, rqs, 0, _RspCld, 0);
_assert (0 == rt, msg, "rt = %d", rt);
rsp = openIsisNRecv (st0, &rdb);
_assert (0 != rsp, msg, "rsp == 0");
_assert (0 != rdb, msg, "rdb == 0");
rt = rInt (rsp, LCOM_RSP_ERR, -1, 0);
_assert (0 == rt, msg, "err = %d", rt);
rt = rInt (rsp, LCOM_RSP_NUMT, -1, 0);
_assert (1 == rt, msg, "numt = %d", rt);
rt = rInt (rsp, LCOM_RSP_ROWID, -1, 0);
_assert (rid == rt, msg, "rowid = %d(%d)", rt, rid);
rec = luti_unwrap (ses1, rsp, 0, LCOM_RSP_DATA, rdb->dbid);
_assert (0 != rec, msg, "rec == 0");
if (0 < rlen) {
_assert (rlen == rec->len, msg, "rec->len = %d(%d)", rec->len, rlen);
}
else {
_assert (0 < rec->len, msg, "rec->len = %d", rec->len);
}
_assert (rdb->dbid == rec->dbid, msg,
"rec->dbid = %d(%d)", rec->dbid, rdb->dbid);
_chk_rec (msg, rec, 0, rid);
mFree (ses1, rec);
}
static int _query ( /* no _rsp_cb */
const char *msg, int flg, int mode, int skip, int size, int exp
) {
Rec **recs;
Db *rdb;
char fbuf[32], mbuf[32], kbuf[32], sbuf[32];
int *rowids;
int rt, rr, j;
alen = 0;
args[alen++] = "type";
args[alen++] = "query";
args[alen++] = "db";
args[alen++] = "cds";
sprintf (mbuf, "%d", mode);
args[alen++] = "mode";
args[alen++] = mbuf;
if (flg) {
sprintf (fbuf, "%d", flg);
args[alen++] = "flags";
args[alen++] = fbuf;
}
if (skip) {
sprintf (kbuf, "%d", skip);
args[alen++] = "skip";
args[alen++] = kbuf;
}
if (size) {
sprintf (sbuf, "%d", size);
args[alen++] = "size";
args[alen++] = sbuf;
}
args[alen++] = "key";
args[alen++] = "Africa$";
rqs = _buildrqs (ses1, alen, args);
_assert (0 != rqs, msg, "rqs == 0");
rt = openIsisNSend (st0, ses1, rqs, 0, _RspCld, 0);
_assert (0 == rt, msg, "rt = %d", rt);
rsp = openIsisNRecv (st0, &rdb);
_assert (0 != rsp, msg, "rsp == 0");
_assert (0 != rdb, msg, "rdb == 0");
rt = rInt (rsp, LCOM_RSP_ERR, -1, 0);
_assert (0 == rt, msg, "err = %d", rt);
rr = openIsisNGetResult (st0, ses0, &rowids, &recs, &rt);
if (0 <= exp) {
_assert (exp == rr, msg, "numr = %d(%d)", rr, exp);
}
else {
_assert (0 <= rr, msg, "numr = %d", rr);
}
_assert (rt >= rr, msg, "numt = %d/%d", rt, rr);
if (recs) {
for (j = 0; rr > j; ++j) {
_chk_rec (msg, recs[j], 0, rowids[j]);
mFree (ses0, recs[j]);
}
mFree (ses0, recs);
}
rt = rr ? rowids[rr - 1] : -1;
mFree (ses0, rowids);
return rt;
}
static int _update (const char *msg,
int rid, int rtag, char *rval, int itag, char *ival, int exp
) {
Rec *rec;
char rbuf[32];
int rt;
alen = 0;
args[alen++] = "type";
args[alen++] = 0 <= rid ? "update" : "insert";
args[alen++] = "db";
args[alen++] = "lstbtest";
_set_tms (msg, "lstbtest", 0);
if (0 <= rid) {
sprintf (rbuf, "%d", rid);
args[alen++] = "rowid";
args[alen++] = rbuf;
}
rqs = _buildrqs (ses1, alen, args);
rec = rSet (ses1, 0, 0, rtag, rval,
88, "summary of the conference in gondor", 0);
rqs = luti_wrap (ses1, rqs, rec, LCOM_RQS_REC);
mFree (ses1, rec);
if (itag) {
rec = rSet (ses1, 0, 0, itag, ival, 0);
rqs = luti_wrap (ses1, rqs, rec, LCOM_RQS_REC);
mFree (ses1, rec);
}
rt = openIsisNSend (st0, ses1, rqs, &_rsp_cb, _RspCld, 0);
_assert (0 == rt, msg, "rt = %d", rt);
rsp = openIsisNRecv (st0, 0);
_assert (0 != rsp, msg, "rsp == 0");
rt = rInt (rsp, LCOM_RSP_ERR, -1, 0);
_assert (0 == rt, msg, "err = %d", rt);
rt = rInt (rsp, LCOM_RSP_ROWID, -1, 0);
if (0 < exp) {
_assert (exp == rt, msg, "rowid = %d(%d)", rt, exp);
}
else {
_assert (0 < rt, msg, "rowid = %d", rt);
}
return rt;
}
static void _rmtstdb () {
char path[1024];
char *p;
int err;
if (! sc0) {
fprintf (stderr, "rmtstdb: st0 == 0\n");
return;
}
if (! sc0->cfg) {
fprintf (stderr, "rmtstdb: st0->cfg == 0\n");
return;
}
if (! rString (sc0->cfg, OPENISIS_SPATH, 0, path, 990)) {
fprintf (stderr, "rmtstdb: path not given\n");
return;
}
p = path + strlen (path);
*p++ = '/';
if (rString (sc0->cfg, OPENISIS_DPATH, 0, p, 10)) {
p += strlen (p);
*p++ = '/';
}
err = 0;
strcpy (p, "lstbtest.mst");
errno = 0;
if (unlink (path)) {
fprintf (stderr, "rmtstdb: unlink mst = %d\n", errno);
err = 1;
}
strcpy (p, "lstbtest.oxi");
errno = 0;
if (unlink (path)) {
fprintf (stderr, "rmtstdb: unlink oxi = %d\n", errno);
err = 1;
}
strcpy (p, "lstbtest.xrf");
errno = 0;
if (unlink (path)) {
fprintf (stderr, "rmtstdb: unlink xrf = %d\n", errno);
err = 1;
}
if (! err) {
log_msg (LOG_INFO, "db lstbtest removed.");
}
}
static void _tstUnwrap () {
Rec *r1, *r2, *r31, *r32, *r4;
r31 = rSet (ses0, 0, 0, 11, "r3111", 12, "r3112", 13, "r3113", 0);
r32 = rSet (ses0, 0, 0, 11, "r3211", 12, "r3212", 13, "r3213", 0);
r2 = rSet (ses0, 0, 0, 21, "r221", 22, "r222", 0);
r2 = luti_wrap (ses0, r2, r31, 89);
r2 = rSet (ses0, r2, 0, 23, "r223", 0);
r2 = luti_wrap (ses0, r2, r32, 89);
r1 = 0;
r1 = luti_wrap (ses0, r1, r2, 79);
r1 = rSet (ses0, r1, 0, 111, "r1111", 112, "r1112", 0);
r4 = luti_getembed (ses0, r1, ".79.89[1]", 0);
_cmp_rec ("unwrap", r4, r32);
mFree (ses0, r1);
mFree (ses0, r2);
mFree (ses0, r31);
mFree (ses0, r32);
mFree (ses0, r4);
}
int main (int argc, char **argv) {
char *argd[64];
int lastid;
int j;
switch (argc) {
case 2:
j = atoi (argv[1]);
goto dfltargs;
case 0:
case 1:
j = 1;
dfltargs:
argc = 0;
/* server side */
argd[argc++] = "syspath";
argd[argc++] = "/opt/openisis/db";
argd[argc++] = "db";
argd[argc++] = "cds";
argd[argc++] = "dbpath";
argd[argc++] = "cds";
/* client side */
/* MMM TBD mapping of client db id 2 schema + srv db id */
argd[argc++] = "schema";
argd[argc++] = "^stonno^htonno^p4343";
argv = argd;
break;
default:
if ((j = atoi (argv[1]))) {
--argc;
++argv;
}
else {
j = 1;
}
--argc;
++argv;
}
cLog (1 == j ? LOG_INFO : LOG_ERROR, 0);
ses1 = cSession (0);
do {
_RspCld[0] = _DelCld[0] = 0;
st0 = st1 = st2 = st3 = 0;
sc0 = sc1 = sc2 = sc3 = 0;
db = 0;
rqs = rsp = 0;
openIsisNInit (0, 0, 0);
openIsisNDeinit ();
memset (&cbdta, 0, sizeof (OpenIsisStubCbData));
cbdta.delcb = &_del_cb;
cbdta.delcld = _DelCld;
_open_srv ("A", argc, (const char**)argv);
openIsisNDeinit ();
_assert (0 == stub0, "B", "stub0 != 0");
_assert (2 == _DelCld[0], "B", "_DelCld = %d", _DelCld[0]);
sc1 = openIsisNSchema (st1);
_assert (sc1 == 0, "B", "sc1 != 0");
_open_srv ("C", argc, (const char**)argv);
openIsisNClose (st3);
_assert (3 == _DelCld[0], "D", "_DelCld = %d", _DelCld[0]);
sc1 = openIsisNSchema (st1);
_assert (sc1 == 0, "D", "sc1 != 0");
_open_db ("E", "cds");
_close_db ("F", "cds", 0);
_open_db ("G", "cds");
_close_db ("H", "cds", 0);
_open_db ("I", "cds");
_assert (0 == _RspCld[0], "I", "_RspCld = %d", _RspCld[0]);
_assert (8 == _DelCld[0], "I", "_DelCld = %d", _DelCld[0]);
_maxrow ("J", "cds", 154);
_read ("K", 94, 7);
_assert (0 == _RspCld[0], "K", "_RspCld = %d", _RspCld[0]);
_assert (9 == _DelCld[0], "K", "_DelCld = %d", _DelCld[0]);
_query ("L", 0, OPENISIS_QRY_KEYAT, 0, 0, 5);
lastid = _query ("M", 0, OPENISIS_QRY_KEYAT, 0, 4, 4);
_query ("N", 0, OPENISIS_QRY_KEYAT, 1 + lastid, 4, 1);
_query ("P", LCOM_QRYF_READREC, OPENISIS_QRY_KEYAT, 0, 0, 5);
lastid = _query ("Q", LCOM_QRYF_READREC, OPENISIS_QRY_KEYAT, 0, 4, 4);
_query ("R", LCOM_QRYF_READREC, OPENISIS_QRY_KEYAT, 1 + lastid, 4, 1);
_assert (0 == _RspCld[0], "R", "_RspCld = %d", _RspCld[0]);
_assert (9 == _DelCld[0], "R", "_DelCld = %d", _DelCld[0]);
#if 0 /*MMMOOO*/
_open_db ("S", "lstbtest");
lastid = _update ("T", -1, 22, "Gandalf", 22, "Frodo", 1);
_update ("U", -1, 23, "Larry", 0, 0, 2);
_update ("V", lastid, 22, "Harry", 0, 0, 1);
_maxrow ("W", "lstbtest", 2);
_close_db ("X", "lstbtest", 1);
_assert (0 == _RspCld[0], "X", "_RspCld = %d", _RspCld[0]);
_assert (15 == _DelCld[0], "X", "_DelCld = %d", _DelCld[0]);
_rmtstdb ();
#endif
_maxrowerr ("Y", "cds");
_read ("Z", 94, 7);
_maxrow ("a", "cds", 154);
_read ("b", 94, 7);
_tstUnwrap ();
openIsisNDeinit ();
#if 0 /*MMMOOO*/
_assert (0 == _RspCld[0], "Z", "_RspCld = %d", _RspCld[0]);
_assert (18 == _DelCld[0], "Z", "_DelCld = %d", _DelCld[0]);
#endif
if (0 == (j % 100)) {
printf ("%d ...\n", j);
}
} while (--j);
printf ("ok.\n");
return 0;
}
lstr.c 0100644 0001130 0001130 00000023050 07630630467 010676 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lstr.c,v 1.12 2003/03/03 10:40:28 kripke Exp $
implementation of record cooking.
*/
#include
#include "lstr.h"
#include "luti.h"
/* ************************************************************
private types
*/
/* ************************************************************
private data
*/
static const char * nameMfc[] = {
"Mfc", "CTLM", "NMFN", "NMFB", "NMFP", "TYPE", "RCNT", "MFX1", "MFX2", "MFX3"
};
static int descMfc[] = {
LSTRSIZE( 9, 0, 0 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT,
LMBRINT,
LMBRINT,
LMBRSHORT,
LMBRSHORT,
LMBRINT,
LMBRINT,
LMBRINT,
LMBRINT
}; /* descMfc */
static const char * nameMfr[] = {
"Mfr", "MFN", "RECL", "BWB", "BWP", "BASE", "NVF", "STAT",
"TAG", "'POS", "LEN"
};
static int descMfr[] = {
LSTRSIZE( 7, 3, 0 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT,
LMBRSHORT,
LMBRINT,
LMBRSHORT,
LMBRSHORT,
LMBRSHORT,
LMBRSHORT,
LMBRSHORT,
LMBRSHORT,
LMBRSHORT
}; /* descMfr */
/* aligned version */
static int descMfrA[] = {
LSTRSIZE( 7, 3, 0 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT,
LMBRSHORT,
LSTRLOFF( LMBRINT, 8 ),
LMBRSHORT,
LMBRSHORT,
LMBRSHORT,
LMBRSHORT,
LMBRSHORT,
LMBRSHORT,
LMBRSHORT
}; /* descMfr */
static const char * nameXrf[] = {
"Xrf", "XPOS", "XREC"
};
static int descXrf[] = {
LSTRSIZE( 1, 1, 127 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT,
LMBRINT
}; /* descXrf */
static const char ** nameMst[LSTR_MST] = {
nameMfc, nameMfr, nameXrf
};
static int* descMst[LSTR_MST] = {
descMfc, /* MST head */
descMfr, /* MST record */
descXrf /* XRF record */
};
static int* descMstA[LSTR_MST] = {
descMfc, /* MST head */
descMfrA, /* MST record */
descXrf /* XRF record */
};
static const char * nameCnt[] = {
"Cnt", "TYPE", "ORDN", "ORDF", "N", "K", "LEV", "POSR", "NMAX", "FMAX", "ABNO"
};
static int descCnt[] = {
LSTRSIZE( 10, 0, 0 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRSHORT, /* TYPE */
LMBRSHORT, /* ORDN */
LMBRSHORT, /* ORDF */
LMBRSHORT, /* N */
LMBRSHORT, /* K */
LMBRSHORT, /* LEV */
LMBRINT, /* POSR */
LMBRINT, /* NMAX */
LMBRINT, /* FMAX */
LMBRSHORT /* ABNO */
}; /* descCnt */
static int descCntA[] = {
LSTRSIZE( 10, 0, 0 ), /* sizes (fix, rep, occ) */
0,28,44,
LMBRSHORT, /* TYPE */
LMBRSHORT, /* ORDN */
LMBRSHORT, /* ORDF */
LMBRSHORT, /* N */
LMBRSHORT, /* K */
LMBRSHORT, /* LEV */
LMBRINT, /* POSR */
LMBRINT, /* NMAX */
LMBRINT, /* FMAX */
LMBRSHORT /* ABNO */
}; /* descCntA */
static const char * nameN0[] = {
"N0", "POS", "OCK", "TYPE", "'KEY", "REF"
};
static int descN01[] = {
LSTRSIZE( 3, 2, 10 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT, /* POS */
LMBRSHORT, /* OCK */
LMBRSHORT, /* TYPE */
10, /* KEY */
LMBRINT /* REF */
};
static int descN01A[] = {
LSTRSIZE( 3, 2, 10 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT, /* POS */
LMBRSHORT, /* OCK */
LMBRSHORT, /* TYPE */
10, /* KEY */
LSTRLOFF( LMBRINT, 12 ) /* REF */
};
static int descN02[] = {
LSTRSIZE( 3, 2, 10 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT, /* POS */
LMBRSHORT, /* OCK */
LMBRSHORT, /* TYPE */
30, /* KEY */
LMBRINT /* REF */
};
static int descN02A[] = {
LSTRSIZE( 3, 2, 10 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT, /* POS */
LMBRSHORT, /* OCK */
LMBRSHORT, /* TYPE */
30, /* KEY */
LSTRLOFF( LMBRINT, 32 ) /* REF */
};
static const char * nameL0[] = {
"L0", "POS", "OCK", "TYPE", "PS", "'KEY", "INFB", "INFP"
};
static int descL01[] = {
LSTRSIZE( 4, 3, 10 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT, /* POS */
LMBRSHORT, /* OCK */
LMBRSHORT, /* TYPE */
LMBRINT, /* PS */
10, /* KEY */
LMBRINT, /* INFB */
LMBRINT /* INFP */
};
static int descL01A[] = {
LSTRSIZE( 4, 3, 10 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT, /* POS */
LMBRSHORT, /* OCK */
LMBRSHORT, /* TYPE */
LMBRINT, /* PS */
10, /* KEY */
LSTRLOFF( LMBRINT, 12 ), /* INFB */
LMBRINT /* INFP */
};
static int descL02[] = {
LSTRSIZE( 4, 3, 10 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT, /* POS */
LMBRSHORT, /* OCK */
LMBRSHORT, /* TYPE */
LMBRINT, /* PS */
30, /* KEY */
LMBRINT, /* INFB */
LMBRINT /* INFP */
};
static int descL02A[] = {
LSTRSIZE( 4, 3, 10 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT, /* POS */
LMBRSHORT, /* OCK */
LMBRSHORT, /* TYPE */
LMBRINT, /* PS */
30, /* KEY */
LSTRLOFF( LMBRINT, 32 ), /* INFB */
LMBRINT /* INFP */
};
static const char * nameIfp[] = {
"IfH", "NXTB", "NXTP", "TOTP", "SEGP", "SEGC", "MFN", "INFO"
};
static int descIfp[] = {
#ifdef CONVERT_POSTINGS
/*
to properly convert postings,
the repeated part needs to be a big-endian 64bit int
however, converting can't be done in a straightforward fashion anyway,
since posting segments have holes of one or two longs at block boundaries
*/
LSTRSIZE( 5, 2, 1 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
LMBRINT, /* */
LMBRINT, /* */
#else
LSTRSIZE( 5, 0, 0 ), /* sizes (fix, rep, occ) */
LSTR_AUTOLENGTHS,
#endif
LMBRINT, /* */
LMBRINT, /* */
LMBRINT, /* */
LMBRINT, /* */
LMBRINT /* */
};
static const char ** nameInv[LSTR_INV] = {
nameCnt,
nameN0,
nameL0,
nameN0,
nameL0,
nameIfp
};
static int* descInv[LSTR_INV] = {
descCnt, /* CNT record */
descN01, /* N01 record */
descL01, /* L01 record */
descN02, /* N02 record */
descL02, /* L02 record */
descIfp /* Ifp record */
};
static int* descInvA[LSTR_INV] = {
descCntA, /* CNT record */
descN01A, /* N01 record */
descL01A, /* L01 record */
descN02A, /* N02 record */
descL02A, /* L02 record */
descIfp /* Ifp record */
};
/* ************************************************************
private functions
*/
/* ************************************************************
package data
*/
const LstrSet lstrlib[LSET_SETS] = {
{ /* Isis 1 MST */
nameMst, { descMst, descMstA }
},
{ /* Isis 1 INV */
nameInv, { descInv, descInvA }
},
}; /* lstrlib */
/* ************************************************************
package functions
*/
/* ************************************************************
public functions
*/
int lstr_auto ( int *str ) {
static int pow2[] = { 1, 2, 4, 8 };
int ret = 0;
int fix,rep,occ,i;
int *xmbrs, txlen, tilen;
int nmbrs, off = 0, xlen[2] = {0,0}, blen[2] = {0,0}, x=0;
if ( NULL == str ) {
#define DOALL( x ) \
for ( i=0; i<(int)(sizeof(x)/sizeof(x[0])); i++ ) \
if ( NULL != x[i] ) ret |= lstr_auto( x[i] )
DOALL( descMst );
DOALL( descInv );
DOALL( descMstA );
DOALL( descInvA );
return ret;
}
fix = LSTRFIX(*str);
rep = LSTRREP(*str);
occ = LSTROCC(*str);
xmbrs = str + LSTR_XMBR;
nmbrs = fix;
for ( x=0; ; x++ ) { /* fix and rep */
for ( i=0; i> 16) )
str[LSTR_XRLO] |= xlen[0] << 16;
else if ( xlen[0] > (short)(str[LSTR_XRLO] >> 16) )
log_msg( LOG_WARN, "short xlen[0] %hd < %d",
(short)(str[LSTR_XRLO] >> 16), xlen[0] );
if ( !(short)str[LSTR_XRLO] )
str[LSTR_XRLO] |= 0xffff & xlen[1];
else if ( xlen[1] > (short)str[LSTR_XRLO] )
log_msg( LOG_WARN, "short xlen[1] %hd < %d",
(short)str[LSTR_XRLO], xlen[1] );
/* build totals applying occ */
txlen = (short)(str[LSTR_XRLO] >> 16)
+ (int)occ*(short)str[LSTR_XRLO];
if ( !str[LSTR_XLEN] )
str[LSTR_XLEN] = txlen;
else if ( txlen > str[LSTR_XLEN] )
log_msg( LOG_WARN, "short xlen %d < %d",
str[LSTR_XLEN], txlen );
tilen = LSTRLEN(*str) + blen[0] + (int)occ*blen[1];
if ( !str[LSTR_ILEN] )
str[LSTR_ILEN] = tilen;
else if ( tilen > str[LSTR_ILEN] )
log_msg( LOG_WARN, "short ilen %d < %d",
str[LSTR_ILEN], tilen );
LOG_DBG( LOG_ALL, "%08x xrlo %08x xl %d il %d",
str[0], str[1], str[2], str[3] );
return ret;
} /* lstr_auto */
lstr.h 0100644 0001130 0001130 00000022631 07630630467 010707 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef LSTR_H
/*
$Id: lstr.h,v 1.14 2003/03/03 10:40:29 kripke Exp $
definition of ISIS internal structures for openisis lib implementation.
This header defines the common internal data structures and their members
and external structure and member definition types.
A internal data structure is the common in-memory representation of
ISIS data; it may be backed by different external (in-file) structures.
An in-memory data structure
is a contigous int* chunk of memory, consisting of the sizes (one int),
an array of members (int) and some buffer (array of char).
A data structures sizes consists of the lengths
fix and rep of the fixed and repeated part, resp.,
and the number of occurences of the repeated part.
It is used in both an abstract description of a data type
and each instance of a data structure.
If the number of occurences of the repeated part is not fixed,
it is 0 in the abstract case.
The lengths are suitable as offsets into the lmbr array.
This structure is designed to fit within 32 bits.
The sizing is the only common meta-data description of a logical
data type. For readability of code, however, the member indexes
are enum-erated. When actually used, the type has to match the code's
assumptions anyway.
The numbers fix and rep are fixed for a given data structure.
Although the logical data type has more properties,
especially the type of each member (numeric, C-string or raw mem),
a representation of this data is needed only when converting
external data and thus is stored (somewhat redundantly)
with the external properties.
There are only two types of members:
numeric and memory (mostly some character data).
both are represented by a int, which, for memory,
denotes an offset (counted in BYTES, not ints) from the records start
(NOT offset into the data buffer, as with the isis mst record).
The member array consists of a fixed number fix of initial
members and a number occ of occurences
of member arrays of fixed length rep .
While occ is usually fixed,
it varies for masterfile records.
Note that this layout somewhat resembles the isis record itself,
which supports a single level of nesting and repetition.
*/
/* id of a structure */
/* construction
*/
#define LSTRID( set, no, vari ) \
(((int)(vari))<<16 | (set)<<8 | (no) )
/* access
*/
#define LSTRNO( l ) (char)(l) /* no of record type in set */
#define LSTRSET( l ) (char)((l)>>8) /* record set number */
#define LSTRVAR( l ) (short)((l)>>16) /* variant */
/* size of a structure */
/* construction
*/
#define LSTRSIZE( fix, rep, occ ) \
(((int)(occ))<<16 | (rep)<<8 | (fix))
/* access
*/
#define LSTRFIX( l ) (char)(l) /* length of fixed part */
#define LSTRREP( l ) (char)((l)>>8) /* length of repeated part */
#define LSTROCC( l ) (short)((l)>>16) /* the members */
/* actual number of members */
#define LSTRMEMS( l ) (LSTRFIX(l) + LSTRREP(l)*LSTROCC(l))
/* in-memory size of members (w/o buffer). */
#define LSTRLEN( t ) ((int)sizeof(int)*(1 + LSTRMEMS(t)))
/**
Description of a external members layout is encoded as a short "len"
and a short offset "off" into the data or repeated part, resp.
offset and len are combined into one int.
If offsets are 0, they are computed based on the assumption,
that members are layouted packed in the declared ordering.
For character data, len just gives the length of the member
in bytes (up to 2**15), with the highest bit cleared.
A length of 0 denotes character data of variable size.
Numerical data is marked with the highest bit set.
For numbers, the lowest two bits give the ld of the length in bytes
(i.e. 0=1,1=2,2=4,3=8 bytes).
The offset and length of the actually used bits within these bytes
are each encoded in six bits, which is enough for 0..63.
A length of 0 (as resulting from 6-bit overflow on '64') means no masking.
The 15th bit, if set, denotes high endian data.
Note that the only reason for stuffing all that into a short is,
that it is sufficient. The amount of space saved is minimal and
depending on the actual processor there may be a slight performance
penalty for the bit operations.
*/
#define LONG2LEN( l ) ((short)(l))
#define LONG2OFF( l ) ((short)((l) >> 16))
/* constants
*/
#define LMBR_FNUM 0x8000 /* flag numeric data */
#define LMBR_FHIE 0x4000 /* flag high endian data */
/* construction
*/
#define LMBRNUM( ld ) (LMBR_FNUM | ld)
/* standard nueric types: the world according to Java :) */
#define LMBRBYTE LMBRNUM(0)
#define LMBRSHORT LMBRNUM(1)
#define LMBRINT LMBRNUM(2)
#define LMBRLONG LMBRNUM(3)
/* bit mask construction for numerical data */
#define LMBRBITS( off, len, ld ) (LMBR_FNUM | (off)<<8 | (len)<<2 | (ld))
#define LMBRBITS4( off, len ) (LMBR_FNUM | (off)<<8 | (len)<<2 | 2)
/* if you have to specify the offset explicitly */
#define LSTRLOFF( len, off ) ((len) | (off)<<16)
/* access
*/
#define LMBRISNUM( f ) (LMBR_FNUM & (f))
#define LMBRLD( f ) (0x3 & (f))
/* bit access for numerical data */
#define LMBRISBITS( f ) (0x3ffc & (f))
#define LMBRBITLEN( f ) (0x003f & ((f) >> 2))
#define LMBRBITOFF( f ) (0x003f & ((f) >> 8))
/**
Description of an external structure layout.
This is usually fixed for a given (implementation of a) type;
for the isis mst record, however, the occurences of the repeated
part and the total external and internal lengths vary for each record.
*/
enum {
LSTR_SIZE, /* sizes */
LSTR_XRLO, /* repeated part byte length and offset <<16; auto */
LSTR_XLEN, /* ext. total length of data; auto */
LSTR_ILEN, /* int. total length of data; auto */
LSTR_XMBR /* members */
};
#define LSTR_AUTOLENGTHS 0,0,0
#define LSTR_LONGS( totmbrs ) (4+(totmbrs))
/* technical variants of a record */
typedef enum {
LVAR_PAC, /* packed little endian base variant */
LVAR_ALI, /* aligned variant */
LVAR_BIG, /* big endian aligned */
LVAR_VARI
} lstr_variant;
/* known record sets */
enum {
LSET_MST, /* isis 1 master file */
LSET_INV, /* isis 1 inverted file */
LSET_SETS
};
typedef struct {
const char ***name; /* an array of names per record type */
int **desc[LVAR_VARI]; /* an array of descs per variant */
} LstrSet;
/* data types of ISIS 1 master file */
typedef enum {
LSTR_MFC, /* MST head */
LSTR_MFR, /* MST record */
LSTR_XRF, /* XRF record */
LSTR_MST /* count */
} lstr_mst;
enum { /* MFC members */
LMFC_CTLM = 1,
LMFC_NMFN, /* next available MFN */
LMFC_NMFB, /* next free block in MF counting from 1 */
LMFC_NMFP, /* next free pos in MF */
LMFC_TYPE, /* "0 for user's bases; 1 for messages" */
LMFC_RCNT, /* record count ? not used by CISIS */
LMFC_MFX1,
LMFC_MFX2, /* LAN lock */
LMFC_MFX3, /* LAN lock */
LMFC__FL
};
enum { /* XRF members */
LXRF_XPOS = 1,
LXRF_XREC
};
/* data types of ISIS 1 inverted file */
typedef enum {
LSTR_CNT, /* CNT record */
LSTR_N01, /* N01 record */
LSTR_L01, /* L01 record */
LSTR_N02, /* N02 record */
LSTR_L02, /* L02 record */
LSTR_IFP, /* IFP record */
LSTR_INV /* count */
} lstr_inv;
enum { /* CNT members */
LCNT_TYPE = 1, /* 1 or 2: N0x */
LCNT_ORDN, /* fix 5 */
LCNT_ORDF, /* fix 5 */
LCNT_N, /* fix 15 */
LCNT_K, /* fix 5 */
LCNT_LEV, /* depth of index */
LCNT_POSR, /* position of root */
LCNT_NMAX,
LCNT_FMAX,
LCNT_ABNO,
LCNT__FL
};
enum { /* N0x members */
LN0X_POS = 1, /* record number */
LN0X_OCK, /* number of keys */
LN0X_TYPE, /* type 1 or 2 */
LN0X__FL, /* offset of repeated part */
LN0X_KEY = 0, /* key chars */
LN0X_REF, /* ref to next node (>0) or leaf (<0) */
LN0X__RL /* length of repeated part */
};
enum { /* L0x members */
LL0X_POS = 1, /* record number */
LL0X_OCK, /* number of keys */
LL0X_TYPE, /* type 1 or 2 */
LL0X_PS, /* type 1 or 2 */
LL0X__FL, /* offset of repeated part */
LL0X_KEY = 0, /* key chars */
LL0X_INFB, /* ifp block */
LL0X_INFP, /* ifp pos */
LL0X__RL /* length of repeated part */
};
enum { /* IFP members */
LIFP_NXTB = 1, /* block of next segment */
LIFP_NXTP, /* pos of next segment */
LIFP_TOTP, /* total postings */
LIFP_SEGP, /* postings in this segment */
LIFP_SEGC, /* postings in this segment */
LIFP__FL, /* offset of repeated part */
LIFP_MFN = 0, /* mfn */
LIFP_ADD, /* additional qualifyers */
LIFP__RL /* length of repeated part */
};
extern const LstrSet lstrlib[LSET_SETS];
/** auto-initialise the given structure or all builtins, iff NULL.
*/
extern int lstr_auto ( int *str );
#define LSTR_H
#endif /* LSTR_H */
lsv.c 0100644 0001130 0001130 00000060621 07642311343 010513 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: lsv.c,v 1.9 2003/04/01 13:23:47 kripke Exp $
OpenIsis server
*/
#include
#include
#include
#include
#include
#include
#include
#include /* memcpy et al */
#include
#include
#include "lsv.h"
static Con *con[FD_SETSIZE]; /* usually 1024, but up to 1<<16 */
static pthread_mutex_t lsv_mutex_init = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t lsv_cond_init = PTHREAD_COND_INITIALIZER;
static pthread_key_t lsv_key_wrk;
static int lsv_term;
static siginfo_t lsv_sig;
static const char * pdg[] = {
"---",
"NEW",
"RED",
"EOF"
};
static const char * stg[] = {
"NEW",
"CON",
"INP",
"SES",
"WRK",
"RUN",
"COM",
"DON",
"RET"
};
static void lsv_sighand ( int sig, siginfo_t *info, void *ucontext )
{
if ( SIGINT == sig || SIGTERM == sig )
lsv_term = !0;
if ( info )
lsv_sig = *info;
else { /* ??? */
lsv_sig.si_signo = sig;
lsv_sig.si_pid = 0;
}
(void)ucontext;
} /* lsv_sighand */
static int lsv_enq ( Que *q, Con *c )
{
c->nxt = 0;
q->len++;
if ( q->tail ) {
q->tail->nxt = c;
q->tail = c;
return 1;
}
q->head = q->tail = c;
return 0;
} /* lsv_enq */
static void lsv_penq ( Srv *srv, Con *c )
{
Pool *work = c->grp ? &srv->pool : &srv->main;
srv->jobs++;
c->stg = LSV_STGWRK;
if ( c->ses )
c->ses->cur = c;
lsv_enq( &work->que, c );
sMsg( srv->ses, LOG_DEBUG, "enq req %d from %s qlen %d",
LIO_FD&c->ios.file, c->nam, work->que.len );
if ( work->wait ) {
Wrk *w = work->wait;
work->wait = w->nxt;
w->nxt = 0;
srv->plen--;
pthread_cond_signal( &w->todo );
sMsg( srv->ses, LOG_DEBUG, "woke %d of %d", w->id, srv->plen );
}
} /* lsv_penq */
static Con *lsv_deq ( Que *q )
{
Con *c = q->head;
if ( c ) {
q->len--;
q->head = c->nxt;
c->nxt = 0;
if ( q->tail == c )
q->tail = 0;
}
return c;
} /* lsv_deq */
Wrk *svCur ()
{
return (Wrk*)pthread_getspecific( lsv_key_wrk );
}
static void *lsv_wrk ( void *arg )
{
Wrk *self = (Wrk*)arg;
Srv *srv = self->srv;
Pool *work = self->id ? &srv->pool : &srv->main;
Con *job = 0;
Session ses;
sigset_t blk;
/* block signals, they should go to the receiver */
sigemptyset( &blk );
sigaddset( &blk, SIGHUP );
sigaddset( &blk, SIGINT );
sigaddset( &blk, SIGTERM );
sigaddset( &blk, SIGPIPE );
pthread_sigmask( SIG_BLOCK, &blk, 0 );
pthread_setspecific( lsv_key_wrk, arg );
if ( self->id )
srv->app( 0, LSV_APPINI );
for (;/*ever*/;) {
int ret;
if ( job ) /* from last turn */
sMsg( job->ses, LOG_DEBUG, "thr %d done job %s", self->id, job->nam );
pthread_mutex_lock( &srv->mut );
if ( job ) { /* from last turn */
sMsg( job->ses, LOG_DEBUG, "thr %d enq job %s pdg %s",
self->id, job->nam, pdg[job->pdg] );
job->stg = LSV_STGRET;
lsv_enq( &srv->recv, job );
if ( job->pdg ) /* wakeup receiver */
write( srv->pip[1], &job->pdg, 1 );
if ( job->ses->que ) { /* immediatly reschedule queued job */
Con *q = job->ses->que;
job->ses->que = q->nxt;
sMsg( srv->ses, LOG_VERBOSE, "dequeing con %s for ses '%s'",
q->nam, job->ses->name );
lsv_penq( srv, q );
}
}
self->cur = 0;
while ( !(job = lsv_deq( &work->que ))
&& !(LSV_SHUTDN & srv->flg)
) {
self->nxt = work->wait;
work->wait = self; /* pls pls lemme wrk */
srv->plen++;
self->waits++;
sMsg( self->ses, LOG_DEBUG, "thr %d waiting", self->id );
pthread_cond_wait( &self->todo, &srv->mut );
}
self->cur = job;
if ( job )
job->stg = LSV_STGRUN;
pthread_mutex_unlock( &srv->mut );
/* end of monday morning meeting blah blah -- start working */
if ( ! job || (LSV_SHUTDN & srv->flg) )
break;
self->jobs++;
if ( ! job->ses )
job->ses = self->ses;
ses = job->ses;
ses->io[1] = &job->ios;
if ( ses->res )
CLRREC( ses->res );
sMsg( ses, LOG_DEBUG, "thr %d run job %s ses '%s'.%d",
self->id, job->nam, job->ses->name, job->ses->accnt );
ret = srv->app( job, LSV_APPRUN );
sMsg( ses, LOG_DEBUG, "thr %d got %d buf %d",
self->id, ret, LIO_SAVAIL( &job->ios ) );
/*
lock only in order to create SMP memory barrier.
on a single processor system, the byte should be safely readable
by receiver without this lock.
TOD: check alternatives:
since receiver is checking the stage of active connections
only in rare cases, it might be more efficient to guard either
this set or the processing up to here with a per thread mutex.
*/
pthread_mutex_lock( &srv->mut );
job->stg = LSV_STGDON;
job->ses->cur = 0;
pthread_mutex_unlock( &srv->mut );
ret = srv->prt( &job->ios, LIO_SCLOSE );
/*
if ( ses->res )
mFree( ses, ses->res );
ses->res = 0;
*/
ses->io[1] = 0;
}
if ( self->id )
srv->app( 0, LSV_APPFIN );
return self;
} /* lsv_wrk */
static void *lsv_rcv ( void *arg )
{
Srv *srv = (Srv*)arg;
fd_set fd;
int fdlen = 0;
int inlock = 0;
int pending = 0;
int ret = 0, sel, i;
FD_ZERO( &fd );
FD_SET( srv->lsn, &fd );
FD_SET( srv->pip[0], &fd );
fdlen = srv->pip[0]+1;
if ( fdlen < srv->lsn+1 )
fdlen = srv->lsn+1;
for (;/*ever*/;) {
struct sockaddr_in npeer; /* of new connection */
int nsock = -1; /* new socket */
Con *c;
fd_set ready = fd;
Tm expire;
/* select on read only; writing is done in workers and
exceptions (i.e. telnet OOB data) are not used
*/
if ( 0 > (sel = select( fdlen, &ready, 0, 0, 0 )) ) {
if ( EINTR == errno ) {
sMsg( srv->ses, LOG_DEBUG, "select woken by signal" );
} else {
ret = sMsg( srv->ses, LOG_SYSERR, "select" );
goto done;
}
}
if ( lsv_sig.si_signo ) {
sMsg( srv->ses, LOG_DEBUG, "SIG%s (%d) from pid %d",
SIGHUP==lsv_sig.si_signo ? "HUP" :
SIGINT==lsv_sig.si_signo ? "INT" :
SIGTERM==lsv_sig.si_signo ? "TERM" :
SIGPIPE==lsv_sig.si_signo ? "PIPE" : "?",
lsv_sig.si_signo, lsv_sig.si_pid );
if ( 1 /*SIGHUP==lsv_sig.si_signo*/ ) {
for ( i=FD_SETSIZE; i--; )
if ( con[i] ) {
c = con[i];
sMsg( srv->ses, LOG_INFO,
"con %3d peer %s stg %s prt %d app %d ios %d %s (0x%x)",
i, c->nam, stg[c->stg], c->prt, c->app,
c->ios.pos + c->ios.b.done,
LIO_SISOPEN(&c->ios) ? "opn" : "clo", c->ios.file );
}
}
if ( lsv_term ) /* even if it wasn't the last signal delivered */
goto done;
lsv_sig.si_signo = 0;
/* must NOT access the set after interrupt,
probably all fds are set
*/
continue;
}
timeUpd( &srv->tim );
timeGtfm( srv->gtm, &srv->tim );
expire.millis = srv->tim.millis - srv->sto*LLL(1000);
pthread_mutex_lock( &srv->mut ); /* back in locked land */
inlock = !0;
/* -- we're the acceptor */
if ( FD_ISSET( srv->lsn, &ready ) ) { /* new connection */
unsigned /*socklen_t is broken*/ alen = sizeof(npeer);
nsock = accept( srv->lsn, (struct sockaddr*)&npeer, &alen );
if ( 0 > nsock ) {
ret = sMsg( srv->ses, LOG_SYSERR, "accept" );
goto done;
}
if ( FD_SETSIZE <= nsock ) {
ret = sMsg( srv->ses, ERR_BADF,
"socket %d >= FD_SETSIZE %d", nsock, FD_SETSIZE );
goto done;
}
if ( sizeof(npeer) != alen ) {
ret = sMsg( srv->ses, ERR_INVAL, "bad peer len %d", alen );
goto done;
}
/* ok */
/* setsockopt
SO_SNDTIMEO fixed value on linux
SO_OOBINLINE ... hmm, don't care
SO_LINGER off
*/
/* prepare con */
if ( !(c = con[nsock]) ) {
c = con[nsock] = mAlloc( srv->ses, sizeof(*c) );
c->ios.file = nsock; /* preliminary -- further setup below */
}
c->pdg = LSV_PDGNEW;
switch ( c->stg ) {
case LSV_STGCON: /* still in control of receiver */
case LSV_STGINP:
sMsg( srv->ses, ERR_IDIOT, "bad recv stage %d on new con", c->stg );
c->stg = LSV_STGNEW;
case LSV_STGNEW: /* clean idle socket -- pass to receiver */
lsv_enq( &srv->recv, c );
break;
case LSV_STGSES: /* still in control of receiver */
case LSV_STGWRK:
sMsg( srv->ses, ERR_IDIOT, "bad worker stage %d on new con", c->stg );
case LSV_STGRUN: /* worker will queue it */
/* hmmm ... should have COM ... */
case LSV_STGCOM:
case LSV_STGDON:
case LSV_STGRET: /* worker has queued it */
pending++;
break;
}
srv->conn++;
FD_CLR( srv->lsn, &ready ); /* skip in fd check */
} /* new connection */
if ( FD_ISSET( srv->pip[0], &ready ) ) { /* pending events */
char trash[LSV_NUMWRK];
read( srv->pip[0], trash, sizeof(trash) );
FD_CLR( srv->pip[0], &ready );
}
/* switching roles -- now we're the receiver */
while ( (c = lsv_deq( &srv->recv )) ) { /* care for ready connections */
int sock = LIO_FD & c->ios.file;
sMsg( srv->ses, LOG_DEBUG, "deq %d from %s pdg %s",
sock, c->nam, pdg[c->pdg] );
if ( c->pdg && c->stg )
pending--;
if ( !(LIO_IN & c->ios.file) && (LIO_RDWR & c->ios.file) ) {
/* cleanup closed socks */
if ( c != con[sock] ) {
ret = sMsg( srv->ses, ERR_IDIOT,
"deq bad sock %d (0x%x)", sock, c->ios.file );
goto done; /* PANIC time */
}
sMsg( srv->ses, LOG_DEBUG, "clos%s sock %d",
(LIO_OUT & c->ios.file) ? "ing" : "ed", sock );
if ( (LIO_OUT & c->ios.file) )
lio_close( &c->ios.file, LIO_INOUT );
c->ios.file &= ~LIO_RDWR; /* clear also, so we're not closing again */
FD_CLR( sock, &fd );
FD_CLR( sock, &ready ); /* probably an EOF to read */
if ( fdlen == sock+1 )
while ( fdlen
/* && !FD_ISSET( fdlen-1, &fd ) risk of muting tmp. disabled */
&& (!con[fdlen-1] || !LIO_SISOPEN(&con[fdlen-1]->ios))
)
fdlen--;
}
if ( LSV_PDGNEW == c->pdg ) { /* new connection */
struct sockaddr_in peer; /* of new connection */
if ( nsock == sock ) /* the one accepted just above */
peer = npeer;
else {
unsigned alen = sizeof(peer);
if ( getpeername( sock, (struct sockaddr*)&peer, &alen )
|| sizeof(peer) != alen
) {
ret = sMsg( srv->ses, LOG_SYSERR, "getpeername alen %d", alen );
goto done;
}
}
c->srv = srv;
c->ses = 0;
{
unsigned host = (unsigned)ntohl( peer.sin_addr.s_addr );
unsigned short port = (unsigned short)ntohs( peer.sin_port );
char *p = c->nam;
c->host = host;
memset( c->nam, 0, sizeof(c->nam) );
p += u2a( p, 0xff & (host >> 24) ); *p++ = '.';
p += u2a( p, 0xff & (host >> 16) ); *p++ = '.';
p += u2a( p, 0xff & (host >> 8) ); *p++ = '.';
p += u2a( p, 0xff & (host ) ); *p++ = ':';
p += u2a( p, port );
/* *p++ = '@'; p += u2a( p, sock ); */
}
c->con++; /* never mind overflow ... */
sMsg( srv->ses, LOG_INFO,
"connect %d on %d from %s", c->con+1, sock, c->nam );
LIO_SINIT( &c->ios, srv->prt, c->nam,
sock | LIO_INOUT|LIO_RDWR|LIO_SOCK );
LIO_BINIT( &c->flt );
FD_SET( sock, &fd );
if ( fdlen < sock+1 )
fdlen = sock+1;
c->bin = 0; /* do not reset on every request */
}
if ( ! (LIO_IN & c->ios.file) )
c->stg = LSV_STGNEW;
else {
c->stg = LSV_STGCON;
if ( ! FD_ISSET( sock, &fd ) ) {
sMsg( srv->ses, LOG_DEBUG, "reenabling %s", c->nam );
FD_SET( sock, &fd );
if ( fdlen < sock+1 )
fdlen = sock+1;
}
}
c->grp = c->pdg = c->prt = c->app = 0;
/* prepare for new request */
c->ios.pos = 0;
c->ios.b.done = c->ios.b.fill = 0;
if ( c->req )
CLRREC( c->req );
else {
/* this is done in the receiver rather than the acceptor
since in multi-receiver model, receiver owns the req
*/
c->req = mAlloc( srv->ses, LSV_BUFSIZ );
OPENISIS_INITREC( c->req, LSV_BUFSIZ, 64 );
}
if ( !(LSV_CONSES & srv->flg) )
c->ses = 0;
} /* prepare connections */
for ( i=fdlen; sel && i--; ) { /* read from the ready */
Field *sid;
int got;
if ( ! FD_ISSET( i, &ready ) )
continue;
sel--;
c = con[i];
if ( ! c ) {
ret = sMsg( srv->ses, ERR_IDIOT, "ran on empty con %d", i );
goto done;
}
if ( LSV_STGINP < c->stg ) {
sMsg( srv->ses, LOG_DEBUG, "con %d busy stage %d!", i, c->stg );
c->pdg = LSV_PDGRED;
pending++;
FD_CLR( i, &fd ); /* mute this */
continue;
}
c->ios.b.done = 0;
got = lio_read( &c->ios.file, c->ios.b.c, sizeof(c->ios.b.c) );
if ( 0 >= got ) {
if ( -ERR_EOF != got )
goto err;
sMsg( srv->ses, LOG_DEBUG, "EOF on 0x%x line %d from %s",
c->ios.file, c->req->len, c->nam );
FD_CLR( i, &fd );
if ( ! c->req->len ) /* initial eof */
goto eof; /* close immediatly */
/* lio_close( &c->ios.file, LIO_IN ); */
c->ios.file &= ~LIO_IN; /* don't shut */
c->pdg = LSV_PDGEOF; /* so it will be dequed and closed */
pending++;
got = 0; /* tell proto to close ... */
}
if ( ! c->req->len ) { /* stage should be CON */
RADDS( srv->ses, c->req, -LSV_PEER, c->nam, 1 );
sMsg( srv->ses, LOG_DEBUG, "req on %d from %s",
LIO_FD&c->ios.file, c->nam );
}
c->stg = LSV_STGINP;
c->ios.b.fill = got;
got = srv->prt( &c->ios, LIO_SPUSH );
sMsg( srv->ses, LOG_DEBUG, "got 0x%x from %s", got, c->nam );
if ( ! got && (LIO_IN & c->ios.file) ) /* ok: more next time */
continue;
if ( 0 > got ) /* nok */
goto err;
got = c->srv->app( c, LSV_APPGOT );
if ( 0 > got ) /* nok */
goto err;
if ( !(LSV_CONSES & srv->flg)
&& c->req && (sid = rGet( c->req, -LSV_SID, 0 ))
) {
c->ses = cSesByName( (char*)sid->val, sid->len, &srv->tim, &expire );
if ( ! c->ses ) { /* sad sad sad */
sMsg( srv->ses, LOG_ERROR, "out of sessions" );
goto err; /* drop connection */
}
if ( ! c->ses->accnt
&& (! c->ses->prop || ! c->ses->prop->len) /* overflow paranoia */
)
got = c->srv->app( c, LSV_APPSES );
if ( 0 > got ) /* nok */
goto err;
}
c->tim = srv->tim;
RADDS( srv->ses, c->req, -LSV_TIME, srv->gtm, 1 );
if ( c->ses ) {
sMsg( srv->ses, LOG_DEBUG, "req %s for ses '%s' %s",
c->nam, c->ses->name, c->ses->cur ? "busy" : "idle" );
c->ses->accnt++;
if ( c->ses->cur || c->ses->que ) { /* oh - busy !? */
Con **cpp = &c->ses->que;
int l = 0;
for ( ; *cpp; cpp = &(*cpp)->nxt )
if ( 4 < ++l ) {
sMsg( srv->ses, LOG_WARN,
"too many con queing on ses '%s' - dropping %s",
c->ses->name, c->nam );
goto err;
}
sMsg( srv->ses, LOG_VERBOSE, "queing con %s for ses '%s'",
c->nam, c->ses->name );
c->nxt = 0;
*cpp = c;
c->stg = LSV_STGSES;
continue;
}
}
/* yo - got it */
lsv_penq( srv, c );
continue;
err:
sMsg( srv->ses, ERR_BADF, "err 0x%x from %s", got, c->nam );
eof:
lio_close( &c->ios.file, LIO_INOUT );
c->stg = LSV_STGNEW;
FD_CLR( i, &fd );
} /* read from the ready */
srv->turn++;
srv->wlen += srv->pool.que.len;
srv->busy += srv->nwr - srv->plen;
sMsg( srv->ses, LOG_INFO,
"\n==== %s\nturn %d: %d jobs for %d idle workers %d pending",
srv->gtm, srv->turn, srv->pool.que.len, srv->plen, pending );
for ( i=srv->nwr; i--; )
if ( srv->wrk[i].cur )
sMsg( srv->ses, LOG_INFO, "thr %d serving %d from %s",
i, LIO_FD&srv->wrk[i].cur->ios.file, srv->wrk[i].cur->nam );
inlock = 0;
pthread_mutex_unlock( &srv->mut );
} /* for(ever) */
done:
if ( inlock )
pthread_mutex_unlock( &srv->mut );
srv->flg |= LSV_SHUTDN;
for ( i=srv->nwr; i--; )
pthread_cond_signal( &srv->wrk[i].todo );
return srv;
} /* lsv_rcv */
int svRun ( Srv *srv, const char *addr )
{
static const int yes = !0;
struct sigaction siga;
struct sockaddr_in sa; /* server address */
int ret = 0, i;
if ( ! srv->ses )
srv->ses = cOpen(0);
srv->mut = lsv_mutex_init;
/* set defaults */
if ( ! srv->sto )
srv->sto = 30*60; /* 30 minutes session timeout */
if ( ! srv->nwr )
srv->nwr = LSV_NUMWRK/2; /* half the max */
else if ( LSV_NUMWRK < srv->nwr )
srv->nwr = LSV_NUMWRK; /* max */
/* prepare signals */
memset( &siga, 0, sizeof(siga) );
siga.sa_sigaction = lsv_sighand;
sigemptyset( &siga.sa_mask );
sigaddset( &siga.sa_mask, SIGHUP );
sigaddset( &siga.sa_mask, SIGINT );
sigaddset( &siga.sa_mask, SIGTERM );
sigaddset( &siga.sa_mask, SIGPIPE );
siga.sa_flags = SA_SIGINFO;
sigaction( SIGHUP, &siga, 0 );
sigaction( SIGINT, &siga, 0 );
sigaction( SIGTERM, &siga, 0 );
sigaction( SIGPIPE, &siga, 0 );
/* prepare socket */
srv->lsn = socket( PF_INET, SOCK_STREAM, IPPROTO_TCP/*6*/ );
/* fcntl( srv->lsn, O_NONBLOCK ); not needed */
sa.sin_family = AF_INET;
sa.sin_port = htons( addr ? a2i( addr, -1 ) : 8080 );
sa.sin_addr.s_addr =
#ifdef LSV_LOCALHOST_ONLY
htonl(INADDR_LOOPBACK);
#else
INADDR_ANY;
#endif
if ( setsockopt( srv->lsn, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes) ) ) {
ret = sMsg( srv->ses, LOG_SYSERR, "setsockopt" );
goto inierr;
}
if ( bind( srv->lsn, (struct sockaddr*)&sa, sizeof(sa) ) ) {
ret = sMsg( srv->ses, LOG_SYSERR, "bind on port %u", htons(sa.sin_port) );
goto inierr;
}
if ( listen( srv->lsn, 64/*backlog*/ ) ) {
ret = sMsg( srv->ses, LOG_SYSERR, "listen" );
goto inierr;
}
/* prepare pending event pipe */
if ( pipe( srv->pip ) ) {
ret = sMsg( srv->ses, LOG_SYSERR, "pipe" );
goto inierr;
}
if ( pthread_key_create( &lsv_key_wrk, 0 ) ) {
ret = sMsg( srv->ses, LOG_SYSERR, "key_create" );
goto inierr;
}
/* setup workers */
memset( srv->wrk, 0, sizeof(srv->wrk) );
for ( i=srv->nwr; i--; ) {
srv->wrk[i].id = i;
srv->wrk[i].srv = srv;
srv->wrk[i].nxt = 0;
srv->wrk[i].ses = cSession(0);
memcpy( srv->wrk[i].ses->name, "wrk", 3 );
u2a( srv->wrk[i].ses->name+3, i );
srv->wrk[i].todo = lsv_cond_init;
if ( i )
pthread_create( &srv->wrk[i].thr, 0, lsv_wrk, srv->wrk+i );
}
/* start the server thread */
pthread_create( &srv->rcv, 0, lsv_rcv, srv );
/* run main worker */
srv->wrk[0].thr = pthread_self();
lsv_wrk( srv->wrk );
/* cleanup */
pthread_join( srv->rcv, 0 );
for ( i=srv->nwr; i--; )
pthread_join( srv->wrk[i].thr, 0 );
for ( i=srv->nwr; i--; )
sMsg( srv->ses, LOG_INFO, "worker %d had %d jobs %d waits",
i, srv->wrk[i].jobs, srv->wrk[i].waits );
sMsg( srv->ses, LOG_INFO,
"server had %d jobs on %d connections %d turns"
" avg queue len %.02f workers %.02f",
srv->jobs, srv->conn, srv->turn,
srv->wlen/srv->turn, srv->busy/srv->turn );
inierr:
close( srv->lsn );
return ret;
} /* svRun */
/** helper, probably should go to lio.
cat l bytes from c to buf b flushing as needed.
if ! c, empty buf b.
*/
int lio_sout ( Ios *s, Buf *b, const char *c, unsigned l )
{
if ( ! b )
b = &s->b;
do {
if ( LIO_BUFSIZ/2 < b->fill && b->done ) { /* purge */
if ( b->done < b->fill )
memmove( b->c, b->c + b->done, b->fill - b->done );
s->pos += b->done;
b->fill -= b->done;
b->done = 0;
}
if ( c && l && b->fill < LIO_BUFSIZ ) {
unsigned cp = LIO_BUFSIZ - b->fill;
if ( cp > l )
cp = l;
memcpy( b->c + b->fill, c, cp );
b->fill += cp;
c += cp;
l -= cp;
}
if ( b->fill > b->done ) {
int wr = lio_write( &s->file, b->c + b->done, b->fill - b->done );
if ( 0 > wr )
return wr;
if ( ! wr )
return -ERR_BUSY;
b->done += wr;
if ( b->done > b->fill ) {
b->done = b->fill = 0;
return -ERR_IDIOT;
}
}
if ( b->done >= b->fill ) {
s->pos += b->done;
b->done = b->fill = 0;
}
} while ( c ? l : (unsigned)(b->fill > b->done) );
return 0;
} /* lio_sout */
#define LF 10 /* LineFeed a.k.a. newline - '\n' isn't really well defined */
#define TAB 9 /* horizontal, that is */
#define VT 11 /* vertical, used as newline replacement */
static const char lf = LF;
static const char tab = TAB;
static const char vt = VT;
/* the plain protocol */
int svPlain ( Ios *s, int op )
{
Field *f;
Con *c = (Con*)s;
switch ( op ) {
case LIO_SPUSH: {
int l = s->b.fill - s->b.done;
unsigned char *b = s->b.c + s->b.done;
unsigned char *end = b+l, *v, *p;
if ( ! l ) { /* EOF: done */
if ( ! c->prt ) /* ok */
return 1;
/* last field wasn't closed by LF */
return sMsg( c->srv->ses, ERR_INVAL, "no EOL from %s", c->nam );
}
if ( c->prt )
RSPACE( c->srv->ses, c->req, l, !0 );
/* add text lines */
while ( bprt ) {
case 0: /* at beginning of line -- start new field */
if ( LF == *b ) /* empty line */
return 1;
if ( TAB != *b || !c->req->len )
RADD( c->srv->ses, c->req, 0,0,end-b, !0 );
else { /* binary mode continuation line */
conti = 1;
if ( ! c->bin ) {
sMsg( c->srv->ses, LOG_INFO,
"detected binary mode on con %s", c->nam );
c->bin = 1;
}
RSPACE( c->srv->ses, c->req, end-b, !0 );
}
if ( ! c->req )
return -ERR_NOMEM;
c->prt = 1;
case 1: /* add to last field */
f = c->req->field + c->req->len-1;
v = (unsigned char*)f->val;
p = v + f->len;
if ( conti ) {
*p++ = LF;
b++;
}
if ( c->bin ) {
for ( ; breq->used += (p - v) - f->len;
f->len = p - v;
if ( LF == b[-1] ) {
int ret = a2il( f->val, f->len, &f->tag );
if ( ret ) {
if ( ret < f->len && TAB == v[ret] )
ret++;
if ( ret < f->len )
memmove( v, v+ret, f->len - ret );
f->len -= ret;
}
ret = c->srv->app( c, LSV_APPARG );
c->prt = 0;
if ( ret )
return ret;
}
sMsg( c->srv->ses, LOG_INFO, "prs from %s: [%2d] %3d = '%.*s'",
c->nam, c->req->len-1, f->tag, f->len, f->val );
}
}
return 0;
} /* case LIO_SPUSH */
case LIO_SFLUSH:
if ( s->b.fill <= s->b.done )
return 0;
case LIO_SCLOSE:
if ( !(LIO_OUT & s->file) )
return -ERR_INVAL;
/*
this switch is not protected, since receiver doesn't care
*/
if ( c->stg < LSV_STGCOM )
c->stg = LSV_STGCOM;
if ( ! s->pos+s->b.done && c->ses->res ) {
char num[32];
int n = c->ses->res->len;
sMsg( c->ses, LOG_INFO, "com %d fields to %s", c->ses->res->len, c->nam );
for ( f = c->ses->res->field; n--; f++ ) {
const char *v = f->val, *nl;
int vl = f->len;
int nlen = i2a( num, f->tag );
num[nlen++] = TAB;
lio_sout( s, &c->flt, num, nlen );
while ( vl && (nl = memchr(v,'\n',vl)) ) {
int ll = 1+(nl-v);
if ( c->bin ) {
lio_sout( s, &c->flt, v, ll );
lio_sout( s, &c->flt, &tab, 1 );
} else {
lio_sout( s, &c->flt, v, ll-1 );
lio_sout( s, &c->flt, &vt, 1 );
}
v += ll;
vl -= ll;
}
if ( vl )
lio_sout( s, &c->flt, v, vl );
lio_sout( s, &c->flt, &lf, 1 );
}
}
if ( s->b.fill )
s->b.done = s->b.fill = 0;
/*
{
lio_sout( s, &c->flt, 0, 0 );
lio_stdio( s, LIO_SFLUSH );
}
*/
if ( LIO_SCLOSE == op ) {
lio_sout( s, &c->flt, &lf, 1 );
lio_sout( s, &c->flt, 0, 0 ); /* finally flush */
}
return 0;
}
return lio_stream( s, op ); /* hmm -- SPURGE ? */
} /* svPlain */
/* the echo application */
int svEcho ( Con *c, int task )
{
static Tm tm = { LLL(50) };
Field *f;
int i;
if ( task ) { /* preparing: we don't care */
if ( LSV_APPGOT == task )
c->grp = 1; /* don't require mainthead */
return 0;
}
/* run it */
sMsg( c->ses, LOG_INFO, "echo %d: %d fields to %s",
svCur()->id, c->req->len, c->nam );
timeSleep( &tm );
for ( i=c->req->len, f = c->req->field; i--; f++ ) {
RADDF( c->ses, c->ses->res, f, !0 );
/*
plain protocol actually does not support plain output ...
sMsg( c->ses, 0, "%d = '%.*s'\n", f->tag, f->len, f->val );
*/
}
return 0;
} /* svEcho */
#if 0
int main ( int argc, const char **argv )
{
Srv echo;
memset( &echo, 0, sizeof(echo) );
echo.prt = lsv_plain;
echo.app = lsv_echo;
cOpen(0);
cLog( LOG_DEBUG, 0 );
return svRun( &echo, 1
#include "loi.h"
#include "lio.h"
#include "lses.h"
/** network service
RECV: loop in receiver thread
- loop selecting on all connections
- dequeue connections from recv queue, clear BUSY
- for every available connection:
- if request is new (no input so far),
prepare req with peer header, preset default flags
- read input (which may clear the LIO_IN flag on EOF)
- call proto to extract input
proto calls check (if any) to preprocess each piece of input
- if done, set BUSY, pass to dispatcher (via queue or in same thread)
DISP: the one and only session master (in separate thread, if multiple recv)
- no input after this point, the req is considered read-only
any data on connection is considered an error (i.e. no pipelining)
- if request contains a SID header, find or create session.
if session is busy, enqueue to session's wait queue
- enqueue to work queue
WORK: (in worker thread)
- once a worker thread dequeued a request, it calls process function
this may add fields to response and/or write to output stream
- before output buffer is flushed first time,
it is set aside and proto is called to output the response
- after process is done, the request is marked in flush phase:
the req is not accessed after this point,
the output is flushed and the response freed
- enqueue to recv queue
events in connection lifetime:
- accept:
if the socket hasn't been used so far, a new connection is created.
an existing connection is reused.
The connection object may still be in use by a worker.
Since the socket file id is reused, the former socket with the
same fid must have been closed, and the LIO_INOUT flags
must have been cleared, so lio_write on that old lio file id will fail.
The connection is marked as new and, if not in use by a worker,
entered into the receive queue.
- dequeue from receive queue:
if the connection is marked new,
it is prepared and added to the select set.
else, if it was closed, it is removed from the select set.
- read events:
if the connection is in use, that is, a previous request hasn't yet
entered the flush phase, there is an attempt to pipeline.
while this is considered an error, for robustness it's handled:
the connection is marked pipeline and the socket is cleared
from the select set, to be reenabled later.
Once the connection is idle, input is read until the protocol
decides on a complete request. Excess data (immediate pipelining)
is considered an error.
*/
/* common header tags -- actually negative tags are used */
enum {
/* nonstandard */
LSV_TEXT, /* +- 0: a small, inlined untyped or text/plain body */
LSV_RID, /* row id, a.k.a MFN */
LSV_SID, /* session id */
LSV_PEER, /* network peer ("remote host") */
LSV_TIME, /* GTF + millis */
LSV_ATTACH, /* wrapped-up attachment, TBD */
/* MIME basics */
LSV_CTYPE,
/* HTTP: CGI/servlet */
LSV_URI,
LSV_QUERY
};
enum {
LSV_NUMWRK = 8,
LSV_BUFSIZ = 0x04000, /* 16384 a.k.a. 16K */
LSV_BUFMSK = 0x03fff /* mask buf lengths */
};
typedef struct Que {
struct Con *head;
struct Con *tail;
int len;
} Que;
/* queue with pool of workers waiting */
typedef struct Pool {
Que que;
struct Wrk *wait; /* stack */
} Pool;
typedef struct Wrk {
pthread_t thr;
unsigned id;
struct Srv *srv;
struct Wrk *nxt;
Session ses;
pthread_cond_t todo; /* work condition waited for by workers */
/* stats */
struct Con *cur; /* current job */
int jobs;
int waits;
} Wrk;
typedef struct Srv {
/*
the protocol stream method
- with LIO_SPUSH, input is pushed to request (len 0 indicates EOF)
return 0=more, >0=done, <0=error
- on LIO_SFLUSH, prt must convert output buffer to flt
- on LIO_SCLOSE, prt must flush and close answer
*/
int (*prt) ( Ios*, int op );
/*
the application
- with task APPARG (called by prt for each field):
check (and typically modify) last field extracted by prt
return 0=more, >0=done, <0=error
- with task APPGOT (after request is complete):
check request, add -LSV_SID field, if needed
(e.g. to extract from an URL or cookie)
- with task APPRUN (0):
process request
*/
int (*app) ( struct Con*, int task );
Fdt *fdt; /* field def */
int flg; /* flags */
unsigned sto; /* config: session timeout (seconds) */
unsigned nwr; /* config: #workers (<= LSV_NUMWRK) */
Session ses; /* usually the default session */
Tm tim; /* tim of last turn */
char gtm[20]; /* generalized time (actually 18 digits) */
Wrk wrk[LSV_NUMWRK];
/*
single receiver model
*/
pthread_t rcv; /* the receiver thread */
pthread_mutex_t mut; /* single mutex held by recv/disp while not in select */
int lsn; /* listening socket */
int pip[2]; /* pipe for pending events */
Que recv; /* enq by worker, deq by receiver */
Pool main; /* enq by dispatcher, deq by workers */
Pool pool; /* enq by dispatcher, deq by workers */
/* stats */
int plen; /* length of pool */
int jobs; /* total #jobs */
int conn; /* total #conn */
int turn; /* total #turn */
float wlen; /* cumulated length of work queue (on each turn) */
float busy; /* cumulated # of active workers (on each turn) */
} Srv;
enum { /* server flags */
LSV_CONSES = 0x01, /* session bound to connection */
LSV_WRKSES = 0x02, /* session bound to worker */
LSV_SHUTDN = 0x100 /* server is shutting down */
};
enum {
/* called in worker thread: */
LSV_APPRUN, /* app called to process request */
LSV_APPINI, /* app called to initialise non-main thread (no con) */
LSV_APPFIN, /* app called to finalize non-main thread (no con) */
/* called in receiver thread: */
LSV_APPARG, /* app called to check parameter */
LSV_APPGOT, /* app called after all input */
LSV_APPSES /* app called on new session */
};
enum { /* connection pending state */
LSV_PDGNON, /* new connection pending */
LSV_PDGNEW, /* new connection pending */
LSV_PDGRED, /* ready to read pending */
LSV_PDGEOF /* EOF pending */
/* when adding, keep array of names in sync */
};
enum { /* connection processing stage */
LSV_STGNEW, /* connection needs to be setup */
LSV_STGCON, /* connected, no request on the way */
LSV_STGINP, /* reading input */
LSV_STGSES, /* waiting for session */
LSV_STGWRK, /* queued for work */
/* set by worker: */
LSV_STGRUN, /* dequeued */
LSV_STGCOM, /* commited (output has started) */
LSV_STGDON, /* done processing, flushing output */
LSV_STGRET /* returning (queued to recv) */
/* when adding, keep array of names in sync */
};
typedef struct Con {
Ios ios; /* raw output */
Buf flt; /* protocol filtered output */
const Srv *srv;
struct Con *nxt;
unsigned char stg; /* processing stage (guarded by mutex) */
unsigned char pdg; /* pending flags set by receiver only */
unsigned char grp; /* thread group (0=main) set by receiver app calls */
unsigned char bin; /* client requested binary mode */
int prt; /* for free use by prt, 0 init */
int app; /* for free use by app, 0 init */
int host;
char nam[64]; /* 123.123.123.123:123456 for IPv4 */
Session ses;
Rec *req; /* request -- owned by srv->ses */
Tm tim; /* time when the request was scheduled */
/* Rec *res; response -- owned by ses */
int con; /* connection counter */
} Con;
extern Wrk *svCur ();
extern int svRun ( Srv *srv, const char *addr );
/* the plain protocol */
extern int svPlain ( Ios *s, int op );
extern int svEcho ( Con *c, int task );
#define LSV_H
#endif /* LSV_H */
luti.c 0100644 0001130 0001130 00000030426 07642121646 010671 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: luti.c,v 1.41 2003/03/31 19:44:00 erikisis Exp $
utilities
*/
#include /* strlen */
#include "openisis.h"
#include "luti.h"
/* ************************************************************
private types
*/
typedef struct {
char *name;
int id;
} LutiLTEnt;
typedef struct OpenIsisLT {
LutiLTEnt *arr;
int siz; /* malloc'ed */
int num; /* used */
} LutiLTNod;
#define LT_NUM 27
#define LT_SIZE (LT_NUM * sizeof (LutiLTNod))
#define LTIDX(name) ( \
('a' <= *(name) && 'z' >= *(name)) ? *(name) - 'a' : ( \
('A' <= *(name) && 'Z' >= *(name)) ? *(name) - 'A' : 26 \
))
#define LT_INCR 2
/* ************************************************************
private data
*/
/* ************************************************************
private functions
*/
/* ************************************************************
package data
*/
const char luti_hex[16] = {
'0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f'
};
/* ************************************************************
package functions
*/
void log_rec (
int level, Rec *rec, const char *msg, const char *delim
) {
Field *F;
int ch = 2;
if (0 > level) {
level = -level;
ch = 1;
}
if ((unsigned)level > log_lev) {
return;
}
if (msg && *msg) {
sMsg (ses0, ch, msg);
}
sMsg (ses0, ch, "%p", rec);
if (rec) {
if (! delim) {
delim = "; ";
}
sMsg (ses0, ch, " #f=%d", (int)rec->len);
for (F = rec->field; rec->len > F - rec->field; ++F) {
sMsg (ses0, ch, delim);
sMsg (ses0, ch, "%d", (int)F->tag);
sMsg (ses0, ch, " %.*s", (int)F->len, F->val);
}
}
sMsg (ses0, ch, "\n");
}
int lprint ( void *buf, int i )
{
char *c = buf;
if ( 0 > i ) {
*c++ = '-';
i = -i;
}
if ( ! i )
*c++ = '0';
else {
int dig = 0, j=i;
char *p;
for ( ; j; j /= 10 )
dig++;
p = c += dig;
for ( ; i; i /= 10 )
*--p = '0' + i % 10;
}
*c = 0;
return c - (char*)buf;
} /* lprint */
int a2il ( const char *p, int l, int *res )
{
const char *s = p;
const char *e = p + (0<=l ? l : (int)strlen(p));
int ret = 0, min = 0;
while ( p < e && ' ' >= (unsigned char)*p )
p++;
if ( p < e && '0' > *p ) {
if ( '+' == *p )
p++;
else if ( '-' == *p ) {
min = 1;
p++;
}
}
if ( p < e && '0' == *p && ++p < e && 'x' == *p ) {
while ( ++p < e ) {
int v = *p;
if ( '0' <= v && v <= '9' )
v -= '0';
else if ( 'a' <= v && v <= 'f' )
v -= 'a'-10;
else if ( 'A' <= v && v <= 'F' )
v -= 'A'-10;
else
break;
ret = (ret<<4) + v;
}
} else {
while ( p < e && '0' <= *p && *p <= '9' )
ret = 10*ret + *p++ - '0';
}
if (0 != res) {
*res = min ? -ret : ret;
}
return p - s;
} /* a2il */
int a2i ( const char *p, int l ) {
int res;
a2il (p, l, &res);
return res;
}
int a2id ( const char *p, int l, int dflt ) {
int res;
if (0 > l) {
l = (int) strlen (p);
}
if (l != a2il (p, l, &res)) {
return dflt;
}
return res;
}
int i2a ( char *p, int i )
{
if ( i ) {
int min, dig = 0, j;
if ( 0 < i )
min = 0;
else {
min = 1;
*p++ = '-';
i = -i;
}
for ( j=i; j; j/=10 )
dig++;
for ( *(p+=dig)=0; i; i/=10 )
*--p = '0' + (i % 10);
return min+dig;
}
*p++ = '0';
*p = 0;
return 1;
} /* i2a */
int u2a ( char *p, unsigned u )
{
if ( u ) {
int dig = 0, j;
for ( j=u; j; j/=10 )
dig++;
for ( *(p+=dig)=0; u; u/=10 )
*--p = '0' + (u % 10);
return dig;
}
*p++ = '0';
*p = 0;
return 1;
} /* u2a */
LutiLT luti_ltnew () {
LutiLT res = (LutiLT) mAlloc (ses0, LT_SIZE);
return res;
}
void luti_ltdel (LutiLT lt) {
LutiLTNod *N;
LutiLTEnt *E;
if (lt) {
for (N = lt + LT_NUM; --N >= lt; ) {
if (N->arr) {
for (E = N->arr + N->num; --E >= N->arr; ) {
if (E->name) {
mFree (ses0, E->name);
}
}
mFree (ses0, N->arr);
}
}
mFree (ses0, lt);
}
}
void luti_ltadd (LutiLT lt, const char *name, int id) {
if (0 != lt) {
LutiLTNod *N;
LutiLTEnt *E;
int idx;
if (!name) {
name = "";
}
idx = LTIDX (name);
N = lt + idx;
if (N->siz == N->num) {
E = mAlloc (ses0,
(LT_INCR + N->siz) * sizeof (LutiLTEnt));
if (N->siz) {
memcpy (E, N->arr, N->siz * sizeof (LutiLTEnt));
mFree (ses0, N->arr);
}
N->arr = E;
N->siz += LT_INCR;
}
N->arr[N->num].name = mDup (ses0, name, -1);
N->arr[N->num].id = id;
++N->num;
}
}
int luti_ltget (LutiLT lt, const char *name) {
if (0 != lt) {
LutiLTNod *N;
LutiLTEnt *E;
int idx;
if (!name) {
name = "";
}
idx = LTIDX (name);
N = lt + idx;
if (N->num) {
for (E = N->arr + N->num; --E >= N->arr; ) {
if (! strcmp (E->name, name)) {
return E->id;
}
}
}
}
return -1;
}
void luti_ltrmv (LutiLT lt, const char *name) {
if (0 != lt) {
LutiLTNod *N;
LutiLTEnt *E;
int idx;
if (!name) {
name = "";
}
idx = LTIDX (name);
N = lt + idx;
if (N->num) {
for (E = N->arr + N->num; --E >= N->arr; ) {
if (! strcmp (E->name, name)) {
int len = N->num - (E - N->arr) - 1;
mFree (ses0, E->name);
if (len) {
memmove (E, E + 1, len * sizeof (LutiLTEnt));
}
--N->num;
return;
}
}
}
log_msg (LOG_WARN, "luti_ltrmv: no such entry <%s>", name);
}
}
static Rec *_unwrap (
Session ses, Rec *env, int *pos, int tag, int dbid, int rdonly
) {
Field *F;
Rec *rec = 0;
int rlen, np;
if (! env || ! ses) {
return 0;
}
if (! pos) {
np = 0;
pos = &np;
}
F = rGet (env, tag, pos);
if (!F) {
return 0;
}
rlen = a2id (F->val, (int)F->len, -1);
if (0 >= rlen) {
sMsg (ses, LOG_ERROR, "luti_unwrap: illegal rec len %d", rlen);
return 0;
}
++F;
if (env->len < *pos + rlen) {
sMsg (ses, LOG_ERROR,
"luti_unwrap: illegal rec len %d, pos %d, env %d",
rlen, *pos, env->len);
return 0;
}
while (0 <= --rlen) {
if (! rdonly) {
RADD (ses, rec, F->tag, F->val, F->len, !0);
if (0 == rec) {
return 0;
}
}
++F;
++(*pos);
}
if (rdonly) {
return (Rec*)1;
}
rec->dbid = dbid;
return rec;
}
Rec *luti_unwrap (Session ses, Rec *env, int *pos, int tag, int dbid) {
return _unwrap (ses, env, pos, tag, dbid, 0);
}
Rec *luti_wrap (Session ses, Rec *env, Rec *rec, int tag) {
Field *F, *E;
if (!rec) {
return env;
}
E = rec->field + rec->len;
env = rAddI (ses, env, tag, rec->len, !0);
for (F = rec->field; env && E > F; ++F) {
RADD (ses, env, F->tag, F->val, F->len, !0);
}
return env;
}
int luti_ptrincr (
Session ses, void *start, int *num, int incr, int siz, int maxn
) {
char **base = (char**)start;
char *arr;
int oldn = *num;
if (0 < maxn && oldn >= maxn) {
return -1;
}
arr = mAlloc (ses, (oldn + incr) * siz);
if (!arr) {
return -1;
}
if (oldn) {
memcpy (arr, *base, oldn * siz);
mFree (ses, *base);
}
memset (arr + oldn * siz, 0, incr * siz);
*num += incr;
*base = arr;
return oldn;
}
void luti_free (Session ses, void **arr, int num) {
if (ses && arr) {
while (0 <= --num) {
if (arr[num]) {
mFree (ses, arr[num]);
}
}
mFree (ses, arr);
}
}
int luti_true (const char *str, int len) {
if (! str || ! *str) {
return -1;
}
if (0 > len) {
len = strlen (str);
}
if (1 == len) {
if ('0' == *str ||
'f' == *str ||
'F' == *str ||
'n' == *str ||
'N' == *str
) {
return 0;
}
if ('1' == *str ||
'y' == *str ||
'Y' == *str ||
't' == *str ||
'T' == *str
) {
return 1;
}
return -1;
}
if (! strncmp ("false", str, len) ||
! strncmp ("no", str, len)) {
return 0;
}
if (! strncmp ("true", str, len) ||
! strncmp ("yes", str, len)) {
return 1;
}
return -1;
}
/* ------------------------------------------------------------------------
* unwrap by path
*/
Fdt* luti_fdt_from_rec (Rec *rec) {
Db *dbh;
if (rec &&
0 <= rec->dbid &&
(dbh = nDbById (rec->dbid))
) {
return dbh->fdt;
}
return 0;
}
const char* luti_parse_path (
const char *path, const Fdt *fdt, Fd **fd, int *tag, int *occ
) {
Fd *_f = 0;
int _t = -1;
int _o = -1;
int num, qsub;
if (fd) {
*fd = 0;
}
if (tag) {
*tag = -1;
}
if (occ) {
*occ = -1;
}
if (! path) {
return 0;
}
if ('.' == *path
|| ('-' == *path && (path[1]<'0' || '9'= _t) {
return 0;
}
*/
if (fdt) {
_f = fdById (fdt, _t);
if (! _f) {
return 0;
}
}
}
else {
char name[OPENISIS_FD_NAMELEN];
if (! fdt) {
return 0;
}
for (num = 0; path[num] && '[' != path[num]; ++num)
;
if (num > OPENISIS_FD_NAMELEN - 1) {
return 0;
}
strncpy (name, path, num) [num] = 0;
_f = fdByName ( fdt, name );
if (! _f) {
return 0;
}
_t = _f->id;
}
path += num;
if (tag) {
*tag = _t;
}
if (fd) {
*fd = _f;
}
if ('[' != *path && '(' != *path) {
return path;
}
qsub = *path;
++path;
num = a2il (path, -1, &_o);
if (! num || 0 > _o) {
return 0;
}
path += num;
if ((']' != *path && '[' == qsub) || (')' != *path && '(' == qsub)) {
return 0;
}
if (occ) {
*occ = _o;
}
return path + 1;
}
Rec* luti_getembed (Session ses, Rec *env, const char *path, const Fdt *fdt) {
Rec *res;
int tag, occ, idx, pos;
if (! ses || ! env || ! path || ! *path) {
return 0;
}
if (! fdt) {
fdt = luti_fdt_from_rec (env);
}
path = luti_parse_path (path, fdt, 0, &tag, &occ);
if (! path) {
return 0;
}
res = 0;
idx = pos = 0;
while (1) {
res = _unwrap (ses, env, &pos, tag, -1, idx < occ);
if (! res) {
return 0;
}
if (idx >= occ) {
break;
}
++idx;
}
if (*path) {
Rec *r2 = luti_getembed (ses, res, path, 0);
mFree (ses, res);
return r2;
}
return res;
}
int lhash ( const char *str, int len )
{
int h = 0;
while ( len-- )
h = 31*h + *str++;
return 0<=h ? h : -h;
}
/* ************************************************************
public functions
*/
char *toHtml ( Session ses, const char *str, int len )
{
int l;
char *p, *e, *ret;
if ( ! str )
return 0;
if ( 0 > len )
len = strlen(str);
l = len + 1;
for ( e = (p=(char*)str) + len; p': l += 3; break;
case '&': l += 4; break;
case '"': l += 5; break;
/* do not replace ' w/ ' -- many clients don't grok it */
}
ret = mAlloc( ses, l );
if ( ! ret )
return 0;
for ( p=ret; str': *p='&'; p[1]='g'; p[2]='t'; p[3]=';'; p+=4; break;
case '&': *p='&'; p[1]='a'; p[2]='m'; p[3]='p'; p[4]=';'; p+=5; break;
case '"': *p='&';p[1]='q';p[2]='u';p[3]='o';p[4]='t';p[5]=';';p+=6;break;
default: *p++ = *str;
}
*p = 0;
return ret;
} /* toHtml */
int utf8Chk ( Session ses, void *mem, int len, int *tof )
{
unsigned char *c = (unsigned char *)mem;
int f = 0; /* expected followers */
int l = len;
if ( tof )
f = *tof;
for ( ; l--; c++ ) {
unsigned char u = *c;
int tofollow = u < 128 ? 0 : u < 192 ? -1 : u < 224 ? 1 : 2;
if ( f ) {
if ( 0 > tofollow ) { f--; continue; }
break;
}
if ( 0 <= tofollow ) { f = tofollow; continue; }
goto croak;
}
if ( tof )
*tof = f;
if ( ! f )
return 0;
if ( 0 > l && tof ) /* exhausted buf, want follower */
return 0;
sMsg( ses, LOG_ERROR, "expected follower at %d got %x", len-l-1, (int)*c );
return len-l;
croak:
sMsg( ses, LOG_ERROR, "expected no follower at %d got %x", len-l-1, (int)*c );
return len-l;
} /* utf8Chk */
luti.h 0100644 0001130 0001130 00000011730 07637671273 010705 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef LUTI_H
/*
$Id: luti.h,v 1.28 2003/03/24 21:03:55 kripke Exp $
common utilities for openisis lib implementation.
*/
#include
#include "loi.h"
extern LogLevel log_lev;
#define LOG_DO( lev ) ((lev) <= log_lev)
#define LOG_DONT( lev ) ((lev) > log_lev)
/**
NOTE: unlike sMsg, this directly and immediatly lio_writes to file 2.
So this is more expensive, but threadsafe.
print some log message to stderr or some configured logdevice.
for errors, use one of the LERR_xxx codes or
a loglevel of SYSERR or IOERR to determine the error from errno.
loglevels FATAL and ERROR, when used w/o error code,
assum IDIOT and INVAL, resp.
other logging levels do not assume an error (i.e. 0 == -LERR_OK).
any error or loglevel code implies stream 2.
@param code errorcode (-LERR_xxx) or loglevel as described above
@return a non-positive code as described above
*/
extern int log_msg ( int code, const char *fmt, ... );
#ifndef NDEBUG
# define LOG_DBG log_msg
#else
# ifdef __GNUC__
# define LOG_DBG( args... )
# else
# define LOG_DBG (void) /* compiler should dispose statement off */
# endif
#endif
#define LOG_OTO( to, args ) do { ret = log_msg args; goto to; } while (0)
/* content logging is usually on level LOG_TRACE */
#define LOG_STR( rec, desc ) log_str( LOG_TRACE, rec, desc )
#define LOG_HEX( mem, len ) log_hex( LOG_TRACE, mem, len )
/**
dump a record
@param level loglevel necessary for output to occurr.
*/
extern void log_str ( LogLevel level, int *rec, const char **desc );
/*
* print fields of record to stderr
* @param level if negative, print msg to stdout
* @param delim field delimiter (default '; ')
*/
extern void log_rec (int level,
Rec *rec, const char *msg, const char *delim);
/**
hexdump memory
@param level loglevel necessary for output to occurr.
*/
extern void log_hex ( LogLevel level, const void *mem, int len );
/**
do a malloc and print error if out of memory.
*/
extern void *lmalloc ( int size );
/* more specific log functions in log.h */
/* printint: optional sign, up to 10 digits, 0 byte.
buf must be at least 12 bytes long.
return length w/o the 0 byte.
*/
extern int lprint ( void *buf, int i );
/* Lookup of id from name */
typedef struct OpenIsisLT *LutiLT;
/* create lookup table */
extern LutiLT luti_ltnew ();
/* destroy lookup table */
extern void luti_ltdel (LutiLT lt);
/* add entry */
extern void luti_ltadd (LutiLT lt, const char *name, int id);
/* get id for name */
extern int luti_ltget (LutiLT lt, const char *name);
/* remove entry */
extern void luti_ltrmv (LutiLT lt, const char *name);
/* get fdt from rec */
extern Fdt* luti_fdt_from_rec (Rec *rec);
/* get fd, tag, occurence from path */
extern const char* luti_parse_path (
const char *path, const Fdt *fdt, Fd **fd, int *tag, int *occ);
/* extract embedded record from envelope */
extern Rec* luti_unwrap (Session ses, Rec *env, int *pos, int tag, int dbid);
/* extract embedded record from envelope */
extern Rec* luti_getembed (
Session ses, Rec *env, const char *path, const Fdt *fdt);
/* append record to envelope */
extern Rec* luti_wrap (Session ses, Rec *env, Rec *rec, int tag);
/** incr base ptr by incr*siz, if 0 < maxnum && *num < maxnum
new allocated range is memsetted 0
NOTE: base is address of base for reallocation (whatever_type**)
@return index of first new allocated address in array or -1 if out of mem
*/
extern int luti_ptrincr (Session ses,
void *base, int *num, int incr, int siz, int maxnum);
/** free pointer array
*/
extern void luti_free (Session ses, void **arr, int num);
/** check whether string represents a boolean value
@return 0 str is false,
1 str is true,
-1 str is something else
*/
extern int luti_true (const char *str, int len);
/**
s[0]*31^(n-1) + s[1]*31^(n-2) + ... + s[n-1]
forced to non-negative
*/
extern int lhash ( const char *str, int len );
extern const char luti_hex[16]; /* '0'..'9','a'..'f' */
#define LUTI_H
#endif /* LUTI_H */
make.mingw 0100664 0001130 0001130 00000000222 07571422633 011522 0 ustar krip krip PATH:=/opt/cross-tools/i386-mingw32msvc/bin/:/opt/cross-tools/bin/:$(PATH)
JNIINC=-I/opt/cross-tools/include/jni/win32 -I/usr/java/jdk1.3/include
mlock.c 0100644 0001130 0001130 00000002422 07473531744 011022 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#include
int main ( int argc, char **argv ) {
size_t s = (size_t)1024*1024*atoi(argv[1]);
char *p = (char *)malloc( s );
if ( ! mlock( p, s ) ) sleep( 10000 );
return 0;
}
oip.h 0100644 0001130 0001130 00000002665 07635354356 010524 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef OIP_H
/*
$Id: oip.h,v 1.1 2003/03/17 14:17:18 kripke Exp $
OpenIsis page
*/
/* standalone stdio version */
#include
#define OIPRUN int main ( int argc, const char **argv ) {
#define OIPEND return 0; }
#define OIPP fprintf
#define OIPF stdout
#define OIPS( s ) fputs( s, stdout )
#define OIP_H
#endif /* OIP_H */
oipc 0100755 0001130 0001130 00000007034 07635354302 010426 0 ustar krip krip #!/usr/bin/perl -w
#/*
# openisis - an open implementation of the ISIS database
# Version 0.8.x (microversion see file Version)
# Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# This library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
#
# This software is dedicated to the memory of Eckart Dietrich.
#
# This software is inspired by (but contains no code of) the iAPI
# Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
# See iAPI.txt for what it contains.
#EOH */
# $Id: oipc,v 1.1 2003/03/17 14:16:34 kripke Exp $
#
# usage
# ./oipc foo.oip gcc %o && ./a.out
# ./oipc foo.oip gcc %o -shared -o foo.oip.so && ./oipl $PWD/foo.oip.so
sub mk {
my $ifile = shift;
my $ofile = $ifile . '.c';
my $buf = ''; # collected strings from output mode
my @code = ('',''); # main and global code
my $code = 0, $state = -1;
# -1: file header
# 0: stuff to output literally
# run code:
# 1: in <%
# 2: in <%=
# global code:
# 3: in <%!
# 4: in <%@
die "could not read $ifile" unless open I, $ifile;
# transform line by line
LIN: while () {
if ( -1 == $state ) { # skipping initial # lines
next if /^#/;
$state = 0; # switch to output state
}
while (1) {
if ( ! $state ) { # output up to <%
my ($o,$s,$m,$x) = /^(.*?)(<%([=!@])?(.*))?$/s;
if ( length $o ) {
$o =~ s/\\/\\\\/g;
$o =~ s/"/\\"/g;
if ( defined $s ) {
$buf .= '"' . $o . '"';
} else {
$buf .= '"' . $o . '\n"' . "\n";
}
}
next LIN unless defined $s;
$code[$code] .= "\tOIPS(\n".$buf."\t);\n" if length $buf;
$buf = '';
($m ||= 0) =~ y/=!@/123/;
$state = 1+$m;
$code = 2 < $state ? 1 : 0;
$_ = $x;
$code[$code] .= "\n#line ".$.."\n";
next LIN unless length $x;
}
my ($x,$s,$m,$o) = /^(.*?)((!)?%>(.*))?$/s;
$buf .= $x;
if ( defined $s ) {
if ( 2 == $state ) { # <%=
if ( 1 == length $buf ) {
if ( $buf =~ /[i-n]/ ) { # integer shorthand
$buf = '"%d",'.$buf;
} elsif ( $buf =~ /[u-w]/ ) { # unsigned shorthand
$buf = '"%u",'.$buf;
} elsif ( $buf =~ /[f-h]/ ) { # OpenIsis Field shorthand
$buf = '"%.*s",'.$buf.'->len,'.$buf.'->val';
}
}
if ( $buf =~ /^\s*"/ ) {
$buf = 'OIPP(OIPF,'.$buf.');';
} else {
$buf = 'OIPS('.$buf.');';
}
}
$code[$code] .= $buf;
# don't reset print code destination if global ends with !%>
$code = 0 unless $code && defined $m;
$state = 0;
if ( length $o ) {
$buf = '';
$_ = $o;
next;
}
$buf = '"\n"';
} else {
$buf .= "\n";
}
last;
}
}
close I;
$code[0] .= "OIPS(\n".$buf.");\n" if length $buf;
die "could not write $ofile" unless open O, '>', $ofile;
print O '#include "oip.h"',"\n";
print O $code[1];
print O "OIPRUN\n";
print O $code[0];
print O "OIPEND\n";
close O;
map s/%o/$ofile/, @_;
die 'exec '.join(' ',@_)." failed\n" unless exec @_;
}
mk @ARGV;
oipl.c 0100644 0001130 0001130 00000003360 07635354330 010654 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: oipl.c,v 1.1 2003/03/17 14:16:56 kripke Exp $
OpenIsis page loader
*/
#include /* dlopen et al. */
#include
int main ( int argc, const char **argv )
{
int (*m)( int, const char ** );
const char *err = 0;
const char *dlfile = argv[1];
void *dl = dlopen( dlfile, RTLD_LAZY );
int ret;
if ( ! dl ) {
fprintf( stderr, "could not open dlfile '%s': %s\n",
dlfile, (err = dlerror()) ? err : "?" );
return 1;
}
m = dlsym( dl, "main" );
if ( ! m ) {
fprintf( stderr, "could not link main in '%s': %s\n",
dlfile, (err = dlerror()) ? err : "?" );
return 2;
}
ret = m( --argc, ++argv );
dlclose( dl );
return ret;
}
ois.c 0100644 0001130 0001130 00000003677 07640414162 010512 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: ois.c,v 1.7 2003/03/24 14:59:30 kripke Exp $
OpenIsis page server
*/
#include /* mem* */
#include "lsv.h"
#if 0
/* the echo application */
static int echo ( Con *c, int task )
{
Field *f;
int i;
if ( task ) /* preparing: we don't care */
return LSV_APPGOT==task;
/* run it */
sMsg( c->ses, LOG_INFO, "echo %d: %d fields to %s",
svCur()->id, c->req->len, c->nam );
for ( i=c->req->len, f = c->req->field; i--; f++ ) {
RADDF( c->ses, c->ses->res, f, !0 );
/*
plain protocol actually does not support plain output ...
sMsg( c->ses, 0, "%d = '%.*s'\n", f->tag, f->len, f->val );
*/
}
return 0;
} /* echo */
#endif
int main ( int argc, const char **argv )
{
Srv echo;
memset( &echo, 0, sizeof(echo) );
echo.prt = svPlain;
echo.app = svEcho;
cOpen(0);
cLog( LOG_DEBUG, 0 );
return svRun( &echo, 1/dev/tcp/$host/$port
local i # running in subshell anyway ...
declare -i i
i=$requests
ses=ses$[id / 4]
while [ 0 != $i ]; do
if [ 0 -eq $[i % 7] ]; then # add to session
echo "-2 $ses
$i $id $i foo
$i $id $i bar
" >&3
else # fetch session info only
echo "-2 $ses
" >&3
fi
while read line; do
# echo $line
[ -z "$line" ] && break;
done <&3
i=i-1
done
echo "$id: `date`"
}
if [ x-trash == "x$1" ]; then
requests=${2:-500}
clients=${3:-8}
port=${4:-8080}
host=${5:-localhost}
declare -i j
echo "starting $clients trashers for $requests requests at `date`"
j=$clients; while [ 0 != $j ]; do time trasher $j & j=j-1; done
echo "started $clients trashers on $host:$port at `date`"
j=$clients; while [ 0 != $j ]; do wait; j=j-1; done
echo "collected $clients trashers at `date`"
exit
fi
# default: run manually
#
exec 3<>/dev/tcp/${2:-localhost}/${1:-8080}
cat <&3 &
cp=$!
cat >&3
kill $cp # copy kill ;/
openisis.c 0100644 0001130 0001130 00000053744 07640345745 011563 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: openisis.c,v 1.59 2003/03/26 15:48:53 kripke Exp $
main file of openisis executable.
*/
#ifdef HAVE_PTHREAD
#define HAVE_THREADS
#endif
#include
#include /* free */
#include /* strcmp */
#ifdef WIN32
# include
# include
# define timeval _timeb
#else
# include /* gettimeofday */
#endif
#include /* gettimeofday */
#ifdef HAVE_PTHREAD
#include /* threaded crashtest */
#endif
#include "openisis.h"
/*
temporary library includes for testing of functions
that should later be accessible via openisis.h
*/
#include "lio.h"
/* ************************************************************
private types
*/
typedef enum { /* what to do */
DO_DUMP,
DO_SCAN, /* simple full scan */
DO_SEARCH, /* basic index based searching */
DO_TERMS, /* list terms */
DO_CHECK, /* check db */
DO_PERF, /* do random reads for performance checking */
DO_CRASH, /* do multi-threaded crashtest */
DO_SPLIT, /* split a field value */
DO_STREAM, /* stream in records */
DO_MFNLIST, /* fetch records by mfn list */
DO_IFLOAD, /* read a lk2-style file from stdin */
DO_IFDUMP, /* dump a lk2-style file to stdout */
DO_VUTF /* validate UTF-8 input */
} todo;
typedef enum {
FMT_MFN, /* rowid only */
FMT_MFNF, /* rowid, 1st field */
FMT_PROP, /* property style */
FMT_ZML /* ZML style */
} format;
typedef enum {
IFM_DUMP,
IFM_TAB,
IFM_OLD, /* dump old index */
IFM_COPY, /* copy old index */
IFM_CHK /* check (new oxi) index */
} ifmode;
/* ************************************************************
private data
*/
static const char *pft;
static const char **term, **val;
static OpenIsisSet *post;
int nterm, nval;
static int db = -1, wdb = -1, append = 0, idxall = 0;
/* ************************************************************
private functions
*/
static int argchk ( OpenIsisSession ses, const char *param, const char *n, const char *v )
{
return strcmp( param, n ) ? 0 : v ? 1 :
(openIsisSMsg( ses, OPENISIS_ERR_INVAL, "no value for param '%s'", param ), 0);
} /* argchk */
static int print ( OpenIsisSession ses, OpenIsisRec *r, int freeit, format f )
{
union { OpenIsisRec r; char buf[10000]; } x;
int i, ret;
if ( ! r )
return -1;
ret = r->rowid;
if ( pft ) {
OpenIsisRec *q;
OPENISIS_INITBUF(x);
q = openIsisFmt( &x.r, pft, r );
if ( freeit )
free( r );
freeit = q != &x.r;
r = q;
}
if ( FMT_MFN == f )
openIsisSMsg( ses, 1, "%d\n", r->rowid );
else if ( FMT_MFNF == f )
openIsisSMsg( ses, 1, "%d %.*s\n", r->rowid,
0 == r->len ? 1 : (int)r->field[0].len,
0 == r->len ? "-" : r->field[0].val );
else if ( FMT_ZML == f ) {
openIsisSMsg( ses, 1, "0\t%d\n", r->rowid );
for ( i=0; ilen; i++ ) {
if ( r->field[i].val )
openIsisSMsg( ses, 1, "%d\t%.*s\n", r->field[i].tag,
(int)r->field[i].len, r->field[i].val );
else
openIsisSMsg( ses, 1, "%d\t%d\n", r->field[i].tag, r->field[i].len );
}
openIsisSMsg( ses, 1, "\014\n" ); /* ^L = FormFeed */
} else for ( i=0; ilen; i++ ) {
if ( ! r->field[i].val ) { /* shouldn't happen -- numeric ? */
openIsisSMsg( ses, 1, "%d.?=%d\n", r->rowid, r->field[i].len );
continue;
}
openIsisSMsg( ses, 1, "%d.%d=%.*s\n", r->rowid, r->field[i].tag,
(int)r->field[i].len, r->field[i].val );
if ( r->field[i].len && '^' == *r->field[i].val ) { /* split subfields */
OpenIsisRec *rf = openIsisReadField( 0, r->field+i );
if ( rf ) {
int j;
for ( j=0; jlen; j++ )
openIsisSMsg( ses, 1, "%d.%d.%c=%.*s\n",
r->rowid, r->field[i].tag,
(0x60 & (int)rf->field[j].tag ) ?
(int)rf->field[j].tag : ' ',
(int)rf->field[j].len, rf->field[j].val );
free( rf );
}
}
}
if ( 0 <= wdb ) {
OpenIsisRec *q = 0;
int ok;
if ( append )
r->rowid = 0;
if ( idxall && r != &x.r ) { /* add index entries for all fields */
OPENISIS_INITBUF(x);
q = &x.r;
for ( i=0; ilen; i++ ) {
OpenIsisField *fld = r->field + i;
OpenIsisHit h;
h.dbn = 0;
h.mfn = r->rowid;
h.tag = fld->tag;
h.occ = i;
h.pos = 1;
OPENISIS_RADD( ses, q, OPENISIS_XHIT, &h, sizeof(h), q != &x.r );
OPENISIS_RADD( ses, q, OPENISIS_XKEY, fld->val, fld->len, q != &x.r );
}
}
ok = openIsisDWritex( ses, wdb, r, q );
openIsisSMsg( ses, 1, "wrote mfn %d (%d)\n", r->rowid, ok );
}
if ( freeit )
free( r );
return ret;
} /* print */
static void printid ( OpenIsisSession ses, int id, format f )
{
if ( FMT_MFN == f && 0 > wdb )
openIsisSMsg( ses, 1, "%d\n", id );
else
print( ses, openIsisReadRow( db, id ), !0, f );
} /* printid */
static int printlk2 ( OpenIsisSession ses, OpenIsisKey *key, OpenIsisHit *hit )
{
if ( key && hit )
/* 30 key BLANK 7 mfn BLANK 5 tag BLANK 4 occ BLANK 4 pos*/
openIsisSMsg( ses, 1, "%-30.*s %7u %5u %4u %4u\n",
key->len, key->byt, hit->mfn, hit->tag, hit->occ, hit->pos );
return 0;
} /* printlk2 */
static int printtab ( OpenIsisSession ses, OpenIsisKey *key, OpenIsisHit *hit )
{
if ( key && hit )
/* 30 key BLANK 7 mfn BLANK 5 tag BLANK 4 occ BLANK 4 pos*/
openIsisSMsg( ses, 1, "%.*s\t%u\t%u\t%u\t%u\n",
key->len, key->byt, hit->mfn, hit->tag, hit->occ, hit->pos );
return 0;
} /* printtab */
/* timing utility. set the timeval, return milliseconds since last call. */
#ifdef WIN32
static int millis ( struct _timeb *tb )
{
struct _timeb otb = *tb;
_ftime( tb );
return (tb->time - otb.time)*1000 + (tb->millitm - otb.millitm);
} /* millis */
#else
static int millis ( struct timeval *tv )
{
struct timeval otv = *tv;
gettimeofday( tv, 0 );
return (tv->tv_sec - otv.tv_sec)*1000 + (tv->tv_usec - otv.tv_usec)/1000;
} /* millis */
#endif
#ifdef HAVE_PTHREAD
int myOpenIsisLockFunc ( int lock )
{
static pthread_mutex_t mut = PTHREAD_MUTEX_INITIALIZER; /* the "fast" kind */
static pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
/*
LOG_DBG( LOG_ERROR, "thread %d op 0x%08x", (int)pthread_self(), lock );
*/
switch ( OPENISIS_WAIT & lock ) {
case OPENISIS_RELE: return pthread_mutex_unlock( &mut );
case OPENISIS_LOCK: return pthread_mutex_lock( &mut );
case OPENISIS_WAKE: return pthread_cond_broadcast( &cond );
case OPENISIS_WAIT: return pthread_cond_wait( &cond, &mut );
}
return -1;
}
#else
# define myOpenIsisLockFunc 0
#endif
typedef struct {
OpenIsisSession ses;
int start;
} threadarg;
/* (p)thread routine */
static void * run ( void *arg )
{
struct timeval tv;
int m;
threadarg *my = (threadarg *)arg;
int start = my->start;
int i;
openIsisSMsg( my->ses, 1, "thread on ses %d\n", *(int*)my->ses );
millis( &tv );
for ( i=start+1; i++!=start; ) {
int j;
if ( i >= nterm )
i = 0;
for ( j=0; jses, 2, "no rec %d\n", mfn );
continue;
}
if ( 0 >= r->len )
openIsisSMsg( my->ses, 2, "no fields for %d\n", mfn );
else if ( r->field[0].len != (int)strlen(val[mfn])
|| memcmp( r->field[0].val, val[mfn], r->field[0].len )
)
openIsisSMsg( my->ses, 2, "mismatch on %d\n", mfn );
free( r );
}
}
m = millis(&tv);
openIsisSMsg( my->ses, 1, "thread %d@%d terminated after %.3f seconds\n",
#ifdef HAVE_THREADS
(int)pthread_self()
#else
0L
#endif
, start, m/1000. );
return (void *)m;
} /* run */
/* multithreaded crashtest */
static int crash ( OpenIsisSession ses, const char *pre )
{
struct timeval tv;
union { OpenIsisRec r; char buf[10000]; } x;
int l;
int p = 0;
int i, pass;
int fd = 2;
threadarg def;
millis( &tv );
x.r.len = 0;
x.r.bytes = sizeof(x);
nterm = 0;
while ( openIsisTerm( &x.r, db, pre ) && x.r.len )
nterm += x.r.len;
openIsisSMsg( ses, fd, "%d terms\n", nterm );
openIsisSMsg( ses, fd, "%.3f sec\n", millis(&tv)/1000. );
term = (const char **)malloc( nterm * sizeof(*term) );
post = (OpenIsisSet *)malloc( nterm * sizeof(*post) );
nterm = 0;
nval = 0;
while ( openIsisTerm( &x.r, db, pre ) && x.r.len ) {
for ( i=0; ilen = 0;
openIsisQuery( set, db, c, OPENISIS_QRY_KEYEQ, 0 );
if ( 0 >= set->len ) {
openIsisSMsg( ses, 2, "no results for '%s'\n", c );
return 1;
}
for ( cnt = 0; cnt < set->len; cnt++ )
if ( nval < set->id[cnt] )
nval = set->id[cnt];
p += set->len;
}
}
openIsisSMsg( ses, fd, "%d postings max mfn %d\n", p, nval );
openIsisSMsg( ses, fd, "%.3f sec\n", millis(&tv)/1000. );
val = (const char **)malloc( (1+nval) * sizeof(*val) );
for ( i=1; i<=nval; i++ ) {
OpenIsisRec *r = openIsisReadRow( db, i );
val[i] = 0;
if ( ! r ) {
openIsisSMsg( ses, 2, "no rec %d\n", i );
continue;
}
if ( 0 < r->len ) {
char *c = malloc( r->field[0].len + 1 );
memcpy( c, r->field[0].val, r->field[0].len );
c[ r->field[0].len ] = 0;
val[i] = c;
}
free( r );
}
l = millis(&tv);
openIsisSMsg( ses, fd, "sequential read %d rows in %.3f seconds %d rows per sec\n",
nval, l/1000., nval*1000/(l?l:1) );
def.ses = ses;
def.start = 0;
i = 0;
run( &def );
l = millis(&tv);
run( &def );
l = millis(&tv);
run( &def );
l = millis(&tv);
openIsisSMsg( ses, fd, "in-thread run in %.3f seconds %d rows per sec\n",
l/1000., p*1000/(l?l:1) );
#ifdef HAVE_THREADS
#define passes 3
{
static int nThreads[] = { 8, 2, 1, 4 };
int res[ sizeof(nThreads)/sizeof(nThreads[0]) ];
pthread_t th[ 8 /* max. nThreads */ ];
threadarg arg[ sizeof(th)/sizeof(th[0]) ];
int j;
for ( i=0; i<(int)(sizeof(nThreads)/sizeof(nThreads[0])); i++ )
res[i] = 0;
for ( j=0; j<(int)(sizeof(th)/sizeof(th[0])); j++ )
if ( ! (arg[j].ses = openIsisSesGet( -1, 0 )) ) {
openIsisSMsg( ses, fd, "could not get %dth session\n", j );
return 1;
}
for ( pass=0; passvalue\n"
);
ret = 1;
goto bye;
}
if ( dowrite && 0 > (wdb = openIsisOpen( dowrite, 0, 0 )) ) {
openIsisSMsg( ses, 2, "could not open write target db '%s'\n", dowrite );
ret = 2;
goto bye;
}
switch ( what ) {
case DO_DUMP: {
int max = openIsisMaxRowid( db );
int rowid;
for ( rowid = 1; rowid <= max; rowid++ )
printid( ses, rowid, fmt );
} break; /* DO_DUMP */
case DO_MFNLIST: {
char *buf = 0;
int l;
while ( 0 <= (l = openIsisSReadln( ses, &buf )) ) {
int id = 0;
while ( l-- )
id = 10*id + *buf++ - '0';
if ( id )
printid( ses, id, fmt );
}
} break; /* DO_MFNLIST */
case DO_SCAN: {
int max = openIsisMaxRowid( db );
int rowid;
for ( rowid = 1; 0 < rowid && rowid <= max; rowid++ )
rowid = print( ses, openIsisScan( db, rowid, 0, search ), !0, fmt );
} break; /* DO_SCAN */
case DO_SEARCH: {
int cnt;
OpenIsisSet set;
set.len = 0;
openIsisQuery( &set, db, search, searchmode, 0 );
if ( 0 >= set.len ) {
openIsisSMsg( ses, 2, "no results for '%s'\n", search );
ret = 1;
goto bye;
}
/* openIsisSMsg( ses, 2, "%d\trows for\t%s\n", set.len, search ); */
for ( cnt = 0; cnt < set.len; cnt++ )
printid( ses, set.id[cnt], fmt );
} break; /* DO_SEARCH */
case DO_TERMS: {
union { OpenIsisRec r; char buf[10000]; } x;
x.r.len = 0;
x.r.bytes = sizeof(x);
while ( openIsisTerm( &x.r, db, search ) && x.r.len ) {
/* openIsisSMsg( ses, 1, "%d terms\n", x.r.len ); */
for ( i=0; ilen; i++ )
openIsisSMsg( ses, 1, "%c=%.*s\n", (int)r->field[i].tag,
(int)r->field[i].len, r->field[i].val );
} break;
case DO_STREAM: {
OpenIsisIos ios;
OpenIsisRecStream rs = { 0, 0, 0, 0, 0 };
LIO_SINIT( &ios, lio_stdio, "stdin", LIO_IN );
rs.in = &ios; /* some gcc versions need it this way */
while ( 0 < (i = openIsisSGetr( ses, &rs )) )
print( ses, rs.rec, 0, fmt );
} break;
case DO_IFLOAD: {
OpenIsisKey key;
OpenIsisHit hit;
OpenIsisIndex idx = openIsisIdxOpen( db, intarg );
char *buf = 0;
int l, lines = 0;
hit.dbn = (-2 == intarg) ? 0xffff : 0; /* secret key for ifdel */
while ( 0 <= (l = openIsisSReadln( ses, &buf )) && buf ) {
char *t = memchr( buf, '\t', l );
if ( t ) { /* tab delimited */
key.len = (unsigned char)(t - buf);
memcpy( key.byt, buf, key.len );
if ( 0 >= (l -= t-buf+1) || !(t = memchr( buf=t+1, '\t', l )) )
continue;
hit.mfn = (unsigned) openIsisA2i( buf, t-buf );
if ( 0 >= (l -= t-buf+1) || !(t = memchr( buf=t+1, '\t', l )) )
continue;
hit.tag = (unsigned short) openIsisA2i( buf, t-buf );
if ( 0 >= (l -= t-buf+1) || !(t = memchr( buf=t+1, '\t', l )) )
continue;
hit.occ = (unsigned short) openIsisA2i( buf, t-buf );
if ( 0 >= (l -= t-buf+1) )
continue;
hit.pos = (unsigned short) openIsisA2i( t+1, l );
} else {
/* 10/30 key BLANK 7 mfn BLANK 5 tag BLANK 4 occ BLANK 4 pos*/
int eok = l - 24; /* pos of blank after key, 10 or 30 */
if ( 54 != l && 34 != l ) {
openIsisSMsg( ses, OPENISIS_ERR_INVAL,
"bad ifload input len %d, want 34 or54 bytes + newline\n", l );
break;
}
for ( i=eok-1; ' ' == buf[i] && i--; )
;
key.len = (unsigned char) (++i);
memcpy( key.byt, buf, key.len );
log_msg( LOG_VERBOSE, "'%.*s'", 7, buf+eok+1 );
hit.mfn = (unsigned) openIsisA2i( buf+eok+1, 7 );
hit.tag = (unsigned short)openIsisA2i( buf+eok+9, 5 );
hit.occ = (unsigned short)openIsisA2i( buf+eok+15, 4 );
hit.pos = (unsigned short)openIsisA2i( buf+eok+20, 4 );
}
log_msg( LOG_VERBOSE, "'%.*s' %d %d %d %d",
key.len, key.byt, hit.mfn, hit.tag, hit.occ, hit.pos );
if ( openIsisIdxAdd( idx, &key, &hit ) )
break;
if ( !(++lines & 0x3ff) )
log_msg( LOG_INFO, "%dK lines", lines >> 10 );
}
openIsisIdxDone( idx );
} break;
case DO_IFDUMP: {
OpenIsisIdxLoop l;
memset( &l, 0, sizeof(l) );
l.flg = idxmode;
switch ( ifm ) {
case IFM_OLD:
l.flg |= OPENISIS_IDXTRAD;
case IFM_DUMP:
l.me = ses;
l.cb = (OpenIsisIdxCb*)printlk2;
break;
case IFM_TAB:
l.me = ses;
l.cb = (OpenIsisIdxCb*)printtab;
break;
case IFM_COPY:
l.flg |= OPENISIS_IDXTRAD;
l.me = openIsisIdxOpen( 0 <= wdb ? wdb : db, 0 );
l.cb = (OpenIsisIdxCb*)openIsisIdxAdd;
break;
case IFM_CHK:
/* nuttin */
break;
}
if ( search )
memcpy( l.key.byt, search,
l.key.len = (unsigned char)strlen( search ) );
if ( idxto )
memcpy( l.to.byt, idxto,
l.to.len = (unsigned char)strlen( idxto ) );
openIsisIdxLoop( db, &l );
if ( IFM_COPY == ifm )
openIsisIdxDone( (OpenIsisIndex)l.me );
} break;
case DO_VUTF: {
char buf[1024];
int t = 0, f = 0, g;
while ( 0 < (g = lio_read( &lio_in, buf, sizeof(buf) )) ) {
int l = openIsisValidUTF8( buf, g, &f );
if ( l ) {
openIsisSMsg( ses, OPENISIS_ERR_INVAL,
"at total %d = %d+%d\n", l-1+t, l-1, t );
ret = 1;
goto bye;
}
t += g;
}
}
} /* switch ( what ) */
bye:
if ( 0 <= db )
openIsisClose( db );
if ( 0 <= wdb )
openIsisClose( wdb );
/* at least with WINE,
atexit cleanup is not performed
unless we explicitly call exit :(
*/
exit( ret );
return ret;
} /* openisis main */
openisis.h 0100644 0001130 0001130 00000116453 07644311511 011551 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
#ifndef OPENISIS_H
/*
$Id: openisis.h,v 1.70 2003/04/07 15:03:37 kripke Exp $
public interface of the openisis lib.
usage samples of most calls are found in openisis.c
*/
#define OPENISIS_VERSION "0.8.7.5"
#ifndef __STDC__
# ifndef const
# define const
# endif
#endif
#ifdef __cplusplus
extern "C" {
#endif
/* ************************************************************
record structure
*/
/** cooked version of an ISIS field.
* values are *NOT* terminated by a 0 byte.
*/
typedef struct OpenIsisField {
int tag;
const char *val;
int len;
} OpenIsisField;
/** cooked version of an ISIS Record.
*/
typedef struct OpenIsisRec {
int dbid;
int rowid;
int bytes; /* total avail bytes */
int used; /* total used bytes (originally BWB) */
int fields; /* avail number of fields (originally BWP) */
int base; /* byte offset of contents area (after fields) */
int len; /* used number of fields */
int state;
OpenIsisField field[1];
} OpenIsisRec;
/* compute base from fields (for both raw and cooked record) */
#define OPENISIS_BASESZ( nfields ) ((int)sizeof(int)*(8 + 3*(nfields)))
/* initialise record of f fields, n bytes */
#define OPENISIS_INITREC( r, n, f ) \
do { \
memset( r, 0, n ); \
(r)->bytes = (n); \
(r)->fields = (f); \
(r)->used = (r)->base = OPENISIS_BASESZ( (r)->fields ); \
} while (0)
/* init with std 1 field per 100 bytes */
#define OPENISIS_INITSTD( r, n ) OPENISIS_INITREC( r, n, (n)/100 )
/* init a buffer structure which has a member r */
#define OPENISIS_INITBUF( x ) OPENISIS_INITSTD( &x.r, sizeof(x) )
#define OPENISIS_CLRREC( r ) \
do { \
(r)->len = (r)->state = 0; \
(r)->used = (r)->base = OPENISIS_BASESZ( (r)->fields ); \
} while (0)
/* check some invariants (for both raw and cooked record) */
#define OPENISIS_RECOK( r ) \
( (r)->base <= (r)->used \
&& (r)->used <= (r)->bytes \
&& (r)->len <= (r)->fields \
&& (r)->base == OPENISIS_BASESZ( (r)->fields ) )
/** field description.
*/
enum { /* field types */
OPENISIS_FTX = 0, /* alphanum */
OPENISIS_FTA = 1, /* STRICTLY alpha */
OPENISIS_FTN = 2, /* numeric */
OPENISIS_FTP = 3, /* pattern */
OPENISIS_FTI = 10, /* ISO */
OPENISIS_FTE = 12, /* enum */
OPENISIS_FTB = 13, /* bool */
OPENISIS_FTT = 14, /* table */
OPENISIS_FTS = 15, /* structure */
OPENISIS_FTF = 16, /* subfield */
OPENISIS_FTV = 31 /* enum value */
};
#define OPENISIS_FD_NAMELEN 32
typedef struct {
int id; /* tag */
char type; /* type */
char rep; /* !0 for repeated */
short len; /* max length or enum value */
char name[OPENISIS_FD_NAMELEN]; /* name, 0 terminated, max 30 */
char subf[21]; /* subfields/pattern, 0 terminated */
char desc[31]; /* description, 0 terminated */
/* TODO: add childs list */
} OpenIsisFd;
/** field description table.
use the CFdt call to create one.
*/
typedef struct {
int len;
OpenIsisFd *fd;
OpenIsisRec *rec; /* additional fields like formats, worksheets ... */
/* additional internal data like a hash might follow */
} OpenIsisFdt;
typedef struct { /* where a key has a hit */
unsigned mfn;
unsigned short tag;
unsigned short occ; /* hit is in occ'th occurence of field tag in row mfn */
unsigned short pos; /* ... as the pos'th word */
unsigned short dbn; /* for multi-db index; ignored by now */
} OpenIsisHit;
typedef struct {
unsigned char len;
unsigned char byt[23];
} OpenIsisVal;
typedef struct {
OpenIsisVal val;
unsigned char len;
unsigned char byt[255];
} OpenIsisKey;
/** Writex tags.
Positive values add entries, negative delete entries.
Most non-zero tags can or must be followed by a key to use.
*/
enum {
OPENISIS_XKEY, /* plain key to use */
OPENISIS_XVAL, /* binary val */
OPENISIS_XHIT, /* binary (12 byte) hit */
OPENISIS_XDOT, /* dotted decimal [[dbn.]mfn.]tag.occ.pos or tag[.occ] hit */
OPENISIS_XFST /* traditional FST entry ^ttag^mmode^ffmt */
/* if the fmt subfield is missing, the following key values are used
(each as one occurrence).
*/
};
/**
get occurence of field.
@param pos if given, the first occ starting at pos is searched
and pos is set to one after the found position (or after end).
may be used to loop all on an int var initialized to 0.
@return the field or 0
*/
extern OpenIsisField *openIsisRGet ( OpenIsisRec *r, int tag, int *pos );
extern OpenIsisField *openIsisROccurence ( OpenIsisRec *r, int tag, int occ );
/** similar to atoi/strtol, but
- string needs not be 0 terminated (unless l < 0)
- cares for hex 0x, but not octal 0.
*/
extern int openIsisA2i ( const char *p, int l );
/** similar to openIsisA2i but
- return number of parsed characters
- put result in *res
*/
extern int openIsisA2il ( const char *p, int l, int *res );
/** similar to openIsisA2i but
- return dflt if less than l characters of p has been successfully parsed
*/
extern int openIsisA2id ( const char *p, int l, int dflt );
/** print 0 terminated int.
p must have 12 bytes space
@return number of chars (up to 10 digits + minus sign)
*/
extern int openIsisI2a ( char *p, int i );
extern int openIsisU2a ( char *p, unsigned u );
/**
get field value as int as of A2i.
boolean values are recognized as 0 and 1.
if not found, return def.
*/
extern int openIsisRInt ( OpenIsisRec *r, int tag, int def, int *pos );
/**
* get field value as string from rec or default rec.
* note: def should be a non-legal value, since a def result in the
* first record causes a lookup in dflt.
*/
extern int openIsisRInt2 (OpenIsisRec *rec,
OpenIsisRec *dflt, int tag, int def);
/**
get field value as enum.
if not found, return def.
*/
extern int openIsisREnum ( OpenIsisFdt *fdt,
OpenIsisRec *r, int tag, int def, int *pos );
/**
get field value as string of length len (including 0-byte).
if not found, return 0.
*/
extern char* openIsisRString (OpenIsisRec *r,
int tag, int *pos, char *buf, int len);
/**
* get field value as string from rec or default rec.
*/
extern char* openIsisRString2 (OpenIsisRec *rec,
OpenIsisRec *dflt, int tag, char *buf, int len);
/** get field description by id (tag). */
extern OpenIsisFd* openIsisFdById ( const OpenIsisFdt *fdt, int id );
/** get field description by name. */
extern OpenIsisFd* openIsisFdByName (const OpenIsisFdt *fdt, const char *name);
enum {
OPENISIS_NOENUM = 0x10000 /* enum NOENUM :) -- not a short */
};
/** lookup enum value for field id.
name must be exact match or unique prefix.
@return the enum value or NOENUM
*/
extern int openIsisFdEnum ( OpenIsisFdt *fdt, int id, const char *name );
/** flatten (serialize) text record
to tag\tval lines ended by a blank line.
Text means the record is assumed to not contain
vertical tabs of any importance,
so newlines in field values are replaced with vertical tabs (^K).
buf must be of size rec->used,
wich has enough room for all values + 24bytes header
+ 12 bytes per field (ok for sign+10digits+tab).
@return # of bytes written
*/
extern int openIsisRSerialize ( char *buf, OpenIsisRec *rec );
/** flatten fully transparently,
replacing newlines with newline-tab.
extern int openIsisRBinSerialize ( char *buf, int len, OpenIsisRec *rec );
*/
/* ************************************************************
session type and utilities
*/
typedef struct OpenIsisSes *OpenIsisSession;
extern void *openIsisMAlloc ( OpenIsisSession s, int size );
extern void openIsisMFree ( OpenIsisSession s, void *mem );
extern void *openIsisMDup ( OpenIsisSession s, const void *str, int size );
/**
convert string to html by replacing the four characters
<,>,& and ".
free the returned pointer after usage.
XML has a fifth standard entity apos,
but many HTML clients don't grok it.
*/
extern char *openIsisToHtml ( OpenIsisSession s, const char *str, int len );
extern int openIsisUtf8Chk ( OpenIsisSession s, void *mem, int len, int *tof );
/* ************************************************************
stream IO
*/
typedef struct OpenIsisIos *OpenIsisStream;
typedef int OpenIsisSFunc ( OpenIsisStream s, int op );
extern int openIsisSOpen ( OpenIsisSession s, const char *name, int flags,
OpenIsisSFunc *type );
/**
error and loglevel codes one nibble each.
*/
enum { /* our very own errno */
OPENISIS_ERR_OK = 0, /* 0 is no error, also read(2)'s EINTR, EAGAIN */
/* errors logged at level VERBOSE */
OPENISIS_ERR_EOF = 0x10000, /* end of file */
/* errors logged at level ERROR */
OPENISIS_ERR_FAULT = 0x20000, /* NULL pointer or bad sized buffer given */
OPENISIS_ERR_INVAL = 0x30000, /* general invalid parameters, any EINVAL errno */
OPENISIS_ERR_BADF = 0x40000, /* bad file, also read(2)'s EINVAL, some of open(2) */
/* errors logged at level SYSERR */
OPENISIS_ERR_IO = 0x50000, /* real IO error, also write(2)'s ENOSPC, EPIPE */
OPENISIS_ERR_NOMEM = 0x60000, /* out of memory, also open(2)'s EMFILE, ENFILE */
OPENISIS_ERR_BUSY = 0x70000, /* object is busy */
/* errors logged at level FATAL */
OPENISIS_ERR_TRASH = 0x80000, /* database internal consistency */
OPENISIS_ERR_IDIOT = 0x90000 /* caught programming error */
};
typedef enum {
OPENISIS_LOG_OFF = 0, /* don't log anything */
OPENISIS_LOG_FATAL = 0x100000, /* fatal internal errors: we can't go on */
OPENISIS_LOG_SYSERR = 0x200000, /* problem with system ressources: bad file, no mem */
OPENISIS_LOG_IOERR = 0x300000, /* problem on IO */
OPENISIS_LOG_ERROR = 0x400000, /* unusable input, database or query */
OPENISIS_LOG_WARN = 0x500000, /* bad input */
OPENISIS_LOG_INFO = 0x600000, /* some major event like opening a db */
OPENISIS_LOG_VERBOSE = 0x700000, /* any event like reading a record */
OPENISIS_LOG_TRACE = 0x800000, /* database content (log_str) */
/* highest levels only with debug built */
OPENISIS_LOG_DEBUG = 0x900000, /* lots of processing details */
OPENISIS_LOG_ALL = 0xa00000 /* just everything, even built-in content */
} OpenIsisLogLevel;
enum {
OPENISIS_ERR_ERRORS = 10,
OPENISIS_ERR_MASK = 0xf0000, /* mask for filtering error code */
OPENISIS_ERR_SHIFT = 16, /* shift to make codes 0,1...ERR_ERRORS */
OPENISIS_LOG_NOCHANGE = -1, /* leave as is when initialising log */
OPENISIS_LOG_LEVELS = 11,
OPENISIS_LOG_MASK = 0xf00000, /* mask for filtering log levels */
OPENISIS_LOG_SHIFT = 20, /* shift to make codes 0,1...LOG_LEVELS */
/*
the following ids are provided just in case
you don't feel comfortable assuming that in,out,err are always 0,1,2 ;)
typically, you don't need any of these, 0 will work ...
*/
OPENISIS_SIN = 0, /* stream id of stdin */
OPENISIS_SOUT = 1, /* stdout */
OPENISIS_SERR = 2, /* stderr */
OPENISIS_SMASK = 0xffff, /* mask of stream id part of to */
OPENISIS_MSG_VA = 0x1000000 /* 1st parameter after fmt IS a va_list */
};
/**
printf to session's stream given by to.
The streams 0,1 and 2 of the default session are usually bound
to the system file descriptors 0,1 and 2 under unixes.
On windows, 2 is redirected to file oisiserr.txt.
0 and 1 are bound to CONIN$ and CONOUT$ for console apps,
to NULL and oisisout.txt for GUI apps.
For server sessions, 0 and 1 are connected to the client
and 2 is redirected to some per-session logging.
@param to
the lower bytes as given by SMASK specify the stream to use.
the higher bytes may specify loglevel and or error code.
if either loglevel or error code is given,
the other will be derived accordingly
and stream defaults to 2 (if stream id is 0).
otherwise, stream defaults to 1,
so you never can print to 0.
@return the negative error code
*/
extern int openIsisSMsg ( OpenIsisSession s, int to, const char *fmt, ... );
/**
get a string delimited by char delim (typically '\n') on stream number fd.
on success, ptr is set to the string
and it's length is returned, not including the delimiting char.
Note that there need not be a delimiting char near end of file.
If there is no string available, ptr is set to 0 (NULL).
If input might be available later, 0 is returned,
else some negative error code.
*/
extern int openIsisSGets ( OpenIsisSession s, int fd, char **ptr, char delim );
#define openIsisSReadln( s, p ) openIsisSGets( s, 0, p, '\n' )
typedef struct OpenIsisRecStream {
OpenIsisStream in; /* input stream to read from */
int flg; /* flags */
OpenIsisRec *buf; /* original buffer */
OpenIsisRec *rec; /* record read */
OpenIsisRec *dict; /* dictionary of field names */
} OpenIsisRecStream;
/*
stream in records in some free-style format suitable for
mails, properties, syspar ...
fields are delimited by CR or NL or CR-NL
within a field, tag stops at blank,TAB,= or :
after a TAB, val starts immediately
after a = or :, val starts at 1st character not a blank
after a blank, val starts at 1st char not one of the above separators
processing
- a field or record separator is recognized anywhere
- within each field, the tag is read up to the first separator
(any "special" char, all controls, whitespace and most punctuation).
upon reaching a separator, the tag is identified.
if it is empty, and the record has a field, now new field is created.
else, if the tag cannot be identified,
a tag of -1 is used and the tag string appended as value.
- the separator consists of any number of leading blanks,
some separator char, and, unless the sepchar is TAB, trailing blanks.
if the current field is not empty, the separator is appended as value.
- the rest of the field is appended as value
- a field with empty tag and value (blank line)
terminates the record in most modes
- a field with empty tag (continuation line)
has it's separator AND value appended to the previous field
- if either a tag or a separator extend over more than 4k,
the stream is considered over limit.
*/
enum {
OPENISIS_STOPONEMPTY = 0x10, /* stop at empty line */
OPENISIS_AUTOCLONE = 0x20, /* clone the record even if not needed */
OPENISIS_TRANSPARENT = 0x40, /* do not convert vtabs to newlines */
OPENISIS_MAIL = 0x100, /* process mail body */
OPENISIS_MBOX = 0x200, /* input stream is mbox format */
OPENISIS_DETACH = 0x400, /* detach body */
OPENISIS_CONVERT = 0x800 /* fix body parts */
};
extern int openIsisSGetr ( OpenIsisSession ses, OpenIsisRecStream *stream );
/* ************************************************************
record utilities requiring a session
*/
/**
clone record or create empty record.
with room >= 0 and rec != NULL,
make sure that there is at least one field and room bytes available.
else, it works in shrink-to-fit mode: exactly the required bytes are used.
@param rec to extend, may be NULL
@param room number of required free bytes (<0 for shrink-to-fit)
@param discard wether the original record should be freed after extension
@return a rec with suitable room
*/
extern OpenIsisRec* openIsisRDup ( OpenIsisSession ses,
OpenIsisRec *r, int room, int discard );
/**
printf a new field of up to 1024 bytes at the end of record.
*/
extern OpenIsisRec* openIsisRMsg ( OpenIsisSession ses,
OpenIsisRec *r, int discard, int tag, const char *fmt, ... );
/**
check for free headspace in a record.
make sure that there is at least one field and room bytes available.
NOTE: this may change recp
@see openIsisRDup
*/
#define OPENISIS_RSPACE( ses, recp, room, disc ) \
do { if ( ! (recp) \
|| (recp)->len >= (recp)->fields \
|| (recp)->bytes < (recp)->used + (room) \
) (recp) = openIsisRDup( (ses), (recp), (room), (disc) ); } while (0)
/**
add a field to a record.
a new field entry and n bytes space are provided.
if the pointer s is not NULL, n bytes are copied from it to the field.
NOTE: this may change recp
@see openIsisRDup
*/
#define OPENISIS_RADD( ses, recp, ntag, s, n, disc ) \
do { \
int _ll = (n); \
OPENISIS_RSPACE( ses, recp, _ll, disc ); \
if ( (recp) ) { \
OpenIsisField *_ff = (recp)->field + (recp)->len++; \
_ff->tag = (ntag); \
_ff->val = ((char*)recp)+(recp)->used; \
if ( ! s ) \
_ff->len = 0; \
else \
memcpy( (char*)_ff->val, s, _ff->len = _ll ); \
(recp)->used += _ff->len; \
} \
} while(0)
/**
append n bytes to the last field of a record.
NOTE: this may change recp
NOTE: this requires the last field to reside at the end of the buffer
@see openIsisRDup
*/
#define OPENISIS_RCAT( ses, recp, s, n, disc ) \
do { \
OpenIsisField *_ff; \
int _ll = (n); \
assert ( (recp) ); \
assert ( (recp)->len ); \
if ( (recp)->bytes < (recp)->used + _ll ) \
(recp) = openIsisRDup( (ses), (recp), _ll, (disc) ); \
_ff = (recp)->field + (recp)->len - 1; \
memcpy( (char*)(_ff->val+_ff->len), s, _ll ); \
_ff->len += _ll; \
(recp)->used += _ll; \
} while(0)
/* like OPENISIS_RADD, but based on strlen */
#define OPENISIS_RADDS( ses, recp, ntag, s, disc ) \
OPENISIS_RADD( ses, recp, ntag, s, strlen(s), disc )
/* like OPENISIS_RADD, but based on field */
#define OPENISIS_RADDF( ses, recp, f, disc ) \
OPENISIS_RADD( ses, recp, (f)->tag, (f)->val, (f)->len, disc )
/* like OPENISIS_RCAT, but based on strlen */
#define OPENISIS_RCATS( ses, recp, s, disc ) \
OPENISIS_RCAT( ses, recp, s, strlen(s), disc )
/**
create a record from a field, i.e. split subfields.
if a rec buffer is given, it is filled up to it's rec->len and returned.
else an appropriately allocated rec is returned,
which must be freed after usage.
NOTE: USE WITH CARE, IT SOMEWHAT BREAKS THE GENERAL CONTRACT OF REC!
DO NOT TRY TO EXTEND THE RETURNED RECORD !!!
The rec does NOT have it's own data buffer,
but instead field values are pointers into the input field's value.
The returned rec will therefore become invalid,
if the inpur field is freed.
@param rec a rec buffer to fill or NULL
@param field some string to be split up at '^' characters
@return an OpenIsisRec or NULL, if splitting failed.
*/
extern OpenIsisRec *openIsisRSplitf ( OpenIsisSession ses,
OpenIsisRec *rec, const OpenIsisField* field );
enum { /* RSet flags */
OPENISIS_RARGC = 0xffff, /* argc mask */
OPENISIS_RFDT = 0x0010000, /* first vararg is the Fdt to use */
OPENISIS_RARGV= 0x0020000, /* next vararg is a char **argv */
OPENISIS_RDIS = 0x0040000, /* discard original record on change */
OPENISIS_RNOC = 0x0080000, /* do not recompact after CHG/DEL op */
OPENISIS_RIGN = 0x0100000, /* ignore unknown fields */
/* default op is to append fields at the end */
OPENISIS_RCHG = 0x1000000, /* change mode: overwrite previous field values */
OPENISIS_RDEL = 0x2000000, /* args is a list of tags to delete */
OPENISIS_ROP = 0xf000000 /* op mask */
};
/**
set fields in a record.
this is going to be the real field setting one-stop-shop.
This is called
EITHER with real varargs tag, val, tag, val ... 0, stopped by a 0 tag,
OR with an char **argv as vararg.
In most modes, the varargs are like tag, val, tag, val ...
Tags may be either numeric (int) or by field name.
Field names are translated to numerical ids by the Fdt,
which may be given explicitly or derived from the rec's dbid.
@param ses current session
@param rec record to modify or 0 to create new one
@param mode bitwise OR of flags and the number of fields (2 byte)
if the number of fields is 0,
args must be zero terminated
@return the same record or a copy, depending on space needs
*/
extern OpenIsisRec *openIsisRSet ( OpenIsisSession ses,
OpenIsisRec *rec, int mode, ... );
/**
add serialized "text" to rec
supports the RDIS, STOPONEMPTY and TRANSPARENT flags
@return # of bytes read
*/
extern int openIsisRDeserialize ( OpenIsisSession ses,
OpenIsisRec **rec, const char *buf, int len, int flg );
/**
append tag with integer value to rec
*/
extern OpenIsisRec *openIsisRAddI (OpenIsisSession ses,
OpenIsisRec *rec, int tag, int value, int discard);
enum {
OPENISIS_FMT_SHARP = 32, /* # */
OPENISIS_FMT_SLASH, /* / */
OPENISIS_FMT_PERCENT, /* % */
OPENISIS_FMT_OPEN, /* { */
OPENISIS_FMT_CLOSE, /* } */
OPENISIS_FMT_ESC, /* ! */
OPENISIS_FMT_B, /* bold */
OPENISIS_FMT_BOX, /* draw a box (0i) */
OPENISIS_FMT_BPICT, /* draw background pict (1sn) */
OPENISIS_FMT_C, /* column (1i) */
OPENISIS_FMT_CL, /* colour (1i) */
OPENISIS_FMT_COLS, /* colour table (s_) - emits a string of hex vals */
OPENISIS_FMT_FONTS, /* font table (s_) */
OPENISIS_FMT_FS, /* font size (1i) */
OPENISIS_FMT_F, /* font (1i) */
OPENISIS_FMT_I, /* italic */
OPENISIS_FMT_LINK, /* link (2s_) */
OPENISIS_FMT_LW, /* line width (1i) */
OPENISIS_FMT_M, /* indentation in twips(1in) */
OPENISIS_FMT_NC, /* new column (0i) */
OPENISIS_FMT_NEWLINE, /* set newline (1s) */
OPENISIS_FMT_NP, /* newpage (0i) */
OPENISIS_FMT_PICT, /* draw pict (1s) */
OPENISIS_FMT_QC, /* center */
OPENISIS_FMT_QJ, /* justify */
OPENISIS_FMT_TAB, /* tab (0i) */
OPENISIS_FMT_UL, /* underline */
OPENISIS_FMT_X /* blanks (1i) */
};
extern OpenIsisRec* openIsisRFmt ( OpenIsisSession ses,
OpenIsisRec *buf, const char *fmt, OpenIsisRec *r );
/* ************************************************************
db access
*/
/**
get highest rowid in db
@param dbid id returned by openIsisOpen
@return positive rowid on success, 0 if unknown, negative on error
*/
extern int openIsisDMaxId ( OpenIsisSession ses, int dbid );
/**
read a cooked row.
free the returned pointer after usage.
@param db id returned by openIsisOpen
@param rowid the mfn
@return an OpenIsisRec or NULL, if row not available.
*/
extern OpenIsisRec *openIsisDRead ( OpenIsisSession ses, int db, int rowid );
/**
for convenience: read and format in one.
*/
extern OpenIsisRec *openIsisDFmt ( OpenIsisSession ses,
OpenIsisRec *buf, const char *fmt, int db, int rowid );
/**
write a record and/or index entries.
If the rowid is 0, the record will be modified by assigning a new rowid.
@param db id returned by openIsisOpen
@param rec the OpenIsisRec to write
@param idx record containing index entries
@return 0 or an error code
*/
extern int openIsisDWritex ( OpenIsisSession ses, int dbid, OpenIsisRec *rec,
OpenIsisRec *idx );
/**
write a record as with DWritex.
index entries should be derived automatically.
*/
extern int openIsisDWrite ( OpenIsisSession ses, int dbid, OpenIsisRec *rec );
#define OPENISIS_DWRITE( ses, rec ) openIsisDWrite( ses, (rec)->dbid, rec )
/**
simple full scan search.
read the first row with rowid >= given rowid
and containing txt in subfield tag (any, if < 0).
free the returned pointer after usage.
@param db id returned by openIsisOpen
@param rowid the mfn
@param tag field in which to look for key, 0 for any
@return an OpenIsisRec or NULL, if row not available.
*/
extern OpenIsisRec *openIsisDScan ( OpenIsisSession ses,
int db, int rowid, int tag, const char *key );
/**
get a term for key w/ or w/o the postings.
prefix mode is determined by a trailing '$' in key.
for prefix mode, the last preset field in the record
gives a starting point, i.e. terms up to and including
it are ignored.
thus, if there are more matching terms than fit in the record,
the search can be continued by passing the same record.
@return an OpenIsisRec or NULL
*/
extern OpenIsisRec* openIsisDTerm ( OpenIsisSession ses,
OpenIsisRec *rec, int db, const char *key );
#ifndef OPENISIS_SETLEN
#define OPENISIS_SETLEN 1000
#endif
/**
record set type as returned from openIsisSearch.
first entry is the set length n, followed by rowids 1..n.
actual array length is thus n+1.
*/
typedef struct {
int len;
int id[OPENISIS_SETLEN];
} OpenIsisSet;
/* query mode */
enum {
/* search given string using index */
OPENISIS_QRY_KEYEQ, /* index scan key equal */
OPENISIS_QRY_KEYPF, /* index scan key prefix */
OPENISIS_QRY_KEYAT, /* index scan key auto (checks for '$') */
/* fulltext scan for given string */
OPENISIS_QRY_SCANE=64, /* fulltext scan for equal */
OPENISIS_QRY_SCANC, /* fulltext scan for contains */
/* parse string as query expression */
OPENISIS_QRY_SIMPLE=128, /* simple left-to-right binding */
OPENISIS_QRY_PROPER /* proper binding using precedence, () */
};
/**
query the db.
@param set set buffer to fill.
set->len must contain the actual buffer length (# of ids).
0 means default length OPENISIS_SETLEN.
@param db id returned by openIsisOpen
@param key keyword or query expression to search for
@param mode query mode, optionally |ed with a field tag<<16.
a field tag affects key and scan modes.
@param skip mfns < skip are ignored, useful to continue a query
@return the number of rowids in set (= set->len) on success, <0 on error.
*/
extern int openIsisDQuery ( OpenIsisSession ses, OpenIsisSet *set, int db,
const char *key, int mode, int skip );
/**
callback for index loop
*/
typedef int OpenIsisDXCb ( void *me, OpenIsisKey *key, OpenIsisHit *hit );
typedef struct {
int flg;
void *me; /* commonly the session */
OpenIsisDXCb *cb;
OpenIsisKey key;
OpenIsisKey to;
} OpenIsisDXLoop;
enum { /* flags */
OPENISIS_IDXPF = 0, /* loop where from is prefix of key */
OPENISIS_IDXEQ = 1, /* loop from == key */
OPENISIS_IDXUPTO = 2, /* loop from <= key < to */
OPENISIS_IDXINCL = 3, /* loop from <= key <= to */
OPENISIS_IDXMODE = 0x03, /* mask for match mode */
OPENISIS_IDXTRAD = 0x10000 /* use traditional index */
};
/**
*/
extern int openIsisDXLoop ( OpenIsisSession ses, int dbid, OpenIsisDXLoop *l );
/* ************************************************************
system control -- session 0 only
*/
enum {
/* system parameters */
OPENISIS_SPATH = 5, /* db dir */
OPENISIS_SLOGF = 700, /* logfile */
OPENISIS_SLOGV = 701, /* verbosity */
OPENISIS_SSCHEMA = 710, /* schema entry */
OPENISIS_SSCNAME = 's', /* subfield schema name */
OPENISIS_SSCHOST = 'h', /* subfield schema host */
OPENISIS_SSCPORT = 'p', /* subfield schema port */
/* database parameters */
OPENISIS_DNAME = 800,
OPENISIS_DTYPE = 801,
OPENISIS_DRO = 802,
OPENISIS_DPATH = 803,
OPENISIS_DDUMP = 804,
OPENISIS_DFMT = 850, /* worksheets */
OPENISIS_DPFT = 851, /* printformats */
OPENISIS_DFST = 852, /* field selections */
/* fdt entries */
OPENISIS_FDID = 860,
OPENISIS_FDTYPE = 861,
OPENISIS_FDREP = 862,
OPENISIS_FDLEN = 863,
OPENISIS_FDNAME = 864,
OPENISIS_FDSUBF = 865,
OPENISIS_FDDESC = 866
};
enum {
OPENISIS_RELE = 0x0000000, /* release, unlock */
OPENISIS_LOCK = 0x1000000,
OPENISIS_WAKE = 0x2000000, /* broadcast signal, notifyall */
OPENISIS_WAIT = 0x3000000,
OPENISIS_COND = 0x0ffffff /* condition mask */
};
/** a function to provide a global lock in a multi threaded process.
@param lock coded action.
the high byte is one of release, lock, wake and wait.
the lower bytes indicate a condition for wake and wait.
the value may be hashed to a small set of conditions,
typically eight or one (ignore).
@return 0 on success
sample with a single condition:
int myOpenIsisLockFunc ( int lock )
{
static pthread_mutex_t mut = PTHREAD_MUTEX_INITIALIZER;
static pthread_cond_t cond = PTHREAD_COND_INITIALIZER;
switch ( OPENISIS_WAIT & lock ) {
case OPENISIS_RELE: return pthread_mutex_unlock( &mut );
case OPENISIS_LOCK: return pthread_mutex_lock( &mut );
case OPENISIS_WAKE: return pthread_cond_broadcast( &cond );
case OPENISIS_WAIT: return pthread_cond_wait( &cond, &mut );
}
return -1;
}
*/
typedef int OpenIsisCLockFunc ( int lock );
/** initialize system with some common parameters and a lock func.
@param lock a function to provide a global lock in a multi threaded process.
*/
extern int openIsisCInit ( int argc, const char **argv,
OpenIsisCLockFunc lock );
/**
set logging level and device.
@param level the logging level
may be given numerical or as ascii char in -ofsewivtda for
nochange(-1), off (0), fatal (1), syserr, error, warn, info, verbose,
trace, debug, all(9)
default is set to error
@param filename specifies, if not NULL, a file to append log to
*/
extern void openIsisCLog ( int level, const char *filename );
/**
create a field definition table.
if fd is not given, definitions are read from rec.
@return a Fdt malloced in the default session
*/
extern OpenIsisFdt *openIsisCFdt (
OpenIsisSession ses, int len, OpenIsisFd *fd, OpenIsisRec *rec );
extern void openIsisCFreeFdt (
OpenIsisSession ses, OpenIsisFdt *fdt);
/** get the default session, possibly adding args.
only one (session master) thread in a process may call this.
@param args initialisation properties (cloned to prop)
*/
extern OpenIsisSession openIsisCOpen ( OpenIsisRec *args );
/** get a new session.
only one (session master) thread in a process may call this.
@param args initialisation properties (cloned to prop)
*/
extern OpenIsisSession openIsisCSession ( OpenIsisRec *args );
/* ************************************************************
schema
*/
#define OPENISIS_SC_NAMELEN 32 /* max length of schema identification */
#define OPENISIS_DB_NAMELEN 32 /* max length of db id relative to schema */
#define OPENISIS_SCIDMSK 0x0ff00 /* schema part of id */
#define OPENISIS_DBIDMSK 0x000ff /* db part of id */
typedef struct {
char name[OPENISIS_DB_NAMELEN];
int dbid;
int mntcnt; /* number of sessions currently using this */
int tms; /* last (re)opening timestamp */
OpenIsisRec *cfg;
OpenIsisFdt *fdt;
} OpenIsisDb;
typedef struct {
char name[OPENISIS_SC_NAMELEN];
int scid;
int mntcnt;
OpenIsisRec *cfg;
int ndbs;
OpenIsisDb **dbs;
} OpenIsisSchema;
extern const OpenIsisFdt *openIsisSyspar;
extern const OpenIsisFdt *openIsisDbpar;
/* ************************************************************
client side
*/
typedef struct OpenIsisStb *OpenIsisStub;
typedef int OpenIsisRspCb (void *client_data,
OpenIsisStub session, OpenIsisRec *response, OpenIsisDb *db);
typedef struct {
/* default response callback */
OpenIsisRspCb *dfltcb;
/* delete callback: cleanup client data cbd that is no longer
used. if 0==cbd, session is going to be deleted.
*/
void (*delcb) (void *cld, OpenIsisStub session, void *cbd);
/* client data for dfltcb */
void *dfltcld;
/* client data for delcb */
void *delcld;
} OpenIsisStubCbData;
extern OpenIsisStub openisis_stub0;
/** create stub for local schema.
*/
extern OpenIsisStub openIsisNInit (
int argc, const char **argv, OpenIsisStubCbData *dta
);
/** destroy local schema.
*/
extern void openIsisNDeinit ();
/** create a new stub.
*/
extern OpenIsisStub openIsisNOpen (
const char *name, int argc, const char **argv, OpenIsisStubCbData *dta
);
/** destroy a stub.
*/
extern void openIsisNClose (OpenIsisStub stub);
/** get schema from stub.
*/
extern OpenIsisSchema* openIsisNSchema (OpenIsisStub stub);
/** send request.
SID,SER,TMS are set in this call.
Note: rqs (or the internal copy) will be freed at the next call
to openIsisNClean or openIsisNSend.
@param stub communication partner
@param ses session to use when later freeing mem of rqs rec
@param rqs request parameter
@param actcb optional async callback for this request
@param actcld optional client data for actcb
@param rdup make a internal copy of rqs
@return 0 on success
*/
extern int openIsisNSend (
OpenIsisStub stub, OpenIsisSession ses, OpenIsisRec *rqs,
OpenIsisRspCb *actcb, void *actcld, int rdup
);
/** get stub's response record.
@param dbid optional db id of embedded result records
*/
extern OpenIsisRec *openIsisNRecv (OpenIsisStub stub, OpenIsisDb **db);
/** clean request, response, session used in last openIsisNSend call.
*/
extern void openIsisNClean (OpenIsisStub stub);
/** util: unwrap embbeded response records.
*/
extern int openIsisNGetResult (OpenIsisStub stub, OpenIsisSession ses,
int **rowids, OpenIsisRec ***recs, int *total);
/** get db by name.
*/
extern OpenIsisDb *openIsisNDbByName (OpenIsisStub stub, const char *dbname);
/** get db by id.
*/
extern OpenIsisDb *openIsisNDbById (int dbid);
/* ************************************************************
db control
*/
/**
open an isis database.
The following options are supported as text-style parameters:
db <name>
dbpath <path>
v <level>
@param dbname basename of database.
the dbhome arg, if given, will be prepended, and the ISIS file
extensions appended to build the actual filename.
Therefore, depending on the value of dbhome and location
of files, this may need to include a path.
db may be NULL, if argv includes a dbname arg.
@param argv array of names and values.
May be NULL.
argv contains parameter names, which may be prefixed
by a dash '-'. Depending on the actual parameter, the next
string may or must contain a corresponding parameter value.
An optional parameter value may be omitted, if at end of
argv or the next name is prefixed with a dash.
To avoid ambiguity, values never start with a dash.
See above for a list of supported parameter names.
Unrecognized parameter names are ignored.
@param argc length of argv.
You may supply -1 to indicate that argv is NULL terminated.
@return db structure or 0 on error
*/
extern OpenIsisDb* openIsisCDOpen ( const char *dbname,
OpenIsisRec *dbpar, OpenIsisRec *syspar );
/**
open by vector
@return some non-negative dbid on success,
some negative errorcode on error.
Errorcodes include -ENOENT, -ENOMEM, -EACCESS and others
returned by file access functions.
*/
extern int openIsisCDOpenv ( const char *dbname, const char **argv, int argc );
enum {
OPENISIS_CHK_ONLY = 0x01, /* check only, do not fix */
OPENISIS_CHK_XRF = 0x02, /* check the xrf */
OPENISIS_CHK_FIX = 0xfe, /* check all and fix */
OPENISIS_CHK_ALL = 0xff /* check all, no fix */
};
/** check and optionally fix the database.
*/
extern int openIsisCDCheck ( int dbid, int flags );
/**
close a database.
@param dbid id returned by openIsisOpen
@return 0 on success, some negative errorcode else.
*/
extern int openIsisCDClose ( int dbid );
/* ************************************************************
low level index access
DO NOT USE unless you know what you're doing !!!!!!!!!
*/
typedef struct OpenIsisIdx *OpenIsisIndex;
/**
get the db's index for loading.
the index may not be access until the loading session is finished.
@param mode lowest byte give percent free for index
*/
extern OpenIsisIndex openIsisCXOpen ( int dbid, int mode );
/**
load a sorted series of keys and hits into index.
@param key may be NULL to append to previous key.
@param hit a value of NULL terminates the loading session
*/
extern int openIsisCXAdd ( OpenIsisIndex idx,
OpenIsisKey *key, OpenIsisHit *hit );
#define openIsisCXClose( idx ) openIsisCXAdd( idx, 0, 0 )
/* ************************************************************
single session and legacy mode pre 0.9
*/
extern OpenIsisSession openisis_ses0;
/** raw version of an ISIS field.
* off gives an offset from the start of the record taken as char*.
* values are *NOT* terminated by a 0 byte.
*/
typedef struct {
/*
although isis-1 dbs have only 2 bytes for field tags (numbers),
we use ints with the lower 3 bytes for the tag.
The highest byte may have special usage in some context.
*/
int tag;
int off;
int len;
} OpenIsisFld;
/** raw version of an ISIS Record. rarely used.
*/
typedef struct {
int dbid;
int rowid;
int bytes; /* total avail bytes */
int used; /* total used bytes (originally BWB) */
int fields; /* avail number of fields (originally BWP) */
int base; /* byte offset of contents area (after fields) */
int len; /* used number of fields */
int state;
OpenIsisFld field[1];
} OpenIsisRaw;
/** read raw. */
extern OpenIsisRaw *openIsisDRaw ( OpenIsisSession ses, int db, int rowid );
#ifndef OPENISIS_NOPRE09
/* auto-initalizing access to the default session */
#define OPENISIS_SES0() \
openisis_ses0 ? openisis_ses0 : openIsisCOpen(0)
/* utilities */
#define openIsis2Html( str, len ) \
openIsisToHtml ( OPENISIS_SES0(), str, len )
#define openIsisValidUTF8( mem, len, tof ) \
openIsisUtf8Chk ( OPENISIS_SES0(), mem, len, tof )
/* record tools */
#define openIsisClone( r, ro, d ) \
openIsisRDup( OPENISIS_SES0(), r, ro, d )
/* GNUC only, and waaarns ...
#define openIsisPrintf( r, d, t, f, a... ) \
openIsisRMsg( OPENISIS_SES0(), r, d, t, f, ## a )
*/
#define OPENISIS_CHKFREE( recp, room, disc ) \
OPENISIS_RSPACE( OPENISIS_SES0(), recp, room, disc )
#define OPENISIS_ADD( recp, ntag, s, n, disc ) \
OPENISIS_RADD( OPENISIS_SES0(), recp, ntag, s, n, disc )
#define OPENISIS_CAT( recp, s, n, disc ) \
OPENISIS_RCAT( OPENISIS_SES0(), recp, s, n, disc )
#define OPENISIS_SADD( recp, ntag, s, disc ) \
OPENISIS_RADDS( OPENISIS_SES0(), recp, ntag, s, disc )
#define OPENISIS_SCAT( recp, s, disc ) \
OPENISIS_RCATS( OPENISIS_SES0(), recp, s, disc )
#define openIsisReadField( rec, f ) \
openIsisRSplitf( OPENISIS_SES0(), rec, f )
#define openIsisFmt( buf, fmt, r ) \
openIsisRFmt( OPENISIS_SES0(), buf, fmt, r )
#define openIsisReadStream( stream ) \
openIsisSGetr( OPENISIS_SES0(), stream )
/* database */
#define openIsisMaxRowid( db ) \
openIsisDMaxId( OPENISIS_SES0(), db )
#define openIsisReadRow( db, rowid ) \
openIsisDRead( OPENISIS_SES0(), db, rowid )
#define openIsisReadRaw( db, rowid ) \
openIsisDRaw( OPENISIS_SES0(), db, rowid )
#define openIsisRead( buf, fmt, db, rowid ) \
openIsisDFmt( OPENISIS_SES0(), buf, fmt, db, rowid )
#define openIsisWrite( db, r ) \
openIsisDWrite( OPENISIS_SES0(), db, r )
#define openIsisWritex( db, r, i ) \
openIsisDWritex( OPENISIS_SES0(), db, r, i )
#define openIsisScan( db, rowid, tag, key ) \
openIsisDScan( OPENISIS_SES0(), db, rowid, tag, key )
#define openIsisTerm( rec, db, key ) \
openIsisDTerm( OPENISIS_SES0(), rec, db, key )
#define openIsisQuery( set, db, key, mode, skip ) \
openIsisDQuery( OPENISIS_SES0(), set, db, key, mode, skip )
/* database index */
#define OpenIsisIdxCb OpenIsisDXCb
#define OpenIsisIdxLoop OpenIsisDXLoop
#define openIsisIdxLoop( dbis, cb ) \
openIsisDXLoop( OPENISIS_SES0(), dbis, cb )
/* system control (main session only) */
#define openIsisLog openIsisCLog
#define OpenIsisLockFunc OpenIsisCLockFunc
#define openIsisInit( argc, argv, lock ) \
openIsisCInit( argc, argv, lock )
#define openIsisSesGet( id, args ) \
(id) ? openIsisCSession( args ) : openIsisCOpen( args )
/* database control (main session only) */
#define openIsisOpen openIsisCDOpenv
#define openIsisCheck openIsisCDCheck
#define openIsisClose openIsisCDClose
/* database index control (main session only) */
#define openIsisIdxOpen openIsisCXOpen
#define openIsisIdxAdd openIsisCXAdd
#define openIsisIdxDone( idx ) openIsisCXClose( idx )
#endif /* OPENISIS_NOPRE09 */
#ifdef __cplusplus
}
#endif /* extern "C" */
#define OPENISIS_H
#endif /* OPENISIS_H */
openjsis.c 0100644 0001130 0001130 00000020622 07631453445 011546 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
/*
$Id: openjsis.c,v 1.16 2003/03/05 19:51:54 istr Exp $
main file of openjsis java native lib.
*/
#include /* free */
#include /* memcpy */
#include "openisis.h"
#define index stupidgcc
/*
java/jdk1.3/include/jni.h:607: warning: declaration of `index' shadows global declaration
jobject (JNICALL *GetObjectArrayElement)
(JNIEnv *env, jobjectArray array, jsize index);
*/
#include "org/openisis/NativeDb.h"
#undef index
/* ************************************************************
private types
*/
/* ************************************************************
private data
*/
/* ************************************************************
private functions
*/
static jobjectArray rec2bytes (JNIEnv* env, OpenIsisRec *r )
{
jobjectArray jarr = 0;
jintArray ia;
jint ji;
jbyteArray ba;
int i;
if ( ! r )
return 0;
jarr = (*env)->NewObjectArray( env, 1+r->len,
(*env)->FindClass( env, "java/lang/Object" ), 0 );
if ( ! jarr )
return 0;
ia = (*env)->NewIntArray( env, 1+r->len );
(*env)->SetObjectArrayElement( env, jarr, 0, ia );
ji = r->rowid;
(*env)->SetIntArrayRegion( env, ia, 0, 1, &ji );
for ( i=0; ilen; i++ ) {
OpenIsisField *f = &r->field[i];
ba = (*env)->NewByteArray( env, f->len );
if ( ! ba )
break;
ji = f->tag;
(*env)->SetIntArrayRegion( env, ia, i+1, 1, &ji );
(*env)->SetByteArrayRegion( env, ba, 0, f->len, (jbyte*)f->val );
(*env)->SetObjectArrayElement( env, jarr, i+1, ba );
}
return jarr;
} /* rec2bytes */
static OpenIsisRec* bytes2rec ( JNIEnv* env, jobjectArray arr ) {
jsize len;
jsize alen;
jintArray ia;
jbyteArray ba;
OpenIsisRec* rec;
OpenIsisSession ses;
int i;
jint ji;
jint* tags;
char *txt;
/* FIXME: real sid */
ses = OPENISIS_SES0();
/* initialize record */
rec = 0;
OPENISIS_RSPACE( ses, rec, 8*1024, -1 );
len = (*env)->GetArrayLength( env, arr );
if ( 1 > len )
return 0;
ia = (*env)->GetObjectArrayElement( env, arr, 0 );
if ( ! ia )
return 0;
(*env)->GetIntArrayRegion( env, ia, 0, 1, &ji );
rec->rowid = ji;
/* strip rowid / type description from length */
--len;
tags = malloc( len * sizeof( jint ) );
(*env)->GetIntArrayRegion( env, ia, 1, len, tags );
for ( i = 0; i < len; i++ ) {
ba = (*env)->GetObjectArrayElement( env, arr, i + 1 );
alen = (*env)->GetArrayLength( env, ba );
txt = malloc( alen );
memset( txt, 0, alen );
(*env)->GetByteArrayRegion( env, ba, 0, alen, (jbyte*)txt );
OPENISIS_RADD( ses, rec, tags[ i ], txt, alen, 0 );
if ( txt )
free( txt );
}
if ( tags )
free( tags );
return rec;
} /* bytes2rec */
/* ************************************************************
package functions
*/
/* ************************************************************
public functions
*/
JNIEXPORT jshort JNICALL Java_org_openisis_NativeDb_nopen
(JNIEnv* env, jclass cls, jstring jdbname, jobjectArray jargs)
{
int db;
const char *dbname = 0;
const char **argv = 0;
int argc = 0;
jsize i;
(void)cls;
dbname = ! jdbname ? 0 :
(*env)->GetStringUTFChars( env, jdbname, 0 );
if ( jargs ) {
argc = (*env)->GetArrayLength( env, jargs );
argv = malloc( argc * sizeof(*argv) );
if ( argv )
for ( i=0; iGetStringUTFChars( env,
(*env)->GetObjectArrayElement( env, jargs, i ), 0 );
}
db = openIsisOpen( dbname, argv, argc );
if ( dbname )
(*env)->ReleaseStringUTFChars( env, jdbname, dbname );
if ( argv )
for ( i=0; iReleaseStringUTFChars( env,
(*env)->GetObjectArrayElement( env, jargs, i ), argv[i] );
return db;
} /* Java_org_openisis_Db_open */
JNIEXPORT jobjectArray JNICALL Java_org_openisis_NativeDb_nreadRow
(JNIEnv* env, jclass cls, jshort db, jint rowid, jint tag, jbyteArray txt )
{
OpenIsisRec *r;
jobjectArray jarr = 0;
(void)cls;
if ( ! txt )
r = openIsisReadRow( db, rowid );
else {
jsize l = (*env)->GetArrayLength( env, txt );
char *search = malloc( l+1 );
(*env)->GetByteArrayRegion( env, txt, 0, l, (jbyte*)search );
search[l] = 0;
r = openIsisScan( db, rowid, tag, search );
free( search );
}
jarr = rec2bytes( env, r );
if ( r ) free( r );
#ifndef NDEBUG
fflush( stderr );
#endif
return jarr;
} /* Java_org_openisis_Db_readRow */
/*
* please leave me in for debugging ;)
* static int print ( OpenIsisSession ses, OpenIsisRec *r )
{
int i;
if ( r ) {
for ( i=0; ilen; i++ ) {
if ( ! r->field[i].val ) {
openIsisSMsg( ses, 1, "%d.?=%d\n", r->rowid, r->field[i].len );
continue;
}
openIsisSMsg( ses, 1, "%d.%d=%.*s\n", r->rowid, r->field[i].tag,
(int)r->field[i].len, r->field[i].val );
if ( r->field[i].len && '^' == *r->field[i].val ) {
OpenIsisRec *rf = openIsisReadField( 0, r->field+i );
if ( rf ) {
int j;
for ( j=0; jlen; j++ )
openIsisSMsg( ses, 1, "%d.%d.%c=%.*s\n",
r->rowid, r->field[i].tag,
(0x60 & (int)rf->field[j].tag ) ?
(int)rf->field[j].tag : ' ',
(int)rf->field[j].len, rf->field[j].val );
free( rf );
}
}
}
}
return ! r ? -1 : r->rowid;
} */
JNIEXPORT jshort JNICALL Java_org_openisis_NativeDb_nwriteRow
(JNIEnv* env, jclass cls, jshort db, jobjectArray arr )
{
OpenIsisRec *r;
jint result;
/* no warning */
(void) cls;
if ( ! arr )
return -1;
r = bytes2rec( env, arr );
if ( ! r )
return -1;
result = openIsisWrite( db, r );
if ( r )
free( r );
return result;
}
JNIEXPORT jshort JNICALL Java_org_openisis_NativeDb_nwriteXRow
(JNIEnv* env, jclass cls, jshort db, jobjectArray arr, jobjectArray idx )
{
OpenIsisRec *r;
OpenIsisRec *i;
jint result;
/* no warning */
(void) cls;
if ( ! arr || ! idx )
return -1;
r = bytes2rec( env, arr );
i = bytes2rec( env, idx );
if ( ! r || ! i )
return -1;
result = openIsisWritex( db, r, i );
if ( r )
free( r );
if ( i )
free( i );
return result;
}
JNIEXPORT jobjectArray JNICALL Java_org_openisis_NativeDb_nTerms
(JNIEnv* env, jclass cls, jshort db, jbyteArray term, jbyteArray start )
{
OpenIsisRec *r;
jobjectArray jarr = 0;
jsize lterm = (*env)->GetArrayLength( env, term );
char *cterm = malloc( lterm+1 );
union { OpenIsisRec r; char buf[4096]; } x;
x.r.len = 0;
x.r.bytes = sizeof(x);
(void)cls;
(*env)->GetByteArrayRegion( env, term, 0, lterm, (jbyte*)cterm );
cterm[lterm] = 0;
if ( start ) {
char *cstart;
jsize lstart = (*env)->GetArrayLength( env, start );
if ( 128 < lstart )
lstart = 128;
cstart = x.buf + sizeof(x.r);
(*env)->GetByteArrayRegion( env, start, 0, lstart, (jbyte*)cstart );
x.r.field[0].val = cstart;
x.r.field[0].len = lstart;
x.r.len = 1;
}
r = openIsisTerm( &x.r, db, cterm );
free( cterm );
jarr = rec2bytes( env, r );
#ifndef NDEBUG
fflush( stderr );
#endif
return jarr;
} /* Java_org_openisis_Db_Terms */
JNIEXPORT jintArray JNICALL Java_org_openisis_NativeDb_nsearch
(JNIEnv * env, jclass cls, jshort db, jbyteArray key, jint mode)
{
jsize l;
char *search;
OpenIsisSet set;
jintArray ia;
(void)cls;
if ( ! key )
return 0;
l = (*env)->GetArrayLength( env, key );
search = malloc( l+1 );
(*env)->GetByteArrayRegion( env, key, 0, l, (jbyte*)search );
search[l] = 0;
set.len = 0;
openIsisQuery( &set, db, search, mode, 0 );
free( search );
if ( 0 >= set.len )
return 0;
ia = (*env)->NewIntArray( env, set.len );
(*env)->SetIntArrayRegion( env, ia, 0, set.len, (jint*)set.id );
return ia;
} /* Java_org_openisis_Db_search */
openjsis.def 0100644 0001130 0001130 00000002605 07631453445 012063 0 ustar krip krip ;
; openisis - an open implementation of the ISIS database
; Version 0.8.3
; Copyright (C) 2001 by Erik Grziwotz, erik@openisis.org
;
; This library is free software; you can redistribute it and/or
; modify it under the terms of the GNU Lesser General Public
; License as published by the Free Software Foundation; either
; version 2.1 of the License, or (at your option) any later version.
;
; This library is distributed in the hope that it will be useful,
; but WITHOUT ANY WARRANTY; without even the implied warranty of
; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
; Lesser General Public License for more details.
;
; You should have received a copy of the GNU Lesser General Public
; License along with this library; if not, write to the Free Software
; Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
;
; This software is dedicated to the memory of Eckart Dietrich.
;
; This software is inspired by (but contains no code of) the iAPI
; Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
; See iAPI.txt for what it contains.
;EOH
; $Id: openjsis.def,v 1.4 2003/03/05 19:52:28 istr Exp $
; export definition file for win32 dll
;
EXPORTS
Java_org_openisis_NativeDb_nopen
Java_org_openisis_NativeDb_nreadRow
Java_org_openisis_NativeDb_nwriteRow
Java_org_openisis_NativeDb_nwriteXRow
Java_org_openisis_NativeDb_nsearch
Java_org_openisis_NativeDb_nTerms
org/openisis/Charset.java 0100644 0001130 0001130 00000013034 07572664272 014431 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
package org.openisis;
import java.io.*;
import java.util.Arrays;
/**
Charset tests and utils for openisis java binding.
$Id: Charset.java,v 1.2 2002/12/02 13:46:34 kripke Exp $
@version $Revision: 1.2 $
@author $Author: kripke $
*/
public class Charset {
static final String[] encodings = {
"ISO-8859-5", // cyrillic
"ISO-8859-7", // greek
"ISO-8859-9", // turkish
};
// bytes 160(=-96) to 255(=-1)
static final byte[] isohigh = new byte[96];
static {
for ( int b=0; b<96; b++ )
isohigh[b] = (byte)(b-96);
}
static void utf8seq ()
throws IOException
{
Writer w = new OutputStreamWriter( System.out, "UTF-8" );
for ( int i=0; i r.getLen() || null == r.getField(0) || null == r.getField(1) ) {
System.err.println( "bad fields on mfn "+n[0]+" for encoding "+e );
continue;
}
Field f = r.getField(0);
if ( 1 != f.tag || !e.equals( f.val ) ) {
System.err.println( "bad field 0 "+f.tag+"="+f.val
+" on mfn "+n[0]+" for encoding "+e );
continue;
}
f = r.getField(1);
if ( 2 != f.tag ) {
System.err.println( "bad field 1 "+f.tag+"="+f.val
+" on mfn "+n[0]+" for encoding "+e );
continue;
}
if ( ! f.val.equals( new String(isohigh,e) ) ) {
System.err.println( "UTF-8 mismatch on mfn "+n[0]+" for encoding "+e );
continue;
}
byte[] b = f.val.getBytes(e);
if ( ! Arrays.equals( b, isohigh ) ) // ok for some encodings
System.err.println( "note: no round-trip for encoding "+e );
}
}
public final String enc;
public Charset ( String encoding ) { enc = encoding; }
public OutputStreamWriter osw ( OutputStream s )
throws IOException
{
return new OutputStreamWriter( s, enc );
}
public void chartab ( OutputStreamWriter w )
throws IOException
{
if ( null == w )
w = new OutputStreamWriter( System.out );
w.write( "... 0 1 2 3 4 5 6 7 8 9 A B C D E F\n" );
byte[] b = new byte[32];
for ( int l=0; l<8; l++ ) {
for ( int j=0; j<16; j++ ) {
b[2*j] = (byte)' ';
b[2*j+1] = (byte)(l*16 + j - 128);
}
String s = new String( b, enc );
w.write( (128+l*16) + s + "\n" );
// char[] c = s.toCharArray[];
}
w.flush();
}
public void codetab ( OutputStreamWriter w )
throws IOException
{
if ( null == w )
w = new OutputStreamWriter( System.out );
byte[] b = new byte[16];
for ( int l=0; l<8; l++ ) {
for ( int j=0; j<16; j++ )
b[j] = (byte)(l*16 + j - 128);
String s = new String( b, enc );
char[] c = s.toCharArray();
for ( int j=0; j\n"
+ "\n"
);
byte[] b = new byte[1];
for ( b[0] = -96; b[0]<-1; b[0]++ ) {
w.write( (256+(int)b[0]) + " = "
+ new String( b, enc ) + "\n" );
}
w.write( " " );
w.flush();
}
public static void main ( String[] args )
throws IOException
{
String enc = "Cp850";
char task = 's';
int a = 0;
while ( a
$Id: Db.java,v 1.5 2003/03/05 19:52:28 istr Exp $
@version $Revision: 1.5 $
@author $Author: istr $
*/
public abstract class Db {
/** the default encoding.
* the constant String value is "ISO-8859-1".
*/
public static final String DEFAULT_ENCODING = "ISO-8859-1";
/* standard empty String returned by several methods */
public static final String EMPTY = "";
/** index scan key equal */
public static final int QRY_KEYEQ = 0;
/** index scan key prefix */
public static final int QRY_KEYPF = 1;
public static final int PREFIX = QRY_KEYPF; // legacy alias
/** index scan key auto (checks for '$') */
public static final int QRY_KEYAT = 2;
/** fulltext scan for equal */
public static final int QRY_SCANE = 64;
/** fulltext scan for contains */
public static final int QRY_SCANC = 65;
/** query expression with simple left-to-right binding */
public static final int QRY_SIMPLE = 128;
/** query expression with proper binding using precedence, () */
public static final int QRY_PROPER = 129;
/** field v: heading mode */
public static final int MHL = 0x10000;
/** field v: data flag (no effect w/o heading) */
public static final int MD = 0x20000;
/** field v: data mode (includes heading flag) */
public static final int MDL = 0x30000;
/** field v: uppercase mode flag */
public static final int MXU = 0x40000;
/** field v: uppercase heading mode */
public static final int MHU = 0x50000;
/** field v: uppercase data mode */
public static final int MDU = 0x70000;
/** field v: index formatting flag */
public static final int MI = 0x80000;
/** field v: 16bit UNICODE HTML. HTML-entities are escaped. */
public static final int HTU = 0x100000;
/** field v: 8bit ISO-8859-1 HTML.
HTML-entities and all chars greater than 255 are escaped.
*/
public static final int HTI = 0x300000;
/** field v: 7bit ASCII HTML.
HTML-entities and all chars greater than 127 are escaped.
*/
public static final int HTA = 0x700000;
/** rec v: repeated prefix plus operator (not on 1st occ) */
public static final int PP = 0x2000000;
/** rec v: repeated suffix plus operator (not on last occ) */
public static final int PS = 0x1000000;
/** rec v: dummy presence selector Dn */
public static final int DS = 0x4000000;
/** rec v: dummy absence selector Nn */
public static final int NS = 0x8000000;
/** rec v: occurence range up to last occ */
public static final int LOCC = 0xffff << 16;
static final Map _dbs = new HashMap();
/** open an isis database.
The following options are supported as text-style parameters:
-db <name>
-dbpath <path>
-v <level>
-encoding <enc>
-puredb
@param dbname basename of database.
the dbhome arg, if given, will be prepended, and the ISIS file
extensions appended to build the actual filename.
Therefore, depending on the value of dbhome and location
of files, this may need to include a path.
db may be null, if argv includes a dbname arg.
@param argv array of names and values.
May be null.
argv contains parameter names, which may be prefixed
by a dash '-'. Depending on the actual parameter, the next
string may or must contain a corresponding parameter value.
An optional parameter value may be omitted, if at end of
argv or the next name is prefixed with a dash.
To avoid ambiguity, values never start with a dash.
See above for a list of supported parameter names.
Unrecognized parameter names are ignored.
@return a new Db (unless some RuntimeException is thrown)
@throws IoException especially FileNotFoundExceptiond
and UnsupportedEncodingException
*/
public static synchronized Db open ( String dbname, String[] args )
throws IOException
{
String enc = DEFAULT_ENCODING;
boolean usePureDb = false;
Db db = (Db) _dbs.get( dbname );
if ( null != db )
return db;
if ( null != args )
for ( int i=0; i
$Id: Fdt.java,v 1.4 2003/03/31 09:52:53 kripke Exp $
@version $Revision: 1.4 $
@author $Author: kripke $
*/
public class Fdt {
/** alphanum field type. */
public static final int FTX = 0;
/** strictly alpha field type. */
public static final int FTA = 1;
/** numeric field type. */
public static final int FTN = 2;
/** pattern field type. */
public static final int FTP = 3;
/** iso field type. */
public static final int FTI = 10;
/** enum field type. */
public static final int FTE = 12;
/** boolean field type. */
public static final int FTB = 13;
/** table field type. */
public static final int FTT = 14;
/** structure field type. */
public static final int FTS = 15;
/** subfield field type. */
public static final int FTF = 16;
/** enum value type. */
public static final int FTV = 31;
/** field description. */
public static class Fd {
/** tag */
public final int id;
/** type */
public final int type;
/** max length or enum value */
public final int len;
/** repeated flag */
public final boolean rep;
/** name */
public final String name;
/** subfields/pattern */
public final String subf;
/** description */
public final String desc;
/** @exception IllegalArgumentException if name is null or ""
*/
public Fd (int id, int type, int len, boolean rep,
String name, String subf, String desc) {
this.id = id;
this.type = type;
this.len = len;
this.rep = rep;
this.name = name;
this.subf = subf;
this.desc = desc;
if (null == name || 0 == name.length ()) {
throw new IllegalArgumentException (
"illegal field desc: id " + id + ", name " + name);
}
}
/** shorthand for Fd (id, type, len, false, name, null, null)
*/
public Fd (int id, int type, int len, String name) {
this (id, type, len, false, name, null, null);
}
public String toString () {
return name + ';' + id + ';' + type;
}
}
public final Fd[] _fd;
/** @exception IllegalArgumentException if fd is null or has length 0
*/
public Fdt (Fd[] fd) {
_fd = fd;
if (null == fd || 0 == fd.length) {
throw new IllegalArgumentException ();
}
}
public Fd fdById (int id) {
for (int j = _fd.length; 0 <= --j; ) {
if (id == _fd[j].id) {
return _fd[j];
}
}
return null;
}
public Fd fdByName (String name) {
if (null == name || 0 == name.length ()) {
return null;
}
char c = name.charAt (0);
if ('0' <= c && '9' >= c) {
try {
int id = Integer.parseInt (name);
return fdById (id);
}
catch (Exception ex) {
}
}
for (int j = _fd.length; 0 <= --j; ) {
if (name.equals (_fd[j].name)) {
return _fd[j];
}
}
return null;
}
public String dump() {
StringBuffer d=new StringBuffer();
for(int i=0;i<_fd.length;i++) {
d.append(_fd[i].toString());
if (_fd[i].subf!=null) {
d.append(',');
d.append(_fd[i].subf);
}
d.append('\n');
}
return d.toString();
}
}
org/openisis/Field.java 0100644 0001130 0001130 00000017452 07642010026 014047 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2003 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
package org.openisis;
import java.util.StringTokenizer; // split
/**
Field implementation of openisis java binding.
A Field is an immutable structure, binding a String value to a tag.
It provides methods for formatted access to the value.
$Id: Field.java,v 1.4 2003/03/31 09:52:54 kripke Exp $
@version $Revision: 1.4 $
@author $Author: kripke $
*/
public class Field {
/**
what a single field can contribute to the V operator.
@param b a StringBuffer to append to
@param mode OR-combination of the field formatting flags
defined in Db and an optional subfield character.
@param plain a String to format
@param len a 16bit length constraint | offset << 16
@return false iff a non-existing subfield was requested, else true
*/
public static boolean v ( StringBuffer b, int mode, String plain, int len ) {
if ( 0 != (Db.MXU & mode) ) // won't change any specials
plain = plain.toUpperCase();
final char[] c = plain.toCharArray();
final int cl=c.length;
char sub = (char)mode;
int o=0, e=cl;
if ( 0 != sub ) {
if ( '*' == sub ) { // first subfield
if ( e > 1 && '^' == c[0] )
o = 2;
} else {
if ( 0 != (Db.MXU & mode) ) // oops - upcased subfield marks ...
sub = Character.toUpperCase(sub);
for (;;) {
if ( cl == o ) return false; // no subfield
if ( '^' != c[o++] ) continue;
if ( cl == o ) return false; // no subfield
if ( sub == c[o++] ) break;
}
}
// got sub starting at o
e = o;
while ( cl > e && '^' != c[e] )
e++;
}
// now the relevant data is at offset o and ends at e
if ( 0 != len ) {
int off = len >>> 16;
len |= 0xffff;
o += off;
if ( 0 != len && e > o+len )
e = o+len;
}
if ( e <= o )
return true; // existing but empty
boolean mh = Db.MHL == (Db.MHL & mode);
boolean mi = Db.MI == (Db.MI & mode);
boolean ht = Db.HTU == (Db.HTU & mode);
boolean ab = false; // inside angel brackets: stop at =
if ( ! (mh || ht) ) {
b.append( c, o, e-o );
return true;
}
final char htlim = // chars > htlim are escaped
( Db.HTA == (Db.HTA & mode) ) ? 127 :
( Db.HTI == (Db.HTI & mode) ) ? 255 :
Character.MAX_VALUE;
int i=o;
char x = 0;
// now we run at least one of mh or ht mode replacements
LOOP: for (;;i++) {
if ( i < e && htlim >= (x = c[i]) )
switch ( x ) {
case '"': case '&': if ( ht ) break; continue;
case '^': if ( mh ) break; continue;
case '=': if ( ab ) break; continue;
case '<': case '>': break; // special in any case
default: continue;
}
if ( i > o ) // flush what we had up to current position
b.append( c, o, i-o );
if ( i == e ) // done
break;
o = i+1; // new offset after this
if ( htlim < x ) { // write decimal value
b.append( "&#" ).append( (int)x ).append( ';' );
continue;
}
// now we have one of the special chars
switch ( x ) {
case '"': b.append( """ ); continue; // ht
case '&': b.append( "&" ); continue; // ht
case '^': // mh
if ( i+1 < e ) {
x = Character.toUpperCase( c[++i] );
if ( 1 < o ) // not on beginning of field
b.append( ('A' == x) ? ';' : ('A'' != c[i] ) if ( e == ++i ) break LOOP; // ran to end
o = i+1;
// don't break, go on checking for ><-pair
case '>':
ab = false;
if ( ! mh )
b.append( ">" );
else if ( i+1
while ( '>' != c[i] && '=' != c[i] )
if ( e == ++i ) break LOOP; // ran to end
o = i+1;
break;
}
}
if ( Db.MDL == (Db.MDL & mode) ) // data mode
switch ( Character.getType( c[e-1] ) ) {
case Character.START_PUNCTUATION:
case Character.END_PUNCTUATION:
case Character.CONNECTOR_PUNCTUATION:
case Character.OTHER_PUNCTUATION:
b.append( " " );
break;
default:
b.append( ". " );
}
return true;
} // v
/** format a String in HTU-mode.
@see #v(StringBuffer,int,String,int)
*/
public static String html ( String plain ) {
StringBuffer b = new StringBuffer( plain.length() + 3 );
v( b, Db.HTU, plain, 0 ); // always true w/o subfield spec
return b.toString();
} // html
//
// so far the statics
//
/** the tag of this field.
For primary fields, this is the field number.
For subfields, this is actually a char.
*/
public final int tag;
public final String val;
public Field ( int tag_, String val_ ) {
tag = tag_;
val = (val_==null)?"":val_;
}
public boolean equals (Object that) {
if (null == that) {
return false;
}
if (getClass () != that.getClass ()) {
return false;
}
String v1 = val; if (null == v1) v1 = "";
String v2 = ((Field)that).val; if (null == v2) v2 = "";
return tag == ((Field)that).tag && v1.equals (v2);
}
public String toString () {
return tag + ";" + val;
}
/** split the Field into subfields.
*/
public Field[] split () {
if ( null == val || 0 == val.length() )
return null;
String tval=val;
int p;
if ((p=tval.indexOf('^'))<0) return null;
if (p>0) tval=tval.substring(p+1,tval.length());
StringTokenizer st = new StringTokenizer( tval, "^" );
Field[] f = new Field[ st.countTokens() ];
for ( int i=0; i
$Id: Mail.java,v 1.3 2002/12/04 01:23:47 kripke Exp $
@version $Revision: 1.3 $
@author $Author: kripke $
*/
public abstract class Mail {
public static final String EMPTY = "";
public static final String MESSAGEID = "message-id";
public static final String DATE = "date";
public static final String TO = "to";
public static final String CC = "cc";
public static final String BCC = "bcc";
public static final String FROM = "from";
public static final String SENDER = "sender";
public static final String REPLYTO = "reply-to";
public static final String SUBJECT = "subject";
public static final String REFERENCES = "references";
public static final String INREPLYTO = "in-reply-to";
public static final String KEYWORDS = "keywords";
public static final String COMMENTS = "comments";
public static final String ENCRYPTED = "encrypted";
public static final String PRECEDENCE = "precedence"; // non-standard
public static final String XPRIORITY = "x-priority"; // non-standard
public static final String RETURNPATH = "return-path";
public static final String DLVRDTO = "delivered-to"; // qmail
public static final String RECEIVED = "received";
public static final String XMAILER = "x-mailer";
public static final String CTYPE = "content-type";
public static final String CTRANSENC = "content-transfer-encoding";
public static final String TEXT_PLAIN = "text/plain";
public static final String TEXT_ISO = "text/plain; charset=\"iso-8859-1\"";
public static final String TEXT_HTML = "text/html";
public static final String MESS_MIME = "message/rfc822";
public static final String MULT_MIXED = "multipart/mixed";
public static final String MULT_ALT = "multipart/alternative";
public static final String MULT_APPLE = "multipart/appledouble";
public static final String MULT_SIGN = "multipart/signed";
public static final String MULT_ANY = "multipart/*";
public static final String APPL_WORD = "application/msword";
public static final String APPL_PDF = "application/pdf";
public static final String APPL_RTF = "application/rtf";
public static final String APPL_BYTES = "application/octet-stream";
public static final String APPL_APPLE = "application/applefile";
public static final String APPL_ANY = "application/*";
public static final String IMAG_GIF = "image/gif";
public static final String IMAG_JPEG = "image/jpeg";
public static final String IMAG_ANY = "image/*";
public static final String SMTP_HOST = "mail.smtp.host";
public static final Session SES;
static {
if ( null == System.getProperty( SMTP_HOST ) )
System.setProperty( SMTP_HOST, "localhost" );
SES = Session.getDefaultInstance( System.getProperties(), null );
}
public static interface Source {
Message[] get () throws Exception;
void done ( boolean expunge ) throws Exception;
}
public static interface Sink {
void put ( Message m ) throws Exception;
void put ( Message[] m ) throws Exception;
}
// //////////////////////////////////////////////////
// utilities
//
public static void del ( Message m )
throws Exception
{
m.setFlag( Flags.Flag.DELETED, true );
} // del
public static void del ( Message[] m )
throws Exception
{
for ( int i=0; i" ) );
}
}
public Dir ( String base )
throws Exception
{
_t = base + File.separator + "tmp";
_n = base + File.separator + "new";
_c = base + File.separator + "cur";
_ndir = new File(_n);
File t = new File(_t);
File c = new File(_c);
if ( ! t.isDirectory() && ! t.mkdirs() )
throw new FileNotFoundException( _t );
if ( ! _ndir.isDirectory() && ! _ndir.mkdirs() )
throw new FileNotFoundException( _n );
if ( ! c.isDirectory() && ! c.mkdirs() )
throw new FileNotFoundException( _c );
}
public String toString () {
return "Dir "+_ndir.getParentFile().getAbsolutePath();
}
public Message[] get () throws Exception
{
File[] l = _ndir.listFiles();
System.err.println( _ndir+": "+l.length );
int i = l.length;
if ( 0 != _lim && i > _lim )
i = _lim;
got = new Msg[i];
while ( 0 < i-- ) {
File f = new File( _t+File.separator+l[i].getName() );
l[i].renameTo( f );
got[i] = new Msg( f );
}
return got;
}
public void done ( boolean expunge ) throws Exception
{
if ( expunge && null != got )
for ( int i=0; i" );
System.err.println( _b+" += "+mid );
Date d = m.getSentDate();
if ( null == d )
d = m.getReceivedDate();
if ( null == d ) {
System.err.println( "warning: no date found in "+mid );
d = new Date();
}
String base = path( d );
String path = base;
File f = new File( _b + path );
File dir = f.getParentFile();
if ( ! dir.isDirectory() && ! dir.mkdirs() )
throw new FileNotFoundException( dir.getAbsolutePath() );
for ( int i=0; ! f.createNewFile(); i++ )
f = new File( _b + (path = base + Integer.toString( i, 36 )) );
String name = f.getName();
// got unique path path
System.out.println( "new message "+mid+" date "+d+" -> "+path );
MimeMessage mm = new MimeMessage( SES );
Enumeration e = m.getAllHeaders();
while ( e.hasMoreElements() ) {
Header h = (Header)e.nextElement();
String n = h.getName().toLowerCase();
if ( OK == OKHDR.get(n) )
mm.addHeader( n, h.getValue() );
}
mm.addHeader( CTYPE, TEXT_ISO );
mm.addHeader( CTRANSENC, "8bit" );
StringBuffer b = new StringBuffer( 4096 );
Stack s = new Stack();
s.push( m );
int i = 0;
int totlines = 0;
List binparts = new ArrayList( 32 );
while ( ! s.isEmpty() ) {
Part p = (Part)s.pop();
String ct = oneline( p.getContentType() );
// ct = null == ct ? TEXT_PLAIN : ct.toLowerCase();
if ( p.isMimeType( MULT_ANY ) ) {
Object content = p.getContent();
if ( ! (content instanceof Multipart) ) {
// ignore
System.err.println( "multipart type "+ct
+" has class "+content.getClass().getName() );
mm.addHeader( COMMENTS, "dropped bad multipart "+ct );
} else {
Multipart mu = (Multipart)content;
int cnt = mu.getCount();
if ( cnt > 8 ) {
System.err.println( "multipart with too many parts: "+cnt );
cnt = 8;
}
Part[] pu = new Part[cnt];
for ( int j=0; j 0; ) // push parts backwards
s.push( pu[j] );
}
continue;
} // multi
i++; // about to create part i
String part = path+'-'+i;
// if ( ! p.isMimeType( TEXT_PLAIN ) ) {
/*
it's so awfully unreliable ... many RTFs go as APPL_WORD ...
if ( p.isMimeType( TEXT_HTML ) )
ext = ".html";
else if ( p.isMimeType( APPL_WORD ) )
ext = ".doc";
else
ext = "";
*/
OutputStream o = new FileOutputStream( _b+part );
p.getDataHandler().writeTo( o );
o.close();
// can we convert it ?
String[] args = new String[] { "any2txt", "-v", name+'-'+i };
Process fix = Runtime.getRuntime().exec( args, null, dir );
BufferedReader err = new BufferedReader( new InputStreamReader(
fix.getErrorStream() ) );
String line;
while ( null != (line = err.readLine()) )
System.err.println( "any2txt:\t"+line );
err.close();
fix.waitFor();
int ret = fix.exitValue();
String ext = EMPTY;
boolean havetext = 0 != (ret & 1) || 0 == ret;
int type = ret & ~1;
switch ( type ) {
case 0: ext = ".txt"; break;
case 2: ext = ".html"; break;
case 4: ext = ".pdf"; break;
case 6: ext = ".rtf"; break;
case 8: ext = ".doc"; break;
case 16: // mail -- reread and start over
try {
s.push( new MimeMessage( SES, new FileInputStream( _b+part ) ) );
// TODO: delete
continue;
} catch (Exception ee) {
System.err.println( "could not reload mail "+mid+"["+i+"] "+ct
+" from "+part );
ee.printStackTrace();
}
ext = ".eml";
break;
case 32: ext = ".gif"; break;
case 34: ext = ".jpg"; break;
case 36: ext = ".bmp"; break;
case 38: ext = ".png"; break;
case 40: ext = ".tiff"; break;
}
System.err.println( mid+"["+i+"] "+ct+" is "+ext+" ("+ret+")" );
if ( EMPTY != ext && 1 != ret ) // rename
(new File(_b+part)).renameTo( new File(_b+part+ext) );
if ( 0 != type ) {
binparts.add( name+'-'+i+ext );
// mm.addHeader( COMMENTS, "part "+i+" "+ct+" is "+part );
if ( ! havetext ) {
if ( ! _zip ) {
if ( 0 < totlines )
b.append( "\r\n---\r\n\r\n" );
b.append( "\r\nget part "+i+" at "+_u+part+ext+"\r\n" );
}
totlines++;
continue;
}
}
// }
// now _b+part.txt contains plaintext
File plain = new File(_b+part+".txt");
BufferedReader br = new BufferedReader(
new InputStreamReader( new FileInputStream( plain ) ) );
int lines = 0;
int max = _maxlines - totlines;
if ( max < _minlines )
max = _minlines;
// String line;
if ( 0 < totlines )
b.append( "\r\n---\r\n\r\n" );
while ( null != (line = br.readLine()) ) {
b.append( line ).append( "\r\n" );
if ( max == ++lines ) {
mm.addHeader( COMMENTS, "plaintext "+i+" continued at "+part );
b.append( "... read more: "+_u+part+".txt ("
+(1 + plain.length()/1024)+" KB)\r\n" );
break;
}
}
if ( 0 != type && !_zip )
b.append( "\r\nget original part "+i+" at "+_u+part+ext+"\r\n" );
totlines += lines;
br.close();
}
int nparts = binparts.size();
if ( _zip && 0 != nparts ) {
String[] args = new String[3+nparts];
int a=0;
args[a++] = "zip";
args[a++] = "-m"; // move
args[a++] = name+".zip";
for ( int p = 0; pInstallation
The shared library "openjsis" must be loadable.
On a UNIX System, make sure that the file libopenjsis.so
resides in a directory listed by the environment variable
LD_LIBRARY_PATH or in /etc/ld.so.conf.
If you just drop it in /usr/lib or similar dir,
you may have to run a command like ldconfig.
On Windows, put openjsis.dll somewhere in the PATH.
$Id: NativeDb.java,v 1.4 2003/03/05 19:52:29 istr Exp $
@version $Revision: 1.4 $
@author $Author: istr $
*/
public class NativeDb extends Db {
private static synchronized native
short nopen ( String dbname, String[] args );
/** read a record by rowid.
* The native implementation gives character data as byte[],
* since we need to employ java encoding mechanisms anyway.
* The tag is packed as four bytes little endian.
* @param db a db number as returned by open
* @param rowid the number (aka mfn) of the row
* @param tag in which to look for txt, negative for any
* @param txt to grep from record, null for no grep
* @return an array of Objects:
* the first is an array of ints filled with rowid an tags
* the others are byte[] with field character data in encoding
* @see #encoding
*/
private static synchronized native
Object[] nreadRow ( short db, int rowid, int tag, byte[] txt );
/** search for terms */
private static synchronized native
Object[] nTerms ( short db, byte[] term, byte[] start );
/** get matching rowids */
private static synchronized native
int[] nsearch ( short db, byte[] key, int mode );
/** plain write */
private static synchronized native
short nwriteRow ( short db, Object[] data );
/**
* Indexed write
* @param db database handle returned by open
* @param data an Object array with an int[] on idx 0 containing
* rowid and tags and with byte[] containing appropriate data for
* the tags; thus a simple sanity check for a properly structured
* data is: data.length == data[ 0 ].length
* @param idx same as data but contains a record that holds index
* information
*/
private static synchronized native
short nwriteXRow ( short db, Object[] data, Object[] idx );
static {
// TODO: try some explicit paths like our jar's location with System.load
// in order to not have to export LD_LIBRARY_PATH or so
System.loadLibrary( "openjsis" );
}
public byte[] cvt ( String txt ) {
try {
return txt.getBytes(encoding);
} catch ( UnsupportedEncodingException e ) {
// shouldn't happen -- our fault
throw new Error( "failed to check for support of encoding "+encoding );
}
}
public String cvt ( byte[] b ) {
try {
// System.err.println( "got "+b.length+" bytes "+b );
for ( int l=b.length; 0 0 ? "0" : // 0... self-contained
b[i] < -64 ? "f" : // 10... six follower bits
b[i] < -32 ? "1" : // 110... five bits, one follower
b[i] < -16 ? "2" : // 1110... four bits, two followers
"b" // bad
) + "("+b[i]+")"
);
return "";
}
return "";
} catch ( UnsupportedEncodingException e ) {
// shouldn't happen -- our fault
throw new Error( "failed to check for support of encoding "+encoding );
}
}
/** Number of this db.
*/
public final short db;
public NativeDb ( String enc, String dbname, String[] args )
throws IOException
{
super( enc );
db = nopen( dbname, args );
if ( 0 > db )
throw new FileNotFoundException();
}
Rec mkRow ( Object[] buf ) {
if ( null == buf )
return null;
int[] ids = (int[])buf[0];
int rowid = ids[0];
Field[] f = new Field[ buf.length-1 ];
for ( int i=1; i= buf.length ) {
buf = null;
return false;
}
nxt = 1;
return true;
}
public Object next () {
if ( ! hasNext() ) throw new NoSuchElementException();
return cvt( (byte[])buf[nxt++] );
}
public void remove () { throw new UnsupportedOperationException(); }
}
public Iterator terms ( String prefix ) {
return new TermIterator( prefix );
}
} // NativeDb
org/openisis/PlainSerializer.java 0100644 0001130 0001130 00000023534 07642121653 016130 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2003 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
package org.openisis;
import java.io.*;
import java.net.Socket;
import java.util.ArrayList;
/**
Serialization of fields by plain protocol.
$Id: PlainSerializer.java,v 1.3 2003/03/31 15:43:36 mawag Exp $
@version $Revision: 1.3 $
@author $Author: mawag $
*/
public class PlainSerializer {
private static class ProtocolViolation extends RuntimeException {
ProtocolViolation () {
super ();
}
ProtocolViolation (String msg) {
super (msg);
}
ProtocolViolation (byte arr[]) {
super ("got <" + new String (arr) + ">");
}
}
private static final byte[] NL = new byte [] { (byte)'\n' };
private static final byte[] TAB = new byte [] { (byte)'\t' };
private static PrintStream _Log = System.err;
private InputStream _in;
private OutputStream _out;
private boolean _dbg = false;
/** where to write the error messages to.
default is stderr.
*/
public static void SetLogStream (PrintStream log) {
if (null != log) {
_Log = log;
}
}
private void warn (String msg, Throwable ex) {
_Log.println ("WARN [" + this + "] " + msg + ": " + ex);
}
private void debug (String rw, byte[] arr, int len) {
_Log.println (rw + " <" + new String (arr, 0, len) + ">");
}
private void _write (byte[] arr) throws Exception {
if (_dbg) {
debug ("w", arr, arr.length);
}
_out.write (arr);
}
private int _read (byte[] arr) throws Exception {
int rd = _in.read (arr);
if (_dbg) {
debug ("r", arr, rd);
}
return rd;
}
/** write end of record to output stream.
@return success
*/
public boolean flush () {
try {
_write (NL);
return true;
}
catch (Exception ex) {
warn ("flush", ex);
}
return false;
}
/** serialize one field.
@return success
*/
public boolean write (Field fld) {
int idx, lst, len;
byte[] val;
// MMM TBD encoding
if (null == fld) {
return false;
}
try {
val = Integer.toString (fld.tag).getBytes ();
_write (val);
_write (TAB);
if (null != fld.val && 0 < (len = fld.val.length ())) {
for (lst = idx = 0;
0 <= (idx = fld.val.indexOf (NL[0], lst));
lst = idx
) {
++idx;
val = fld.val.substring (lst, idx).getBytes ();
_write (val);
_write (TAB);
}
if (lst < len) {
val = fld.val.substring (lst).getBytes ();
_write (val);
}
}
_write (NL);
return true;
}
catch (Exception ex) {
warn ("serializing " + fld, ex);
}
return false;
}
/** serialize fields.
@return success
*/
public boolean write (Field[] flds) {
if (null == flds) {
return false;
}
for (int j = 0; flds.length > j; ++j) {
if (! write (flds[j])) {
return false;
}
}
return flush ();
}
/** deserialize fields.
@return array of fields read from input stream, null on error
*/
public Field[] read () {
byte[] buf = new byte [1024];
StringBuffer str = new StringBuffer (1024);
ArrayList lst = new ArrayList ();
Field fld;
int rd, pos, tag, vs, val;
boolean rdt; // reading tag
boolean lnl; // last byte was NL
boolean neg; // tag negative
// MMM TBD encoding
try {
tag = 0;
rdt = true;
neg = false;
lnl = false;
all:
while (true) {
rd = _read (buf);
if (0 >= rd) {
throw new ProtocolViolation ("no more bytes available");
}
vs = pos = 0;
while (pos < rd) {
if (rdt) {
if (TAB[0] == buf[pos]) {
rdt = false;
vs = ++pos;
continue;
}
if (0 == tag && '-' == buf[pos]) {
neg = true;
++pos;
continue;
}
val = buf[pos] - '0';
if (0 > val || 9 < val) {
if (0 == pos &&
NL[0] == buf[0] &&
0 == lst.size ()) {
warn ("", new ProtocolViolation (
"empty response"));
break;
}
throw new ProtocolViolation (buf);
}
tag = 10 * tag + val;
++pos;
continue;
}
if (lnl) {
lnl = false;
if (TAB[0] == buf[pos]) {
str.append ("\n");
vs = ++pos;
continue;
}
fld = new Field (neg ? - tag : tag, str.toString ());
lst.add (fld);
if (NL[0] == buf[pos]) {
if (pos != rd - 1) {
throw new ProtocolViolation ("trailing bytes");
}
break all;
}
str.setLength (0);
rdt = true;
neg = false;
tag = 0;
continue;
}
if (NL[0] == buf[pos]) {
if (pos > vs) {
str.append (new String (buf, vs, pos - vs));
}
lnl = true;
}
++pos;
}
if (! rdt && rd > vs) {
str.append (new String (buf, vs, rd - vs));
}
}
return (Field[]) lst.toArray (new Field [lst.size ()]);
}
catch (Exception ex) {
ex.printStackTrace (_Log);
warn ("read", ex);
}
return null;
}
/** Constructor with read and write stream.
*/
public PlainSerializer (InputStream in, OutputStream out) {
_in = in;
_out = out;
}
/** Constructor for a socket connection to port@host.
@exception all exceptions a socket might throw
*/
public PlainSerializer (String hostname, int port) throws Exception {
try {
Socket sock = new Socket (hostname, port);
_in = sock.getInputStream ();
_out = sock.getOutputStream ();
}
catch (Exception ex) {
warn ("(" + hostname + ';' + port + ")", ex);
throw ex;
}
}
/** Close communication channels.
*/
public void close () {
if (null != _in) {
try {
_in.close ();
}
catch (Exception ex) {
}
_in = null;
}
if (null != _out) {
try {
_out.close ();
}
catch (Exception ex) {
}
_out = null;
}
}
// //////////////////////////////////////////////////////////////////////
private class PSWriter extends Thread {
Field[] _rqs;
int _numr;
PSWriter (Field[] rqs, int numr) {
_rqs = rqs;
_numr = numr;
}
public void run () {
for (int j = 0; _numr > j; ++j) {
if (! write (_rqs)) {
warn ("error writing rqs #" + j, null);
System.exit (1);
}
try {
synchronized (this) {
wait ();
}
}
catch (Exception ex) {
warn ("wait", ex);
}
}
}
}
/** Tester.
[ []]] | -p []");
System.exit (1);
}
}
}
org/openisis/PureDb.java 0100644 0001130 0001130 00000004604 07631453453 014215 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
package org.openisis;
import java.io.IOException;
import java.io.FileNotFoundException;
import java.util.Iterator;
/**
This class is the pure java implementation of class Db.
Unfortunately, it's still empty :(
$Id: PureDb.java,v 1.3 2003/03/05 19:52:29 istr Exp $
@version $Revision: 1.3 $
@author $Author: istr $
*/
public class PureDb extends Db {
public PureDb ( String enc, String dbname, String[] args )
throws IOException
{
super( enc );
throw new UnsupportedOperationException( "sorry, not implemented" );
}
public Rec readRow ( int rowid ) {
throw new UnsupportedOperationException( "sorry, not implemented" );
} // readrow
public short writeRow ( Rec row ) {
throw new UnsupportedOperationException( "sorry, not implemented" );
} // writerow
public short writeXRow ( Rec row, Rec idx ) {
throw new UnsupportedOperationException( "sorry, not implemented" );
} // writerow
public Rec scanRow ( int rowid, int tag, String txt ) {
throw new UnsupportedOperationException( "sorry, not implemented" );
} // scanrow
public int[] search ( String key, int mode ) {
throw new UnsupportedOperationException( "sorry, not implemented" );
} // search
public Iterator terms ( String prefix ) {
throw new UnsupportedOperationException( "sorry, not implemented" );
}
} // PureDb
org/openisis/Rec.java 0100644 0001130 0001130 00000016135 07642010026 013532 0 ustar krip krip /*
openisis - an open implementation of the ISIS database
Version 0.8.x (microversion see file Version)
Copyright (C) 2001-2002 by Erik Grziwotz, erik@openisis.org
This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
This software is dedicated to the memory of Eckart Dietrich.
This software is inspired by (but contains no code of) the iAPI
Copyright (C) 2000 by Robert Janusz, rj@jezuici.krakow.pl.
See iAPI.txt for what it contains.
EOH */
package org.openisis;
/**
Record implementation of openisis java binding.
$Id: Rec.java,v 1.3 2003/03/31 09:52:54 kripke Exp $
@version $Revision: 1.3 $
@author $Author: kripke $
*/
public class Rec {
private static final int APPEND_LENGTH=10;
private int myRowId;
private Field[] myFields;
private int myLength;
public Rec ( int rowid, Field[] fields ) {
myRowId = rowid;
myFields = fields;
myLength=fields.length;
}
public Rec (int rowid) {
myRowId=rowid;
myFields=new Field[0];
myLength=0;
}
public int getRowId () { return myRowId; }
public int getLen () { return myLength; }
public Field getField ( int i ) { return myFields[i]; }
public Field[] getFields () {
int len = getLen();
Field[] clone = new Field[ len ];
System.arraycopy( myFields, 0, clone, 0, len );
return clone;
}
public Field[] getFields(int offset,int len) {
Field[] clone=new Field[len];
System.arraycopy(myFields,offset,clone,0,len);
return clone;
}
public boolean append(int tag,String value) {
return append(new Field(tag,value));
}
public boolean append(Field field) {
if (field==null) {
System.err.println("FIELD==null");
field=new Field(0,"");
}
try {
if (myFields.length<=myLength) {
if (myFields.length0) {
for(int i=0;i
* organization
> UeberSicht der OpenIsis Verein
> OpenSource open source
; > InfoSys information systems
> AboutUs about us
* software
> OpenIsis OpenIsis
> FactSheet at a glance
> Status current status
> SeeIt sample databases
> GetIt download
> http://sourceforge.net/projects/isis/ @ sourceforge
* documentation
> IsisIntro Isis principles
> whatabout Isis compared
> unirec the Isis record
> HowTo installation
> History version trace
> MikiDoc doc structure
> formatting
> fst field selection table
> Meta FDT/metadata
> threading multi-threading
> xmlisis XML and RDF
> Struct embedded structures
> btree new B-L-Tree index
> Concurrency
> Api the API
> RdbConv ISIS and RDBs
> Client
> Server
> Bindings languages
> Tcl
> Java
> /doc/charsets.html Unicode
> CsTables Latin1 and Cp850
> Syspar
> IsisLinks links ...
doc/AboutUs.txt 0100664 0001130 0001130 00000002313 07633345144 012435 0 ustar krip krip The board of the OpenIsis society is
- Erik Grziwotz (chair) pgp
- Gabi Rohmann pgp
- Ingo Struck pgp
- Thomas Sonnemann
OpenIsis.org is sponsored by
> http://www.sferix.com/ 
and
> http://www.allmaxx.de/ 
As a student's site, allmaxx supports
open software with a focus on education and knowledge management.
We closely cooperate with the
> http://www.oc4s.org/ open community for science
.
Currently the site is maintained by
> mailto:erik@openisis.org Erik Grziwotz
and
> mailto:paul@openisis.org Klaus Paul Ripke.
The new PHP version was written by
> mailto:braulio@openisis.org Braulio José Solano Rojas.
Volunteers are very welcome.
Openisis sources are available via CVS at
> http://sourceforge.net/projects/isis/ 
Open Source can be one of the different tools used for human development.
Since Software Development is not an extensive activity but an intensive one,
it can be very productive with less resources. There is a very strong cicle
between learning and development when we participate in Open Source projects.
The feature "Open" can attract people from around the world to our development.
If this is true also for large scale projects we must use this approach for our
Latinamerican countries (we need human development).
* GRAN lección aprendida (mi propia experiencia)
- El desarrollo de Código Abierto puede llevar al desarrollo humano
- http://www.bsolano.com/images/opensource.gif
El Código Abierto puede ser una de las diferentes herramientas utilizadas para
el desarrollo humano. Dado el Desarrollo de Software no es una actividad
extensiva, sino más bien una actividad intensiva, puede ser muy productiva con
menos recursos. Hay un fuerte ciclo entre el aprendizaje y el desarrollo de
software cuando se participa en un proyecto de Código Abierto. La
característica "Abierto" puede atraer muchas personas de alrededor del mundo
hacia nuestros desarrollos. Si esto es cierto también para proyectos a gran
escala debemos utilizar este enfoque en nuestros países latinoamericanos
(necesitamos desarrollo humano).
---
$Id: CubaPHP.txt,v 1.2 2003/02/10 19:51:43 kripke Exp $
doc/FactSheet.txt 0100664 0001130 0001130 00000004126 07632113116 012714 0 ustar krip krip the OpenIsis database at a glance
* characteristics
- highly flexible data structure:
potentially unlimited number of data fields ("columns") in record
- highly efficient storage:
unused data fields consume no space
- natural modelling - ultra fast access:
logically related data that would be artificially separated in
a relational DB is stored in a single record
- highly flexible index structure:
index entries associated with a record are under full application control,
can even be derived from associated text documents of any format.
* standards
- Z39.2 (IIF) based data structure
record structure corresponds to ANSI Z39.2
(ISO 2709, "Information Interchange Format")
- Z39.50 based retrieval
the query language resembles Z39.50 "Type-1" queries as used by
all major libraries (known from WAIS and most search engines).
- all bibliographic data models
like MAB and the various MARC standards can be represented in IIF
and thus in OpenIsis
- XML data
is canonically mapped to IIF for highly efficient storage and retrieval
* interfaces
- variety of programming languages
Java, Perl, Tcl, PHP, C-library for all compiler languages
- web applications
using Java, Perl or PHP as wanted
- graphical user interface
for Windows and Unix (incl. Mac OS X) based on Tk
- usage in-process or as server
- Z39.50 protocol
server planned using indexdata's Yaz
* performance
benchmarks on an 800MHz Pentium III with standard IDE disk
- 100.000 records/sec
read in random order
- 50.000 entries/sec
index lookup
- 15.000 records/sec
write access with no index entries
- 3.000 records/sec
write access with ten index entries per record
* limits
There is a choice between traditional and 21st century database formats.
- size of data field and record
traditional: 32KB, 21st: no limit
- number of records and database size
traditional: 16 million records of 500 MB total size, 21st: no limit.
logical database can span several physical databases (virtually no limit).
- index size
traditional: ~1 TB, 21st: 32 TB per physical index.
logical index can span several physical indexes (virtually no limit).
doc/GetIt.txt 0100644 0001130 0001130 00000003405 07627223104 012062 0 ustar krip krip Available for download:
* Sources
The
> /tar/openisis.0.8.7.4.tgz Current version 0.8.7.4
includes native win32 support, own streaming buffered IO and Tcl
(not yet final).
Also available are the older versions
> /tar/openisis.0.8.7.tgz 0.8.7
(linux only),
> /tar/openisis.0.8.6.tgz 0.8.6 stable
and
> /tar/openisis.0.8.5.2.tgz 0.8.5.2
> /tar/PHP-OpenIsis.1.0.tar.gz PHP-OpenIsis.1.0
has sources for the PHP binding.
All are in tar.gz format and include some documentation.
See
> HowTo installation notes
Alternative download from
> http://sourceforge.net/project/showfiles.php?group_id=11257 sourceforge
There also is a more or less 0.8.6-based version
> /tar/VisualC_adapted.zip adapted for M$ VC
by
> mailto:dubyk@lsl.lviv.ua Serhij Dubyk
* Databases
> /tar/cds.tgz CDS
(~53KB, 160 rows, needed for tests),
> /tar/unesb.tgz unesb
(about 15 MB, 58.000 rows)
* Binaries
> http://sourceforge.net/project/showfiles.php?group_id=11257&release_id=136977 PHP-OpenISIS.zip
contains windows binaries of Braulio's release 1.0 of the PHP-binding,
including openisis.dll 0.8.6
> /tar/builds/win32/openisis-0860-win.zip 0.8.6/win32
(mingw32 cross-build) includes the openisis.dll and Java stuff.
should work on all win32 platforms from win95b on.
Note: If you want to use java you will need the JDK1.3.x
The current windows version is not yet thread-safe, so using it
with a Java servlet engine under windows requires some care.
> /tar/Perl_Win32_library.zip A perl DLL for Win32
(based on 0.8.6, I think) by
> mailto:dubyk@lsl.lviv.ua Serhij Dubyk
> /tar/builds/solaris/solaris-20020212.tar.gz 0.8.4/solaris
( build on solaris 5.8 ). Please
> mailto:erik@openisis.org mail
us if you need a more up-to-date solaris binary.
Note: the JDK shipped with solaris 5.7 / 5.8 ( java 1.2 ) should
work.
doc/History.txt 0100664 0001130 0001130 00000014117 07627131600 012512 0 ustar krip krip * Perl improvements February 2003
Try 'make pdemo' to see encoding support (Cp850 to Latin1,
for other see perl/encoding.pl), a sample standalone perl install tree,
support for 'indicators' (initial unnamed subfields)
and a non-nonsense way of spitting out XML.
* 0.8.7.4 January 2003
Experimental FDT support and preliminary (old-style),
but pretty usable Tcl bindung by Matthias included
(see tcl/test for how it works,
please contact us before using it for serious work).
Of course several fixes...
* 0.8.7.2/3 December 2002
No longer uses stdio but direct threadsafe buffered io (*snprintf still used).
Every session has their own buffered in, out and err streams,
which may be accessed w/o interlocking.
* 0.8.7.1 November 2002
The IO module is rewritten to use native windows calls for WIN32.
This also fixes some problems with read-only databases.
Error checking needs to be added; however, under wine,
it passes the standard test.
adjust make.mingw to fit your MINGW installation, then use
$
make WIN32=sigh test
$
* October 19th 2002
The OpenIsis society ("Verein") has been founded with 15 members.
Chairman is Erik Grziwotz, other board members are Gabi Rohmann,
Ingo Struck and Thomas Sonnemann.
* 0.8.7 October 2002
Version 0.8.7 supports writing of DOS/WinIsis masterfiles and xrf.
This currently works fine for a single process on Linux. See
[17]writing.txt for details. Current TODOs: interlock multiprocess
writing (PHP, Perl CGI) and fix the windows and solaris versions.
Besides writing support, there is a new streaming record reader,
which groks various formats like the SYSPAR.PAR, email headers and
property files, so you can fill your db from such textfiles. Next
step: new indexing engine.
* August 2002
No new software yet. Still very busy doing metawork. We are
preparing to set up an organisation to support OpenIsis
development with much more momentum and a company for professional
services like help on large scale ISIS installations. paperwork on
[18]the universal ISIS record
* July 2002
some paperwork on [19]What is it about ISIS that makes it ISIS?
* 0.8.6 June 2002
This version supports basic formatting. While most, especially
graphical, features of WinISIS or CISIS formatting are not yet
supported (which are typically not used in a web environment
anyway), there is support for repeated subfields as declared by
MARC for many fields. See [20]formatting notes for details. The
perl binding supports formatting (see the test.pl), enhanced
versions for all those languages are to follow soon.
* PHP June 2002
[21]Braulio José Solano Rojas from Costa Rica created a PHP
binding, which can be seen in action at [22]galileo. Available as
[23]download or from sourceforge module php-openisis. Also, the
[24]Institut Teknologi Bandung of Indonesia switched it's web
index to a PHP/OpenIsis based version.
* 0.8.5.2 March 2002
Much enhanced PERL binding. See the OpenIsis.pm included in the
sources.
* 0.8.5.1 March 2002
Java now has support for basic formatting modes like MHL, various
HTML-safety modes (like escaping all non-ASCIIs), a
[25]Vn-field-selector-style method and several nice utils.
Indentation is not properly handled, since there is no easy common
solution in HTML. Will build tools for a choice of standard
strategies later ...
* 0.8.5 March 2002
Finally some implementation of the query language ([26]demo). All
the operators are there (including /(tag), but not /(t1,t2...)).
Every attempt is made to limit the potential costs of even
extremely stupid queries like "$"^"$", so no historical (#n) or
intermediate results (for precedence) are stored. Queries are
processed strictly left to right on a buffer of 1000 hits.
* 0.8.4 January 2002
Nearly complete rewrite of search code with support for NEAR
conditions. Fixed alignment problems in IFP, now works with the
unesb db (-format aligned) and the cds db as distributed with
winisis.
The cds db we had with previous versions (from an old CDS
distribution) has a mixed format: aligned leaf files, others
unaligned. Although support for a mixed format is easily achieved
with openisis, it will not be included unless somebody has a need
for it (send us mail).
The JSP demo now supports [27]searching the unesb db with over
58.000 rows (note the hosting server is a 500MHz Celeron).
Searching is limited to 1000 postings, usually resulting in a
somewhat lower number of rows (where rows have multiple matching
postings). The lowest row number (MFN) that was cut off is
recorded, and it is possible (not yet in the JSP) to repeat the
search starting from that rowid.
* 0.8.3 October 2001
First truly usable release, since we have true search by index now
(as prefix or complete word). Search gives a list (array) of
sorted MFNs; arithmetic on those lists (and, or, not) is
straightforward. The JSP [28]demo shows how a query is refined
(narrowed) iteratively by ANDing it with a second query. And
thanks to Veronica Lencinas and colleagues, we have a version of
this document [29]en espanol.
* September 2001
Not a new release yet, but maintenance and testing. New structure
logging mechanism. Sources are available via CVS at
[30]sourceforge. Windows version openisis.exe running.
* 0.8.2 August 2001
openisis now under [31]LGPL, no legacy code. Conversion from file
structures governed by abstract dynamic description rather than
C-structs, so we can support different file layouts, much larger
databases, big endian processors and more. Simple full-scan search
available in C-Lib and Java. Given a random read throughput of
about 30.000 Records/sec on a lame 300 MHz Notebook this seems to
be of practic use. jsp demo under [32]development ;).
* 0.8.1 June 2001
Java native interface version available. Java package org.openisis
has Db, Rec, Field, Test. NativeDb implemented in libopenjsis.so.
Subfield splitting and htmlifying in pure java.
* 0.8.0 May 2001
Subfield splitting and htmlifying. Everything also available from
perl as an xsub. Record shows up as hash, handy but no repeated
fields. Makefile.PL, test.pl etc.
* 0.7.9 May 2001
First version: static C-Library libopenisis.a for reading records
by rowid(Mfn). Executable "openisis" does test. Logging,
argumentparsing, Makefile, demo etc.
... ToDo
doc/HowTo.txt 0100664 0001130 0001130 00000005436 07566146163 012131 0 ustar krip krip
Start by downloading the Software.
Unpack everything in some arbitrary directory.
For the tests, you will also need some isis database,
which must be located as files db/cds/cds.*, see downloads.
Make sure filenames are lowercase.
If you are on Windows, you should either get yourself the cygwin
environment with tools like gmake and gcc or volunteer as a porter and
start writing the Makefile for your make and compiler. Erik has build
a Windows version using mingw and Linux gcc as crosscompiler. If you
are on Linux, everything is fine. Ports to MAC OS X and other UNIXes
should be no problem.
Type "make" and enjoy the compiler messages.
(If your make complains, e.g. on BSD, try "gmake").
Type "make demo" and enjoy your first open isis record.
(You installed a db/cds/cds.*, didn't you? It has more than 42 rows?)
Type "make run" and watch the guts of your db passing by.
Type "make test", there should be no difference between the testout.txt and
the testres.txt as provided (using the cds database from winisis
and the 15 MB 58.000+ row unesb db, see downloads).
Type "make time" to measure performance,
subsequent tries are usually much faster.
My 800 MHz P3 random-reads more than 179.000 records a second,
once the files are in the system cache.
Typical values:
$
time ./openisis -perf 1000000 -db db/cds/cds >/dev/null
real 0m5.655s
user 0m3.650s
sys 0m2.000s
time ./openisis -perf 100000 -db db/unesb/unesb >/dev/null
real 0m0.991s
user 0m0.670s
sys 0m0.320s
time ./openisis -fmt mfnf -search 'k$' -db db/unesb/unesb >/dev/null
860 rows for k
real 0m0.044s
user 0m0.040s
sys 0m0.000s
$
Type "make perl" to build the perl stuff;
some perl 5.* must be installed beforehand.
Type "make java" or, if you just can't get enough, "make jdump",
to see it all happen in your shiny new JDK1.3 Java VM.
Some 1.2.* JDKs should do, but tell the Makefile to not look
in /usr/java/jdk1.3 by setting JAVAHOME.
* installing openisis
libopenisis.a can be linked with your code; no installation necessary.
You may wan't to install the 'openisis' binary somewhere in your path
for the fun of it; go ahead, just copy, no magic registry entries.
To install the perl stuff for general availability in your
/usr/lib/perl5 or whatever, cd to the perl subdir (after "make perl")
and issue "make install" (as root or otherwise legitimated).
After that, try "perldoc OpenIsis" and the demo.pl script.
Java, like perl, needs to dynamically slurp both some stuff in the own
language and a native shared object. The former is openisis.jar, set
your CLASSPATH to include it, or specify when invoking java like in
the Makefile. The latter is libopenjsis.so on linux (yes, it's jsis).
The system dynamic linker must be able to find it;
see
> /jdoc/org/openisis/NativeDb.html NativeDb.java
for details.
... GetIt
doc/IsisIntro.txt 0100664 0001130 0001130 00000005207 07566146163 013010 0 ustar krip krip * what is isis
Isis is a simple, yet powerful database system with a large installed
base since the 80s. Since it's well suited for bibliographic data,
it's commonly used in libraries, and since it's very low cost,
especially in those running on a low budget.
* introduction to the isis db
An isis DB is a list of rows of unspecified structure, each identified
by a unique number, the rowid (a.k.a. mfn). Each row is a list of
fields, and each field has number (tag) and a string value. Within a
row there may be zero, one or more fields with a given tag. While the
field's value usually is a textual representation of data in one or
the other character encoding (commonly one of the IBM/DOS code pages),
it may actually contain arbitrary bytes. This is closely modelled
after ISO2709 "Information Interchange Format" (IIF, a.k.a. ANSI/NISO
> http://www.niso.org/standards/resources/Z39-2.pdf Z39.2
)
* subfields
There is a convention to encode multiple fields in one by separating
them with a '^' followed by one character tagging the subfield. So the
field value '^afoo^bbar^bbaz' represents a field having one 'a'
subfield with value 'foo' and two 'b' subfields 'bar' and 'baz'. An
other separator char may be used, e.g. ASCII character 31 ("Unit
Separator") is used in the
> http://www.loc.gov/marc/specifications/specrecstruc.html MARC standard.
* formatting
There is a formatting language, with literal text, field and subfield
variables, if-else branches (on field existance) and for loops (over
field repetitions) (roughly speaking).
* indexing
An index is build by converting a row into a list of words (optionally
applying formats) and stuffing every word, qualified by the position
of it's occurence in the row, into a B+-Tree (which is actually spread
to six files). Searching for a word or word prefix is possible with or
without qualifying the position (field). Since all fields can be
combined into one index, it is usually not necessary (but possible) to
set up multiple indexes.
* queries
A query language allows for combination of word lookups using and, or
and not(without) operators. This is very similar to the "Type-1" query of
> ftp://ftp.loc.gov/pub/z3950/official/part1.txt Z39.50.
* usage
While isis lacks most features of RDBMS like complex relations between
different entities, it's flexibility comes in handy for many
catalogues and directories with highly varying records and one single
level of substructure, which today are usually modelled in XML
documents rather than table rows. In other words, isis is an ideal
storage for many XML applications. The flexible indexing mechanism
combines the best of full text searching and structured retrieval.
doc/IsisLinks.txt 0100664 0001130 0001130 00000004742 07627661776 013012 0 ustar krip krip > http://sourceforge.net/projects/isis/ openisis and PHP isis at sourceforge
* isis core sites:
> http://www.unesco.org/webworld/isis/isis.htm Unesco
> http://www.unesco.org/webworld/isis/ new Unesco site
> http://www.bireme.br/isis/ Bireme
* documentation:
> http://www.cindoc.csic.es/isis/indice.htm THE BOOK
CDS/ISIS reference manual incl. data formats (en espanol),
> http://www.vista.gov.vn/VistaEnglish/VistaWeb/learn/Isis/Eng/ WinISIS manual
* standards:
- ISO2709 "Information Interchange Format", a.k.a. ANSI/NISO
> http://www.niso.org/standards/resources/Z39-2.pdf Z39.2
- Machine Readable Catalogues
> http://www.loc.gov/marc/specifications/specrecstruc.html (US) MARC 21
,
> http://lcweb.loc.gov/marc/ overview
, references at the
> http://www.oasis-open.org/cover/marc.html Cover Pages
- Z39.50
> ftp://ftp.loc.gov/pub/z3950/official/ official spec
, overview at
> http://www.oclcpica.org/content/45/pdf/z3950_handbook_paper.pdf OCLC|Pica
, links at
> http://indexdata.dk/z3950/ indexdata
, makers of excellent free Z39.50 software.
* people and projects:
> http://www.jezuici.krakow.pl/soft/iapi/ Robert Janusz' iAPI
, Kafkas Caprazli's
> http://members.aol.com/cdsisis/ EVERYTHING about CDS/ISIS
> http://www.oss4lib.org/ open source software for libraries
> http://web.tiscali.it/javaisis/ javaisis
> http://isisonline.lib.itb.ac.id/ Institut Teknologi Bandung
IsisOnline in Indonesia
* user groups:
> http://www.bib.wau.nl/isis/ Netherlands / international
> http://www.axp.mdx.ac.uk/~alan2/isisplus.htm UK (ISIS PLUS)
> http://www.isisnetz.de/ Germany (isisnetz)
* non-ISIS
If you do not care for ISIS compatibility
but need a fully integrated library system, then
> http://koha.org/ koha
might be for you.
Unlike OpenIsis, which provides flexible tools and components
to be used in a wide variety of applications, koha is strictly
focused on everything a library needs, including circulation
and the like.
However, it has a plain table structure based on MySQL,
which can not support sophisticated queries as ISIS can.
There are
> http://www.zlb.de/bibliothek/fachbereiche/infdienste/fremdspr/englisch/ lots of mailinglists
on library issues in many languages.
* yet other ways to find a book
Not all books are living in libraries! You may try to
> http://bookcrossing.com/ catch one
that has been set free. 
* and don't forget ...
... to visit the
> http://moffem.de/ Museum of Faulty Error Messages
:)
doc/Java.txt 0100644 0001130 0001130 00000004371 07606625131 011735 0 ustar krip krip review of the OpenIsis Java interface
* Java, Server and Sessions
Approaching version 0.9 of OpenIsis,
giving the concepts needed for a multisession server,
we also have to revise the options for the Java binding.
When speaking about Java, we mostly have Java Servlet Engines in mind.
Therefore we assume that we are running multithreaded,
and that we have little control about the selection of threads.
A client's request already has a thread dedicated to it,
and this thread cannot do anything else before this request is served,
i.e. requests can't be somehow queued or postponed.
Therefore asynchronous access to a database is mostly pointless.
All we can do is to synchronize access by blocking one thread
while another accesses the same ressource.
To not reduce concurrency to complete mutual exclusion,
we clearly need database sessions.
Moreover we have to assume that we have multiple users,
which might be represented by Http sessions.
Where database sessions are configured to keep results sets,
they should be bound to Http sessions.
Otherwise we might pick an unused database session from a pool.
* Java on a local db
The situation of a local database accessed from a servlet engine
is similar to that of a dedicated database server in that we
have multiple sessions and multiple threads.
The multiplexer, ensuring that there are no two
requests running in parallel on a given session,
is implemented in Java by synchronizing on the Java
object representing each session.
The creation of a session is synchronized on the default session.
A servlet engine accessing a local db may not only
create cute looking HTML pages (by means of JSPs),
but may also provide a general purpose database server,
speaking a HTTP wrapped variant of the server protocol.
* Java as client of a remote db
On the other hand, a database session might be a client session
bound to some OpenIsis server.
Since the protocol used for communication with a server is pretty simple,
and we will use synchronous access anyway,
we are planning to implement this in pure Java.
This will remove the need for JNI, which has some problems
especially where servlet engines deploy multiple class loaders
for what they think are "web apps".
---
$Id: Java.txt,v 1.1 2003/01/07 19:28:25 kripke Exp $
doc/Meta.txt 0100644 0001130 0001130 00000011011 07612074137 011730 0 ustar krip krip description of OpenIsis metadata
* the FDT
According to the CDS/ISIS manual,
the field definition table is displayed like
$
44 Serie 300 X R vz
$
and contains for each field tag:
- a field description (up to 30 characters)
- maximum length (1 to 1650)
- field type:
X for alphanum, A for strictly alpha (not including space),
N for numeric (decimal digits), P for pattern (see below).
- repeatability indicator
R meaning field is repeatable, N else
- format (subfield list or pattern)
For a P field, this gives a COBOL-PIC-style pattern
consisting of X (alnum), A (alpha), 9 (num) and literal letters,
e.g. 99-999/AA to allow for input like 35-674/XE.
For other field types, this is a list of legal subfields,
e.g. vz to allow for input like foo^vbar^zbaz.
Patterns are not supported by winisis as of Version 1.4.
The actual file format looks like
$
Series vz 44 300 0 1
$
which is 30 characters name, 20 characters subfield/pattern,
the field tag, maximum length, a number encoding the type
(0 alphanum, 1 alpha, 2 numeric, 3 pattern),
and a number 1 for repeatable / 0 for non repeatable fields.
Moreover, the FDT defines the available
worksheets W (.fmt), printformats F (.pft) and field selections S (.fst).
Each such definition is on a line starting with the
key letter and a colon, and followed by (blank padded)
6 character fields of file basenames.
$
F:THES THES1 THES2
$
* field names
It is common practice to use long field descriptions
like "Corporate Bodies" or "Govt. Publications No.",
which may be nice looking but are not well suited
for technical use.
Therefore the OpenIsis metadata contains an extra field name
in it's metadata record (see below).
When sourcing metadata from an FDT file,
OpenIsis will derive field names by lowercasing the descriptions
and replacing all runs of non-alphanums by a single '_',
yielding "govt_publications_no".
* ISO
A special kind of alphanum, using the ISO2709/Z39.2 subfield delimiter
ASCII 31 (CTRL-_) has symbolic name I and code 10.
* enum
A special kind of numeric, where the possible values are named,
has symbolic name E and code 12.
* table
Table fields use a fixed set of subfields separated by TAB
without the subfield letters.
It has symbolic name T and code 14.
* structures
In order to describe
> Struct substructures
(i.e. embedded records),
we introduce a new field type with symbolic name S and code 15.
Since fields introducing structures may contain subfields
(attributes), the other type properties are interpreted
just like for 'X'.
* subfield names and descriptions
As an extension to be able to give names to subfields,
which is especially required when converting from/to
SGML-style long attributes,
we use a second set of fields by adding 16 to the type.
The tag is the tag of the field for which this description
is relevant, or 0, if it could apply to several fields.
For the subfields, the first letter of the subfield spec
is the subfield described. The rest may be a pattern for type 19 (= 16+3).
Subfield specs may have length and repeatability.
* enum values
Since subfields may be anything, but not structures,
we can use code 31 for the values attached to an enum
with symbolic name V.
The tag and (first char of) subfield are
the tag (and subfield) for which this description
is relevant, or 0, if it could apply to several fields.
The numeric value is held in the length.
* compatibility
When exporting the metadata to FDT format,
S fields are written as X, E as N,
and F and V entries are omitted.
* representation as record
The metadata record uses TAB as subfield delimiter.
One field of metadata is used to describe each field.
It has the following subfields:
- i id (tag)
- t type (as digit)
- n name
- s subfields
- l length
- r repeatability (not present if not repeatable)
- d description
- c child legal child id (repeatable)
There are other tags reserved for (repeatable) fields to hold
the names of available worksheets, printformats and field selections.
* example
A simple description for HTML's td tag could look like
$
td abchnrvw 99 4000 15 1
align a 99 1 30 0
center a 99 0 31 0
left a 99 1 31 0
..
bgcolor b 99 1 1 0
..
valign v 99 1 30 0
..
$
Note the list of
> Builtin builtin metadata
---
$Id: Meta.txt,v 1.4 2003/01/17 21:40:47 kripke Exp $
doc/MikiDoc.txt 0100644 0001130 0001130 00000001716 07576120307 012374 0 ustar krip krip Mikidoc is a mini pseudo-
> http://c2.com/cgi/wiki WikiWiki
-style documentation convention.
Unlike real Wikis, it is much simpler and less oriented towards
the web and web-editing of content, more towards the content.
Mikidoc currently support the following constructs:
Empty (really empty) lines give a <br>, lines starting with
- $ toggle code mode
- * are a head
- --- give a <hr>
- - start a list topic
- TAB continue a list topic
- > are links;
the link body may follow the ref after a TAB
- ... denote a footnote document displayed to the right
- ! set break-after-every-line mode (pseudo pre)
- ; are ignored
Todo: support *bold*, _italic_ and MikiLinks.
Write converter to texinfo (left as an exercise to the reader).
That's all, folks!
> http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/isis/openisis/doc.jsp?rev=HEAD&content-type=text/vnd.viewcvs-markup get the code ...
(4 KB :)
---
$Id: MikiDoc.txt,v 1.2 2002/12/12 14:44:23 kripke Exp $
doc/News.txt 0100644 0001130 0001130 00000002713 07642332322 011763 0 ustar krip krip > History more ...
* April 2003
detailled description of the
> Serialized
record format
* PoC March 2003
nearly all ideas now have passed their
> Status#devel Proof-of-Concept
* idxall March 2003
now supports writing with application-controlled index update at
> WritePerf reasonable performance
* Ukrainian translation
of the 0.8.6
> /index-uk.html index page
by
> mailto:dubyk@lsl.lviv.ua Serhij Dubyk
, who also did some
> GetIt Windows stuff.
* Perl improvements Feb 2003
The reworked perl demo provides
encoding support and no-nonsense-XML.
* new index improvements Feb 2003
The B-L-Tree now supports update/delete
at about 30.000 ops/second.
* X. Reunion Feb 2003
> XReunion X. regional meeting
of ISIS distributors in la Havana.
* 0.8.7.4 January 2003
Experimental FDT support and preliminary old-style,
but pretty usable Tcl binding by Matthias.
* January 2003
Papers on
> Server
,
> Client
,
> Java
,
> Tcl
and all
> Bindings
,
> Meta metadata
and
> Struct embedded structures
.
* 0.8.7.2/3 December 2002
No longer uses stdio but direct threadsafe buffered io.
Experimental Tcl binding included.
* December 2002
Papers on
> Concurrency
and classification of the
> Api
.
Session handling partly implemented.
* 0.8.7.1 November 2002
The IO module is rewritten to use native windows calls for WIN32.
This also fixes some problems with read-only databases.
Error checking needs to be added; however, under wine,
it passes the standard test.
> History more ...
doc/OpenIsis.txt 0100664 0001130 0001130 00000006341 07614265012 012603 0 ustar krip krip By now, OpenIsis is mainly used for web publishing of Isis databases.
Web developers use the Java, Perl or PHP binding for this.
There also is a command line version which serves as a demonstration
and test of the various features and can be used as a utility for
data import/export and similar tasks.
A graphical, standalone user interface similar to UNESCO's WinIsis
will be developed during 2003, after the Tcl/Tk binding is finished.
See
> Status current status
* developing OpenIsis
In general, there are no plans to reimplement every piece of code ever
written for isis. To be of practical value, OpenIsis has to maintain
compatibility in the format of the database files anyway. So, one may
use winisis or whatever existing import scripts to create and maintain
the database, yet deploy OpenIsis' perl interface to run powerful
reports and the Java Native Interface to allow queries from a Servlet
based web application.
OpenIsis will focus on providing tools rather than applications. For example,
there will be no attempt to mirror the full functionality of winisis.
To achieve this, OpenIsis provides access from the most important
programming languages: Java and PHP for the web, Perl for the scripts
and Tcl/Tk for platform independent GUIs (partly DONE).
All others can, of course, link the lib.
* Roadmap for the development of OpenIsis -- as of September 2002
After more than a year of OpenIsis development,
experimenting with various options and environments
and some feedback, we learned something about what
is important to the community and what is not.
* findings and goals
- A fairly portable C library that can be plugged into Java, Perl, PHP
or Tcl proved to be valuable and estimated by several developers.
There was not much demand for a pure Java version.
- There is high demand for a GUI version similar to WinISIS
(cross-plattform, of course) with a safe multi-user capability.
- This in turn requires a simple but fast and robust server.
Multithreading can greatly improve the throughput.
- Users are used to incompatible file formats, formatting and so on.
There is little of a standard to follow.
So building a next generation system clearly is more
important than sticking to the (which?) legacy too much.
- The power buried in the similarity between ISIS records
and other structures like, for example, E-Mails is not leveraged.
- The existing versions of the formatting language are the result
of a process of growth, adding feature after feature.
The logic is mixed with the presentation (e.g. HTML literals).
* conclusions
- The multiformat support will be dropped in favour of an efficient
support of the "DOS/WinISIS" (packed little endian) structures.
- The database kernel has to support basic thread-safety
for use in Java and other server environments.
- Tcl will be the scripting language of choice: Tcl/Tk for portable GUIs
and a limited set of Tcl commands for server side scripting.
* flavours
- The core library will not contain any particular external technology,
so it is as lightweight as possible and can be linked into Java,
Perl, PHP, Tcl and the like.
- There are binaries in plain, with Tcl and with Tcl/Tk.
Any of these can act standalone, as client, as server, or both.
volunteers are welcome !
... ToDo
doc/OpenSource.txt 0100664 0001130 0001130 00000005723 07566146163 013152 0 ustar krip krip So why are we writing the openisis software? Because Isis is not open
source software, it's not even free software, and that leads to a
whole bunch of problems.
* problems resulting from closed software
- Availability (in theory)
Versions of the software exist for some operating systems, library
versions and languages. For other environments, there is no
version of the software, and there is not much one can do about
it.
- Availability (in practice)
You may download most the software, but it's partly protected with
passwords, which you have to order at some national distributor.
You have to pay some fee and/or declare some good reasons, why you
want to use the software. Then you have to wait. In germany, for
example, it didn't work at all for some time, until the newly
founded
> http://www.isisnetz.de/ IsisNetz
remedied the situation.
- Availability (in legal terms)
Some parts of the software are accompanied by different documents
stating some license terms, others are not. Terms seem to be
pretty different between countries. One can not easily figure out,
what exactly might be allowed usage.
- Availability (of documentation)
Some documentation is available in english, some only in
portugese, espanol or italiano. Only a small part is downloadable
at all, most is paperware.
- Bugfixing
There is no way one can fix a bug, and not much one can do about
having somebody fix it.
- Extending
The only way one could write a Binding for perl or Java would be
using the isis.dll. There are problems with regard to required
additional libraries (especially some C++ stuff), there are no
statements about thread safety, unicode compatibility and so on.
As a consequence, it's practically impossible to write a
state-of-the-art web application based on an isis db.
- Improving
Many users develop useful ideas for improvements from practice.
Their expertise is lost as they are not able to turn it into
improved software.
- Enabling
While open source software enables people all over the world to
shape their tools themselves, closed software lets them in
dependency.
* benefits of open software
To address these problems we feel the need for an open source
implementation of isis. Of course it would be best to have all of the
existing isis code under one or the other form of open license (GPL,
LGPL, artistic or similar as appropriate).
On the other hand, an independent secondary implementation has
advantages in it's own right. It may have a different focus and
develop strengths in one aspect while another aproach performs better
in other situations. For example, openisis will have some support for
multithreading and unicode, which is paid for by a certain overhead. A
rewrite by developers with a different background might introduce new
ideas which finally, after having had their indepent test bed, help
improve the standard.
OpenIsis as a software to access isis databases is and will be freely
available for everybody with full sourcecode, no fee, no restrictions.
doc/OverView.txt 0100644 0001130 0001130 00000004711 07624470764 012632 0 ustar krip krip The OpenIsis society was founded in October 2002 to further the
development and dissemination of open source information systems.
We support libraries, archives and other public organizations
like social movements to manage and retrieve information in a
high quality way according to bibliographical principles.
Most important is the
> OpenIsis
database system which was developed by our members since May 2001.
OpenIsis is the open source member of the
> IsisIntro CDS/ISIS
> CdsIsis software family.
* who needs quality information systems?
Well ... everybody who has quality information.
Traditional examples are libraries, where librarians invest much
knowledge and experience to make you find the book you need.
While the large libraries have a long track of using and developing
bibliographic databases, many organizations like worker's unions,
charity organizations and social movements have smaller archives
of their publications, which ought to be as well accessible.
Even for the purely virtual web-archives, newsgroups and mailing lists,
it is increasingly understood, that the power of search engines needs
to be augmented by quality metadata to be retrieved reliably.
Dublin Core and the Resource Description Framework are examples
of such approaches based on bibliographical principles.
* who needs open source information systems?
Many individuals, organizations and institutions can't afford
the commercial bibliographic database systems,
where even small site licenses start at about 10.000 US$.
Therefore,
> http://www.unesco.org/webworld/isis/ UNESCO's CDS/ISIS
, which is available for a nominal fee, is in widespread use,
where the budgets are low.
Our
> http://www.gnu.org/philosophy/free-sw.html free software
is not only free of charge, but also free of restrictions.
It enables the users to build on it, to improve it,
to adopt it to their needs rather than being dependent
on the original developers.
To close the "digital gap", access to *both* the content
and the software is needed.
* what about the "Information Society"?
Even the techies and librarians at rather well-equiped universities
in the US or Europe, which are working on new ways to publish
and manage free content, like the
> http://www.openarchives.org/ Open Archives Initiative
or the
> http://oc4s.org/ Open Community for Science
, do not have access to the code of existing commercial systems.
Following the Linux Economy, we may find that open source
is the basis for progress ...
... News
doc/PHP.txt 0100644 0001130 0001130 00000006031 07623000707 011471 0 ustar krip krip considerations on the design of a PHP interface
This paper is based on the requirements of
> Concurrency
* why not to export sessions
From the C-API's point of view, the use of sessions is a MUST
in a multithreaded environment, which PHP may be.
This, however, is the server-side of the PHP engine.
The PHP page, as seen by the programmer, is a client to the database:
it does NOT handle multiple requests, but one!
In other words, the page is already executing within the context of
a given request, including a given thread.
Since a request is not meant to operate on several sessions,
but rather has one session as an attribute of it's context,
nothing would be gained by making the session visible ("export")
to the PHP programmer by calls like read(ses,...) or ses->read(...).
Moreover, the currently most used environments, i.e. MP Apache 1.x/Unix
and some Windoze boxes, do not allow/require the use of multiple sessions,
anyway.
Finally, the only reason to explicitly select a session based on a user's
session id, would be to access this session's state as #1-style result sets
or authentication information. Both will not be implemented in the near
future and both will rarely be needed in web environments.
Yet, it might be made available when needed by means of an additional
call that "selects" that session as the one to be associated with the
request context.
* strategy
Therefore, I would suggest the following strategy:
- In the first step, stick to the default session by using a static
session variable like openisis_ses0 and auto-initializing it like
the OPENISIS_SES0 macro does.
- For the MP version register a function using register_shutdown_function
(is there a corresponding C-API to register a C function ?)
that closes all open DBs in order to release their flocks.
(Berlin will provide such a "close all").
- On Windoze and Apache 2, make sure to actually run single-threaded
by using EnterCriticalSection/pthread_mutex_lock
on a static CRITICAL_SECTION/PTHREAD_MUTEX_INITIALIZER
and leaving/unlocking it in a registered shutdown_function
- Some day, in a better age, replace the static session variable
with a thread specific one by using pthread_key_create/pthread_setspecific
or maybe PHP's TSRM.
C.f. man pthread_mutex_lock,
> http://msdn.microsoft.com/library/en-us/dllproc/base/critical_section_objects.asp critical Windoze
* notes
- is there a call in the PHP API that can be relied upon
to be the first one used, in order to do initialization?
Probably opening a DB?
- In the MT version, closing databases is not enforced,
thus successive requests will open already opened databases.
This is ok with the C-API's DOpen, which will happily return
the same handle.
- Once additional worker sessions/threads are supported,
we'll need additional open/close calls, which do neither
actually open or close a database, but access and release
(and possibly flush). The MT and MP usage could then be made
to look very similarm with details governed by a few config
settings.
---
$Id: PHP.txt,v 1.1 2003/02/13 20:36:23 kripke Exp $
doc/RdbConv.txt 0100644 0001130 0001130 00000013275 07570723757 012431 0 ustar krip krip Representing ISIS data in a relational database
If you consider setting up a table with columns "author", "title"
and so on, don't do it. The following structure gives you at least
some of the benefits of ISIS, because it's shaped more closely
to the ISIS data model.
* table structure
It is not very difficult to think of a table structure
for lossless representation of the contents of an ISIS masterfile:
$
CREATE TABLE Mst (
mfn number, -- master file number
fld number, -- counting fields in this masterfile record
tag number, -- the field tag
val varchar(2048) -- field's content
)
$
Notes:
- a row in this table does NOT hold an ISIS record,
but one field of an ISIS record
- each ISIS record is represented by all table rows
with the same mfn
- you may omit the column "fld" if you don't care for the ordering
- you may choose other datatypes for the field tag,
for example number(3) or char(3),
depending on the compatibility required (e.g. MARC)
- you may choose other dimensioning for the val column.
ISIS can hold up to 32KB (signed short) in one field,
Oracle has a maximum of 4KB on the varchar type.
* accessing the data
Retrieving row 123 is easily done with
$
SELECT tag, val WHERE mfn = 123 ORDER BY fld
$
It is suggested that you fill the val column with native
ISIS data as of "MPL" proof mode. Then you have to extract
subfields and perform other substitutions for output.
This can be achieved for example using the v-method of the
> /jdoc/org/openisis/Field.html Field
Java class (it's a static method of about 100 lines,
which you may grab and port to your system).
An alternative would be to pre-split the subfields
into a structure like
$
CREATE TABLE Mst (
mfn number,
fld number,
tag number,
sub char(1), -- subfield code
val varchar(2048) -- subfield's content
)
$
A typical keywords field like
$