This appendix gives a brief description of most of the commands available in the low-level interface to the Okapi BSS. Commands intended specifically for users wishing to implement relevance feedback systems are not included as they are regarded as being at a higher level; it may be better to construct your own functions using the low-level commands.

1. The BSS Library.

There are three functions in the library:

1. iinit()
2. iexit()
3. i0()

1.1. iinit()

1.2. iexit()

1.3. i0()

return_code = i0(command, response)

The C source code for a straightforward example of the use of i0() is given in Appendix L . A more complex example of how this can be done can be seen in the source code for the GUI to the BSS described in The Interface to the BSS.

Other error messages are written to Err_file, and this may be output to stderr, written to a file or discarded, depending on the compilation. They will sometimes be useful to a system developer. It must be emphasised that the error-trapping system is not well developed. It is hoped that it will improve with time and experience.

2. Notes on commands and arguments

<argument name>=value

There are a few cases where the <name>= is unnecessary and a few where it is invalid (e.g. set defaults).

White space is ignored around the '='.

Most arguments have default values. Some of these are settable using a SET command ; the current value of all settable defaults can be obtained by using the SET command with no arguments. In previous versions of the interface some arguments were implicitly set by being used in a command. This is now true ONLY of current set number (s=..., set=...) and current record number (rec=..., record=...). These are automatically set to the most recently produced, or SHOWn set; and to the next, or first record of the current set.

3. Notes on errors

      -29, interface syntax error, no database specified

"System"-type failures (e.g. shortage of memory, missing files, shortage of file descriptors) are scarcely handled at all; this may be done sometime.

One error which can arise from most of the commands is NO_DB_OPEN.

4. The Commands.

4.1. INFO   |   INF <topic>

responds with a line containing "<n> database(s)" followed by a line for each available database consisting of the database name followed by a very brief description.

Other INFO topics are described below.

4.2. CHOOSE   |   CH

Successfully choosing a database resets all values to their defaults (see SET) and destroys any existing search sets.

There is an optional argument dbflags=<value> or db_flags=<value> which affects the facilities which are loaded when the database is opened. The current status of this is unknown, and you are advised not to use it.

4.3. PARSE   |   P

Parses the string into individual terms in a form suitable for looking up in the database. Used to determine what terms to look for with a FIND(1) command.

For description of the "attr" qualifier see under FIND(1) . The response is:

<number of non-stopped terms>   [t=<term>]   ..

The number may of course be 0. Only indexed terms are output. An output term may be a GSL token of the form @nnnn

4.4. SUPERPARSE   |   SP

superparse   [attr   |   a=<attribute mnemonic>]   t=<string>

As PARSE except that more information is given, and terms are output even if they are not indexed (in which case they have GSL class H or F (see Appendix D: The GSL File. ). The response is:

<number of terms>   [t=<term>   c=< gsl class >   s=<source>]   ..

<source> is the portion of the input which gave rise to this term.

4.5. STEM

Applies the named function to the given term. There is no default stem function. The response is:

t=<stemmed term>

This utility function is independent of the current database (if any). Even if there is an open database the stemmed term may not occur in any of the database's indexes. Names of the available functions may be obtained using the DISPLAY command. Note that the <term> is treated by the stem function as if it were a single word.

Errors: an unrecognized stem function gives NO_SUCH_STEMFUNCTION

4.6. FIND   |   F

4.6.1. FIND (1)

NB one term only, and the "t=<term>" must come last. The response is:

[ s<n> ]   np=<np>   t=<term found>

<term found> will be empty (nothing following the "t=") if type=0 and specified term is not in the index or type is nonzero and an extremity of the index has been reached.

The system remembers the last term found by a FIND (1) command. This is its notion of "current term" (initially empty). If no term is specified in a FIND command the current term will be looked up. This is not useful unless the "type" qualifier (see below) is 2 or -2. If type is 2 the "empty" FIND command will find the next term in the index, if type is -2 the previous term. For example, to look at all the terms starting with "dog":

set type=2 save=0
find type=0 t=dog
np=3650 t=dog
find
np=1 t=dogaggress
find
np=12 t=dogan
[etc]

save

If save=y or 1 (default) a set is made, otherwise no set is made. save=n may be faster if you only want to find the number of postings for a term, and there will be no unwanted set to be deleted.

type

An invalid type value causes a NO_SUCH_SEARCHTYPE error.

a | attr | attribute

This determines the type of data which will be searched, i.e. which index, and, for some kinds of index, the class of objects within that index. There is always a default attribute, which may be referred to as "default" (e.g. in a SET command). Typically, this is some kind of general keyword attribute. It is defined by the first entry in the database's search_groups file. Usually there also exists an attribute called "dn" (document number). Apart from these two, the only way to determine what attributes there are for a database is to look at the search_groups parameter file for the database, although if the database manager is sensible they will have fairly self-evident names like "ti", "au", "ab".

A non-existent attribute mnemonic causes a NO_SUCH_ATTRIBUTE error.

4.6.2. FIND (2)

NB sets and weights must come before options

<setnum> must be the number of a set which has been created and not deleted. Set numbers run from zero. There is a limit on the number of sets, currently about 1000. Deleted setnumbers are reused, always lowest free setnumber first. A non-existent set causes a NO_SUCH_SET error.

If the operation is of the best match type a weight must be given for each set which is not itself the result of a best match operation. Missing weight will give a SET_NO_WEIGHT error. If a weight is given for a set which is the result of a best match operation it is ignored (arguably this is wrong, and may in the future be changed).

Normally, weights should be determined by using the weight command. If weights are derived in some other way they must be smallish integers, typically in the range 1-200; weight "overflow" (> 32767) is not detected and will give spurious ordering of retrieved documents.

The response is:

• non-weighted operation:

  [S<n> ]   np=<np>

• weighted operation:

  [S<n> ]   np=<np>   maxwt=<maxwt>   nmaxwt=<# with max wt>   ngw=<ngw>   mpw=<max poss wt>   mpw=<# with max poss wt>

Options

Options may be given in any order (but must follow all sets or set/weight pairs). Redundant options are ignored. If an option is repeated the last occurrence is used.

Special options

These cannot be used in conjunction with any other options (or with each other), and they only apply to a single set.

op

The following boolean and quasi-boolean operations work on all indexes.

'and2' and 'not2' are referred to as "Robertson limits".

The following group work on all indexes not of types 0 or 2 (all keyword indexes of current databases are OK)

Weighted operations

4.7. WEIGHT   |   W

The response is a relevance weight as a 16-bit integer (10 * a log to base 2). n is the number of postings for the term (<np> from a FIND command). No default. Not settable.

Note that the big-N argument (number of indexed documents) required by the Robertson/Sparck Jones functions cannot be supplied by the user; it is determined automatically. If there is a need for it a facility may be provided for allowing the user process to supply a big-N value.

Weight errors

A function other than 0-2 will be reported as NO_SUCH_WEIGHTFUNC; failure to supply a value for n gives SYNTAX; phoney values of n, r, bigr, rload and bigrload don't generate an error (perhaps they should), weight is usually returned as 0 or 1.

4.8. SHOW   |   S

1. show [format   |   f=<fmt #>] rn   |   recnum=<database record number>

   A direct request by primary key (normally a sequential record number which runs from 1 to the number of records in the database). No set need have been formed. No default. Not settable. Error: RECORD_OUT_OF_RANGE_DB

2. show [format   |   f=<fmt #>] [n=<num>] [r=<record>] [s=<set #>]

   A request for a record from a set. In this case records are numbered from zero to one less than the number in the set. <num> is the number of records requested. It defaults to 1. The set defaults to the most recent set formed or from which a record has been shown. <record> defaults to the next one, or the first if it is the first show command on a new default set. If set or record is specified they become the new default values. Set and record are not settable, except implicitly. Errors: NO_SUCH_SET, NO_SUCH_RECORD.

See also under MISSING COMMANDS.

There are a number of predefined formats, some only applicable to specific databases. The initial default format is 1, which works with any database, and delivers a complete record in the form

Record <n>
Weight <w>
<fdnum>: <contents>
................

Empty fields are not shown. Where fields end with a linefeed (databases of type "text") there may be one or more empty lines between fields. Other databases (abstracting and indexing databases) do not normally contain linefeeds, and one linefeed is inserted at the end of each nonempty field. Other SHOW formats are shown in the following table. You should refer to the source code of the "okapi" gui (particularly hitlist.c, header.c and defs.h) for an example of the use of format 259.

At present there are always either zero or two passage records. If there are two, one is for the whole record and the other is for the "best" passage; the one with the higher weight appears first, and it is this weight which appears in the entire record's "weight" field.

Highlight records will be ordered by increasing offset from start of data.

Adding 256 to the format number highlights the terms by which the record has been retrieved if possible. The highlighting codes are vt100 at present, but this might in the future become settable, or be replaced by some universal code which the client will replace by whatever is locally required.

Error: NO_SUCH_FORMAT

4.9. DELETE | DEL

Deletes the sets given by number, or all sets if "all". There are no errors, even if invalid set number.

4.10. SET

4.12. INFO sets

Not supported

4.13. DISPLAY | DISP | DI | D

Exhibits various things.

See also STEM.

5. SPECIAL PURPOSE OR PROVISIONAL COMMANDS

5.1. DISPLAYSTATS

displaystats   |   ds [s=<setnumber]

Outputs "sum=<sum of weights in set> sumsq=<sum of squares of weights in set". If no set is specified the "current" set is assumed. Using this command does not alter the system's notion of current set.

This command may be used on any set, but at present the only sets which may give nonzero statistics are ones obtained by using a Robertson limit (op=and2 or op=not2 (see above). In addition the environment variable BSS_COMBINE_DO_STATS (see ENVIRONMENT VARIABLES below) must be set and nonzero.

Errors: NO_SUCH_SET

6. MISSING commands.

6.1. ACCEPT   <length of longest acceptable response>

6.2. PREFER   <length of reasonable response>

6.3. INFO

The last info command applies mainly to databases of type "text".

6.4. SHOW

show	[ highlight ] fd=<fdnum> [offset=<byte> length=<bytes> ] | para=<start para> paras=<num> ] ... rn=<recnum> | s=<setnum> r=<record>

The "para" and "paras" qualifiers would only apply to databases of type "text".

7. Sizes, Problems and Limitations

The maximum number of sets is 16384 (numbered 0-16383). Each set occupies at least 232 bytes of memory, and may also have one or two files in the $BSS_TEMPPATH directory. If a FIND operation would result in two many sets it fails with a "NO_FREE_SETS" error. The memory for sets is now dynamically allocated, but this memory is never freed until there is a successful subsequent CHOOSE command.

The maximum number of sets which can be combined in a single FIND operation is operating-system dependent. On the Sun OS 4.1 machines you are recommended not to exceed about 110 (it may appear to work with up to about 238, but results are likely to be spurious). With SOLARIS it should be OK to go up to about 238.

There have been problems PARSE-ing long input. This seems to have been cured by using flex instead of lex to compile the command parser. (But note that the command parser will always stop at a linefeed; you may need to replace linefeeds by blanks before using either of the PARSE commands; we haven't found a way of getting round this.)

8. ENVIRONMENT VARIABLES

Three environment variables are read by the iinit() function. These are:

The low-level interface program ( i1+ ) looks for additional environmentvariables. These are

NOTE: if BSS_ATTRIBUTE is set without BSS_DB, there will be a "NO_DB_OPEN" error.

The main use of these environment variables is in scripts such as the following

#!/bin/sh
BSS_DB=trec123_94
BSS_ATTRIBUTE=dn
export BSS_DB BSS_ATTRIBUTE
i1+

which loads the low level interface with the database set to search on index "dn" before handing it over to the user.

A number of other environment variables are looked for by several BSS functions. Some of them control the parameters for passage searching, or for logistic regression (not documented here). Another is BSS_COMBINE_DO_STATS , which must be nonzero for the DISPLAYSTATS command to produce nonzero results.