[ TOC ]
The SWISH-E program is controlled by command line arguments (called switches). Often, SWISH-E is run manually from a shell (command prompt), or from a program such as a CGI script that passes the command line arguments to swish.
Note: A number of the command line switches may be specified in the SWISH-E
configuration file specified with the -c
command line argument. Please see SWISH-CONFIG for a complete description of available configuration file directives.
There are two basic operating modes of SWISH-E: indexing and searching. There are command line arguments that are unique to each mode, and others that apply to both (yet may have different meaning depending on the operating mode). These command line arguments are listed below, grouped by:
INDEXING -- describes the command line arguments used while indexing.
SEARCHING -- lists the command line arguments used while searching.
OTHER SWITCHES -- lists switches that don't apply to searching or indexing.
Beginning with SWISH-E version 2.1, you may embed the SWISH-E search engine into your applications. Please see SWISH-LIBRARY.
[ TOC ]
Swish indexing is initiated by passing command line arguments to swish. The command line arguments used for searching are described in SEARCHING. Also, see SWISH-SEARCH for examples of searching with SWISH-E.
SWISH-E usage:
swish-e [-i dir file ... ] [-c file] [-f file] [-l] \ [-v (num)] [-S method(fs|http|prog)] [-N path] |
The -h
switch (help) will list the available SWISH-E command line arguments:
swish-e -h |
Typically, most if not all indexing settings are placed in a configuration
file (specified with the -c
switch). Once the configuration file is setup indexing is initiated as:
swish-e -c /path/to/config/file |
See SWISH-CONFIG for information on the configuration file.
Security Note: If the swish binary is named swish-search
then swish will not allow any operation that would cause swish to write to
the index file.
When indexing it may be advisable to index to a temporary file, and then after indexing has successfully completed rename the file to the final location. This is especially important when replacing an index that is currently in use.
swish-e -c swish.config -f index.tmp [check return code from swish or look for err: output] mv index.tmp index.swish-e |
[ TOC ]
This specifies the directories and/or files to index. Directories will be indexed recursively. This is typically specified in the configuration file with the IndexDir directive instead of on the command line. Use of this switch overrides the configuration file settings.
This specifies the method to use for indexing. Can be either fs
for local indexing (the default),
http
for spidering, or prog for reading documents from an external program.
The fs
method simply reads files from a local (or networked) drive. This is the
most common way to index documents with Swish.
The http
method is used to spider web servers. It uses an included helper program
called swishspider
located in the src directory.
The prog method is new to SWISH-E version 2.2. It's designed as a general purpose
method to feed documents to swish from an external program. For example,
the program can read a database (e.g. MySQL), spider a web server, or
convert documents from one format to another (e.g. pdf to html). The
program name is passed to swish either by the IndexDir
directive, or via the -i
option. A few example programs are provided in the swish-e distribution
located in the prog-bin
directory.
The prog method bypasses some of the configuration parameters available to the file
system method -- settings such as IndexOnly
and FileRules
are ignored when using the prog method. It's expected that these operations are better accomplished in the
external program before passing the document onto swish. In other words,
when using the prog method, only send the documents to swish that you want indexed.
You may use swish's filter feature with the prog method, but performance will be better if you run filtering programs from within your external program.
Located in the conf directory are example configuration files that demonstrate indexing with the different document source methods.
If you are indexing, this specifies the file to save the generated index in, and you can only specify one file. See also IndexFile in the configuration file.
If you are searching, this specifies the index files (one or more) to search from. The default index file is index.swish-e in the current directory.
Specify the configuration file(s)
to use for indexing. This
file contains many directives that control how SWISH-E proceeds. See SWISH-CONFIG for a complete listing of configuration file directives.
Example:
swish-e -c docs.conf |
If you specify a directory to index, an index file, or the verbose option on the command-line, these values will override any specified in the configuration file.
You can specify multiple configuration files. For example, you may have one configuration file that has common site-wide settings, and another for a specific index.
Examples:
1) swish-e -c swish-e.conf 2) swish-e -i /usr/local/www -f index.swish-e -v -c swish-e.conf 3) swish-e -c swish-e.conf stopwords.conf |
The settings in the configuration file will be used to index a site.
These command-line options will override anything in the configuration file.
The variables in swish-e.conf will be read, then the variable in stopwords.conf will be read. Note that if the same variables occur in both files, older values may be written over.
For large sites indexing may require more RAM than is available. The -e
switch tells swish to use disk space to store data structures while
indexing, saving memory. This option is recommended if swish uses so much
RAM that the computer begins to swap excessively.
Specifying this option tells swish to follow symbolic links when indexing. The configuration file value FollowSymLinks will override the command-line value.
The default is not to follow symlinks. A small improvement in indexing time my result from enabling FollowSymLinks.
The -N
option takes a path to a file, and only files newer that the listed file will be indexed. This is helpful for creating
incremental indexes -- that is, indexes that contain just files added since
the last full index was created of all files.
Example (bad example)
swish-e -c config.file -N index.swish-e -f index.new This will index as normal, but only files with a modified date newer than C<index.swish-e> will be indexed. |
This is a bad example because it uses index.swish-e
which one might assume was the date of last indexing. The problem is that
files might have been added between the time indexing read the directory
and when the index.swish-e
file was created -- which can be quite a bit of time for very large
indexing jobs.
The only solution is to prevent any new file additions while full indexing is running. If this is impossible then it will be slightly better to do this:
Full indexing:
touch indexing_time.file swish-e -c config.file -f index.tmp mv index.tmp index.full |
Incremental indexing:
swish-e -c config.file -N indexing_time.file -f index.tmp mv index.tmp index.incremental |
Then search with
swish-e -w foo -f index.full index.incremental |
or merge the indexes
swish-e -M index.full index.incremental index.tmp mv index.tmp index.swish-e swish-e -w foo |
The -v
option can take a numerical value from 0 to 4. Specify 0 for completely
silent operation and 3 for detailed reports. A level of 4 and above is used
to generate mostly debugging output. If no value is given then 3 is
assumed. See also IndexReport in the configuration file.
[ TOC ]
The following command line arguments are available when searching with SWISH-E. These switches are used to select the index to search, what fields to search, and how and what to print as results.
This section just lists the available command line arguments and their usage. Please see SWISH-SEARCH for detailed searching instructions.
Warning: If using SWISH-E via a CGI interface, please see CGI Danger!
Security Note: If the swish binary is named swish-search
then swish will not allow any operation that would cause swish to write to
the index file.
[ TOC ]
This performs a case-insensitive search using a number of keywords. If no
index file to search is specified (via the -f
switch), swish-e will try to search a file called index.swish-e in the
current directory.
swish-e -w word |
Phrase searching is accomplished by placing the quote delimiter (a double-quote by default) around the search phrase.
swish-e -w 'word or "this phrase"' |
Search would should be protected from the shell by quotes. Typically, this is single quotes when running under Unix.
Under Windows command.com you may not need to use quotes, but you will need to backslash the quotes used to delimit phrases:
swish-e -w \"a phrase\" |
The phrase delimiter can be set with the -P
switch.
The search may be limited to a MetaName. For example:
swish-e -w meta1=(foo or baz) |
will only search within the meta1 tag.
Please see SWISH-SEARCH for a description of MetaNames.
Specifies the index file(s)
used while searching. More than
one file may be listed, and each file will be searched. If no -f
switch is specified then the file index.swish-e in the current directory will be used as the index file.
While searching, this specifies the maximum number of results to return. The default is to return all results.
This switch is often used in conjunction with the -b
switch to return results one page at a time (strongly recommended for large
indexes).
Sets the begining search result to return (records are numbered from 1). This switch can be
used with the -m
switch to return results in groups or pages.
Example:
swish-e -w 'word' -b 1 -m 20 # first 'page' swish-e -w 'word' -b 21 -m 20 # second 'page' |
The -t
option allows you to search for words that exist only in specific HTML
tags. Each character in the string you specify in the argument to this
option represents a different tag in which to search for the word. H means
all HEAD tags, B stands for BODY tags, t is all TITLE tags, h is H1 to H6
(header) tags, e is emphasized tags (this may be B, I, EM, or STRONG), and
c is HTML comment tags
search only in header (<H*>) tags
swish-c -w word -t h |
Set the delimiter used when printing results. By default, SWISH-E separates
the output fields by a space, and places double-quotes around the document
title. This output may be hard to parse, so it is recommended to use -d
to specify a character or string used as a separator between fields.
The string dq
means ``double-quotes''.
swish-e -w word -d , # single char swish-e -w word -d :: # string swish-e -w word -d '"' # double quotes under Unix swish-e -w word -d \" # double quotes under Windows swish-e -w word -d dq # double quotes |
The following control characters may also be specified: \t \r \n \f
.
Sets the delimiter used for phrase searches. The default is double quotes "
.
Some examples under bash: (be careful about you shell metacharacters)
swish-e -P ^ -w 'title=^words in a phrase^' swish-e -P \' -w "title='words in a pharse"' |
This causes swish to print the listed property in the search results. The
properties are returned in the order they are listed in the -p
argument.
Properties are defined by the ProperNames directive in the configuration file (see SWISH-CONFIG) and properties must also be defined in MetaNames. Swish stores the text of the meta name as a property, and then will return this text while searching if this option is used.
This feature is very useful for returning data included in a source documnet without having to re-read the source document while searching. For example, this could be used to return a short document description. See also see Document Summeries in SWISH-CONFIG.
To return the subject and category properties while indexing.
swish-e -w word -p subject category |
NOTE: it is necessary to have indexed with the proper PropertyNames directive in the user config file in order to use this option.
Normally, search results are printed out in order of relevancy, with the
most relevant listed first. The -s
sort switch allows you to sort results in order of a specified property, where a property
was defined using the MetaNames and PropertyNames directives during indexing (see SWISH-CONFIG).
The string passed can include the strings asc
and desc
to specify the sort order, and more than one property may be specified to
sort on more than one key.
Examples:
sort by title property ascending order
-s title |
sort descending by title, ascending by name
-s title desc name asc |
This is an experimental feature!
The -L switch can be used to limit search results to a range of properties
(properties are defined with the PropertyNames* family of directives).
Properties must be presorted for this feature to work. (Presorted
properties is the default swish behavior.) This feature will not work with swishrank
or swishdbfile
properties.
Example:
swish-e -w foo -L swishtitle a m |
finds all documents that contain the word foo
, and where the document's title is in the range of a
to m
, inclusive. Limiting may be done with user-defined properties, as well.
For example, if you indexed documents that contain a created timestamp in a meta tag:
<meta name="created_on" content="982648324"> |
Then you tell Swish that you have a property called created_on
, and that it's a timestamp.
PropertyNamesDate created_on |
After indexing you will be able to limit documents to a range of timestamps:
-w foo -L created_on 946684800 949363199 |
will find documents containing the word foo and that have a created_on date from the start of Jan 1, 2000 to the end of Jan 31, 2000.
Note: swish currently does not parse dates; Unix timestamps must be used.
Two special formats can be used:
-L swishtitle <= m -L swishtitle >= m |
Finds titles less than or equal, or grater than or equal to the letter m
. Case is ignored when comparing strings.
This is an experimental feature, and its use and interface are subject to change.
The -x
switch defines the output format string. The format string can contain
plain text and property names (including swish-defined internal property
names) and is used to generate the output for every result. In addition,
the output format of the property name can be controlled with C-like printf
format strings. This feature overrides the cmdline switches -d
and -p
, and a warning will be generated if -d
or -p
are used with -x
.
For example, to return just the title, one per line, in the search results:
swish-e -w ... -x '<swishtitle>\n' ... |
Note: the \n
may need to be protected from your shell.
See also the ResultExtFormatName
in SWISH-CONFIG for a way to define named
format strings in the swish configuration file.
Format of "formatstring":
"text<propertyname>text<propertyname fmt=propfmtstr>text..." |
Where propertyname is:
the name of a user property as specified with the config file directive ``PropertyNames''
the name of a swish Auto property (see below). These properties are defined automatically by swish -- you do not need to specify them with PropertyNames directive. (This may change in the future.)
propertynames must be placed within ``<'' and ``>''.
User properties:
SWISH-E allows you to specify certain META tags within your documents that can be used as document properties. The contents of any META tag that has been identified as a document property can be returned as part of the search results. Doucment properties must be defined while indexing using the PropertyNames configuration directive (see SWISH-CONFIG).
Examples of user-defined PropertyNames:
<keywords> <author> <deliveredby> <reference> <id> |
Note: In swish-e 2.2 all user properties are String type. This may change in future...
Auto properties:
Swish defines a number of ``Auto'' properties for each document indexed.
These are available for output when using the -x
format.
Name Type Contents swishreccount Integer Result record counter swishtitle String Document title (html only) swishrank Integer Result rank for this hit swishdocpath String URL or filepath to document swishdocsize Integer Document size in bytes swishlastmodified Date Last mod. date of document swishdescription String Description of document (see:StoreDescription) swishstartpos Integer [not yet used] swishdbfile String Path of swish database indexfile |
The Auto properties can also be specified using shortcuts:
%c = <swishreccount> (c-ount) %d = <swishdescription> %D = <swishlastmodified> (D-ate) %I = <swishdbfile> (I-ndex) %p = <swishdocpath> (P-ath) %r = <swishrank> %l = <swishdocsize> (lentgh) %S = <swishstartpos> %t = <swishtitle> %% = % |
Formatstrings of properties:
Properties listed in an -x
format string can include format control strings. These ``propertyformats''
are used to control how the contents of the associated property are
printed. Property formats are used like C-language printf formats. The
property format is specified by including the attribute ``fmt'' within the
property tag.
General syntax:
-x '<propertyname fmt="propfmtstr">' |
where subfmt
controls the output format of propertyname
.
Examples of property format strings:
date type: ...fmt='%d.%m.%Y' string type: ...fmt='%-40.20s' integer type: ...fmt=/%8.8d/ |
Please see the manual pages for strftime(3)
and
sprintf(3)
for an explanation of format strings.
Rainer -- are strftime and sprintf formats fully implemented?
The first character of a property format string defines the delimiter for this format string. For example,
swish -x "<author fmt=\"%20s\"> ...\n" swish -x "<author fmt='%20s'> ...\n" swish -x "<author fmt=/%20s/> ...\n" |
Standard predefined formats:
If you ommit the sub-format, the following formats are used:
String type: "%s" (like printf char *) Integer type: "%d" (like printf int) Float type: "%f" (like printf double) Date type: "%Y-%m-%d %H:%M:%S" (like strftime) special: fixed format string "%ld" = print seconds since epoch |
Text in "formatstring" or "propfmtstr":
Text will be output as-is in format strings (and property format strings). Special characters can be escaped with a backslash. To get a new line for each result hit, you have to include the Newline-Character ``\n'' at the end of ``fmtstr''.
-x "<swishreccount>|<swishrank>|<swishdocpath>\n" -x "Count=<swishreccount>, Rank=<swishrank>\n" -x "\<p\>Title=\<b\><swishtitle>\<p\>" -x 'Date: <swishlastmodified fmt="%m/%d/%Y">\n' -x 'Date in seconds: <swishlastmodified fmt=/%ld/>\n' |
Control/Escape charcters:
you can use C-like control escapes in the format string:
known controls: \a, \b, \f, \n, \r, \t, \v, digit escapes: \xhexdigits \0octaldigits character escapes: \anychar |
Example,
swish -x "%c\t%r\t%p\t\"<swishtitle fmt=/%40s/>\"\n" |
Examples of -x format strings:
-x "%c|%r|%p|%t|%D|%d\n" -x "%c|%r|%p|%t|<swishdate fmt=/%A, %d. %B %Y/>|%d\n" -x "<swishrank>\t<swishdocpath>\t<swishtitle>\t<keywords>\n -x "xml_out: \<title\><swishtitle>\>\</title\>\n" -x "xml_out: <swishtitle fmt='<title>%s</title>'>\n" |
The -H n
switch generates extened header output. This is most useful when searching more than one index file at a
time, either by specifying more than one index file with the -f
switch, or when searching a merged index file. In these cases, -H 2
will generate a set of headers specific to each index file. This gives
access to the settings used to generate each index file.
Even when searching a single index file, -H n
will provided additional information about the index file, how it was
indexed, and how swish is interperting the query.
-H 0 : print no header information, output only search result entries. -H 1 : print standard result header (default). -H 2 : print additional header information for each searched index file. -H 3 : enhanced header output (e.g. print stopwords). -H 9 : print diagnostic information in the header of the results (changed from: C<-v 4>) |
[ TOC ]
Print the current version.
The -k
switch is used for testing and will cause swish to print out all keywords
in the index beginning with that letter. You may enter -k '*'
to generate a list of all words indexed by swish.
This option is provided so you can check the word, file, and maintenance
information in index files. You can specify multiple files to decode. Use
with -v 4
for even more output. You must specify an index file with -f
, even if it's the default index file index.swish-e.
A lot of information is printed, so use with caution.
For example here's a few of the interesting parts from -D
(with -v 4
):
swish-e -D -v 4 -f test.index |
-----> WORD INFO <----- apples: Meta:1 ./test.html Strct:25 Freq:1 Pos:54 |
This tells you that the word apples
is found in ./test.html, and is *not* associated with any MetaNames (meta tag number one is the
default index). The Strct:25
is used to limit searches by context (see -t
above). Freq:1
says the word was found one time in the document at position
Pos:54
. This position data is used for phrase searches. If the word was found in
other meta tags you will see other Meta:
items listed. And the entire structure will be repeated, on the same line,
for every file that contains this word.
The next section:
<-----> FILES <-----> |
Lists the files, and their associated properties (including last modified date, title and file size).
-----> METANAMES <----- swishfilename"2 swishtitle"2 swishfiledate"10 swishstart"6 swishsize"6 swishsummary"2 meta1"3 meta2"3 meta3"3 |
(The above will be all on one line.)
The METANAME section lists the meta names and properties that are assigned internally within swish, and the names you specify in the configuration file with MetaNames and PropertyNames. The number that follow is a decimal number where each bit represents an attribute of that meta name. The bits are AND'ed together to make the attribute.
Bit Bit Represents ---------- -------------------------------------- 0000000001 META_INDEX (it is indexed -- MetaNames) 0000000010 META_PROP (it is a property -- PropertyNames) 0000000100 META_NUMBER (it is binary number.Eg: swishsize) 0000001000 META_DATE(it is a binary date. Eg: swishfiledate) |
So, for swishfiledate value 10 means META_PROP+META_DATE, or 2 (property and binary date).
Currently, the indexing of numbers (searching by range) is not supported.
[ TOC ]
At times it can be useful to merge different index files into one file for searching. This could be because you want to keep separate site indexes and a common one for a global search, or because your site is very large and Swish-e runs out of memory if you try to index it directly.
You should try to merge only indexes that were indexed with common settings. Otherwise results may be confusing (e.g. don't mix stemming and non-stemming indexes, or indexes with different WordCharacter settings).
usage: swish-e [-v (num)] [-c file] -M index1 index2 ... outputfile |
This allows you to merge two or more index files - the last file you specify on the list will be the output file.
Merging removes all redundant file and word data. To estimate how much memory the operation will need, sum up the sizes of the files to be merged and divide by two. That's about the maximum amount of memory that will be used.
You can use the -v
option to produce feedback while merging and the -c
option with a configuration file to include new administrative information
in the new index file.
Specify a configuration file while indexing to add administrative information to the output index file.
[ TOC ]
$Id: SWISH-RUN.pod,v 1.12 2001/06/17 04:13:33 whmoseley Exp $
. [ TOC ]