[ TOC ]
This document contains list of bug fixes and feature additions to SWISH-E.
[ TOC ]
Release Date: ???????
Many large changes were made internally in the code, some for performance reasons, some for feature changes and additions, and some to prepare for new features in later versions of SWISH-E.
Documentation is now included in the source distribution as .pod (perldoc) files, and as HTML files. In addition, the distribution can now generate PDF, postscript, and unix man pages from the source .pod files. See README for more information.
Pre 2.2 some internal data was stored in fixed locations within the index, namely the file name, file size, and title. 2.2 introduced new internal data such as the last modified date, and document summaries. This data is considered meta data since it is data about a document.
Instead of adding new data to the internal structure of the index file, it
was decided to use the MetaNames and PropertyNames feature of swish to
store this meta information. This allows for new meta data to be added at a
later time (e.g. Content-type), and provides an easy and customizable way
to print results with the -p
switch and the new -x
switch. In addition, this data can now be sorted by this internal data.
For example, to sort by the rank and title:
swish-e -w foo -s swishrank desc swishtitle asc |
If you are parsing output headers in a program then you may need to adjust your code. There's a new switch <-H> to control the level of header output when searching.
Swish now merges (and sorts) the results from multiple indexes when using -f
to specify more than one index. This change effects the way maxhits (-m
) works. Here's a summary of the way swish works for the different
versions.
1.3.2 - MaxHits returns first N results starting from the first index. e.g. maxhits=20; 15 hits Index1, 40 hits Index2 All 15 from Index1 plus first five from Index2 = 20 hits. |
2.0.0 - MaxHits returns first N results from each index. e.g. Maxhits=20; 15 hits Index1, 40 hits Index2 All 15 from Index1 plus 15 from Index2 = 40 hits |
2.2.0 - Results are merged and first N results are returned. e.g. Maxhits=20; 15 hits Index1, 40 hits Index2 Results are merged from each index and sorted (rank is the default sort) and only the first 20 are returned. |
A number of improvements in indexing and searching speed were made, especially for searching very large indexes and sorting by properties.
You can now use -S prog to use an external program to supply documents to
swish. This program can be used to spider web servers, index databases, or
to convert any type of document into html, xml, or text, so it can be
indexed by swish. Examples are given in the prog-bin
directory.
TranslateCharacters now is done before WordCharacters is checked. For example,
WordCharacters abcdefghijklmnopqrstuvwxyz TranslateCharacters ñ n |
Now El Niño
will be indexed as El Nino (el and nino), even though ñ
is not listed in WordCharacters.
Previously, stopwords were checked after stemming and soundex conversions, as well as most of the other word checks (WordCharacters, min/max length and so on). This meant that the stopword list probably didn't work as expected when using stemming. This has been corrected.
The search parser was rewritten to correct a number of logic errors. Previously, swish did not differentiate between meta names, swish operators and search words when parsing the query. This meant, for example, that metanames might be broken up by the WordCharacters setting, and that it could be stemmed. This has been fixed.
Swish operator characters "*()=
can now be searched by escaping with a backslash. For example:
./swish-e -w 'this\=odd\)word' |
will end up searching for the word this=odd)word
. To search for a backslash character preceed it with a backslash.
Currently, searching for:
./swish-e -w 'this\*' |
is the same as a wildcard search. This may be fixed in the future.
Searching for buzzwords with those characters will still require backslashing. This also may change to allow some un-escaped operator characters, but some will always need to be escaped (e.g. the double-quote phrase character).
Previous versions of swish included a configuration file called user.config
which contained examples of all directives. This has been replace by a
series of example configuration files located in the conf
directory. The configuration directives are now described in SWISH-CONFIG.
David Norris has included the files required to build swish under Windows.
In addition, a binary version of swish for Win32 is now included in the
distribution. See src/win32
.
Jean-François Piéronne has provided the files required to build swish under
OpenVMS. See src/vms
for more information.
Changes to Configuration File Directives. Please see SWISH-CONFIG for more info.
The IndexContents directive assigns internal Swish document parsers to files based on their file type. The DefaultContents directive assigns a parser to be used on file that are not assigned a parser with IndexContents.
This describes what to do when a meta tag is found in a document that is not listed in the MetaNames directive.
Will ignore listed tags in XML files
Passes words listed to the external swish-e program when running with
-S prog
document source method.
Controls parsing and conversion of HTML entities.
The word position is now bumped when a new metatag is found -- this is to prevent phrases from matching across meta tags. This directive will disable this behavior for the listed tags.
This currently works only with XML parsing.
This has been changed such that comments are not indexed by default.
The builtin list of stop words has been removed. Use of the SwishDefault
word will generate a warning. A sample file stopwords.txt
has been included if you wish to use these default words using IgnoreWords File: stopwords.txt
The default is now ``yes''.
Buzzwords are words that should be indexed as-is, without checking for
stopwords, word length, WordCharacters, or any other of the word limiting
features. This allows indexing of things like C++
when ``+'' is not listed in WordCharacters.
Currenly, IgnoreFirstChar and IgnoreLastChar will be stripped before processing Buzzwords.
In the future we may use separate IgnoreFirst/Last settings for buzzwords
since, for example, you may wish to index all +
within swish words, but strip +
from the start/end of swish words, but not from the buzzword C++
.
Before Swish 2.2 all user-defined document properties were stored in the index as strings. PropertyNamesNumeric and PropertyNamesDate tells swish that a property should be stored in binary format. This allows for correct sorting of numeric properties.
Currenly, only integers can be stored, such as a unix timestamp. (Swish
uses strtoul
to convert the number to an unsigned long internally.)
PropertyNamesDate only indicates to swish that a number is a unix timestamp, and to display the property as a formatted time when printing results. Swish can not currently parse date strings.
Changes to command line arguments. See SWISH-RUN for documentation on these switches.
Controls the level (verbosity) of header information printed with search results.
Provides additional header output and allows for a format string to describe what data to print.
Prints words stored in the swish index.
Provides a way to do incremental indexing by comparing last modification
dates. You pass -N
a path to a file and only files newer than the last modified date of that
file will be indexed.
When -D
is used with -v 4
additional debugging data is printed.
Now -d can accept some back-slashed characters to be used as output separators.
Now -P sets the phrase delimiter character in searches.
Swish 2.2 contains an experimental feature to limit results by a range of property values. This behavior of this feature may change in the future.