prefix matching using this facility.
Teodor Sigaev and Oleg Bartunov
-<!-- $PostgreSQL: pgsql/doc/src/sgml/datatype.sgml,v 1.226 2008/03/30 04:08:14 neilc Exp $ -->
+<!-- $PostgreSQL: pgsql/doc/src/sgml/datatype.sgml,v 1.227 2008/05/16 16:31:01 tgl Exp $ -->
<chapter id="datatype">
<title id="datatype-title">Data Types</title>
SELECT 'a fat cat sat on a mat and ate a fat rat'::tsvector;
tsvector
----------------------------------------------------
- 'a' 'on' 'and' 'ate' 'cat' 'fat' 'mat' 'rat' 'sat'
+ 'a' 'and' 'ate' 'cat' 'fat' 'mat' 'on' 'rat' 'sat'
</programlisting>
- (As the example shows, the sorting is first by length and then
- alphabetically, but that detail is seldom important.) To represent
+ To represent
lexemes containing whitespace or punctuation, surround them with quotes:
<programlisting>
SELECT $$the lexeme ' ' contains spaces$$::tsvector;
tsvector
-------------------------------------------
- 'the' ' ' 'lexeme' 'spaces' 'contains'
+ ' ' 'contains' 'lexeme' 'spaces' 'the'
</programlisting>
(We use dollar-quoted string literals in this example and the next one,
SELECT $$the lexeme 'Joe''s' contains a quote$$::tsvector;
tsvector
------------------------------------------------
- 'a' 'the' 'Joe''s' 'quote' 'lexeme' 'contains'
+ 'Joe''s' 'a' 'contains' 'lexeme' 'quote' 'the'
</programlisting>
Optionally, integer <firstterm>position(s)</>
SELECT 'a:1 fat:2 cat:3 sat:4 on:5 a:6 mat:7 and:8 ate:9 a:10 fat:11 rat:12'::tsvector;
tsvector
-------------------------------------------------------------------------------
- 'a':1,6,10 'on':5 'and':8 'ate':9 'cat':3 'fat':2,11 'mat':7 'rat':12 'sat':4
+ 'a':1,6,10 'and':8 'ate':9 'cat':3 'fat':2,11 'mat':7 'on':5 'rat':12 'sat':4
</programlisting>
A position normally indicates the source word's location in the
select 'The Fat Rats'::tsvector;
tsvector
--------------------
- 'Fat' 'The' 'Rats'
+ 'Fat' 'Rats' 'The'
</programlisting>
For most English-text-searching applications the above words would
</programlisting>
</para>
+ <para>
+ Also, lexemes in a <type>tsquery</type> can be labeled with <literal>*</>
+ to specify prefix matching:
+<programlisting>
+SELECT 'super:*'::tsquery;
+ tsquery
+-----------
+ 'super':*
+</programlisting>
+ This query will match any word in a <type>tsvector</> that begins
+ with <quote>super</>.
+ </para>
+
<para>
Quoting rules for lexemes are the same as described above for
lexemes in <type>tsvector</>; and, as with <type>tsvector</>,
-<!-- $PostgreSQL: pgsql/doc/src/sgml/gin.sgml,v 2.14 2008/04/14 17:05:32 tgl Exp $ -->
+<!-- $PostgreSQL: pgsql/doc/src/sgml/gin.sgml,v 2.15 2008/05/16 16:31:01 tgl Exp $ -->
<chapter id="GIN">
<title>GIN Indexes</title>
</para>
<para>
- All it takes to get a <acronym>GIN</acronym> access method working
- is to implement four user-defined methods, which define the behavior of
+ All it takes to get a <acronym>GIN</acronym> access method working is to
+ implement four (or five) user-defined methods, which define the behavior of
keys in the tree and the relationships between keys, indexed values,
and indexable queries. In short, <acronym>GIN</acronym> combines
extensibility with generality, code reuse, and a clean interface.
</para>
<para>
- The four methods that an index operator class for
+ The four methods that an operator class for
<acronym>GIN</acronym> must provide are:
</para>
</varlistentry>
<varlistentry>
- <term>Datum* extractValue(Datum inputValue, int32 *nkeys)</term>
+ <term>Datum *extractValue(Datum inputValue, int32 *nkeys)</term>
<listitem>
<para>
Returns an array of keys given a value to be indexed. The
</varlistentry>
<varlistentry>
- <term>Datum* extractQuery(Datum query, int32 *nkeys,
- StrategyNumber n)</term>
+ <term>Datum *extractQuery(Datum query, int32 *nkeys,
+ StrategyNumber n, bool **pmatch)</term>
<listitem>
<para>
Returns an array of keys given a value to be queried; that is,
to consult <literal>n</> to determine the data type of
<literal>query</> and the key values that need to be extracted.
The number of returned keys must be stored into <literal>*nkeys</>.
- If number of keys is equal to zero then <function>extractQuery</>
- should store 0 or -1 into <literal>*nkeys</>. 0 means that any
- row matches the <literal>query</> and sequence scan should be
- produced. -1 means nothing can satisfy <literal>query</>.
- Choice of value should be based on semantics meaning of operation with
- given strategy number.
+ If the query contains no keys then <function>extractQuery</>
+ should store 0 or -1 into <literal>*nkeys</>, depending on the
+ semantics of the operator. 0 means that every
+ value matches the <literal>query</> and a sequential scan should be
+ produced. -1 means nothing can match the <literal>query</>.
+ <literal>pmatch</> is an output argument for use when partial match
+ is supported. To use it, <function>extractQuery</> must allocate
+ an array of <literal>*nkeys</> booleans and store its address at
+ <literal>*pmatch</>. Each element of the array should be set to TRUE
+ if the corresponding key requires partial match, FALSE if not.
+ If <literal>*pmatch</> is set to NULL then GIN assumes partial match
+ is not required. The variable is initialized to NULL before call,
+ so this argument can simply be ignored by operator classes that do
+ not support partial match.
</para>
+
</listitem>
</varlistentry>
</variablelist>
+ <para>
+ Optionally, an operator class for
+ <acronym>GIN</acronym> can supply a fifth method:
+ </para>
+
+ <variablelist>
+
+ <varlistentry>
+ <term>int comparePartial(Datum partial_key, Datum key, StrategyNumber n)</term>
+ <listitem>
+ <para>
+ Compare a partial-match query to an index key. Returns an integer
+ whose sign indicates the result: less than zero means the index key
+ does not match the query, but the index scan should continue; zero
+ means that the index key does match the query; greater than zero
+ indicates that the index scan should stop because no more matches
+ are possible. The strategy number <literal>n</> of the operator
+ that generated the partial match query is provided, in case its
+ semantics are needed to determine when to end the scan.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ <para>
+ To support <quote>partial match</> queries, an operator class must
+ provide the <function>comparePartial</> method, and its
+ <function>extractQuery</> method must set the <literal>pmatch</>
+ parameter when a partial-match query is encountered. See
+ <xref linkend="gin-partial-match"> for details.
+ </para>
+
</sect1>
<sect1 id="gin-implementation">
list of heap pointers (PL, posting list) if the list is small enough.
</para>
+ <sect2 id="gin-partial-match">
+ <title>Partial match algorithm</title>
+
+ <para>
+ GIN can support <quote>partial match</> queries, in which the query
+ does not determine an exact match for one or more keys, but the possible
+ matches fall within a reasonably narrow range of key values (within the
+ key sorting order determined by the <function>compare</> support method).
+ The <function>extractQuery</> method, instead of returning a key value
+ to be matched exactly, returns a key value that is the lower bound of
+ the range to be searched, and sets the <literal>pmatch</> flag true.
+ The key range is then searched using the <function>comparePartial</>
+ method. <function>comparePartial</> must return zero for an actual
+ match, less than zero for a non-match that is still within the range
+ to be searched, or greater than zero if the index key is past the range
+ that could match.
+ </para>
+
+ <para>
+ During a partial-match scan, all <literal>itemPointer</>s for matching keys
+ are OR'ed into a <literal>TIDBitmap</>.
+ The scan fails if the <literal>TIDBitmap</> becomes lossy.
+ In this case an error message will be reported with advice
+ to increase <literal>work_mem</>.
+ </para>
+ </sect2>
+
</sect1>
<sect1 id="gin-tips">
</para>
<para>
- <acronym>GIN</acronym> searches keys only by equality matching. This might
- be improved in future.
+ It is possible for an operator class to circumvent the restriction against
+ full index scan. To do that, <function>extractValue</> must return at least
+ one (possibly dummy) key for every indexed value, and
+ <function>extractQuery</function> must convert an unrestricted search into
+ a partial-match query that will scan the whole index. This is inefficient
+ but might be necessary to avoid corner-case failures with operators such
+ as LIKE. Note however that failure could still occur if the intermediate
+ <literal>TIDBitmap</> becomes lossy.
</para>
</sect1>
<para>
The <productname>PostgreSQL</productname> source distribution includes
<acronym>GIN</acronym> operator classes for <type>tsvector</> and
- for one-dimensional arrays of all internal types. The following
- <filename>contrib</> modules also contain <acronym>GIN</acronym>
- operator classes:
+ for one-dimensional arrays of all internal types. Prefix searching in
+ <type>tsvector</> is implemented using the <acronym>GIN</> partial match
+ feature.
+ The following <filename>contrib</> modules also contain
+ <acronym>GIN</acronym> operator classes:
</para>
<variablelist>
-<!-- $PostgreSQL: pgsql/doc/src/sgml/textsearch.sgml,v 1.43 2008/04/14 17:05:32 tgl Exp $ -->
+<!-- $PostgreSQL: pgsql/doc/src/sgml/textsearch.sgml,v 1.44 2008/05/16 16:31:01 tgl Exp $ -->
<chapter id="textsearch">
<title id="textsearch-title">Full Text Search</title>
'fat' | 'rat':AB
</programlisting>
+ Also, <literal>*</> can be attached to a lexeme to specify prefix matching:
+
+<programlisting>
+SELECT to_tsquery('supern:*A & star:A*B');
+ to_tsquery
+--------------------------
+ 'supern':*A & 'star':*AB
+</programlisting>
+
+ Such a lexeme will match any word in a <type>tsvector</> that begins
+ with the given string.
+ </para>
+
+ <para>
<function>to_tsquery</function> can also accept single-quoted
phrases. This is primarily useful when the configuration includes a
thesaurus dictionary that may trigger on such phrases.
</programlisting>
Note that <function>plainto_tsquery</> cannot
- recognize either Boolean operators or weight labels in its input:
+ recognize Boolean operators, weight labels, or prefix-match labels
+ in its input:
<programlisting>
SELECT plainto_tsquery('english', 'The Fat & Rats:C');
-<!-- $PostgreSQL: pgsql/doc/src/sgml/xindex.sgml,v 1.62 2008/04/14 17:05:32 tgl Exp $ -->
+<!-- $PostgreSQL: pgsql/doc/src/sgml/xindex.sgml,v 1.63 2008/05/16 16:31:01 tgl Exp $ -->
<sect1 id="xindex">
<title>Interfacing Extensions To Indexes</title>
<entry>consistent - determine whether value matches query condition</entry>
<entry>4</entry>
</row>
+ <row>
+ <entry>comparePartial - (optional method) compare partial key from
+ query and key from index, and return an integer less than zero, zero,
+ or greater than zero, indicating whether GIN should ignore this index
+ entry, treat the entry as a match, or stop the index scan</entry>
+ <entry>5</entry>
+ </row>
</tbody>
</tgroup>
</table>
* Portions Copyright (c) 1994, Regents of the University of California
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/access/gin/ginget.c,v 1.15 2008/05/12 00:00:44 alvherre Exp $
+ * $PostgreSQL: pgsql/src/backend/access/gin/ginget.c,v 1.16 2008/05/16 16:31:01 tgl Exp $
*-------------------------------------------------------------------------
*/
#include "catalog/index.h"
#include "miscadmin.h"
#include "storage/bufmgr.h"
+#include "utils/datum.h"
#include "utils/memutils.h"
+
+/*
+ * Tries to refind previously taken ItemPointer on page.
+ */
static bool
findItemInPage(Page page, ItemPointer item, OffsetNumber *off)
{
}
/*
- * Start* functions setup state of searches: find correct buffer and locks it,
- * Stop* functions unlock buffer (but don't release!)
+ * Goes to the next page if current offset is outside of bounds
+ */
+static bool
+moveRightIfItNeeded( GinBtreeData *btree, GinBtreeStack *stack )
+{
+ Page page = BufferGetPage(stack->buffer);
+
+ if ( stack->off > PageGetMaxOffsetNumber(page) )
+ {
+ /*
+ * We scanned the whole page, so we should take right page
+ */
+ stack->blkno = GinPageGetOpaque(page)->rightlink;
+
+ if ( GinPageRightMost(page) )
+ return false; /* no more pages */
+
+ LockBuffer(stack->buffer, GIN_UNLOCK);
+ stack->buffer = ReleaseAndReadBuffer(stack->buffer, btree->index, stack->blkno);
+ LockBuffer(stack->buffer, GIN_SHARE);
+ stack->off = FirstOffsetNumber;
+ }
+
+ return true;
+}
+
+/*
+ * Does fullscan of posting tree and saves ItemPointers
+ * in scanEntry->partialMatch TIDBitmap
+ */
+static void
+scanForItems( Relation index, GinScanEntry scanEntry, BlockNumber rootPostingTree )
+{
+ GinPostingTreeScan *gdi;
+ Buffer buffer;
+ Page page;
+ BlockNumber blkno;
+
+ gdi = prepareScanPostingTree(index, rootPostingTree, TRUE);
+
+ buffer = scanBeginPostingTree(gdi);
+ IncrBufferRefCount(buffer); /* prevent unpin in freeGinBtreeStack */
+
+ freeGinBtreeStack(gdi->stack);
+ pfree(gdi);
+
+ /*
+ * Goes through all leaves
+ */
+ for(;;)
+ {
+ page = BufferGetPage(buffer);
+
+ if ((GinPageGetOpaque(page)->flags & GIN_DELETED) == 0 && GinPageGetOpaque(page)->maxoff >= FirstOffsetNumber )
+ {
+ tbm_add_tuples( scanEntry->partialMatch,
+ (ItemPointer)GinDataPageGetItem(page, FirstOffsetNumber),
+ GinPageGetOpaque(page)->maxoff, false);
+ scanEntry->predictNumberResult += GinPageGetOpaque(page)->maxoff;
+ }
+
+ blkno = GinPageGetOpaque(page)->rightlink;
+ if ( GinPageRightMost(page) )
+ {
+ UnlockReleaseBuffer(buffer);
+ return; /* no more pages */
+ }
+
+ LockBuffer(buffer, GIN_UNLOCK);
+ buffer = ReleaseAndReadBuffer(buffer, index, blkno);
+ LockBuffer(buffer, GIN_SHARE);
+ }
+}
+
+/*
+ * Collects all ItemPointer into the TIDBitmap struct
+ * for entries partially matched to search entry.
+ *
+ * Returns true if done, false if it's needed to restart scan from scratch
+ */
+static bool
+computePartialMatchList( GinBtreeData *btree, GinBtreeStack *stack, GinScanEntry scanEntry )
+{
+ Page page;
+ IndexTuple itup;
+ Datum idatum;
+ bool isnull;
+ int32 cmp;
+
+ scanEntry->partialMatch = tbm_create( work_mem * 1024L );
+
+ for(;;)
+ {
+ /*
+ * stack->off points to the interested entry, buffer is already locked
+ */
+ if ( moveRightIfItNeeded(btree, stack) == false )
+ return true;
+
+ page = BufferGetPage(stack->buffer);
+ itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, stack->off));
+ idatum = index_getattr(itup, 1, btree->ginstate->tupdesc, &isnull);
+ Assert(!isnull);
+
+ /*----------
+ * Check of partial match.
+ * case cmp == 0 => match
+ * case cmp > 0 => not match and finish scan
+ * case cmp < 0 => not match and continue scan
+ *----------
+ */
+ cmp = DatumGetInt32(FunctionCall3(&btree->ginstate->comparePartialFn,
+ scanEntry->entry,
+ idatum,
+ UInt16GetDatum(scanEntry->strategy)));
+
+ if ( cmp > 0 )
+ return true;
+ else if ( cmp < 0 )
+ {
+ stack->off++;
+ continue;
+ }
+
+ if ( GinIsPostingTree(itup) )
+ {
+ BlockNumber rootPostingTree = GinGetPostingTree(itup);
+ Datum newDatum,
+ savedDatum = datumCopy (
+ idatum,
+ btree->ginstate->tupdesc->attrs[0]->attbyval,
+ btree->ginstate->tupdesc->attrs[0]->attlen
+ );
+ /*
+ * We should unlock current page (but not unpin) during
+ * tree scan to prevent deadlock with vacuum processes.
+ *
+ * We save current entry value (savedDatum) to be able to refind
+ * our tuple after re-locking
+ */
+ LockBuffer(stack->buffer, GIN_UNLOCK);
+ scanForItems( btree->index, scanEntry, rootPostingTree );
+
+ /*
+ * We lock again the entry page and while it was unlocked
+ * insert might occured, so we need to refind our position
+ */
+ LockBuffer(stack->buffer, GIN_SHARE);
+ page = BufferGetPage(stack->buffer);
+ if ( !GinPageIsLeaf(page) )
+ {
+ /*
+ * Root page becomes non-leaf while we unlock it. We
+ * will start again, this situation doesn't cause
+ * often - root can became a non-leaf only one per
+ * life of index.
+ */
+
+ return false;
+ }
+
+ for(;;)
+ {
+ if ( moveRightIfItNeeded(btree, stack) == false )
+ elog(ERROR, "lost saved point in index"); /* must not happen !!! */
+
+ page = BufferGetPage(stack->buffer);
+ itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, stack->off));
+ newDatum = index_getattr(itup, FirstOffsetNumber, btree->ginstate->tupdesc, &isnull);
+
+ if ( compareEntries(btree->ginstate, newDatum, savedDatum) == 0 )
+ {
+ /* Found! */
+ if ( btree->ginstate->tupdesc->attrs[0]->attbyval == false )
+ pfree( DatumGetPointer(savedDatum) );
+ break;
+ }
+
+ stack->off++;
+ }
+ }
+ else
+ {
+ tbm_add_tuples( scanEntry->partialMatch, GinGetPosting(itup), GinGetNPosting(itup), false);
+ scanEntry->predictNumberResult += GinGetNPosting(itup);
+ }
+
+ /*
+ * Ok, we save ItemPointers, go to the next entry
+ */
+ stack->off++;
+ }
+
+ return true;
+}
+
+/*
+ * Start* functions setup begining state of searches: finds correct buffer and pins it.
*/
static void
startScanEntry(Relation index, GinState *ginstate, GinScanEntry entry)
entry->offset = InvalidOffsetNumber;
entry->list = NULL;
entry->nlist = 0;
+ entry->partialMatch = NULL;
+ entry->partialMatchResult = NULL;
entry->reduceResult = FALSE;
entry->predictNumberResult = 0;
- if (btreeEntry.findItem(&btreeEntry, stackEntry))
+ if ( entry->isPartialMatch )
+ {
+ /*
+ * btreeEntry.findItem points to the first equal or greater value
+ * than needed. So we will scan further and collect all
+ * ItemPointers
+ */
+ btreeEntry.findItem(&btreeEntry, stackEntry);
+ if ( computePartialMatchList( &btreeEntry, stackEntry, entry ) == false )
+ {
+ /*
+ * GIN tree was seriously restructured, so we will
+ * cleanup all found data and rescan. See comments near
+ * 'return false' in computePartialMatchList()
+ */
+ if ( entry->partialMatch )
+ {
+ tbm_free( entry->partialMatch );
+ entry->partialMatch = NULL;
+ }
+ LockBuffer(stackEntry->buffer, GIN_UNLOCK);
+ freeGinBtreeStack(stackEntry);
+
+ startScanEntry(index, ginstate, entry);
+ return;
+ }
+
+ if ( entry->partialMatch && !tbm_is_empty(entry->partialMatch) )
+ {
+ tbm_begin_iterate(entry->partialMatch);
+ entry->isFinished = FALSE;
+ }
+ }
+ else if (btreeEntry.findItem(&btreeEntry, stackEntry))
{
IndexTuple itup = (IndexTuple) PageGetItem(page, PageGetItemId(page, stackEntry->off));
GinPostingTreeScan *gdi;
Page page;
+ /*
+ * We should unlock entry page before make deal with
+ * posting tree to prevent deadlocks with vacuum processes.
+ * Because entry is never deleted from page and posting tree is
+ * never reduced to the posting list, we can unlock page after
+ * getting BlockNumber of root of posting tree.
+ */
LockBuffer(stackEntry->buffer, GIN_UNLOCK);
needUnlock = FALSE;
gdi = prepareScanPostingTree(index, rootPostingTree, TRUE);
*/
entry->list = (ItemPointerData *) palloc( BLCKSZ );
entry->nlist = GinPageGetOpaque(page)->maxoff;
- memcpy( entry->list, GinDataPageGetItem(page, FirstOffsetNumber),
+ memcpy( entry->list, GinDataPageGetItem(page, FirstOffsetNumber),
GinPageGetOpaque(page)->maxoff * sizeof(ItemPointerData) );
LockBuffer(entry->buffer, GIN_UNLOCK);
return;
for (i = 0; i < key->nentries; i++)
+ {
startScanEntry(index, ginstate, key->scanEntry + i);
+ /*
+ * Copy strategy number to each entry of key to
+ * use in comparePartialFn call
+ */
+ key->scanEntry[i].strategy = key->strategy;
+ }
memset(key->entryRes, TRUE, sizeof(bool) * key->nentries);
key->isFinished = FALSE;
* Found position equal to or greater than stored
*/
entry->nlist = GinPageGetOpaque(page)->maxoff;
- memcpy( entry->list, GinDataPageGetItem(page, FirstOffsetNumber),
+ memcpy( entry->list, GinDataPageGetItem(page, FirstOffsetNumber),
GinPageGetOpaque(page)->maxoff * sizeof(ItemPointerData) );
LockBuffer(entry->buffer, GIN_UNLOCK);
- if ( !ItemPointerIsValid(&entry->curItem) ||
+ if ( !ItemPointerIsValid(&entry->curItem) ||
compareItemPointers( &entry->curItem, entry->list + entry->offset - 1 ) == 0 )
{
/*
break;
}
-
+
/*
* Find greater than entry->curItem position, store it.
*/
entry->isFinished = entry->master->isFinished;
entry->curItem = entry->master->curItem;
}
+ else if ( entry->partialMatch )
+ {
+ do
+ {
+ if ( entry->partialMatchResult == NULL || entry->offset >= entry->partialMatchResult->ntuples )
+ {
+ entry->partialMatchResult = tbm_iterate( entry->partialMatch );
+
+ if ( entry->partialMatchResult == NULL )
+ {
+ ItemPointerSet(&entry->curItem, InvalidBlockNumber, InvalidOffsetNumber);
+ entry->isFinished = TRUE;
+ break;
+ }
+ else if ( entry->partialMatchResult->ntuples < 0 )
+ {
+ /* bitmap became lossy */
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("not enough memory to store result of partial match operator" ),
+ errhint("Increase the \"work_mem\" parameter.")));
+ }
+ entry->offset = 0;
+ }
+
+ ItemPointerSet(&entry->curItem,
+ entry->partialMatchResult->blockno,
+ entry->partialMatchResult->offsets[ entry->offset ]);
+ entry->offset ++;
+
+ } while (entry->isFinished == FALSE && entry->reduceResult == TRUE && dropItem(entry));
+ }
else if (!BufferIsValid(entry->buffer))
{
entry->offset++;
return entry->isFinished;
}
+/*
+ * restart from saved position. Actually it's needed only for
+ * partial match. function is called only by ginrestpos()
+ */
+void
+ginrestartentry(GinScanEntry entry)
+{
+ ItemPointerData stopItem = entry->curItem;
+ bool savedReduceResult;
+
+ if ( entry->master || entry->partialMatch == NULL )
+ return; /* entry is slave or not a partial match type*/
+
+ if ( entry->isFinished )
+ return; /* entry was finished before ginmarkpos() call */
+
+ if ( ItemPointerGetBlockNumber(&stopItem) == InvalidBlockNumber )
+ return; /* entry wasn't began before ginmarkpos() call */
+
+ /*
+ * Reset iterator
+ */
+ tbm_begin_iterate( entry->partialMatch );
+ entry->partialMatchResult = NULL;
+ entry->offset = 0;
+
+ /*
+ * Temporary reset reduceResult flag to guarantee refinding
+ * of curItem
+ */
+ savedReduceResult = entry->reduceResult;
+ entry->reduceResult = FALSE;
+
+ do
+ {
+ /*
+ * We can use null instead of index because
+ * partial match doesn't use it
+ */
+ if ( entryGetItem( NULL, entry ) == false )
+ elog(ERROR, "cannot refind scan position"); /* must not be here! */
+ } while( compareItemPointers( &stopItem, &entry->curItem ) != 0 );
+
+ Assert( entry->isFinished == FALSE );
+
+ entry->reduceResult = savedReduceResult;
+}
+
/*
* Sets key->curItem to new found heap item pointer for one scan key
* Returns isFinished, ie TRUE means we did NOT get a new item pointer!
bool res;
if (dir != ForwardScanDirection)
- elog(ERROR, "Gin doesn't support other scan directions than forward");
+ elog(ERROR, "GIN doesn't support other scan directions than forward");
if (GinIsNewKey(scan))
newScanKey(scan);
* Portions Copyright (c) 1994, Regents of the University of California
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/access/gin/ginscan.c,v 1.13 2008/05/12 00:00:44 alvherre Exp $
+ * $PostgreSQL: pgsql/src/backend/access/gin/ginscan.c,v 1.14 2008/05/16 16:31:01 tgl Exp $
*-------------------------------------------------------------------------
*/
static void
fillScanKey(GinState *ginstate, GinScanKey key, Datum query,
- Datum *entryValues, uint32 nEntryValues, StrategyNumber strategy)
+ Datum *entryValues, bool *partial_matches, uint32 nEntryValues,
+ StrategyNumber strategy)
{
uint32 i,
j;
key->scanEntry[i].buffer = InvalidBuffer;
key->scanEntry[i].list = NULL;
key->scanEntry[i].nlist = 0;
+ key->scanEntry[i].isPartialMatch = ( ginstate->canPartialMatch && partial_matches )
+ ? partial_matches[i] : false;
/* link to the equals entry in current scan key */
key->scanEntry[i].master = NULL;
key->scanEntry[j].buffer = InvalidBuffer;
key->scanEntry[j].list = NULL;
key->scanEntry[j].nlist = 0;
+ key->scanEntry[j].partialMatch = NULL;
+ key->scanEntry[j].partialMatchResult = NULL;
}
}
}
ReleaseBuffer(key->scanEntry[j].buffer);
if (removeRes && key->scanEntry[j].list)
pfree(key->scanEntry[j].list);
+ if (removeRes && key->scanEntry[j].partialMatch)
+ tbm_free(key->scanEntry[j].partialMatch);
}
if (removeRes)
{
Datum *entryValues;
int32 nEntryValues;
+ bool *partial_matches = NULL;
- if (scankey[i].sk_flags & SK_ISNULL)
- elog(ERROR, "Gin doesn't support NULL as scan key");
Assert(scankey[i].sk_attno == 1);
- entryValues = (Datum *) DatumGetPointer(
- FunctionCall3(
+ /* XXX can't we treat nulls by just setting isVoidRes? */
+ /* This would amount to assuming that all GIN operators are strict */
+ if (scankey[i].sk_flags & SK_ISNULL)
+ elog(ERROR, "GIN doesn't support NULL as scan key");
+
+ entryValues = (Datum *) DatumGetPointer(FunctionCall4(
&so->ginstate.extractQueryFn,
scankey[i].sk_argument,
PointerGetDatum(&nEntryValues),
- UInt16GetDatum(scankey[i].sk_strategy)
- )
- );
+ UInt16GetDatum(scankey[i].sk_strategy),
+ PointerGetDatum(&partial_matches)));
if (nEntryValues < 0)
{
/*
so->isVoidRes = true;
break;
}
+
+ /*
+ * extractQueryFn signals that everything matches
+ */
if (entryValues == NULL || nEntryValues == 0)
/* full scan... */
continue;
fillScanKey(&so->ginstate, &(so->keys[nkeys]), scankey[i].sk_argument,
- entryValues, nEntryValues, scankey[i].sk_strategy);
+ entryValues, partial_matches, nEntryValues, scankey[i].sk_strategy);
nkeys++;
}
}
static GinScanKey
-copyScanKeys(GinScanKey keys, uint32 nkeys)
+copyScanKeys(GinScanKey keys, uint32 nkeys, bool restart)
{
GinScanKey newkeys;
uint32 i,
newkeys[i].scanEntry[j].master = newkeys[i].scanEntry + masterN;
}
+
+ if ( restart )
+ ginrestartentry( &keys[i].scanEntry[j] );
}
}
GinScanOpaque so = (GinScanOpaque) scan->opaque;
freeScanKeys(so->markPos, so->nkeys, FALSE);
- so->markPos = copyScanKeys(so->keys, so->nkeys);
+ so->markPos = copyScanKeys(so->keys, so->nkeys, FALSE);
PG_RETURN_VOID();
}
GinScanOpaque so = (GinScanOpaque) scan->opaque;
freeScanKeys(so->keys, so->nkeys, FALSE);
- so->keys = copyScanKeys(so->markPos, so->nkeys);
+ so->keys = copyScanKeys(so->markPos, so->nkeys, TRUE);
PG_RETURN_VOID();
}
* Portions Copyright (c) 1994, Regents of the University of California
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/access/gin/ginutil.c,v 1.14 2008/05/12 00:00:44 alvherre Exp $
+ * $PostgreSQL: pgsql/src/backend/access/gin/ginutil.c,v 1.15 2008/05/16 16:31:01 tgl Exp $
*-------------------------------------------------------------------------
*/
fmgr_info_copy(&(state->consistentFn),
index_getprocinfo(index, 1, GIN_CONSISTENT_PROC),
CurrentMemoryContext);
+
+ /*
+ * Check opclass capability to do partial match.
+ */
+ if ( index_getprocid(index, 1, GIN_COMPARE_PARTIAL_PROC) != InvalidOid )
+ {
+ fmgr_info_copy(&(state->comparePartialFn),
+ index_getprocinfo(index, 1, GIN_COMPARE_PARTIAL_PROC),
+ CurrentMemoryContext);
+
+ state->canPartialMatch = true;
+ }
+ else
+ {
+ state->canPartialMatch = false;
+ }
}
/*
*
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/optimizer/path/indxpath.c,v 1.229 2008/04/13 20:51:20 tgl Exp $
+ * $PostgreSQL: pgsql/src/backend/optimizer/path/indxpath.c,v 1.230 2008/05/16 16:31:01 tgl Exp $
*
*-------------------------------------------------------------------------
*/
* expand_indexqual_opclause --- expand a single indexqual condition
* that is an operator clause
*
- * The input is a single RestrictInfo, the output a list of RestrictInfos
+ * The input is a single RestrictInfo, the output a list of RestrictInfos.
+ *
+ * In the base case this is just list_make1(), but we have to be prepared to
+ * expand special cases that were accepted by match_special_index_operator().
*/
static List *
expand_indexqual_opclause(RestrictInfo *rinfo, Oid opfamily)
Const *prefix = NULL;
Const *rest = NULL;
Pattern_Prefix_Status pstatus;
- List *result;
+ /*
+ * LIKE and regex operators are not members of any btree index opfamily,
+ * but they can be members of opfamilies for more exotic index types such
+ * as GIN. Therefore, we should only do expansion if the operator is
+ * actually not in the opfamily. But checking that requires a syscache
+ * lookup, so it's best to first see if the operator is one we are
+ * interested in.
+ */
switch (expr_op)
{
- /*
- * LIKE and regex operators are not members of any index opfamily,
- * so if we find one in an indexqual list we can assume that it
- * was accepted by match_special_index_operator().
- */
case OID_TEXT_LIKE_OP:
case OID_BPCHAR_LIKE_OP:
case OID_NAME_LIKE_OP:
case OID_BYTEA_LIKE_OP:
- pstatus = pattern_fixed_prefix(patt, Pattern_Type_Like,
- &prefix, &rest);
- result = prefix_quals(leftop, opfamily, prefix, pstatus);
+ if (!op_in_opfamily(expr_op, opfamily))
+ {
+ pstatus = pattern_fixed_prefix(patt, Pattern_Type_Like,
+ &prefix, &rest);
+ return prefix_quals(leftop, opfamily, prefix, pstatus);
+ }
break;
case OID_TEXT_ICLIKE_OP:
case OID_BPCHAR_ICLIKE_OP:
case OID_NAME_ICLIKE_OP:
- /* the right-hand const is type text for all of these */
- pstatus = pattern_fixed_prefix(patt, Pattern_Type_Like_IC,
- &prefix, &rest);
- result = prefix_quals(leftop, opfamily, prefix, pstatus);
+ if (!op_in_opfamily(expr_op, opfamily))
+ {
+ /* the right-hand const is type text for all of these */
+ pstatus = pattern_fixed_prefix(patt, Pattern_Type_Like_IC,
+ &prefix, &rest);
+ return prefix_quals(leftop, opfamily, prefix, pstatus);
+ }
break;
case OID_TEXT_REGEXEQ_OP:
case OID_BPCHAR_REGEXEQ_OP:
case OID_NAME_REGEXEQ_OP:
- /* the right-hand const is type text for all of these */
- pstatus = pattern_fixed_prefix(patt, Pattern_Type_Regex,
- &prefix, &rest);
- result = prefix_quals(leftop, opfamily, prefix, pstatus);
+ if (!op_in_opfamily(expr_op, opfamily))
+ {
+ /* the right-hand const is type text for all of these */
+ pstatus = pattern_fixed_prefix(patt, Pattern_Type_Regex,
+ &prefix, &rest);
+ return prefix_quals(leftop, opfamily, prefix, pstatus);
+ }
break;
case OID_TEXT_ICREGEXEQ_OP:
case OID_BPCHAR_ICREGEXEQ_OP:
case OID_NAME_ICREGEXEQ_OP:
- /* the right-hand const is type text for all of these */
- pstatus = pattern_fixed_prefix(patt, Pattern_Type_Regex_IC,
- &prefix, &rest);
- result = prefix_quals(leftop, opfamily, prefix, pstatus);
+ if (!op_in_opfamily(expr_op, opfamily))
+ {
+ /* the right-hand const is type text for all of these */
+ pstatus = pattern_fixed_prefix(patt, Pattern_Type_Regex_IC,
+ &prefix, &rest);
+ return prefix_quals(leftop, opfamily, prefix, pstatus);
+ }
break;
case OID_INET_SUB_OP:
case OID_INET_SUBEQ_OP:
- result = network_prefix_quals(leftop, expr_op, opfamily,
- patt->constvalue);
- break;
-
- default:
- result = list_make1(rinfo);
+ if (!op_in_opfamily(expr_op, opfamily))
+ {
+ return network_prefix_quals(leftop, expr_op, opfamily,
+ patt->constvalue);
+ }
break;
}
- return result;
+ /* Default case: just make a list of the unmodified indexqual */
+ return list_make1(rinfo);
}
/*
*
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/tsearch/to_tsany.c,v 1.11 2008/03/25 22:42:43 tgl Exp $
+ * $PostgreSQL: pgsql/src/backend/tsearch/to_tsany.c,v 1.12 2008/05/16 16:31:01 tgl Exp $
*
*-------------------------------------------------------------------------
*/
static int
compareWORD(const void *a, const void *b)
{
- if (((ParsedWord *) a)->len == ((ParsedWord *) b)->len)
+ int res;
+
+ res = tsCompareString(
+ ((ParsedWord *) a)->word, ((ParsedWord *) a)->len,
+ ((ParsedWord *) b)->word, ((ParsedWord *) b)->len,
+ false );
+
+ if (res == 0)
{
- int res = strncmp(
- ((ParsedWord *) a)->word,
- ((ParsedWord *) b)->word,
- ((ParsedWord *) b)->len);
+ if (((ParsedWord *) a)->pos.pos == ((ParsedWord *) b)->pos.pos)
+ return 0;
- if (res == 0)
- {
- if (((ParsedWord *) a)->pos.pos == ((ParsedWord *) b)->pos.pos)
- return 0;
-
- return (((ParsedWord *) a)->pos.pos > ((ParsedWord *) b)->pos.pos) ? 1 : -1;
- }
- return res;
+ res = (((ParsedWord *) a)->pos.pos > ((ParsedWord *) b)->pos.pos) ? 1 : -1;
}
- return (((ParsedWord *) a)->len > ((ParsedWord *) b)->len) ? 1 : -1;
+
+ return res;
}
static int
* and different variants are ORred together.
*/
static void
-pushval_morph(Datum opaque, TSQueryParserState state, char *strval, int lenval, int2 weight)
+pushval_morph(Datum opaque, TSQueryParserState state, char *strval, int lenval, int2 weight, bool prefix)
{
int4 count = 0;
ParsedText prs;
while (count < prs.curwords && pos == prs.words[count].pos.pos && variant == prs.words[count].nvariant)
{
- pushValue(state, prs.words[count].word, prs.words[count].len, weight);
+ pushValue(state, prs.words[count].word, prs.words[count].len, weight,
+ ( (prs.words[count].flags & TSL_PREFIX) || prefix ) ? true : false );
pfree(prs.words[count].word);
if (cnt)
pushOperator(state, OP_AND);
*
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/tsearch/ts_parse.c,v 1.7 2008/01/01 19:45:52 momjian Exp $
+ * $PostgreSQL: pgsql/src/backend/tsearch/ts_parse.c,v 1.8 2008/05/16 16:31:01 tgl Exp $
*
*-------------------------------------------------------------------------
*/
prs->words[prs->curwords].len = strlen(ptr->lexeme);
prs->words[prs->curwords].word = ptr->lexeme;
prs->words[prs->curwords].nvariant = ptr->nvariant;
+ prs->words[prs->curwords].flags = ptr->flags & TSL_PREFIX;
prs->words[prs->curwords].alen = 0;
prs->words[prs->curwords].pos.pos = LIMITPOS(prs->pos);
ptr++;
for (i = 0; i < query->size; i++)
{
if (item->type == QI_VAL &&
- item->operand.length == buflen &&
- strncmp(GETOPERAND(query) + item->operand.distance, buf, buflen) == 0)
+ tsCompareString( GETOPERAND(query) + item->operand.distance, item->operand.length,
+ buf, buflen, item->operand.prefix ) == 0 )
{
if (word->item)
{
*
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/utils/adt/tsginidx.c,v 1.11 2008/04/14 17:05:33 tgl Exp $
+ * $PostgreSQL: pgsql/src/backend/utils/adt/tsginidx.c,v 1.12 2008/05/16 16:31:01 tgl Exp $
*
*-------------------------------------------------------------------------
*/
#include "utils/builtins.h"
+Datum
+gin_cmp_tslexeme(PG_FUNCTION_ARGS)
+{
+ text *a = PG_GETARG_TEXT_P(0);
+ text *b = PG_GETARG_TEXT_P(1);
+ int cmp;
+
+ cmp = tsCompareString(
+ VARDATA(a), VARSIZE(a) - VARHDRSZ,
+ VARDATA(b), VARSIZE(b) - VARHDRSZ,
+ false );
+
+ PG_FREE_IF_COPY(a,0);
+ PG_FREE_IF_COPY(b,1);
+ PG_RETURN_INT32( cmp );
+}
+
+Datum
+gin_cmp_prefix(PG_FUNCTION_ARGS)
+{
+ text *a = PG_GETARG_TEXT_P(0);
+ text *b = PG_GETARG_TEXT_P(1);
+#ifdef NOT_USED
+ StrategyNumber strategy = PG_GETARG_UINT16(2);
+#endif
+ int cmp;
+
+ cmp = tsCompareString(
+ VARDATA(a), VARSIZE(a) - VARHDRSZ,
+ VARDATA(b), VARSIZE(b) - VARHDRSZ,
+ true );
+
+ if ( cmp < 0 )
+ cmp = 1; /* prevent continue scan */
+
+ PG_FREE_IF_COPY(a,0);
+ PG_FREE_IF_COPY(b,1);
+ PG_RETURN_INT32( cmp );
+}
+
Datum
gin_extract_tsvector(PG_FUNCTION_ARGS)
{
TSQuery query = PG_GETARG_TSQUERY(0);
int32 *nentries = (int32 *) PG_GETARG_POINTER(1);
/* StrategyNumber strategy = PG_GETARG_UINT16(2); */
+ bool **ptr_partialmatch = (bool**) PG_GETARG_POINTER(3);
Datum *entries = NULL;
+ bool *partialmatch;
*nentries = 0;
j = 0,
len;
QueryItem *item;
+ bool use_fullscan=false;
item = clean_NOT(GETQUERY(query), &len);
if (!item)
- ereport(ERROR,
- (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
- errmsg("query requires full scan, which is not supported by GIN indexes")));
+ {
+ use_fullscan = true;
+ *nentries = 1;
+ }
item = GETQUERY(query);
(*nentries)++;
entries = (Datum *) palloc(sizeof(Datum) * (*nentries));
+ partialmatch = *ptr_partialmatch = (bool*) palloc(sizeof(bool) * (*nentries));
for (i = 0; i < query->size; i++)
if (item[i].type == QI_VAL)
txt = cstring_to_text_with_len(GETOPERAND(query) + val->distance,
val->length);
+ partialmatch[j] = val->prefix;
entries[j++] = PointerGetDatum(txt);
}
+
+ if ( use_fullscan )
+ entries[j++] = PointerGetDatum(cstring_to_text_with_len("", 0));
}
else
*nentries = -1; /* nothing can be found */
*
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/utils/adt/tsgistidx.c,v 1.8 2008/04/14 17:05:33 tgl Exp $
+ * $PostgreSQL: pgsql/src/backend/utils/adt/tsgistidx.c,v 1.9 2008/05/16 16:31:01 tgl Exp $
*
*-------------------------------------------------------------------------
*/
/* Loop invariant: StopLow <= val < StopHigh */
+ /*
+ * we are not able to find a a prefix by hash value
+ */
+ if ( val->prefix )
+ return true;
+
while (StopLow < StopHigh)
{
StopMiddle = StopLow + (StopHigh - StopLow) / 2;
static bool
checkcondition_bit(void *checkval, QueryOperand *val)
{
+ /*
+ * we are not able to find a a prefix in signature tree
+ */
+ if ( val->prefix )
+ return true;
return GETBIT(checkval, HASHVAL(val->valcrc));
}
*
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/utils/adt/tsquery.c,v 1.17 2008/04/11 22:52:05 tgl Exp $
+ * $PostgreSQL: pgsql/src/backend/utils/adt/tsquery.c,v 1.18 2008/05/16 16:31:01 tgl Exp $
*
*-------------------------------------------------------------------------
*/
#define WAITSINGLEOPERAND 4
/*
- * subroutine to parse the weight part, like ':1AB' of a query.
+ * subroutine to parse the modifiers (weight and prefix flag currently)
+ * part, like ':1AB' of a query.
*/
static char *
-get_weight(char *buf, int16 *weight)
+get_modifiers(char *buf, int16 *weight, bool *prefix)
{
*weight = 0;
+ *prefix = false;
if (!t_iseq(buf, ':'))
return buf;
case 'D':
*weight |= 1;
break;
+ case '*':
+ *prefix = true;
+ break;
default:
return buf;
}
static ts_tokentype
gettoken_query(TSQueryParserState state,
int8 *operator,
- int *lenval, char **strval, int16 *weight)
+ int *lenval, char **strval, int16 *weight, bool *prefix)
{
+ *weight = 0;
+ *prefix = false;
+
while (1)
{
switch (state->state)
reset_tsvector_parser(state->valstate, state->buf);
if (gettoken_tsvector(state->valstate, strval, lenval, NULL, NULL, &state->buf))
{
- state->buf = get_weight(state->buf, weight);
+ state->buf = get_modifiers(state->buf, weight, prefix);
state->state = WAITOPERATOR;
return PT_VAL;
}
}
static void
-pushValue_internal(TSQueryParserState state, pg_crc32 valcrc, int distance, int lenval, int weight)
+pushValue_internal(TSQueryParserState state, pg_crc32 valcrc, int distance, int lenval, int weight, bool prefix)
{
QueryOperand *tmp;
tmp = (QueryOperand *) palloc0(sizeof(QueryOperand));
tmp->type = QI_VAL;
tmp->weight = weight;
+ tmp->prefix = prefix;
tmp->valcrc = (int32) valcrc;
tmp->length = lenval;
tmp->distance = distance;
* of the string.
*/
void
-pushValue(TSQueryParserState state, char *strval, int lenval, int2 weight)
+pushValue(TSQueryParserState state, char *strval, int lenval, int2 weight, bool prefix)
{
pg_crc32 valcrc;
INIT_CRC32(valcrc);
COMP_CRC32(valcrc, strval, lenval);
FIN_CRC32(valcrc);
- pushValue_internal(state, valcrc, state->curop - state->op, lenval, weight);
+ pushValue_internal(state, valcrc, state->curop - state->op, lenval, weight, prefix);
/* append the value string to state.op, enlarging buffer if needed first */
while (state->curop - state->op + lenval + 1 >= state->lenop)
int8 opstack[STACKDEPTH];
int lenstack = 0;
int16 weight = 0;
+ bool prefix;
/* since this function recurses, it could be driven to stack overflow */
check_stack_depth();
- while ((type = gettoken_query(state, &operator, &lenval, &strval, &weight)) != PT_END)
+ while ((type = gettoken_query(state, &operator, &lenval, &strval, &weight, &prefix)) != PT_END)
{
switch (type)
{
case PT_VAL:
- pushval(opaque, state, strval, lenval, weight);
+ pushval(opaque, state, strval, lenval, weight, prefix);
while (lenstack && (opstack[lenstack - 1] == OP_AND ||
opstack[lenstack - 1] == OP_NOT))
{
static void
pushval_asis(Datum opaque, TSQueryParserState state, char *strval, int lenval,
- int16 weight)
+ int16 weight, bool prefix)
{
- pushValue(state, strval, lenval, weight);
+ pushValue(state, strval, lenval, weight, prefix);
}
/*
char *op = in->op + curpol->distance;
int clen;
- RESIZEBUF(in, curpol->length * (pg_database_encoding_max_length() + 1) + 2 + 5);
+ RESIZEBUF(in, curpol->length * (pg_database_encoding_max_length() + 1) + 2 + 6);
*(in->cur) = '\'';
in->cur++;
while (*op)
}
*(in->cur) = '\'';
in->cur++;
- if (curpol->weight)
+ if (curpol->weight || curpol->prefix)
{
*(in->cur) = ':';
in->cur++;
+ if ( curpol->prefix )
+ {
+ *(in->cur) = '*';
+ in->cur++;
+ }
if (curpol->weight & (1 << 3))
{
*(in->cur) = 'A';
* uint8 type, QI_VAL
* uint8 weight
* operand text in client encoding, null-terminated
+ * uint8 prefix
*
* For each operator:
* uint8 type, QI_OPR
{
case QI_VAL:
pq_sendint(&buf, item->operand.weight, sizeof(uint8));
+ pq_sendint(&buf, item->operand.prefix, sizeof(uint8));
pq_sendstring(&buf, GETOPERAND(query) + item->operand.distance);
break;
case QI_OPR:
{
size_t val_len; /* length after recoding to server encoding */
uint8 weight;
+ uint8 prefix;
const char *val;
pg_crc32 valcrc;
weight = (uint8) pq_getmsgint(buf, sizeof(uint8));
+ prefix = (uint8) pq_getmsgint(buf, sizeof(uint8));
val = pq_getmsgstring(buf);
val_len = strlen(val);
FIN_CRC32(valcrc);
item->operand.weight = weight;
+ item->operand.prefix = (prefix) ? true : false;
item->operand.valcrc = (int32) valcrc;
item->operand.length = val_len;
item->operand.distance = datalen;
*
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/utils/adt/tsquery_util.c,v 1.8 2008/01/01 19:45:53 momjian Exp $
+ * $PostgreSQL: pgsql/src/backend/utils/adt/tsquery_util.c,v 1.9 2008/05/16 16:31:01 tgl Exp $
*
*-------------------------------------------------------------------------
*/
return (ao->valcrc > bo->valcrc) ? -1 : 1;
}
- if (ao->length == bo->length)
- return strncmp(an->word, bn->word, ao->length);
- else
- return (ao->length > bo->length) ? -1 : 1;
+ return tsCompareString( an->word, ao->length, bn->word, bo->length, false);
}
}
*
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/utils/adt/tsrank.c,v 1.12 2008/01/01 19:45:53 momjian Exp $
+ * $PostgreSQL: pgsql/src/backend/utils/adt/tsrank.c,v 1.13 2008/05/16 16:31:01 tgl Exp $
*
*-------------------------------------------------------------------------
*/
return len;
}
-static int
-WordECompareQueryItem(char *eval, char *qval, WordEntry *ptr, QueryOperand *item)
-{
- if (ptr->len == item->length)
- return strncmp(
- eval + ptr->pos,
- qval + item->distance,
- item->length);
- return (ptr->len > item->length) ? 1 : -1;
-}
+#define WordECompareQueryItem(e,q,p,i,m) \
+ tsCompareString((q) + (i)->distance, (i)->length, \
+ (e) + (p)->pos, (p)->len, (m))
+
/*
- * Returns a pointer to a WordEntry corresponding 'item' from tsvector 't'. 'q'
- * is the TSQuery containing 'item'. Returns NULL if not found.
+ * Returns a pointer to a WordEntry's array corresponding to 'item' from
+ * tsvector 't'. 'q' is the TSQuery containing 'item'.
+ * Returns NULL if not found.
*/
static WordEntry *
-find_wordentry(TSVector t, TSQuery q, QueryOperand *item)
+find_wordentry(TSVector t, TSQuery q, QueryOperand *item, int32 *nitem)
{
WordEntry *StopLow = ARRPTR(t);
WordEntry *StopHigh = (WordEntry *) STRPTR(t);
- WordEntry *StopMiddle;
+ WordEntry *StopMiddle = StopHigh;
int difference;
- /* Loop invariant: StopLow <= item < StopHigh */
+ *nitem=0;
+ /* Loop invariant: StopLow <= item < StopHigh */
while (StopLow < StopHigh)
{
StopMiddle = StopLow + (StopHigh - StopLow) / 2;
- difference = WordECompareQueryItem(STRPTR(t), GETOPERAND(q), StopMiddle, item);
+ difference = WordECompareQueryItem(STRPTR(t), GETOPERAND(q), StopMiddle, item, false);
if (difference == 0)
- return StopMiddle;
- else if (difference < 0)
+ {
+ StopHigh = StopMiddle;
+ *nitem=1;
+ break;
+ }
+ else if (difference > 0)
StopLow = StopMiddle + 1;
else
StopHigh = StopMiddle;
}
- return NULL;
+ if ( item->prefix == true )
+ {
+ if ( StopLow >= StopHigh )
+ StopMiddle = StopHigh;
+
+ *nitem=0;
+
+ while( StopMiddle < (WordEntry *) STRPTR(t) &&
+ WordECompareQueryItem(STRPTR(t), GETOPERAND(q), StopMiddle, item, true) == 0 )
+ {
+ (*nitem)++;
+ StopMiddle++;
+ }
+ }
+
+ return ( *nitem > 0 ) ? StopHigh : NULL;
}
QueryOperand *qa = (*(QueryOperand **) a);
QueryOperand *qb = (*(QueryOperand **) b);
- if (qa->length == qb->length)
- return strncmp(operand + qa->distance,
- operand + qb->distance,
- qb->length);
-
- return (qa->length > qb->length) ? 1 : -1;
+ return tsCompareString(operand + qa->distance, qa->length,
+ operand + qb->distance, qb->length,
+ false);
}
/*
k,
l,
p;
- WordEntry *entry;
+ WordEntry *entry,
+ *firstentry;
WordEntryPos *post,
*ct;
int4 dimt,
lenct,
- dist;
+ dist,
+ nitem;
float res = -1.0;
QueryOperand **item;
int size = q->size;
for (i = 0; i < size; i++)
{
- entry = find_wordentry(t, q, item[i]);
+ firstentry = entry = find_wordentry(t, q, item[i], &nitem);
if (!entry)
continue;
- if (entry->haspos)
- pos[i] = _POSVECPTR(t, entry);
- else
- pos[i] = &POSNULL;
-
-
- dimt = pos[i]->npos;
- post = pos[i]->pos;
- for (k = 0; k < i; k++)
+ while( entry - firstentry < nitem )
{
- if (!pos[k])
- continue;
- lenct = pos[k]->npos;
- ct = pos[k]->pos;
- for (l = 0; l < dimt; l++)
+ if (entry->haspos)
+ pos[i] = _POSVECPTR(t, entry);
+ else
+ pos[i] = &POSNULL;
+
+ dimt = pos[i]->npos;
+ post = pos[i]->pos;
+ for (k = 0; k < i; k++)
{
- for (p = 0; p < lenct; p++)
+ if (!pos[k])
+ continue;
+ lenct = pos[k]->npos;
+ ct = pos[k]->pos;
+ for (l = 0; l < dimt; l++)
{
- dist = Abs((int) WEP_GETPOS(post[l]) - (int) WEP_GETPOS(ct[p]));
- if (dist || (dist == 0 && (pos[i] == &POSNULL || pos[k] == &POSNULL)))
+ for (p = 0; p < lenct; p++)
{
- float curw;
-
- if (!dist)
- dist = MAXENTRYPOS;
- curw = sqrt(wpos(post[l]) * wpos(ct[p]) * word_distance(dist));
- res = (res < 0) ? curw : 1.0 - (1.0 - res) * (1.0 - curw);
+ dist = Abs((int) WEP_GETPOS(post[l]) - (int) WEP_GETPOS(ct[p]));
+ if (dist || (dist == 0 && (pos[i] == &POSNULL || pos[k] == &POSNULL)))
+ {
+ float curw;
+
+ if (!dist)
+ dist = MAXENTRYPOS;
+ curw = sqrt(wpos(post[l]) * wpos(ct[p]) * word_distance(dist));
+ res = (res < 0) ? curw : 1.0 - (1.0 - res) * (1.0 - curw);
+ }
}
}
}
+
+ entry++;
}
}
pfree(pos);
static float
calc_rank_or(float *w, TSVector t, TSQuery q)
{
- WordEntry *entry;
+ WordEntry *entry,
+ *firstentry;
WordEntryPos *post;
int4 dimt,
j,
- i;
+ i,
+ nitem;
float res = 0.0;
QueryOperand **item;
int size = q->size;
wjm;
int4 jm;
- entry = find_wordentry(t, q, item[i]);
+ firstentry = entry = find_wordentry(t, q, item[i], &nitem);
if (!entry)
continue;
- if (entry->haspos)
- {
- dimt = POSDATALEN(t, entry);
- post = POSDATAPTR(t, entry);
- }
- else
+ while( entry - firstentry < nitem )
{
- dimt = POSNULL.npos;
- post = POSNULL.pos;
- }
+ if (entry->haspos)
+ {
+ dimt = POSDATALEN(t, entry);
+ post = POSDATAPTR(t, entry);
+ }
+ else
+ {
+ dimt = POSNULL.npos;
+ post = POSNULL.pos;
+ }
- resj = 0.0;
- wjm = -1.0;
- jm = 0;
- for (j = 0; j < dimt; j++)
- {
- resj = resj + wpos(post[j]) / ((j + 1) * (j + 1));
- if (wpos(post[j]) > wjm)
+ resj = 0.0;
+ wjm = -1.0;
+ jm = 0;
+ for (j = 0; j < dimt; j++)
{
- wjm = wpos(post[j]);
- jm = j;
+ resj = resj + wpos(post[j]) / ((j + 1) * (j + 1));
+ if (wpos(post[j]) > wjm)
+ {
+ wjm = wpos(post[j]);
+ jm = j;
+ }
}
- }
/*
- limit (sum(i/i^2),i->inf) = pi^2/6
- resj = sum(wi/i^2),i=1,noccurence,
- wi - should be sorted desc,
- don't sort for now, just choose maximum weight. This should be corrected
- Oleg Bartunov
+ limit (sum(i/i^2),i->inf) = pi^2/6
+ resj = sum(wi/i^2),i=1,noccurence,
+ wi - should be sorted desc,
+ don't sort for now, just choose maximum weight. This should be corrected
+ Oleg Bartunov
*/
- res = res + (wjm + resj - wjm / ((jm + 1) * (jm + 1))) / 1.64493406685;
+ res = res + (wjm + resj - wjm / ((jm + 1) * (jm + 1))) / 1.64493406685;
+
+ entry++;
+ }
}
if (size > 0)
res = res / size;
get_docrep(TSVector txt, QueryRepresentation *qr, int *doclen)
{
QueryItem *item = GETQUERY(qr->query);
- WordEntry *entry;
+ WordEntry *entry,
+ *firstentry;
WordEntryPos *post;
int4 dimt,
j,
- i;
+ i,
+ nitem;
int len = qr->query->size * 4,
cur = 0;
DocRepresentation *doc;
if (QR_GET_OPERAND_EXISTS(qr, &item[i]))
continue;
- entry = find_wordentry(txt, qr->query, curoperand);
+ firstentry = entry = find_wordentry(txt, qr->query, curoperand, &nitem);
if (!entry)
continue;
- if (entry->haspos)
- {
- dimt = POSDATALEN(txt, entry);
- post = POSDATAPTR(txt, entry);
- }
- else
- {
- dimt = POSNULL.npos;
- post = POSNULL.pos;
- }
-
- while (cur + dimt >= len)
+ while( entry - firstentry < nitem )
{
- len *= 2;
- doc = (DocRepresentation *) repalloc(doc, sizeof(DocRepresentation) * len);
- }
-
- for (j = 0; j < dimt; j++)
- {
- if (j == 0)
+ if (entry->haspos)
+ {
+ dimt = POSDATALEN(txt, entry);
+ post = POSDATAPTR(txt, entry);
+ }
+ else
{
- int k;
+ dimt = POSNULL.npos;
+ post = POSNULL.pos;
+ }
- doc[cur].nitem = 0;
- doc[cur].item = (QueryItem **) palloc(sizeof(QueryItem *) * qr->query->size);
+ while (cur + dimt >= len)
+ {
+ len *= 2;
+ doc = (DocRepresentation *) repalloc(doc, sizeof(DocRepresentation) * len);
+ }
- for (k = 0; k < qr->query->size; k++)
+ for (j = 0; j < dimt; j++)
+ {
+ if (j == 0)
{
- QueryOperand *kptr = &item[k].operand;
- QueryOperand *iptr = &item[i].operand;
-
- if (k == i ||
- (item[k].type == QI_VAL &&
- compareQueryOperand(&kptr, &iptr, operand) == 0))
+ int k;
+
+ doc[cur].nitem = 0;
+ doc[cur].item = (QueryItem **) palloc(sizeof(QueryItem *) * qr->query->size);
+
+ for (k = 0; k < qr->query->size; k++)
{
- /*
- * if k == i, we've already checked above that it's
- * type == Q_VAL
- */
- doc[cur].item[doc[cur].nitem] = item + k;
- doc[cur].nitem++;
- QR_SET_OPERAND_EXISTS(qr, item + k);
+ QueryOperand *kptr = &item[k].operand;
+ QueryOperand *iptr = &item[i].operand;
+
+ if (k == i ||
+ (item[k].type == QI_VAL &&
+ compareQueryOperand(&kptr, &iptr, operand) == 0))
+ {
+ /*
+ * if k == i, we've already checked above that it's
+ * type == Q_VAL
+ */
+ doc[cur].item[doc[cur].nitem] = item + k;
+ doc[cur].nitem++;
+ QR_SET_OPERAND_EXISTS(qr, item + k);
+ }
}
}
+ else
+ {
+ doc[cur].nitem = doc[cur - 1].nitem;
+ doc[cur].item = doc[cur - 1].item;
+ }
+ doc[cur].pos = WEP_GETPOS(post[j]);
+ doc[cur].wclass = WEP_GETWEIGHT(post[j]);
+ cur++;
}
- else
- {
- doc[cur].nitem = doc[cur - 1].nitem;
- doc[cur].item = doc[cur - 1].item;
- }
- doc[cur].pos = WEP_GETPOS(post[j]);
- doc[cur].wclass = WEP_GETWEIGHT(post[j]);
- cur++;
+
+ entry++;
}
}
*
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/utils/adt/tsvector.c,v 1.13 2008/03/10 12:57:05 tgl Exp $
+ * $PostgreSQL: pgsql/src/backend/utils/adt/tsvector.c,v 1.14 2008/05/16 16:31:01 tgl Exp $
*
*-------------------------------------------------------------------------
*/
const WordEntryIN *b = (const WordEntryIN *) vb;
char *BufferStr = (char *) arg;
- if (a->entry.len == b->entry.len)
- {
- return strncmp(&BufferStr[a->entry.pos],
- &BufferStr[b->entry.pos],
- a->entry.len);
- }
-
- return (a->entry.len > b->entry.len) ? 1 : -1;
+ return tsCompareString( &BufferStr[a->entry.pos], a->entry.len,
+ &BufferStr[b->entry.pos], b->entry.len,
+ false );
}
/*
*
*
* IDENTIFICATION
- * $PostgreSQL: pgsql/src/backend/utils/adt/tsvector_op.c,v 1.15 2008/04/08 18:20:29 tgl Exp $
+ * $PostgreSQL: pgsql/src/backend/utils/adt/tsvector_op.c,v 1.16 2008/05/16 16:31:01 tgl Exp $
*
*-------------------------------------------------------------------------
*/
{
return (aptr->haspos > bptr->haspos) ? -1 : 1;
}
- else if (aptr->len != bptr->len)
- {
- return (aptr->len > bptr->len) ? -1 : 1;
- }
- else if ((res = strncmp(STRPTR(a) + aptr->pos, STRPTR(b) + bptr->pos, bptr->len)) != 0)
+ else if ( (res=tsCompareString( STRPTR(a) + aptr->pos, aptr->len, STRPTR(b) + bptr->pos, bptr->len, false)) !=0 )
{
return res;
}
PG_RETURN_POINTER(out);
}
-static int
-compareEntry(char *ptra, WordEntry *a, char *ptrb, WordEntry *b)
-{
- if (a->len == b->len)
- {
- return strncmp(
- ptra + a->pos,
- ptrb + b->pos,
- a->len);
- }
- return (a->len > b->len) ? 1 : -1;
-}
+#define compareEntry(pa, a, pb, b) \
+ tsCompareString((pa) + (a)->pos, (a)->len, \
+ (pb) + (b)->pos, (b)->len, \
+ false)
/*
* Add positions from src to dest after offsetting them by maxpos.
}
/*
- * compare 2 string values
+ * Compare two strings by tsvector rules.
+ * if isPrefix = true then it returns not-zero value if b has prefix a
*/
-static int4
-ValCompare(CHKVAL *chkval, WordEntry *ptr, QueryOperand *item)
+int4
+tsCompareString(char *a, int lena, char *b, int lenb, bool prefix)
{
- if (ptr->len == item->length)
- return strncmp(
- &(chkval->values[ptr->pos]),
- &(chkval->operand[item->distance]),
- item->length);
+ int cmp;
+
+ if ( lena == 0 )
+ {
+ if ( prefix )
+ cmp = 0; /* emtry string is equal to any if a prefix match */
+ else
+ cmp = (lenb>0) ? -1 : 0;
+ }
+ else if ( lenb == 0 )
+ {
+ cmp = (lena>0) ? 1 : 0;
+ }
+ else
+ {
+ cmp = memcmp(a, b, Min(lena, lenb));
- return (ptr->len > item->length) ? 1 : -1;
+ if ( prefix )
+ {
+ if ( cmp == 0 && lena > lenb )
+ {
+ /*
+ * b argument is not beginning with argument a
+ */
+ cmp=1;
+ }
+ }
+ else if ( (cmp == 0) && (lena != lenb) )
+ {
+ cmp = (lena < lenb) ? -1 : 1;
+ }
+ }
+
+ return cmp;
}
/*
CHKVAL *chkval = (CHKVAL *) checkval;
WordEntry *StopLow = chkval->arrb;
WordEntry *StopHigh = chkval->arre;
- WordEntry *StopMiddle;
- int difference;
+ WordEntry *StopMiddle = StopHigh;
+ int difference = -1;
+ bool res=false;
/* Loop invariant: StopLow <= val < StopHigh */
-
while (StopLow < StopHigh)
{
StopMiddle = StopLow + (StopHigh - StopLow) / 2;
- difference = ValCompare(chkval, StopMiddle, val);
+ difference = tsCompareString( chkval->operand + val->distance, val->length,
+ chkval->values + StopMiddle->pos, StopMiddle->len,
+ false);
+
if (difference == 0)
- return (val->weight && StopMiddle->haspos) ?
+ {
+ res = (val->weight && StopMiddle->haspos) ?
checkclass_str(chkval, StopMiddle, val) : true;
- else if (difference < 0)
+ break;
+ }
+ else if (difference > 0)
StopLow = StopMiddle + 1;
else
StopHigh = StopMiddle;
}
- return (false);
+ if ( res == false && val->prefix == true )
+ {
+ /*
+ * there was a failed exact search, so we should scan further to find
+ * a prefix match.
+ */
+ if ( StopLow >= StopHigh )
+ StopMiddle = StopHigh;
+
+ while( res == false && StopMiddle < chkval->arre &&
+ tsCompareString( chkval->operand + val->distance, val->length,
+ chkval->values + StopMiddle->pos, StopMiddle->len,
+ true) == 0 )
+ {
+ res = (val->weight && StopMiddle->haspos) ?
+ checkclass_str(chkval, StopMiddle, val) : true;
+
+ StopMiddle++;
+ }
+ }
+
+ return res;
}
/*
return num;
}
-static WordEntry **
-SEI_realloc(WordEntry **in, uint32 *len)
-{
- if (*len == 0 || in == NULL)
- {
- *len = 8;
- in = palloc(sizeof(WordEntry *) * (*len));
- }
- else
- {
- *len *= 2;
- in = repalloc(in, sizeof(WordEntry *) * (*len));
- }
- return in;
-}
+#define compareStatWord(a,e,s,t) \
+ tsCompareString(STATSTRPTR(s) + (a)->pos, (a)->len, \
+ STRPTR(t) + (e)->pos, (e)->len, \
+ false)
-static int
-compareStatWord(StatEntry *a, WordEntry *b, tsstat *stat, TSVector txt)
+typedef struct WordEntryMark
{
- if (a->len == b->len)
- return strncmp(
- STATSTRPTR(stat) + a->pos,
- STRPTR(txt) + b->pos,
- a->len
- );
- return (a->len > b->len) ? 1 : -1;
-}
+ WordEntry *newentry;
+ StatEntry *pos;
+} WordEntryMark;
static tsstat *
-formstat(tsstat *stat, TSVector txt, WordEntry **entry, uint32 len)
+formstat(tsstat *stat, TSVector txt, List *entries)
{
- tsstat *newstat;
- uint32 totallen,
- nentry;
- uint32 slen = 0;
- WordEntry **ptr = entry;
- char *curptr;
- StatEntry *sptr,
- *nptr;
-
- while (ptr - entry < len)
+ tsstat *newstat;
+ uint32 totallen,
+ nentry,
+ len = list_length(entries);
+ uint32 slen = 0;
+ WordEntry *ptr;
+ char *curptr;
+ StatEntry *sptr,
+ *nptr;
+ ListCell *entry;
+ StatEntry *PosSE = STATPTR(stat),
+ *prevPosSE;
+ WordEntryMark *mark;
+
+ foreach( entry, entries )
{
- slen += (*ptr)->len;
- ptr++;
+ mark = (WordEntryMark*)lfirst(entry);
+ slen += mark->newentry->len;
}
nentry = stat->size + len;
memcpy(STATSTRPTR(newstat), STATSTRPTR(stat), STATSTRSIZE(stat));
curptr = STATSTRPTR(newstat) + STATSTRSIZE(stat);
- ptr = entry;
sptr = STATPTR(stat);
nptr = STATPTR(newstat);
- if (len == 1)
+ foreach(entry, entries)
{
- StatEntry *StopLow = STATPTR(stat);
- StatEntry *StopHigh = (StatEntry *) STATSTRPTR(stat);
+ prevPosSE = PosSE;
- while (StopLow < StopHigh)
+ mark = (WordEntryMark*)lfirst(entry);
+ ptr = mark->newentry;
+ PosSE = mark->pos;
+
+ /*
+ * Copy missed entries
+ */
+ if ( PosSE > prevPosSE )
{
- sptr = StopLow + (StopHigh - StopLow) / 2;
- if (compareStatWord(sptr, *ptr, stat, txt) < 0)
- StopLow = sptr + 1;
- else
- StopHigh = sptr;
+ memcpy( nptr, prevPosSE, sizeof(StatEntry) * (PosSE-prevPosSE) );
+ nptr += PosSE-prevPosSE;
}
- nptr = STATPTR(newstat) + (StopLow - STATPTR(stat));
- memcpy(STATPTR(newstat), STATPTR(stat), sizeof(StatEntry) * (StopLow - STATPTR(stat)));
- if ((*ptr)->haspos)
- nptr->nentry = (stat->weight) ? check_weight(txt, *ptr, stat->weight) : POSDATALEN(txt, *ptr);
+
+ /*
+ * Copy new entry
+ */
+ if (ptr->haspos)
+ nptr->nentry = (stat->weight) ? check_weight(txt, ptr, stat->weight) : POSDATALEN(txt, ptr);
else
nptr->nentry = 1;
nptr->ndoc = 1;
- nptr->len = (*ptr)->len;
- memcpy(curptr, STRPTR(txt) + (*ptr)->pos, nptr->len);
+ nptr->len = ptr->len;
+ memcpy(curptr, STRPTR(txt) + ptr->pos, nptr->len);
nptr->pos = curptr - STATSTRPTR(newstat);
- memcpy(nptr + 1, StopLow, sizeof(StatEntry) * (((StatEntry *) STATSTRPTR(stat)) - StopLow));
- }
- else
- {
- while (sptr - STATPTR(stat) < stat->size && ptr - entry < len)
- {
- if (compareStatWord(sptr, *ptr, stat, txt) < 0)
- {
- memcpy(nptr, sptr, sizeof(StatEntry));
- sptr++;
- }
- else
- {
- if ((*ptr)->haspos)
- nptr->nentry = (stat->weight) ? check_weight(txt, *ptr, stat->weight) : POSDATALEN(txt, *ptr);
- else
- nptr->nentry = 1;
- nptr->ndoc = 1;
- nptr->len = (*ptr)->len;
- memcpy(curptr, STRPTR(txt) + (*ptr)->pos, nptr->len);
- nptr->pos = curptr - STATSTRPTR(newstat);
- curptr += nptr->len;
- ptr++;
- }
- nptr++;
- }
+ curptr += nptr->len;
+ nptr++;
- memcpy(nptr, sptr, sizeof(StatEntry) * (stat->size - (sptr - STATPTR(stat))));
-
- while (ptr - entry < len)
- {
- if ((*ptr)->haspos)
- nptr->nentry = (stat->weight) ? check_weight(txt, *ptr, stat->weight) : POSDATALEN(txt, *ptr);
- else
- nptr->nentry = 1;
- nptr->ndoc = 1;
- nptr->len = (*ptr)->len;
- memcpy(curptr, STRPTR(txt) + (*ptr)->pos, nptr->len);
- nptr->pos = curptr - STATSTRPTR(newstat);
- curptr += nptr->len;
- ptr++;
- nptr++;
- }
+ pfree(mark);
}
+ if ( PosSE < (StatEntry *) STATSTRPTR(stat) )
+ memcpy(nptr, PosSE, sizeof(StatEntry) * (stat->size - (PosSE - STATPTR(stat))));
+
return newstat;
}
{
tsstat *newstat;
TSVector txt = DatumGetTSVector(data);
- WordEntry **newentry = NULL;
- uint32 len = 0,
- cur = 0;
StatEntry *sptr;
WordEntry *wptr;
int n = 0;
+ List *newentries=NIL;
+ StatEntry *StopLow;
if (stat == NULL)
{ /* Init in first */
sptr = STATPTR(stat);
wptr = ARRPTR(txt);
+ StopLow = STATPTR(stat);
- if (stat->size < 100 * txt->size)
- { /* merge */
- while (sptr - STATPTR(stat) < stat->size && wptr - ARRPTR(txt) < txt->size)
- {
- int cmp = compareStatWord(sptr, wptr, stat, txt);
+ while (wptr - ARRPTR(txt) < txt->size)
+ {
+ StatEntry *StopHigh = (StatEntry *) STATSTRPTR(stat);
+ int cmp;
+
+ /*
+ * We do not set StopLow to begin of array because tsvector is ordered
+ * with the sames rule, so we can search from last stopped position
+ */
- if (cmp < 0)
- sptr++;
- else if (cmp == 0)
+ while (StopLow < StopHigh)
+ {
+ sptr = StopLow + (StopHigh - StopLow) / 2;
+ cmp = compareStatWord(sptr, wptr, stat, txt);
+ if (cmp == 0)
{
if (stat->weight == 0)
{
sptr->ndoc++;
sptr->nentry += n;
}
- sptr++;
- wptr++;
+ break;
}
+ else if (cmp < 0)
+ StopLow = sptr + 1;
else
- {
- if (stat->weight == 0 || check_weight(txt, wptr, stat->weight) != 0)
- {
- if (cur == len)
- newentry = SEI_realloc(newentry, &len);
- newentry[cur] = wptr;
- cur++;
- }
- wptr++;
- }
+ StopHigh = sptr;
}
- while (wptr - ARRPTR(txt) < txt->size)
- {
+ if (StopLow >= StopHigh)
+ { /* not found */
if (stat->weight == 0 || check_weight(txt, wptr, stat->weight) != 0)
{
- if (cur == len)
- newentry = SEI_realloc(newentry, &len);
- newentry[cur] = wptr;
- cur++;
- }
- wptr++;
- }
- }
- else
- { /* search */
- while (wptr - ARRPTR(txt) < txt->size)
- {
- StatEntry *StopLow = STATPTR(stat);
- StatEntry *StopHigh = (StatEntry *) STATSTRPTR(stat);
- int cmp;
+ WordEntryMark *mark = (WordEntryMark*)palloc(sizeof(WordEntryMark));
- while (StopLow < StopHigh)
- {
- sptr = StopLow + (StopHigh - StopLow) / 2;
- cmp = compareStatWord(sptr, wptr, stat, txt);
- if (cmp == 0)
- {
- if (stat->weight == 0)
- {
- sptr->ndoc++;
- sptr->nentry += (wptr->haspos) ? POSDATALEN(txt, wptr) : 1;
- }
- else if (wptr->haspos && (n = check_weight(txt, wptr, stat->weight)) != 0)
- {
- sptr->ndoc++;
- sptr->nentry += n;
- }
- break;
- }
- else if (cmp < 0)
- StopLow = sptr + 1;
- else
- StopHigh = sptr;
- }
+ mark->newentry = wptr;
+ mark->pos = StopLow;
+ newentries = lappend( newentries, mark );
- if (StopLow >= StopHigh)
- { /* not found */
- if (stat->weight == 0 || check_weight(txt, wptr, stat->weight) != 0)
- {
- if (cur == len)
- newentry = SEI_realloc(newentry, &len);
- newentry[cur] = wptr;
- cur++;
- }
}
- wptr++;
}
+ wptr++;
}
-
- if (cur == 0)
+ if (list_length(newentries) == 0)
{ /* no new words */
if (txt != (TSVector) DatumGetPointer(data))
pfree(txt);
return stat;
}
- newstat = formstat(stat, txt, newentry, cur);
- pfree(newentry);
+ newstat = formstat(stat, txt, newentries);
+ list_free(newentries);
if (txt != (TSVector) DatumGetPointer(data))
pfree(txt);
*
* Copyright (c) 2006-2008, PostgreSQL Global Development Group
*
- * $PostgreSQL: pgsql/src/include/access/gin.h,v 1.19 2008/05/12 00:00:53 alvherre Exp $
+ * $PostgreSQL: pgsql/src/include/access/gin.h,v 1.20 2008/05/16 16:31:01 tgl Exp $
*--------------------------------------------------------------------------
*/
#include "access/itup.h"
#include "access/relscan.h"
#include "fmgr.h"
+#include "nodes/tidbitmap.h"
#include "storage/block.h"
#include "storage/buf.h"
#include "storage/off.h"
#define GIN_EXTRACTVALUE_PROC 2
#define GIN_EXTRACTQUERY_PROC 3
#define GIN_CONSISTENT_PROC 4
-#define GINNProcs 4
+#define GIN_COMPARE_PARTIAL_PROC 5
+#define GINNProcs 5
/*
* Page opaque data in a inverted index page.
FmgrInfo extractValueFn;
FmgrInfo extractQueryFn;
FmgrInfo consistentFn;
+ FmgrInfo comparePartialFn; /* optional method */
+ bool canPartialMatch; /* can opclass perform partial
+ * match (prefix search)? */
TupleDesc tupdesc;
} GinState;
/* current ItemPointer to heap */
ItemPointerData curItem;
+ /* partial match support */
+ bool isPartialMatch;
+ TIDBitmap *partialMatch;
+ TBMIterateResult *partialMatchResult;
+ StrategyNumber strategy;
+
/* used for Posting list and one page in Posting tree */
ItemPointerData *list;
uint32 nlist;
extern Datum gingetbitmap(PG_FUNCTION_ARGS);
extern Datum gingettuple(PG_FUNCTION_ARGS);
+extern void ginrestartentry(GinScanEntry entry);
/* ginvacuum.c */
extern Datum ginbulkdelete(PG_FUNCTION_ARGS);
* Portions Copyright (c) 1996-2008, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
- * $PostgreSQL: pgsql/src/include/catalog/catversion.h,v 1.459 2008/05/15 00:17:40 tgl Exp $
+ * $PostgreSQL: pgsql/src/include/catalog/catversion.h,v 1.460 2008/05/16 16:31:01 tgl Exp $
*
*-------------------------------------------------------------------------
*/
*/
/* yyyymmddN */
-#define CATALOG_VERSION_NO 200805141
+#define CATALOG_VERSION_NO 200805161
#endif
* Portions Copyright (c) 1996-2008, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
- * $PostgreSQL: pgsql/src/include/catalog/pg_am.h,v 1.55 2008/04/10 22:25:25 tgl Exp $
+ * $PostgreSQL: pgsql/src/include/catalog/pg_am.h,v 1.56 2008/05/16 16:31:01 tgl Exp $
*
* NOTES
* the genbki.sh script reads this file and generates .bki
DATA(insert OID = 783 ( gist 0 7 f f t t t t t t gistinsert gistbeginscan gistgettuple gistgetbitmap gistrescan gistendscan gistmarkpos gistrestrpos gistbuild gistbulkdelete gistvacuumcleanup gistcostestimate gistoptions ));
DESCR("GiST index access method");
#define GIST_AM_OID 783
-DATA(insert OID = 2742 ( gin 0 4 f f f f f f t f gininsert ginbeginscan gingettuple gingetbitmap ginrescan ginendscan ginmarkpos ginrestrpos ginbuild ginbulkdelete ginvacuumcleanup gincostestimate ginoptions ));
+DATA(insert OID = 2742 ( gin 0 5 f f f f f f t f gininsert ginbeginscan gingettuple gingetbitmap ginrescan ginendscan ginmarkpos ginrestrpos ginbuild ginbulkdelete ginvacuumcleanup gincostestimate ginoptions ));
DESCR("GIN index access method");
#define GIN_AM_OID 2742
* Portions Copyright (c) 1996-2008, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
- * $PostgreSQL: pgsql/src/include/catalog/pg_amproc.h,v 1.71 2008/03/27 03:57:34 tgl Exp $
+ * $PostgreSQL: pgsql/src/include/catalog/pg_amproc.h,v 1.72 2008/05/16 16:31:01 tgl Exp $
*
* NOTES
* the genbki.sh script reads this file and generates .bki
DATA(insert ( 2745 1025 1025 2 2743 ));
DATA(insert ( 2745 1025 1025 3 2774 ));
DATA(insert ( 2745 1025 1025 4 2744 ));
-DATA(insert ( 3659 3614 3614 1 360 ));
+DATA(insert ( 3659 3614 3614 1 3724 ));
DATA(insert ( 3659 3614 3614 2 3656 ));
DATA(insert ( 3659 3614 3614 3 3657 ));
DATA(insert ( 3659 3614 3614 4 3658 ));
+DATA(insert ( 3659 3614 3614 5 2700 ));
DATA(insert ( 3626 3614 3614 1 3622 ));
DATA(insert ( 3683 3615 3615 1 3668 ));
* Portions Copyright (c) 1996-2008, PostgreSQL Global Development Group
* Portions Copyright (c) 1994, Regents of the University of California
*
- * $PostgreSQL: pgsql/src/include/catalog/pg_proc.h,v 1.499 2008/05/15 00:17:40 tgl Exp $
+ * $PostgreSQL: pgsql/src/include/catalog/pg_proc.h,v 1.500 2008/05/16 16:31:01 tgl Exp $
*
* NOTES
* The script catalog/genbki.sh reads this file and generates .bki
/* GIN array support */
DATA(insert OID = 2743 ( ginarrayextract PGNSP PGUID 12 1 0 f f t f i 2 2281 "2277 2281" _null_ _null_ _null_ ginarrayextract - _null_ _null_ ));
DESCR("GIN array support");
-DATA(insert OID = 2774 ( ginqueryarrayextract PGNSP PGUID 12 1 0 f f t f i 3 2281 "2277 2281 21" _null_ _null_ _null_ ginqueryarrayextract - _null_ _null_ ));
+DATA(insert OID = 2774 ( ginqueryarrayextract PGNSP PGUID 12 1 0 f f t f i 4 2281 "2277 2281 21 2281" _null_ _null_ _null_ ginqueryarrayextract - _null_ _null_ ));
DESCR("GIN array support");
DATA(insert OID = 2744 ( ginarrayconsistent PGNSP PGUID 12 1 0 f f t f i 4 16 "2281 21 2281 2281" _null_ _null_ _null_ ginarrayconsistent - _null_ _null_ ));
DESCR("GIN array support");
DATA(insert OID = 3656 ( gin_extract_tsvector PGNSP PGUID 12 1 0 f f t f i 2 2281 "3614 2281" _null_ _null_ _null_ gin_extract_tsvector - _null_ _null_ ));
DESCR("GIN tsvector support");
-DATA(insert OID = 3657 ( gin_extract_tsquery PGNSP PGUID 12 1 0 f f t f i 3 2281 "3615 2281 21" _null_ _null_ _null_ gin_extract_tsquery - _null_ _null_ ));
+DATA(insert OID = 3657 ( gin_extract_tsquery PGNSP PGUID 12 1 0 f f t f i 4 2281 "3615 2281 21 2281" _null_ _null_ _null_ gin_extract_tsquery - _null_ _null_ ));
DESCR("GIN tsvector support");
DATA(insert OID = 3658 ( gin_tsquery_consistent PGNSP PGUID 12 1 0 f f t f i 4 16 "2281 21 3615 2281" _null_ _null_ _null_ gin_tsquery_consistent - _null_ _null_ ));
DESCR("GIN tsvector support");
+DATA(insert OID = 3724 ( gin_cmp_tslexeme PGNSP PGUID 12 1 0 f f t f i 2 23 "25 25" _null_ _null_ _null_ gin_cmp_tslexeme - _null_ _null_ ));
+DESCR("GIN tsvector support");
+DATA(insert OID = 2700 ( gin_cmp_prefix PGNSP PGUID 12 1 0 f f t f i 3 23 "25 25 21" _null_ _null_ _null_ gin_cmp_prefix - _null_ _null_ ));
+DESCR("GIN tsvector support");
DATA(insert OID = 3662 ( tsquery_lt PGNSP PGUID 12 1 0 f f t f i 2 16 "3615 3615" _null_ _null_ _null_ tsquery_lt - _null_ _null_ ));
DESCR("less-than");
*
* Copyright (c) 1998-2008, PostgreSQL Global Development Group
*
- * $PostgreSQL: pgsql/src/include/tsearch/ts_public.h,v 1.8 2008/01/01 19:45:59 momjian Exp $
+ * $PostgreSQL: pgsql/src/include/tsearch/ts_public.h,v 1.9 2008/05/16 16:31:02 tgl Exp $
*
*-------------------------------------------------------------------------
*/
} TSLexeme;
#define TSL_ADDPOS 0x01
+#define TSL_PREFIX 0x02
/*
* Struct for supporting complex dictionaries like thesaurus.
*
* Copyright (c) 1998-2008, PostgreSQL Global Development Group
*
- * $PostgreSQL: pgsql/src/include/tsearch/ts_type.h,v 1.10 2008/01/01 19:45:59 momjian Exp $
+ * $PostgreSQL: pgsql/src/include/tsearch/ts_type.h,v 1.11 2008/05/16 16:31:02 tgl Exp $
*
*-------------------------------------------------------------------------
*/
* bitmask of allowed weights. if it =0 then
* any weight are allowed. Weights and bit
* map: A: 1<<3 B: 1<<2 C: 1<<1 D: 1<<0 */
+ bool prefix; /* true if it's a prefix search */
int32 valcrc; /* XXX: pg_crc32 would be a more appropriate
* data type, but we use comparisons to signed
* integers in the code. They would need to be
*
* Copyright (c) 1998-2008, PostgreSQL Global Development Group
*
- * $PostgreSQL: pgsql/src/include/tsearch/ts_utils.h,v 1.14 2008/04/21 00:26:47 tgl Exp $
+ * $PostgreSQL: pgsql/src/include/tsearch/ts_utils.h,v 1.15 2008/05/16 16:31:02 tgl Exp $
*
*-------------------------------------------------------------------------
*/
typedef void (*PushFunction) (Datum opaque, TSQueryParserState state,
char *token, int tokenlen,
- int2 tokenweights /* bitmap as described
+ int2 tokenweights, /* bitmap as described
* in QueryOperand
- struct */ );
+ * struct */
+ bool prefix);
extern TSQuery parse_tsquery(char *buf,
PushFunction pushval,
/* Functions for use by PushFunction implementations */
extern void pushValue(TSQueryParserState state,
- char *strval, int lenval, int2 weight);
+ char *strval, int lenval, int2 weight, bool prefix);
extern void pushStop(TSQueryParserState state);
extern void pushOperator(TSQueryParserState state, int8 operator);
*/
uint16 *apos;
} pos;
+ uint16 flags; /* currently, only TSL_PREFIX */
char *word;
uint32 alen;
} ParsedWord;
* to_ts* - text transformation to tsvector, tsquery
*/
extern TSVector make_tsvector(ParsedText *prs);
+extern int32 tsCompareString(char *a, int lena, char *b, int lenb, bool prefix);
extern Datum to_tsvector_byid(PG_FUNCTION_ARGS);
extern Datum to_tsvector(PG_FUNCTION_ARGS);
*/
extern Datum gin_extract_tsvector(PG_FUNCTION_ARGS);
+extern Datum gin_cmp_tslexeme(PG_FUNCTION_ARGS);
+extern Datum gin_cmp_prefix(PG_FUNCTION_ARGS);
extern Datum gin_extract_tsquery(PG_FUNCTION_ARGS);
extern Datum gin_tsquery_consistent(PG_FUNCTION_ARGS);
-- Detect missing pg_amproc entries: should have as many support functions
-- as AM expects for each datatype combination supported by the opfamily.
+-- GIN is a special case because it has an optional support function.
SELECT p1.amname, p2.opfname, p3.amproclefttype, p3.amprocrighttype
FROM pg_am AS p1, pg_opfamily AS p2, pg_amproc AS p3
WHERE p2.opfmethod = p1.oid AND p3.amprocfamily = p2.oid AND
+ p1.amname <> 'gin' AND
p1.amsupport != (SELECT count(*) FROM pg_amproc AS p4
WHERE p4.amprocfamily = p2.oid AND
p4.amproclefttype = p3.amproclefttype AND
--------+---------+----------------+-----------------
(0 rows)
+-- Similar check for GIN, allowing one optional proc
+SELECT p1.amname, p2.opfname, p3.amproclefttype, p3.amprocrighttype
+FROM pg_am AS p1, pg_opfamily AS p2, pg_amproc AS p3
+WHERE p2.opfmethod = p1.oid AND p3.amprocfamily = p2.oid AND
+ p1.amname = 'gin' AND
+ p1.amsupport - 1 > (SELECT count(*) FROM pg_amproc AS p4
+ WHERE p4.amprocfamily = p2.oid AND
+ p4.amproclefttype = p3.amproclefttype AND
+ p4.amprocrighttype = p3.amprocrighttype);
+ amname | opfname | amproclefttype | amprocrighttype
+--------+---------+----------------+-----------------
+(0 rows)
+
-- Also, check if there are any pg_opclass entries that don't seem to have
--- pg_amproc support.
+-- pg_amproc support. Again, GIN has to be checked separately.
SELECT amname, opcname, count(*)
FROM pg_am am JOIN pg_opclass op ON opcmethod = am.oid
LEFT JOIN pg_amproc p ON amprocfamily = opcfamily AND
amproclefttype = amprocrighttype AND amproclefttype = opcintype
+WHERE am.amname <> 'gin'
GROUP BY amname, amsupport, opcname, amprocfamily
HAVING count(*) != amsupport OR amprocfamily IS NULL;
amname | opcname | count
--------+---------+-------
(0 rows)
+SELECT amname, opcname, count(*)
+FROM pg_am am JOIN pg_opclass op ON opcmethod = am.oid
+ LEFT JOIN pg_amproc p ON amprocfamily = opcfamily AND
+ amproclefttype = amprocrighttype AND amproclefttype = opcintype
+WHERE am.amname = 'gin'
+GROUP BY amname, amsupport, opcname, amprocfamily
+HAVING count(*) < amsupport - 1 OR amprocfamily IS NULL;
+ amname | opcname | count
+--------+---------+-------
+(0 rows)
+
-- Unfortunately, we can't check the amproc link very well because the
-- signature of the function may be different for different support routines
-- or different base data types.
SELECT to_tsvector('ispell_tst', 'Booking the skies after rebookings for footballklubber from a foot');
to_tsvector
----------------------------------------------------------------------------------------------------
- 'sky':3 'ball':7 'book':1,5 'foot':7,10 'booking':1,5 'klubber':7 'football':7 'footballklubber':7
+ 'ball':7 'book':1,5 'booking':1,5 'foot':7,10 'football':7 'footballklubber':7 'klubber':7 'sky':3
(1 row)
SELECT to_tsquery('ispell_tst', 'footballklubber');
SELECT to_tsvector('hunspell_tst', 'Booking the skies after rebookings for footballklubber from a foot');
to_tsvector
----------------------------------------------------------------------------------------------------
- 'sky':3 'ball':7 'book':1,5 'foot':7,10 'booking':1,5 'klubber':7 'football':7 'footballklubber':7
+ 'ball':7 'book':1,5 'booking':1,5 'foot':7,10 'football':7 'footballklubber':7 'klubber':7 'sky':3
(1 row)
SELECT to_tsquery('hunspell_tst', 'footballklubber');
SELECT to_tsvector('synonym_tst', 'Most common mistake is to write Gogle instead of Google');
to_tsvector
----------------------------------------------------------
- 'googl':7,10 'write':6 'common':2 'mistak':3 'instead':8
+ 'common':2 'googl':7,10 'instead':8 'mistak':3 'write':6
(1 row)
-- test thesaurus in configuration
SELECT to_tsvector('thesaurus_tst', 'Supernovae star is very new star and usually called supernovae (abbrevation SN)');
to_tsvector
-------------------------------------------------------------
- 'sn':1,9,11 'new':4 'call':8 'star':5 'usual':7 'abbrev':10
+ 'abbrev':10 'call':8 'new':4 'sn':1,9,11 'star':5 'usual':7
(1 row)
SELECT to_tsvector('thesaurus_tst', 'Booking tickets is looking like a booking a tickets');
to_tsvector
-------------------------------------------------------
- 'card':3,10 'like':6 'look':5 'invit':2,9 'order':1,8
+ 'card':3,10 'invit':2,9 'like':6 'look':5 'order':1,8
(1 row)
39
(1 row)
+SELECT count(*) FROM test_tsvector WHERE a @@ 'w:*|q:*';
+ count
+-------
+ 494
+(1 row)
+
create index wowidx on test_tsvector using gist (a);
SET enable_seqscan=OFF;
SELECT count(*) FROM test_tsvector WHERE a @@ 'wr|qh';
39
(1 row)
+SELECT count(*) FROM test_tsvector WHERE a @@ 'w:*|q:*';
+ count
+-------
+ 494
+(1 row)
+
RESET enable_seqscan;
DROP INDEX wowidx;
CREATE INDEX wowidx ON test_tsvector USING gin (a);
39
(1 row)
+SELECT count(*) FROM test_tsvector WHERE a @@ 'w:*|q:*';
+ count
+-------
+ 494
+(1 row)
+
RESET enable_seqscan;
INSERT INTO test_tsvector VALUES ('???', 'DFG:1A,2B,6C,10 FGH');
<i <b> wow < jqw <> qwerty');
to_tsvector
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
- 'ad':17 'dw':19 'jf':39 '234':61 '345':1 '4.2':54,55,56 '455':31 'jqw':64 'qwe':2,18,27,28,35 'wer':36 'wow':63 '-4.2':58,60 'asdf':37 'ewr1':43 'qwer':38 'sdjk':40 '5.005':32 'efd.r':3 'ewri2':44 'hjwer':42 'qwqwe':29 'wefjn':48 'gist.c':52 'gist.h':50 'qwerti':65 '234.435':30 'qwe-wer':34 'readlin':53,57,59 'www.com':4 '+4.0e-10':26 'gist.h.c':51 'rewt/ewr':47 '/?ad=qwe&dw':7,10,14,22 '/wqe-324/ewr':49 'aew.werc.ewr':6 '1aew.werc.ewr':9 '2aew.werc.ewr':11 '3aew.werc.ewr':13 '4aew.werc.ewr':15 '/usr/local/fff':45 '/awdf/dwqe/4325':46 'teodor@stack.net':33 '/?ad=qwe&dw=%20%32':25 '5aew.werc.ewr:8100':16 '6aew.werc.ewr:8100':21 '7aew.werc.ewr:8100':24 'aew.werc.ewr/?ad=qwe&dw':5 '1aew.werc.ewr/?ad=qwe&dw':8 '3aew.werc.ewr/?ad=qwe&dw':12 '6aew.werc.ewr:8100/?ad=qwe&dw':20 '7aew.werc.ewr:8100/?ad=qwe&dw=%20%32':23
+ '+4.0e-10':26 '-4.2':58,60 '/?ad=qwe&dw':7,10,14,22 '/?ad=qwe&dw=%20%32':25 '/awdf/dwqe/4325':46 '/usr/local/fff':45 '/wqe-324/ewr':49 '1aew.werc.ewr':9 '1aew.werc.ewr/?ad=qwe&dw':8 '234':61 '234.435':30 '2aew.werc.ewr':11 '345':1 '3aew.werc.ewr':13 '3aew.werc.ewr/?ad=qwe&dw':12 '4.2':54,55,56 '455':31 '4aew.werc.ewr':15 '5.005':32 '5aew.werc.ewr:8100':16 '6aew.werc.ewr:8100':21 '6aew.werc.ewr:8100/?ad=qwe&dw':20 '7aew.werc.ewr:8100':24 '7aew.werc.ewr:8100/?ad=qwe&dw=%20%32':23 'ad':17 'aew.werc.ewr':6 'aew.werc.ewr/?ad=qwe&dw':5 'asdf':37 'dw':19 'efd.r':3 'ewr1':43 'ewri2':44 'gist.c':52 'gist.h':50 'gist.h.c':51 'hjwer':42 'jf':39 'jqw':64 'qwe':2,18,27,28,35 'qwe-wer':34 'qwer':38 'qwerti':65 'qwqwe':29 'readlin':53,57,59 'rewt/ewr':47 'sdjk':40 'teodor@stack.net':33 'wefjn':48 'wer':36 'wow':63 'www.com':4
(1 row)
SELECT length(to_tsvector('english', '345 qwe@efd.r '' http://www.com/ http://aew.werc.ewr/?ad=qwe&dw 1aew.werc.ewr/?ad=qwe&dw 2aew.werc.ewr http://3aew.werc.ewr/?ad=qwe&dw http://4aew.werc.ewr http://5aew.werc.ewr:8100/? ad=qwe&dw 6aew.werc.ewr:8100/?ad=qwe&dw 7aew.werc.ewr:8100/?ad=qwe&dw=%20%32 +4.0e-10 qwe qwe qwqwe 234.435 455 5.005 teodor@stack.net qwe-wer asdf <fr>qwer jf sdjk<we hjwer <werrwe> ewr1> ewri2 <a href="qwe<qwe>">
SELECT to_tsvector('SKIES My booKs');
to_tsvector
----------------------------
- 'my':2 'books':3 'skies':1
+ 'books':3 'my':2 'skies':1
(1 row)
SELECT plainto_tsquery('SKIES My booKs');
SELECT to_tsvector('SKIES My booKs');
to_tsvector
------------------
- 'sky':1 'book':3
+ 'book':3 'sky':1
(1 row)
SELECT plainto_tsquery('SKIES My booKs');
SELECT E'''1 \\''2''3'::tsvector;
tsvector
-------------
- '3' '1 ''2'
+ '1 ''2' '3'
(1 row)
SELECT E'''1 \\''2'' 3'::tsvector;
tsvector
-------------
- '3' '1 ''2'
+ '1 ''2' '3'
(1 row)
SELECT E'''1 \\''2'' '' 3'' 4 '::tsvector;
tsvector
------------------
- '4' ' 3' '1 ''2'
+ ' 3' '1 ''2' '4'
(1 row)
SELECT $$'\\as' ab\c ab\\c AB\\\c ab\\\\c$$::tsvector;
tsvector
----------------------------------------
- '\\as' 'abc' 'AB\\c' 'ab\\c' 'ab\\\\c'
+ 'AB\\c' '\\as' 'ab\\\\c' 'ab\\c' 'abc'
(1 row)
SELECT tsvectorin(tsvectorout($$'\\as' ab\c ab\\c AB\\\c ab\\\\c$$::tsvector));
tsvectorin
----------------------------------------
- '\\as' 'abc' 'AB\\c' 'ab\\c' 'ab\\\\c'
+ 'AB\\c' '\\as' 'ab\\\\c' 'ab\\c' 'abc'
(1 row)
SELECT '''w'':4A,3B,2C,1D,5 a:8';
SELECT setweight('w:12B w:13* w:12,5,6 a:1,3* a:3 w asd:1dc asd zxc:81,567,222A'::tsvector, 'c');
setweight
----------------------------------------------------------
- 'a':1C,3C 'w':5C,6C,12C,13C 'asd':1C 'zxc':81C,222C,567C
+ 'a':1C,3C 'asd':1C 'w':5C,6C,12C,13C 'zxc':81C,222C,567C
(1 row)
SELECT strip('w:12B w:13* w:12,5,6 a:1,3* a:3 w asd:1dc asd'::tsvector);
strip
---------------
- 'a' 'w' 'asd'
+ 'a' 'asd' 'w'
(1 row)
--Base tsquery test
'\\as'
(1 row)
+SELECT 'a:* & nbb:*ac | doo:a* | goo'::tsquery;
+ tsquery
+------------------------------------------
+ ( 'a':* & 'nbb':*AC | 'doo':*A ) | 'goo'
+(1 row)
+
SELECT 'a' < 'b & c'::tsquery as "true";
true
------
t
(1 row)
+SELECT 'a b:89 ca:23A,64b d:34c'::tsvector @@ 'd:AC & c:*C' as "false";
+ false
+-------
+ f
+(1 row)
+
+SELECT 'a b:89 ca:23A,64b d:34c'::tsvector @@ 'd:AC & c:*CB' as "true";
+ true
+------
+ t
+(1 row)
+
+SELECT 'a b:89 ca:23A,64b cb:80c d:34c'::tsvector @@ 'd:AC & c:*C' as "true";
+ true
+------
+ t
+(1 row)
+
+SELECT 'a b:89 ca:23A,64c cb:80b d:34c'::tsvector @@ 'd:AC & c:*C' as "true";
+ true
+------
+ t
+(1 row)
+
+SELECT 'a b:89 ca:23A,64c cb:80b d:34c'::tsvector @@ 'd:AC & c:*B' as "true";
+ true
+------
+ t
+(1 row)
+
+SELECT 'supernova'::tsvector @@ 'super'::tsquery AS "false";
+ false
+-------
+ f
+(1 row)
+
+SELECT 'supeanova supernova'::tsvector @@ 'super'::tsquery AS "false";
+ false
+-------
+ f
+(1 row)
+
+SELECT 'supeznova supernova'::tsvector @@ 'super'::tsquery AS "false";
+ false
+-------
+ f
+(1 row)
+
+SELECT 'supernova'::tsvector @@ 'super:*'::tsquery AS "true";
+ true
+------
+ t
+(1 row)
+
+SELECT 'supeanova supernova'::tsvector @@ 'super:*'::tsquery AS "true";
+ true
+------
+ t
+(1 row)
+
+SELECT 'supeznova supernova'::tsvector @@ 'super:*'::tsquery AS "true";
+ true
+------
+ t
+(1 row)
+
SELECT ts_rank(' a:1 s:2C d g'::tsvector, 'a | s');
ts_rank
-----------
0.0911891
(1 row)
+SELECT ts_rank(' a:1 sa:2C d g'::tsvector, 'a | s');
+ ts_rank
+-----------
+ 0.0303964
+(1 row)
+
+SELECT ts_rank(' a:1 sa:2C d g'::tsvector, 'a | s:*');
+ ts_rank
+-----------
+ 0.0911891
+(1 row)
+
+SELECT ts_rank(' a:1 sa:2C d g'::tsvector, 'a | sa:*');
+ ts_rank
+-----------
+ 0.0911891
+(1 row)
+
SELECT ts_rank(' a:1 s:2B d g'::tsvector, 'a | s');
ts_rank
----------
0.3
(1 row)
+SELECT ts_rank_cd(' a:1 sa:2C d g'::tsvector, 'a | s');
+ ts_rank_cd
+------------
+ 0.1
+(1 row)
+
+SELECT ts_rank_cd(' a:1 sa:2C d g'::tsvector, 'a | s:*');
+ ts_rank_cd
+------------
+ 0.3
+(1 row)
+
+SELECT ts_rank_cd(' a:1 sa:2C d g'::tsvector, 'a | sa:*');
+ ts_rank_cd
+------------
+ 0.3
+(1 row)
+
+SELECT ts_rank_cd(' a:1 sa:3C sab:2c d g'::tsvector, 'a | sa:*');
+ ts_rank_cd
+------------
+ 0.5
+(1 row)
+
SELECT ts_rank_cd(' a:1 s:2B d g'::tsvector, 'a | s');
ts_rank_cd
------------
-- Detect missing pg_amproc entries: should have as many support functions
-- as AM expects for each datatype combination supported by the opfamily.
+-- GIN is a special case because it has an optional support function.
SELECT p1.amname, p2.opfname, p3.amproclefttype, p3.amprocrighttype
FROM pg_am AS p1, pg_opfamily AS p2, pg_amproc AS p3
WHERE p2.opfmethod = p1.oid AND p3.amprocfamily = p2.oid AND
+ p1.amname <> 'gin' AND
p1.amsupport != (SELECT count(*) FROM pg_amproc AS p4
WHERE p4.amprocfamily = p2.oid AND
p4.amproclefttype = p3.amproclefttype AND
p4.amprocrighttype = p3.amprocrighttype);
+-- Similar check for GIN, allowing one optional proc
+
+SELECT p1.amname, p2.opfname, p3.amproclefttype, p3.amprocrighttype
+FROM pg_am AS p1, pg_opfamily AS p2, pg_amproc AS p3
+WHERE p2.opfmethod = p1.oid AND p3.amprocfamily = p2.oid AND
+ p1.amname = 'gin' AND
+ p1.amsupport - 1 > (SELECT count(*) FROM pg_amproc AS p4
+ WHERE p4.amprocfamily = p2.oid AND
+ p4.amproclefttype = p3.amproclefttype AND
+ p4.amprocrighttype = p3.amprocrighttype);
+
-- Also, check if there are any pg_opclass entries that don't seem to have
--- pg_amproc support.
+-- pg_amproc support. Again, GIN has to be checked separately.
SELECT amname, opcname, count(*)
FROM pg_am am JOIN pg_opclass op ON opcmethod = am.oid
LEFT JOIN pg_amproc p ON amprocfamily = opcfamily AND
amproclefttype = amprocrighttype AND amproclefttype = opcintype
+WHERE am.amname <> 'gin'
GROUP BY amname, amsupport, opcname, amprocfamily
HAVING count(*) != amsupport OR amprocfamily IS NULL;
+SELECT amname, opcname, count(*)
+FROM pg_am am JOIN pg_opclass op ON opcmethod = am.oid
+ LEFT JOIN pg_amproc p ON amprocfamily = opcfamily AND
+ amproclefttype = amprocrighttype AND amproclefttype = opcintype
+WHERE am.amname = 'gin'
+GROUP BY amname, amsupport, opcname, amprocfamily
+HAVING count(*) < amsupport - 1 OR amprocfamily IS NULL;
+
-- Unfortunately, we can't check the amproc link very well because the
-- signature of the function may be different for different support routines
-- or different base data types.
SELECT count(*) FROM test_tsvector WHERE a @@ 'eq|yt';
SELECT count(*) FROM test_tsvector WHERE a @@ '(eq&yt)|(wr&qh)';
SELECT count(*) FROM test_tsvector WHERE a @@ '(eq|yt)&(wr|qh)';
+SELECT count(*) FROM test_tsvector WHERE a @@ 'w:*|q:*';
create index wowidx on test_tsvector using gist (a);
SELECT count(*) FROM test_tsvector WHERE a @@ 'eq|yt';
SELECT count(*) FROM test_tsvector WHERE a @@ '(eq&yt)|(wr&qh)';
SELECT count(*) FROM test_tsvector WHERE a @@ '(eq|yt)&(wr|qh)';
+SELECT count(*) FROM test_tsvector WHERE a @@ 'w:*|q:*';
RESET enable_seqscan;
SELECT count(*) FROM test_tsvector WHERE a @@ 'eq|yt';
SELECT count(*) FROM test_tsvector WHERE a @@ '(eq&yt)|(wr&qh)';
SELECT count(*) FROM test_tsvector WHERE a @@ '(eq|yt)&(wr|qh)';
+SELECT count(*) FROM test_tsvector WHERE a @@ 'w:*|q:*';
RESET enable_seqscan;
INSERT INTO test_tsvector VALUES ('???', 'DFG:1A,2B,6C,10 FGH');
SELECT '1&(2&(4&(5|!6)))'::tsquery;
SELECT E'1&(''2''&('' 4''&(\\|5 | ''6 \\'' !|&'')))'::tsquery;
SELECT $$'\\as'$$::tsquery;
+SELECT 'a:* & nbb:*ac | doo:a* | goo'::tsquery;
SELECT 'a' < 'b & c'::tsquery as "true";
SELECT 'a' > 'b & c'::tsquery as "false";
SELECT 'a b:89 ca:23A,64b d:34c'::tsvector @@ 'd:AC & ca:A' as "true";
SELECT 'a b:89 ca:23A,64b d:34c'::tsvector @@ 'd:AC & ca:C' as "false";
SELECT 'a b:89 ca:23A,64b d:34c'::tsvector @@ 'd:AC & ca:CB' as "true";
+SELECT 'a b:89 ca:23A,64b d:34c'::tsvector @@ 'd:AC & c:*C' as "false";
+SELECT 'a b:89 ca:23A,64b d:34c'::tsvector @@ 'd:AC & c:*CB' as "true";
+SELECT 'a b:89 ca:23A,64b cb:80c d:34c'::tsvector @@ 'd:AC & c:*C' as "true";
+SELECT 'a b:89 ca:23A,64c cb:80b d:34c'::tsvector @@ 'd:AC & c:*C' as "true";
+SELECT 'a b:89 ca:23A,64c cb:80b d:34c'::tsvector @@ 'd:AC & c:*B' as "true";
+
+SELECT 'supernova'::tsvector @@ 'super'::tsquery AS "false";
+SELECT 'supeanova supernova'::tsvector @@ 'super'::tsquery AS "false";
+SELECT 'supeznova supernova'::tsvector @@ 'super'::tsquery AS "false";
+SELECT 'supernova'::tsvector @@ 'super:*'::tsquery AS "true";
+SELECT 'supeanova supernova'::tsvector @@ 'super:*'::tsquery AS "true";
+SELECT 'supeznova supernova'::tsvector @@ 'super:*'::tsquery AS "true";
SELECT ts_rank(' a:1 s:2C d g'::tsvector, 'a | s');
+SELECT ts_rank(' a:1 sa:2C d g'::tsvector, 'a | s');
+SELECT ts_rank(' a:1 sa:2C d g'::tsvector, 'a | s:*');
+SELECT ts_rank(' a:1 sa:2C d g'::tsvector, 'a | sa:*');
SELECT ts_rank(' a:1 s:2B d g'::tsvector, 'a | s');
SELECT ts_rank(' a:1 s:2 d g'::tsvector, 'a | s');
SELECT ts_rank(' a:1 s:2C d g'::tsvector, 'a & s');
SELECT ts_rank(' a:1 s:2 d g'::tsvector, 'a & s');
SELECT ts_rank_cd(' a:1 s:2C d g'::tsvector, 'a | s');
+SELECT ts_rank_cd(' a:1 sa:2C d g'::tsvector, 'a | s');
+SELECT ts_rank_cd(' a:1 sa:2C d g'::tsvector, 'a | s:*');
+SELECT ts_rank_cd(' a:1 sa:2C d g'::tsvector, 'a | sa:*');
+SELECT ts_rank_cd(' a:1 sa:3C sab:2c d g'::tsvector, 'a | sa:*');
SELECT ts_rank_cd(' a:1 s:2B d g'::tsvector, 'a | s');
SELECT ts_rank_cd(' a:1 s:2 d g'::tsvector, 'a | s');
SELECT ts_rank_cd(' a:1 s:2C d g'::tsvector, 'a & s');
projects / postgresql.git / commitdiff