From dbaec70c153239224c0288d865b96c2f939fbdf5 Mon Sep 17 00:00:00 2001 From: Tom Lane Date: Tue, 23 Oct 2007 20:46:12 +0000 Subject: Rename and slightly redefine the default text search parser's "word" categories, as per discussion. asciiword (formerly lword) is still ASCII-letters-only, and numword (formerly word) is still the most general mixed-alpha-and-digits case. But word (formerly nlword) is now any-group-of-letters-with-at-least-one-non-ASCII, rather than all-non-ASCII as before. This is no worse than before for parsing mixed Russian/English text, which seems to have been the design center for the original coding; and it should simplify matters for parsing most European languages. In particular it will not be necessary for any language to accept strings containing digits as being regular "words". The hyphenated-word categories are adjusted similarly. --- doc/src/sgml/func.sgml | 8 +- doc/src/sgml/textsearch.sgml | 365 ++++++++++++++++++++++--------------------- 2 files changed, 193 insertions(+), 180 deletions(-) (limited to 'doc/src') diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 368673c66e6..c9d665376e4 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -1,4 +1,4 @@ - + Functions and Operators @@ -7861,7 +7861,7 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple setof record test a configuration ts_debug('english', 'The Brightest supernovaes') - (lword,"Latin word",The,{english_stem},english_stem,{}) ... + (asciiword,"Word, all ASCII",The,{english_stem},english_stem,{}) ... ts_lexize(dict regdictionary, token text) @@ -7889,14 +7889,14 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple setof record get token types defined by parser ts_token_type('default') - (1,lword,"Latin word") ... + (1,asciiword,"Word, all ASCII") ... ts_token_type(parser_oid oid, OUT tokid integer, OUT alias text, OUT description text) setof record get token types defined by parser ts_token_type(3722) - (1,lword,"Latin word") ... + (1,asciiword,"Word, all ASCII") ... ts_stat(sqlquery text, weights text, OUT word text, OUT ndoc integer, OUT nentry integer) diff --git a/doc/src/sgml/textsearch.sgml b/doc/src/sgml/textsearch.sgml index 81b54d8e174..64a5929e1a7 100644 --- a/doc/src/sgml/textsearch.sgml +++ b/doc/src/sgml/textsearch.sgml @@ -1,4 +1,4 @@ - + Full Text Search @@ -1775,119 +1775,120 @@ LIMIT 10; - lword - Latin word (only ASCII letters) + asciiword + Word, all ASCII letters foo - nlword - Non-latin word (only non-ASCII letters) - + word + Word, all letters + føø - word - Word (other cases) + numword + Word, letters and digits beta1 - lhword - Latin hyphenated word + asciihword + Hyphenated word, all ASCII foo-bar - nlhword - Non-latin hyphenated word - + hword + Hyphenated word, all letters + føø-bar - hword - Hyphenated word + numhword + Hyphenated word, letters and digits foo-beta1 - lpart_hword - Latin part of hyphenated word + hword_asciipart + Hyphenated word part, all ASCII foo or bar in the context - foo-bar + foo-bar - nlpart_hword - Non-latin part of hyphenated word - + hword_part + Hyphenated word part, all letters + føø in the context + føø-bar - part_hword - Part of hyphenated word + hword_numpart + Hyphenated word part, letters and digits beta1 in the context - foo-beta1 + foo-beta1 - email + email Email address foo@bar.com - protocol + protocol Protocol head http:// - url + url URL foo.com/stuff/index.html - host + host Host foo.com - uri + uri URI /stuff/index.html, in the context of a URL - file + file File or path name /usr/local/foo.txt, if not within a URL - sfloat + sfloat Scientific notation -1.234e56 - float + float Decimal notation -1.234 - int + int Signed integer -1234 - uint + uint Unsigned integer 1234 - version + version Version number 8.3.0 - tag - HTML Tag + tag + HTML tag <A HREF="dictionaries.html"> - entity - HTML Entity + entity + HTML entity &amp; - blank + blank Space symbols (any whitespace or punctuation not otherwise recognized) @@ -1895,6 +1896,17 @@ LIMIT 10; + + + The parser's notion of a letter is determined by the server's + locale setting, specifically lc_ctype. Words containing + only the basic ASCII letters are reported as a separate token type, + since it is sometimes useful to distinguish them. In most European + languages, token types word and asciiword + should always be treated alike. + + + It is possible for the parser to produce overlapping tokens from the same piece of text. As an example, a hyphenated word will be reported both @@ -1902,14 +1914,14 @@ LIMIT 10; SELECT alias, description, token FROM ts_debug('foo-bar-beta1'); - alias | description | token --------------+-------------------------------+--------------- - hword | Hyphenated word | foo-bar-beta1 - lpart_hword | Latin part of hyphenated word | foo - blank | Space symbols | - - lpart_hword | Latin part of hyphenated word | bar - blank | Space symbols | - - part_hword | Part of hyphenated word | beta1 + alias | description | token +-----------------+------------------------------------------+--------------- + numhword | Hyphenated word, letters and digits | foo-bar-beta1 + hword_asciipart | Hyphenated word part, all ASCII | foo + blank | Space symbols | - + hword_asciipart | Hyphenated word part, all ASCII | bar + blank | Space symbols | - + hword_numpart | Hyphenated word part, letters and digits | beta1 This behavior is desirable since it allows searches to work for both @@ -2045,13 +2057,13 @@ SELECT alias, description, token FROM ts_debug('http://foo.com/stuff/index.html' a Snowball stemmer or simple, which recognizes everything. For example, for an astronomy-specific search (astro_en configuration) one could bind token type - lword (Latin word) to a synonym dictionary of astronomical + asciiword (ASCII word) to a synonym dictionary of astronomical terms, a general English dictionary and a Snowball English stemmer: ALTER TEXT SEARCH CONFIGURATION astro_en - ADD MAPPING FOR lword WITH astrosyn, english_ispell, english_stem; + ADD MAPPING FOR asciiword WITH astrosyn, english_ispell, english_stem; @@ -2187,9 +2199,9 @@ SELECT ts_lexize('public.simple_dict','The'); SELECT * FROM ts_debug('english', 'Paris'); - alias | description | token | dictionaries | dictionary | lexemes --------+-------------+-------+----------------+--------------+--------- - lword | Latin word | Paris | {english_stem} | english_stem | {pari} + alias | description | token | dictionaries | dictionary | lexemes +-----------+-----------------+-------+----------------+--------------+--------- + asciiword | Word, all ASCII | Paris | {english_stem} | english_stem | {pari} CREATE TEXT SEARCH DICTIONARY my_synonym ( TEMPLATE = synonym, @@ -2197,12 +2209,12 @@ CREATE TEXT SEARCH DICTIONARY my_synonym ( ); ALTER TEXT SEARCH CONFIGURATION english - ALTER MAPPING FOR lword WITH my_synonym, english_stem; + ALTER MAPPING FOR asciiword WITH my_synonym, english_stem; SELECT * FROM ts_debug('english', 'Paris'); - alias | description | token | dictionaries | dictionary | lexemes --------+-------------+-------+---------------------------+------------+--------- - lword | Latin word | Paris | {my_synonym,english_stem} | my_synonym | {paris} + alias | description | token | dictionaries | dictionary | lexemes +-----------+-----------------+-------+---------------------------+------------+--------- + asciiword | Word, all ASCII | Paris | {my_synonym,english_stem} | my_synonym | {paris} @@ -2293,7 +2305,7 @@ the one a two : swsw2 uses these assignments to check if it should handle the next word or stop accumulation. The thesaurus dictionary must be configured carefully. For example, if the thesaurus dictionary is assigned to handle - only the lword token, then a thesaurus dictionary + only the asciiword token, then a thesaurus dictionary definition like one 7 will not work since token type uint is not assigned to the thesaurus dictionary. @@ -2353,7 +2365,7 @@ CREATE TEXT SEARCH DICTIONARY thesaurus_simple ( ALTER TEXT SEARCH CONFIGURATION russian - ADD MAPPING FOR lword, lhword, lpart_hword WITH thesaurus_simple; + ADD MAPPING FOR asciiword, asciihword, hword_asciipart WITH thesaurus_simple; @@ -2382,7 +2394,7 @@ CREATE TEXT SEARCH DICTIONARY thesaurus_astro ( ); ALTER TEXT SEARCH CONFIGURATION russian - ADD MAPPING FOR lword, lhword, lpart_hword WITH thesaurus_astro, english_stem; + ADD MAPPING FOR asciiword, asciihword, hword_asciipart WITH thesaurus_astro, english_stem; Now we can see how it works. @@ -2633,12 +2645,13 @@ CREATE TEXT SEARCH DICTIONARY english_ispell ( ); - Now we can set up the mappings for Latin words for configuration + Now we can set up the mappings for words in configuration pg: ALTER TEXT SEARCH CONFIGURATION pg - ALTER MAPPING FOR lword, lhword, lpart_hword + ALTER MAPPING FOR asciiword, asciihword, hword_asciipart, + word, hword, hword_part WITH pg_dict, english_ispell, english_stem; @@ -2778,32 +2791,32 @@ SHOW default_text_search_config; SELECT * FROM ts_debug('english','a fat cat sat on a mat - it ate a fat rats'); - alias | description | token | dictionaries | dictionary | lexemes --------+---------------+-------+----------------+--------------+--------- - lword | Latin word | a | {english_stem} | english_stem | {} - blank | Space symbols | | {} | | - lword | Latin word | fat | {english_stem} | english_stem | {fat} - blank | Space symbols | | {} | | - lword | Latin word | cat | {english_stem} | english_stem | {cat} - blank | Space symbols | | {} | | - lword | Latin word | sat | {english_stem} | english_stem | {sat} - blank | Space symbols | | {} | | - lword | Latin word | on | {english_stem} | english_stem | {} - blank | Space symbols | | {} | | - lword | Latin word | a | {english_stem} | english_stem | {} - blank | Space symbols | | {} | | - lword | Latin word | mat | {english_stem} | english_stem | {mat} - blank | Space symbols | | {} | | - blank | Space symbols | - | {} | | - lword | Latin word | it | {english_stem} | english_stem | {} - blank | Space symbols | | {} | | - lword | Latin word | ate | {english_stem} | english_stem | {ate} - blank | Space symbols | | {} | | - lword | Latin word | a | {english_stem} | english_stem | {} - blank | Space symbols | | {} | | - lword | Latin word | fat | {english_stem} | english_stem | {fat} - blank | Space symbols | | {} | | - lword | Latin word | rats | {english_stem} | english_stem | {rat} + alias | description | token | dictionaries | dictionary | lexemes +-----------+-----------------+-------+----------------+--------------+--------- + asciiword | Word, all ASCII | a | {english_stem} | english_stem | {} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | fat | {english_stem} | english_stem | {fat} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | cat | {english_stem} | english_stem | {cat} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | sat | {english_stem} | english_stem | {sat} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | on | {english_stem} | english_stem | {} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | a | {english_stem} | english_stem | {} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | mat | {english_stem} | english_stem | {mat} + blank | Space symbols | | {} | | + blank | Space symbols | - | {} | | + asciiword | Word, all ASCII | it | {english_stem} | english_stem | {} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | ate | {english_stem} | english_stem | {ate} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | a | {english_stem} | english_stem | {} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | fat | {english_stem} | english_stem | {fat} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | rats | {english_stem} | english_stem | {rat} @@ -2824,23 +2837,23 @@ CREATE TEXT SEARCH DICTIONARY english_ispell ( ); ALTER TEXT SEARCH CONFIGURATION public.english - ALTER MAPPING FOR lword WITH english_ispell, english_stem; + ALTER MAPPING FOR asciiword WITH english_ispell, english_stem; SELECT * FROM ts_debug('public.english','The Brightest supernovaes'); - alias | description | token | dictionaries | dictionary | lexemes --------+---------------+-------------+-------------------------------+----------------+------------- - lword | Latin word | The | {english_ispell,english_stem} | english_ispell | {} - blank | Space symbols | | {} | | - lword | Latin word | Brightest | {english_ispell,english_stem} | english_ispell | {bright} - blank | Space symbols | | {} | | - lword | Latin word | supernovaes | {english_ispell,english_stem} | english_stem | {supernova} + alias | description | token | dictionaries | dictionary | lexemes +-----------+-----------------+-------------+-------------------------------+----------------+------------- + asciiword | Word, all ASCII | The | {english_ispell,english_stem} | english_ispell | {} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | Brightest | {english_ispell,english_stem} | english_ispell | {bright} + blank | Space symbols | | {} | | + asciiword | Word, all ASCII | supernovaes | {english_ispell,english_stem} | english_stem | {supernova} In this example, the word Brightest was recognized by the - parser as a Latin word (alias lword). + parser as an ASCII word (alias asciiword). For this token type the dictionary list is english_ispell and english_stem. The word was recognized by @@ -2868,13 +2881,13 @@ SELECT * FROM ts_debug('public.english','The Brightest supernovaes'); SELECT alias, token, dictionary, lexemes FROM ts_debug('public.english','The Brightest supernovaes'); - alias | token | dictionary | lexemes --------+-------------+----------------+------------- - lword | The | english_ispell | {} - blank | | | - lword | Brightest | english_ispell | {bright} - blank | | | - lword | supernovaes | english_stem | {supernova} + alias | token | dictionary | lexemes +-----------+-------------+----------------+------------- + asciiword | The | english_ispell | {} + blank | | | + asciiword | Brightest | english_ispell | {bright} + blank | | | + asciiword | supernovaes | english_stem | {supernova} @@ -2935,31 +2948,31 @@ SELECT * FROM ts_parse('default', '123 - a number'); SELECT * FROM ts_token_type('default'); - tokid | alias | description --------+--------------+----------------------------------- - 1 | lword | Latin word - 2 | nlword | Non-latin word - 3 | word | Word - 4 | email | Email - 5 | url | URL - 6 | host | Host - 7 | sfloat | Scientific notation - 8 | version | VERSION - 9 | part_hword | Part of hyphenated word - 10 | nlpart_hword | Non-latin part of hyphenated word - 11 | lpart_hword | Latin part of hyphenated word - 12 | blank | Space symbols - 13 | tag | HTML Tag - 14 | protocol | Protocol head - 15 | hword | Hyphenated word - 16 | lhword | Latin hyphenated word - 17 | nlhword | Non-latin hyphenated word - 18 | uri | URI - 19 | file | File or path name - 20 | float | Decimal notation - 21 | int | Signed integer - 22 | uint | Unsigned integer - 23 | entity | HTML Entity + tokid | alias | description +-------+-----------------+------------------------------------------ + 1 | asciiword | Word, all ASCII + 2 | word | Word, all letters + 3 | numword | Word, letters and digits + 4 | email | Email address + 5 | url | URL + 6 | host | Host + 7 | sfloat | Scientific notation + 8 | version | Version number + 9 | hword_numpart | Hyphenated word part, letters and digits + 10 | hword_part | Hyphenated word part, all letters + 11 | hword_asciipart | Hyphenated word part, all ASCII + 12 | blank | Space symbols + 13 | tag | HTML tag + 14 | protocol | Protocol head + 15 | numhword | Hyphenated word, letters and digits + 16 | asciihword | Hyphenated word, all ASCII + 17 | hword | Hyphenated word, all letters + 18 | uri | URI + 19 | file | File or path name + 20 | float | Decimal notation + 21 | int | Signed integer + 22 | uint | Unsigned integer + 23 | entity | HTML entity @@ -3304,27 +3317,27 @@ EXPLAIN SELECT * FROM apod WHERE textsearch @@ to_tsquery('supernovae'); => \dF+ russian Text search configuration "pg_catalog.russian" Parser: "pg_catalog.default" - Token | Dictionaries ---------------+-------------- - email | simple - file | simple - float | simple - host | simple - hword | russian_stem - int | simple - lhword | english_stem - lpart_hword | english_stem - lword | english_stem - nlhword | russian_stem - nlpart_hword | russian_stem - nlword | russian_stem - part_hword | russian_stem - sfloat | simple - uint | simple - uri | simple - url | simple - version | simple - word | russian_stem + Token | Dictionaries +-----------------+-------------- + asciihword | english_stem + asciiword | english_stem + email | simple + file | simple + float | simple + host | simple + hword | russian_stem + hword_asciipart | english_stem + hword_numpart | simple + hword_part | russian_stem + int | simple + numhword | simple + numword | simple + sfloat | simple + uint | simple + uri | simple + url | simple + version | simple + word | russian_stem @@ -3389,32 +3402,32 @@ Parser: "pg_catalog.default" Get headline | prsd_headline | Get token types | prsd_lextype | - Token types for parser "pg_catalog.default" - Token name | Description ---------------+----------------------------------- - blank | Space symbols - email | Email - entity | HTML Entity - file | File or path name - float | Decimal notation - host | Host - hword | Hyphenated word - int | Signed integer - lhword | Latin hyphenated word - lpart_hword | Latin part of hyphenated word - lword | Latin word - nlhword | Non-latin hyphenated word - nlpart_hword | Non-latin part of hyphenated word - nlword | Non-latin word - part_hword | Part of hyphenated word - protocol | Protocol head - sfloat | Scientific notation - tag | HTML Tag - uint | Unsigned integer - uri | URI - url | URL - version | VERSION - word | Word + Token types for parser "pg_catalog.default" + Token name | Description +-----------------+------------------------------------------ + asciihword | Hyphenated word, all ASCII + asciiword | Word, all ASCII + blank | Space symbols + email | Email address + entity | HTML entity + file | File or path name + float | Decimal notation + host | Host + hword | Hyphenated word, all letters + hword_asciipart | Hyphenated word part, all ASCII + hword_numpart | Hyphenated word part, letters and digits + hword_part | Hyphenated word part, all letters + int | Signed integer + numhword | Hyphenated word, letters and digits + numword | Word, letters and digits + protocol | Protocol head + sfloat | Scientific notation + tag | HTML tag + uint | Unsigned integer + uri | URI + url | URL + version | Version number + word | Word, all letters (23 rows) -- cgit v1.2.3