Baoding-FenShuaJiang
/
Green_GoodERP18_FSJ_Standard
關註 1
收藏 0
Wiki 捐赠
gooderp18绿色标准版
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
目錄樹: 0f75e31513
分支列表 標籤列表
${ item.name }
Create branch ${ searchTerm }
from '0f75e31513'
${ noResults }
Green_GoodERP18_FSJ_Standard/PostgreSQL/doc/postgresql/html/textsearch-intro.html

339 line
24KB
原始文件 Blame 文件歷史 

  1. <?xml version="1.0" encoding="UTF-8" standalone="no"?>

  2. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>12.1. Introduction</title><link rel="stylesheet" type="text/css" href="stylesheet.css" /><link rev="made" href="pgsql-docs@lists.postgresql.org" /><meta name="generator" content="DocBook XSL Stylesheets V1.79.1" /><link rel="prev" href="textsearch.html" title="Chapter 12. Full Text Search" /><link rel="next" href="textsearch-tables.html" title="12.2. Tables and Indexes" /></head><body><div xmlns="http://www.w3.org/TR/xhtml1/transitional" class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="5" align="center">12.1. Introduction</th></tr><tr><td width="10%" align="left"><a accesskey="p" href="textsearch.html" title="Chapter 12. Full Text Search">Prev</a> </td><td width="10%" align="left"><a accesskey="u" href="textsearch.html" title="Chapter 12. Full Text Search">Up</a></td><th width="60%" align="center">Chapter 12. Full Text Search</th><td width="10%" align="right"><a accesskey="h" href="index.html" title="PostgreSQL 12.4 Documentation">Home</a></td><td width="10%" align="right"> <a accesskey="n" href="textsearch-tables.html" title="12.2. Tables and Indexes">Next</a></td></tr></table><hr></hr></div><div class="sect1" id="TEXTSEARCH-INTRO"><div class="titlepage"><div><div><h2 class="title" style="clear: both">12.1. Introduction</h2></div></div></div><div class="toc"><dl class="toc"><dt><span class="sect2"><a href="textsearch-intro.html#TEXTSEARCH-DOCUMENT">12.1.1. What Is a Document?</a></span></dt><dt><span class="sect2"><a href="textsearch-intro.html#TEXTSEARCH-MATCHING">12.1.2. Basic Text Matching</a></span></dt><dt><span class="sect2"><a href="textsearch-intro.html#TEXTSEARCH-INTRO-CONFIGURATIONS">12.1.3. Configurations</a></span></dt></dl></div><p>

  3.    Full Text Searching (or just <em class="firstterm">text search</em>) provides

  4.    the capability to identify natural-language <em class="firstterm">documents</em> that

  5.    satisfy a <em class="firstterm">query</em>, and optionally to sort them by

  6.    relevance to the query.  The most common type of search

  7.    is to find all documents containing given <em class="firstterm">query terms</em>

  8.    and return them in order of their <em class="firstterm">similarity</em> to the

  9.    query.  Notions of <code class="varname">query</code> and

  10.    <code class="varname">similarity</code> are very flexible and depend on the specific

  11.    application. The simplest search considers <code class="varname">query</code> as a

  12.    set of words and <code class="varname">similarity</code> as the frequency of query

  13.    words in the document.

  14.   </p><p>

  15.    Textual search operators have existed in databases for years.

  16.    <span class="productname">PostgreSQL</span> has

  17.    <code class="literal">~</code>, <code class="literal">~*</code>, <code class="literal">LIKE</code>, and

  18.    <code class="literal">ILIKE</code> operators for textual data types, but they lack

  19.    many essential properties required by modern information systems:

  20.   </p><div class="itemizedlist"><ul class="itemizedlist compact" style="list-style-type: bullet; "><li class="listitem" style="list-style-type: disc"><p>

  21.      There is no linguistic support, even for English.  Regular expressions

  22.      are not sufficient because they cannot easily handle derived words, e.g.,

  23.      <code class="literal">satisfies</code> and <code class="literal">satisfy</code>. You might

  24.      miss documents that contain <code class="literal">satisfies</code>, although you

  25.      probably would like to find them when searching for

  26.      <code class="literal">satisfy</code>. It is possible to use <code class="literal">OR</code>

  27.      to search for multiple derived forms, but this is tedious and error-prone

  28.      (some words can have several thousand derivatives).

  29.     </p></li><li class="listitem" style="list-style-type: disc"><p>

  30.      They provide no ordering (ranking) of search results, which makes them

  31.      ineffective when thousands of matching documents are found.

  32.     </p></li><li class="listitem" style="list-style-type: disc"><p>

  33.      They tend to be slow because there is no index support, so they must

  34.      process all documents for every search.

  35.     </p></li></ul></div><p>

  36.    Full text indexing allows documents to be <span class="emphasis"><em>preprocessed</em></span>

  37.    and an index saved for later rapid searching. Preprocessing includes:

  38.   </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: none; "><li class="listitem" style="list-style-type: none"><p>

  39.      <span class="emphasis"><em>Parsing documents into <em class="firstterm">tokens</em></em></span>. It is

  40.      useful to identify various classes of tokens, e.g., numbers, words,

  41.      complex words, email addresses, so that they can be processed

  42.      differently.  In principle token classes depend on the specific

  43.      application, but for most purposes it is adequate to use a predefined

  44.      set of classes.

  45.      <span class="productname">PostgreSQL</span> uses a <em class="firstterm">parser</em> to

  46.      perform this step.  A standard parser is provided, and custom parsers

  47.      can be created for specific needs.

  48.     </p></li><li class="listitem" style="list-style-type: none"><p>

  49.      <span class="emphasis"><em>Converting tokens into <em class="firstterm">lexemes</em></em></span>.

  50.      A lexeme is a string, just like a token, but it has been

  51.      <em class="firstterm">normalized</em> so that different forms of the same word

  52.      are made alike.  For example, normalization almost always includes

  53.      folding upper-case letters to lower-case, and often involves removal

  54.      of suffixes (such as <code class="literal">s</code> or <code class="literal">es</code> in English).

  55.      This allows searches to find variant forms of the

  56.      same word, without tediously entering all the possible variants.

  57.      Also, this step typically eliminates <em class="firstterm">stop words</em>, which

  58.      are words that are so common that they are useless for searching.

  59.      (In short, then, tokens are raw fragments of the document text, while

  60.      lexemes are words that are believed useful for indexing and searching.)

  61.      <span class="productname">PostgreSQL</span> uses <em class="firstterm">dictionaries</em> to

  62.      perform this step.  Various standard dictionaries are provided, and

  63.      custom ones can be created for specific needs.

  64.     </p></li><li class="listitem" style="list-style-type: none"><p>

  65.      <span class="emphasis"><em>Storing preprocessed documents optimized for

  66.      searching</em></span>.  For example, each document can be represented

  67.      as a sorted array of normalized lexemes. Along with the lexemes it is

  68.      often desirable to store positional information to use for

  69.      <em class="firstterm">proximity ranking</em>, so that a document that

  70.      contains a more <span class="quote">“<span class="quote">dense</span>”</span> region of query words is

  71.      assigned a higher rank than one with scattered query words.

  72.     </p></li></ul></div><p>

  73.    Dictionaries allow fine-grained control over how tokens are normalized.

  74.    With appropriate dictionaries, you can:

  75.   </p><div class="itemizedlist"><ul class="itemizedlist compact" style="list-style-type: bullet; "><li class="listitem" style="list-style-type: disc"><p>

  76.      Define stop words that should not be indexed.

  77.     </p></li><li class="listitem" style="list-style-type: disc"><p>

  78.      Map synonyms to a single word using <span class="application">Ispell</span>.

  79.     </p></li><li class="listitem" style="list-style-type: disc"><p>

  80.      Map phrases to a single word using a thesaurus.

  81.     </p></li><li class="listitem" style="list-style-type: disc"><p>

  82.      Map different variations of a word to a canonical form using

  83.      an <span class="application">Ispell</span> dictionary.

  84.     </p></li><li class="listitem" style="list-style-type: disc"><p>

  85.      Map different variations of a word to a canonical form using

  86.      <span class="application">Snowball</span> stemmer rules.

  87.     </p></li></ul></div><p>

  88.    A data type <code class="type">tsvector</code> is provided for storing preprocessed

  89.    documents, along with a type <code class="type">tsquery</code> for representing processed

  90.    queries (<a class="xref" href="datatype-textsearch.html" title="8.11. Text Search Types">Section 8.11</a>).  There are many

  91.    functions and operators available for these data types

  92.    (<a class="xref" href="functions-textsearch.html" title="9.13. Text Search Functions and Operators">Section 9.13</a>), the most important of which is

  93.    the match operator <code class="literal">@@</code>, which we introduce in

  94.    <a class="xref" href="textsearch-intro.html#TEXTSEARCH-MATCHING" title="12.1.2. Basic Text Matching">Section 12.1.2</a>.  Full text searches can be accelerated

  95.    using indexes (<a class="xref" href="textsearch-indexes.html" title="12.9. GIN and GiST Index Types">Section 12.9</a>).

  96.   </p><div class="sect2" id="TEXTSEARCH-DOCUMENT"><div class="titlepage"><div><div><h3 class="title">12.1.1. What Is a Document?</h3></div></div></div><a id="id-1.5.11.4.10.2" class="indexterm"></a><p>

  97.     A <em class="firstterm">document</em> is the unit of searching in a full text search

  98.     system; for example, a magazine article or email message.  The text search

  99.     engine must be able to parse documents and store associations of lexemes

  100.     (key words) with their parent document. Later, these associations are

  101.     used to search for documents that contain query words.

  102.    </p><p>

  103.     For searches within <span class="productname">PostgreSQL</span>,

  104.     a document is normally a textual field within a row of a database table,

  105.     or possibly a combination (concatenation) of such fields, perhaps stored

  106.     in several tables or obtained dynamically. In other words, a document can

  107.     be constructed from different parts for indexing and it might not be

  108.     stored anywhere as a whole. For example:

  109. 

  110. </p><pre class="programlisting">

  111. SELECT title || ' ' ||  author || ' ' ||  abstract || ' ' || body AS document

  112. FROM messages

  113. WHERE mid = 12;

  114. 

  115. SELECT m.title || ' ' || m.author || ' ' || m.abstract || ' ' || d.body AS document

  116. FROM messages m, docs d

  117. WHERE m.mid = d.did AND m.mid = 12;

  118. </pre><p>

  119.    </p><div class="note"><h3 class="title">Note</h3><p>

  120.      Actually, in these example queries, <code class="function">coalesce</code>

  121.      should be used to prevent a single <code class="literal">NULL</code> attribute from

  122.      causing a <code class="literal">NULL</code> result for the whole document.

  123.     </p></div><p>

  124.     Another possibility is to store the documents as simple text files in the

  125.     file system. In this case, the database can be used to store the full text

  126.     index and to execute searches, and some unique identifier can be used to

  127.     retrieve the document from the file system.  However, retrieving files

  128.     from outside the database requires superuser permissions or special

  129.     function support, so this is usually less convenient than keeping all

  130.     the data inside <span class="productname">PostgreSQL</span>.  Also, keeping

  131.     everything inside the database allows easy access

  132.     to document metadata to assist in indexing and display.

  133.    </p><p>

  134.     For text search purposes, each document must be reduced to the

  135.     preprocessed <code class="type">tsvector</code> format.  Searching and ranking

  136.     are performed entirely on the <code class="type">tsvector</code> representation

  137.     of a document — the original text need only be retrieved

  138.     when the document has been selected for display to a user.

  139.     We therefore often speak of the <code class="type">tsvector</code> as being the

  140.     document, but of course it is only a compact representation of

  141.     the full document.

  142.    </p></div><div class="sect2" id="TEXTSEARCH-MATCHING"><div class="titlepage"><div><div><h3 class="title">12.1.2. Basic Text Matching</h3></div></div></div><p>

  143.     Full text searching in <span class="productname">PostgreSQL</span> is based on

  144.     the match operator <code class="literal">@@</code>, which returns

  145.     <code class="literal">true</code> if a <code class="type">tsvector</code>

  146.     (document) matches a <code class="type">tsquery</code> (query).

  147.     It doesn't matter which data type is written first:

  148. 

  149. </p><pre class="programlisting">

  150. SELECT 'a fat cat sat on a mat and ate a fat rat'::tsvector @@ 'cat &amp; rat'::tsquery;

  151.  ?column?

  152. ----------

  153.  t

  154. 

  155. SELECT 'fat &amp; cow'::tsquery @@ 'a fat cat sat on a mat and ate a fat rat'::tsvector;

  156.  ?column?

  157. ----------

  158.  f

  159. </pre><p>

  160.    </p><p>

  161.     As the above example suggests, a <code class="type">tsquery</code> is not just raw

  162.     text, any more than a <code class="type">tsvector</code> is.  A <code class="type">tsquery</code>

  163.     contains search terms, which must be already-normalized lexemes, and

  164.     may combine multiple terms using AND, OR, NOT, and FOLLOWED BY operators.

  165.     (For syntax details see <a class="xref" href="datatype-textsearch.html#DATATYPE-TSQUERY" title="8.11.2. tsquery">Section 8.11.2</a>.)  There are

  166.     functions <code class="function">to_tsquery</code>, <code class="function">plainto_tsquery</code>,

  167.     and <code class="function">phraseto_tsquery</code>

  168.     that are helpful in converting user-written text into a proper

  169.     <code class="type">tsquery</code>, primarily by normalizing words appearing in

  170.     the text.  Similarly, <code class="function">to_tsvector</code> is used to parse and

  171.     normalize a document string.  So in practice a text search match would

  172.     look more like this:

  173. 

  174. </p><pre class="programlisting">

  175. SELECT to_tsvector('fat cats ate fat rats') @@ to_tsquery('fat &amp; rat');

  176.  ?column? 

  177. ----------

  178.  t

  179. </pre><p>

  180. 

  181.     Observe that this match would not succeed if written as

  182. 

  183. </p><pre class="programlisting">

  184. SELECT 'fat cats ate fat rats'::tsvector @@ to_tsquery('fat &amp; rat');

  185.  ?column? 

  186. ----------

  187.  f

  188. </pre><p>

  189. 

  190.     since here no normalization of the word <code class="literal">rats</code> will occur.

  191.     The elements of a <code class="type">tsvector</code> are lexemes, which are assumed

  192.     already normalized, so <code class="literal">rats</code> does not match <code class="literal">rat</code>.

  193.    </p><p>

  194.     The <code class="literal">@@</code> operator also

  195.     supports <code class="type">text</code> input, allowing explicit conversion of a text

  196.     string to <code class="type">tsvector</code> or <code class="type">tsquery</code> to be skipped

  197.     in simple cases.  The variants available are:

  198. 

  199. </p><pre class="programlisting">

  200. tsvector @@ tsquery

  201. tsquery  @@ tsvector

  202. text @@ tsquery

  203. text @@ text

  204. </pre><p>

  205.    </p><p>

  206.     The first two of these we saw already.

  207.     The form <code class="type">text</code> <code class="literal">@@</code> <code class="type">tsquery</code>

  208.     is equivalent to <code class="literal">to_tsvector(x) @@ y</code>.

  209.     The form <code class="type">text</code> <code class="literal">@@</code> <code class="type">text</code>

  210.     is equivalent to <code class="literal">to_tsvector(x) @@ plainto_tsquery(y)</code>.

  211.    </p><p>

  212.     Within a <code class="type">tsquery</code>, the <code class="literal">&amp;</code> (AND) operator

  213.     specifies that both its arguments must appear in the document to have a

  214.     match.  Similarly, the <code class="literal">|</code> (OR) operator specifies that

  215.     at least one of its arguments must appear, while the <code class="literal">!</code> (NOT)

  216.     operator specifies that its argument must <span class="emphasis"><em>not</em></span> appear in

  217.     order to have a match.

  218.     For example, the query <code class="literal">fat &amp; ! rat</code> matches documents that

  219.     contain <code class="literal">fat</code> but not <code class="literal">rat</code>.

  220.    </p><p>

  221.     Searching for phrases is possible with the help of

  222.     the <code class="literal">&lt;-&gt;</code> (FOLLOWED BY) <code class="type">tsquery</code> operator, which

  223.     matches only if its arguments have matches that are adjacent and in the

  224.     given order.  For example:

  225. 

  226. </p><pre class="programlisting">

  227. SELECT to_tsvector('fatal error') @@ to_tsquery('fatal &lt;-&gt; error');

  228.  ?column? 

  229. ----------

  230.  t

  231. 

  232. SELECT to_tsvector('error is not fatal') @@ to_tsquery('fatal &lt;-&gt; error');

  233.  ?column? 

  234. ----------

  235.  f

  236. </pre><p>

  237. 

  238.     There is a more general version of the FOLLOWED BY operator having the

  239.     form <code class="literal">&lt;<em class="replaceable"><code>N</code></em>&gt;</code>,

  240.     where <em class="replaceable"><code>N</code></em> is an integer standing for the difference between

  241.     the positions of the matching lexemes.  <code class="literal">&lt;1&gt;</code> is

  242.     the same as <code class="literal">&lt;-&gt;</code>, while <code class="literal">&lt;2&gt;</code>

  243.     allows exactly one other lexeme to appear between the matches, and so

  244.     on.  The <code class="literal">phraseto_tsquery</code> function makes use of this

  245.     operator to construct a <code class="literal">tsquery</code> that can match a multi-word

  246.     phrase when some of the words are stop words.  For example:

  247. 

  248. </p><pre class="programlisting">

  249. SELECT phraseto_tsquery('cats ate rats');

  250.        phraseto_tsquery        

  251. -------------------------------

  252.  'cat' &lt;-&gt; 'ate' &lt;-&gt; 'rat'

  253. 

  254. SELECT phraseto_tsquery('the cats ate the rats');

  255.        phraseto_tsquery        

  256. -------------------------------

  257.  'cat' &lt;-&gt; 'ate' &lt;2&gt; 'rat'

  258. </pre><p>

  259.    </p><p>

  260.     A special case that's sometimes useful is that <code class="literal">&lt;0&gt;</code>

  261.     can be used to require that two patterns match the same word.

  262.    </p><p>

  263.     Parentheses can be used to control nesting of the <code class="type">tsquery</code>

  264.     operators.  Without parentheses, <code class="literal">|</code> binds least tightly,

  265.     then <code class="literal">&amp;</code>, then <code class="literal">&lt;-&gt;</code>,

  266.     and <code class="literal">!</code> most tightly.

  267.    </p><p>

  268.     It's worth noticing that the AND/OR/NOT operators mean something subtly

  269.     different when they are within the arguments of a FOLLOWED BY operator

  270.     than when they are not, because within FOLLOWED BY the exact position of

  271.     the match is significant.  For example, normally <code class="literal">!x</code> matches

  272.     only documents that do not contain <code class="literal">x</code> anywhere.

  273.     But <code class="literal">!x &lt;-&gt; y</code> matches <code class="literal">y</code> if it is not

  274.     immediately after an <code class="literal">x</code>; an occurrence of <code class="literal">x</code>

  275.     elsewhere in the document does not prevent a match.  Another example is

  276.     that <code class="literal">x &amp; y</code> normally only requires that <code class="literal">x</code>

  277.     and <code class="literal">y</code> both appear somewhere in the document, but

  278.     <code class="literal">(x &amp; y) &lt;-&gt; z</code> requires <code class="literal">x</code>

  279.     and <code class="literal">y</code> to match at the same place, immediately before

  280.     a <code class="literal">z</code>.  Thus this query behaves differently from

  281.     <code class="literal">x &lt;-&gt; z &amp; y &lt;-&gt; z</code>, which will match a

  282.     document containing two separate sequences <code class="literal">x z</code> and

  283.     <code class="literal">y z</code>.  (This specific query is useless as written,

  284.     since <code class="literal">x</code> and <code class="literal">y</code> could not match at the same place;

  285.     but with more complex situations such as prefix-match patterns, a query

  286.     of this form could be useful.)

  287.    </p></div><div class="sect2" id="TEXTSEARCH-INTRO-CONFIGURATIONS"><div class="titlepage"><div><div><h3 class="title">12.1.3. Configurations</h3></div></div></div><p>

  288.     The above are all simple text search examples.  As mentioned before, full

  289.     text search functionality includes the ability to do many more things:

  290.     skip indexing certain words (stop words), process synonyms, and use

  291.     sophisticated parsing, e.g., parse based on more than just white space.

  292.     This functionality is controlled by <em class="firstterm">text search

  293.     configurations</em>.  <span class="productname">PostgreSQL</span> comes with predefined

  294.     configurations for many languages, and you can easily create your own

  295.     configurations.  (<span class="application">psql</span>'s <code class="command">\dF</code> command

  296.     shows all available configurations.)

  297.    </p><p>

  298.     During installation an appropriate configuration is selected and

  299.     <a class="xref" href="runtime-config-client.html#GUC-DEFAULT-TEXT-SEARCH-CONFIG">default_text_search_config</a> is set accordingly

  300.     in <code class="filename">postgresql.conf</code>.  If you are using the same text search

  301.     configuration for the entire cluster you can use the value in

  302.     <code class="filename">postgresql.conf</code>.  To use different configurations

  303.     throughout the cluster but the same configuration within any one database,

  304.     use <code class="command">ALTER DATABASE ... SET</code>.  Otherwise, you can set

  305.     <code class="varname">default_text_search_config</code> in each session.

  306.    </p><p>

  307.     Each text search function that depends on a configuration has an optional

  308.     <code class="type">regconfig</code> argument, so that the configuration to use can be

  309.     specified explicitly.  <code class="varname">default_text_search_config</code>

  310.     is used only when this argument is omitted.

  311.    </p><p>

  312.     To make it easier to build custom text search configurations, a

  313.     configuration is built up from simpler database objects.

  314.     <span class="productname">PostgreSQL</span>'s text search facility provides

  315.     four types of configuration-related database objects:

  316.    </p><div class="itemizedlist"><ul class="itemizedlist compact" style="list-style-type: bullet; "><li class="listitem" style="list-style-type: disc"><p>

  317.      <em class="firstterm">Text search parsers</em> break documents into tokens

  318.      and classify each token (for example, as words or numbers).

  319.     </p></li><li class="listitem" style="list-style-type: disc"><p>

  320.      <em class="firstterm">Text search dictionaries</em> convert tokens to normalized

  321.      form and reject stop words.

  322.     </p></li><li class="listitem" style="list-style-type: disc"><p>

  323.      <em class="firstterm">Text search templates</em> provide the functions underlying

  324.      dictionaries.  (A dictionary simply specifies a template and a set

  325.      of parameters for the template.)

  326.     </p></li><li class="listitem" style="list-style-type: disc"><p>

  327.      <em class="firstterm">Text search configurations</em> select a parser and a set

  328.      of dictionaries to use to normalize the tokens produced by the parser.

  329.     </p></li></ul></div><p>

  330.     Text search parsers and templates are built from low-level C functions;

  331.     therefore it requires C programming ability to develop new ones, and

  332.     superuser privileges to install one into a database.  (There are examples

  333.     of add-on parsers and templates in the <code class="filename">contrib/</code> area of the

  334.     <span class="productname">PostgreSQL</span> distribution.)  Since dictionaries and

  335.     configurations just parameterize and connect together some underlying

  336.     parsers and templates, no special privilege is needed to create a new

  337.     dictionary or configuration.  Examples of creating custom dictionaries and

  338.     configurations appear later in this chapter.

  339.    </p></div></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="textsearch.html">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="textsearch.html">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="textsearch-tables.html">Next</a></td></tr><tr><td width="40%" align="left" valign="top">Chapter 12. Full Text Search </td><td width="20%" align="center"><a accesskey="h" href="index.html">Home</a></td><td width="40%" align="right" valign="top"> 12.2. Tables and Indexes</td></tr></table></div></body></html>
上海开阖软件有限公司 沪ICP备12045867号-1