diff options
Diffstat (limited to 'src/backend/optimizer')
47 files changed, 6587 insertions, 3304 deletions
diff --git a/src/backend/optimizer/README b/src/backend/optimizer/README index 1aa0dbcdbb..88cdbc3294 100644 --- a/src/backend/optimizer/README +++ b/src/backend/optimizer/README @@ -297,9 +297,6 @@ planner() handle UNION/INTERSECT/EXCEPT, GROUP BY, HAVING, aggregates, ORDER BY, DISTINCT, LIMIT --query_planner() - pull out constant quals, which can be used to gate execution of the - whole plan (if any are found, we make a top-level Result node - to do the gating) make list of base relations used in query split up the qual into restrictions (a=1) and joins (b=c) find qual clauses that enable merge and hash joins @@ -318,11 +315,11 @@ planner() Back at standard_join_search(), apply set_cheapest() to extract the cheapest path for each newly constructed joinrel. Loop back if this wasn't the top join level. - Back at query_planner: - put back any constant quals by adding a Result node Back at grouping_planner: + convert Path tree returned by query_planner into a Plan tree do grouping(GROUP) do aggregates + do window functions make unique(DISTINCT) make sort(ORDER BY) make limit(LIMIT/OFFSET) @@ -372,10 +369,10 @@ generated during the optimization process are marked with their sort order It is also possible to avoid an explicit sort step to implement a user's ORDER BY clause if the final path has the right ordering already, so the -sort ordering is of interest even at the top level. query_planner() will +sort ordering is of interest even at the top level. grouping_planner() will look for the cheapest path with a sort order matching the desired order, -and grouping_planner() will compare its cost to the cost of using the -cheapest-overall path and doing an explicit sort. +then compare its cost to the cost of using the cheapest-overall path and +doing an explicit sort on that. When we are generating paths for a particular RelOptInfo, we discard a path if it is more expensive than another known path that has the same or better @@ -589,16 +586,6 @@ since they are easily compared to the pathkeys of a potential candidate path. So, SortGroupClause lists are turned into pathkeys lists for use inside the optimizer. -Because we have to generate pathkeys lists from the sort clauses before -we've finished EquivalenceClass merging, we cannot use the pointer-equality -method of comparing PathKeys in the earliest stages of the planning -process. Instead, we generate "non canonical" PathKeys that reference -single-element EquivalenceClasses that might get merged later. After we -complete EquivalenceClass merging, we replace these with "canonical" -PathKeys that reference only fully-merged classes, and after that we make -sure we don't generate more than one copy of each "canonical" PathKey. -Then it is safe to use pointer comparison on canonical PathKeys. - An additional refinement we can make is to insist that canonical pathkey lists (sort orderings) do not mention the same EquivalenceClass more than once. For example, in all these cases the second sort column is redundant, @@ -651,12 +638,11 @@ mergejoinable clauses found in the quals. At the end of this process, we know all we can know about equivalence of different variables, so subsequently there will be no further merging of EquivalenceClasses. At that point it is possible to consider the EquivalenceClasses as -"canonical" and build canonical PathKeys that reference them. Before -we reach that point (actually, before entering query_planner at all) -we also ensure that we have constructed EquivalenceClasses for all the -expressions used in the query's ORDER BY and related clauses. These -classes might or might not get merged together, depending on what we -find in the quals. +"canonical" and build canonical PathKeys that reference them. At this +time we construct PathKeys for the query's ORDER BY and related clauses. +(Any ordering expressions that do not appear elsewhere will result in +the creation of new EquivalenceClasses, but this cannot result in merging +existing classes, so canonical-ness is not lost.) Because all the EquivalenceClasses are known before we begin path generation, we can use them as a guide to which indexes are of interest: @@ -789,7 +775,7 @@ a nestloop that provides parameters to the lower join's inputs). While we do not ignore merge joins entirely, joinpath.c does not fully explore the space of potential merge joins with parameterized inputs. Also, add_path treats parameterized paths as having no pathkeys, so that they compete -only on cost and rowcount; they don't get preference for producing a +only on total cost and rowcount; they don't get preference for producing a special sort order. This creates additional bias against merge joins, since we might discard a path that could have been useful for performing a merge without an explicit sort step. Since a parameterized path must @@ -799,4 +785,32 @@ output order of a query --- they only make it harder to use a merge join at a lower level. The savings in planning work justifies that. +LATERAL subqueries +------------------ + +As of 9.3 we support SQL-standard LATERAL references from subqueries in +FROM (and also functions in FROM). The planner implements these by +generating parameterized paths for any RTE that contains lateral +references. In such cases, *all* paths for that relation will be +parameterized by at least the set of relations used in its lateral +references. (And in turn, join relations including such a subquery might +not have any unparameterized paths.) All the other comments made above for +parameterized paths still apply, though; in particular, each such path is +still expected to enforce any join clauses that can be pushed down to it, +so that all paths of the same parameterization have the same rowcount. + +We also allow LATERAL subqueries to be flattened (pulled up into the parent +query) by the optimizer, but only when this does not introduce lateral +references into JOIN/ON quals that would refer to relations outside the +lowest outer join at/above that qual. The semantics of such a qual would +be unclear. Note that even with this restriction, pullup of a LATERAL +subquery can result in creating PlaceHolderVars that contain lateral +references to relations outside their syntactic scope. We still evaluate +such PHVs at their syntactic location or lower, but the presence of such a +PHV in the quals or targetlist of a plan node requires that node to appear +on the inside of a nestloop join relative to the rel(s) supplying the +lateral reference. (Perhaps now that that stuff works, we could relax the +pullup restriction?) + + -- bjm & tgl diff --git a/src/backend/optimizer/geqo/geqo_copy.c b/src/backend/optimizer/geqo/geqo_copy.c index bed7aa046f..1b9d8d7382 100644 --- a/src/backend/optimizer/geqo/geqo_copy.c +++ b/src/backend/optimizer/geqo/geqo_copy.c @@ -2,7 +2,7 @@ * * geqo_copy.c * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * src/backend/optimizer/geqo/geqo_copy.c diff --git a/src/backend/optimizer/geqo/geqo_cx.c b/src/backend/optimizer/geqo/geqo_cx.c index afae948a61..9f6d5e478a 100644 --- a/src/backend/optimizer/geqo/geqo_cx.c +++ b/src/backend/optimizer/geqo/geqo_cx.c @@ -47,7 +47,6 @@ int cx(PlannerInfo *root, Gene *tour1, Gene *tour2, Gene *offspring, int num_gene, City *city_table) { - int i, start_pos, curr_pos; diff --git a/src/backend/optimizer/geqo/geqo_eval.c b/src/backend/optimizer/geqo/geqo_eval.c index 1e352fd3b5..de2a6709dd 100644 --- a/src/backend/optimizer/geqo/geqo_eval.c +++ b/src/backend/optimizer/geqo/geqo_eval.c @@ -3,7 +3,7 @@ * geqo_eval.c * Routines to evaluate query trees * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * src/backend/optimizer/geqo/geqo_eval.c @@ -56,6 +56,7 @@ geqo_eval(PlannerInfo *root, Gene *tour, int num_gene) MemoryContext mycontext; MemoryContext oldcxt; RelOptInfo *joinrel; + Path *best_path; Cost fitness; int savelength; struct HTAB *savehash; @@ -81,11 +82,11 @@ geqo_eval(PlannerInfo *root, Gene *tour, int num_gene) * not already contain some entries. The newly added entries will be * recycled by the MemoryContextDelete below, so we must ensure that the * list is restored to its former state before exiting. We can do this by - * truncating the list to its original length. NOTE this assumes that any + * truncating the list to its original length. NOTE this assumes that any * added entries are appended at the end! * * We also must take care not to mess up the outer join_rel_hash, if there - * is one. We can do this by just temporarily setting the link to NULL. + * is one. We can do this by just temporarily setting the link to NULL. * (If we are dealing with enough join rels, which we very likely are, a * new hash table will get built and used locally.) * @@ -99,14 +100,16 @@ geqo_eval(PlannerInfo *root, Gene *tour, int num_gene) /* construct the best path for the given combination of relations */ joinrel = gimme_tree(root, tour, num_gene); + best_path = joinrel->cheapest_total_path; /* * compute fitness * * XXX geqo does not currently support optimization for partial result - * retrieval --- how to fix? + * retrieval, nor do we take any cognizance of possible use of + * parameterized paths --- how to fix? */ - fitness = joinrel->cheapest_total_path->total_cost; + fitness = best_path->total_cost; /* * Restore join_rel_list to its former state, and put back original @@ -214,7 +217,7 @@ gimme_tree(PlannerInfo *root, Gene *tour, int num_gene) * Merge a "clump" into the list of existing clumps for gimme_tree. * * We try to merge the clump into some existing clump, and repeat if - * successful. When no more merging is possible, insert the clump + * successful. When no more merging is possible, insert the clump * into the list, preserving the list ordering rule (namely, that * clumps of larger size appear earlier). * @@ -265,7 +268,7 @@ merge_clump(PlannerInfo *root, List *clumps, Clump *new_clump, bool force) /* * Recursively try to merge the enlarged old_clump with - * others. When no further merge is possible, we'll reinsert + * others. When no further merge is possible, we'll reinsert * it into the list. */ return merge_clump(root, clumps, old_clump, force); @@ -276,7 +279,7 @@ merge_clump(PlannerInfo *root, List *clumps, Clump *new_clump, bool force) /* * No merging is possible, so add new_clump as an independent clump, in - * proper order according to size. We can be fast for the common case + * proper order according to size. We can be fast for the common case * where it has size 1 --- it should always go at the end. */ if (clumps == NIL || new_clump->size == 1) diff --git a/src/backend/optimizer/geqo/geqo_main.c b/src/backend/optimizer/geqo/geqo_main.c index 3dda183e41..579f94042a 100644 --- a/src/backend/optimizer/geqo/geqo_main.c +++ b/src/backend/optimizer/geqo/geqo_main.c @@ -4,7 +4,7 @@ * solution to the query optimization problem * by means of a Genetic Algorithm (GA) * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * src/backend/optimizer/geqo/geqo_main.c diff --git a/src/backend/optimizer/geqo/geqo_misc.c b/src/backend/optimizer/geqo/geqo_misc.c index 2666c83305..8c546732de 100644 --- a/src/backend/optimizer/geqo/geqo_misc.c +++ b/src/backend/optimizer/geqo/geqo_misc.c @@ -3,7 +3,7 @@ * geqo_misc.c * misc. printout and debug stuff * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * src/backend/optimizer/geqo/geqo_misc.c diff --git a/src/backend/optimizer/geqo/geqo_pool.c b/src/backend/optimizer/geqo/geqo_pool.c index 0a97328059..f60f3df21e 100644 --- a/src/backend/optimizer/geqo/geqo_pool.c +++ b/src/backend/optimizer/geqo/geqo_pool.c @@ -3,7 +3,7 @@ * geqo_pool.c * Genetic Algorithm (GA) pool stuff * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * src/backend/optimizer/geqo/geqo_pool.c diff --git a/src/backend/optimizer/geqo/geqo_px.c b/src/backend/optimizer/geqo/geqo_px.c index 808ff6a14c..99289bc11f 100644 --- a/src/backend/optimizer/geqo/geqo_px.c +++ b/src/backend/optimizer/geqo/geqo_px.c @@ -46,7 +46,6 @@ void px(PlannerInfo *root, Gene *tour1, Gene *tour2, Gene *offspring, int num_gene, City *city_table) { - int num_positions; int i, pos, diff --git a/src/backend/optimizer/geqo/geqo_random.c b/src/backend/optimizer/geqo/geqo_random.c index 7743715eaf..13b16cea28 100644 --- a/src/backend/optimizer/geqo/geqo_random.c +++ b/src/backend/optimizer/geqo/geqo_random.c @@ -3,7 +3,7 @@ * geqo_random.c * random number generator * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * src/backend/optimizer/geqo/geqo_random.c diff --git a/src/backend/optimizer/geqo/geqo_selection.c b/src/backend/optimizer/geqo/geqo_selection.c index fbdcc5ff0c..3d4d948bf3 100644 --- a/src/backend/optimizer/geqo/geqo_selection.c +++ b/src/backend/optimizer/geqo/geqo_selection.c @@ -3,7 +3,7 @@ * geqo_selection.c * linear selection scheme for the genetic query optimizer * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * src/backend/optimizer/geqo/geqo_selection.c diff --git a/src/backend/optimizer/path/Makefile b/src/backend/optimizer/path/Makefile index 07938dbe57..6864a62132 100644 --- a/src/backend/optimizer/path/Makefile +++ b/src/backend/optimizer/path/Makefile @@ -13,6 +13,6 @@ top_builddir = ../../../.. include $(top_builddir)/src/Makefile.global OBJS = allpaths.o clausesel.o costsize.o equivclass.o indxpath.o \ - joinpath.o joinrels.o orindxpath.o pathkeys.o tidpath.o + joinpath.o joinrels.o pathkeys.o tidpath.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 525a659007..99b6147465 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -8,7 +8,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * Portions Copyright (c) 2012-2014, TransLattice, Inc. - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -24,6 +24,7 @@ #include "catalog/pg_namespace.h" #include "catalog/pg_class.h" +#include "catalog/pg_operator.h" #include "foreign/fdwapi.h" #include "nodes/nodeFuncs.h" #ifdef OPTIMIZER_DEBUG @@ -84,6 +85,9 @@ static void set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, static void generate_mergeappend_paths(PlannerInfo *root, RelOptInfo *rel, List *live_childrels, List *all_child_pathkeys); +static Path *get_cheapest_parameterized_child_path(PlannerInfo *root, + RelOptInfo *rel, + Relids required_outer); static List *accumulate_append_subpath(List *subpaths, Path *path); static void set_dummy_rel_pathlist(RelOptInfo *rel); static void set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, @@ -98,13 +102,14 @@ static void set_worktable_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte); static RelOptInfo *make_rel_from_joinlist(PlannerInfo *root, List *joinlist); static bool subquery_is_pushdown_safe(Query *subquery, Query *topquery, - bool *differentTypes); + bool *unsafeColumns); static bool recurse_pushdown_safe(Node *setOp, Query *topquery, - bool *differentTypes); + bool *unsafeColumns); +static void check_output_expressions(Query *subquery, bool *unsafeColumns); static void compare_tlist_datatypes(List *tlist, List *colTypes, - bool *differentTypes); + bool *unsafeColumns); static bool qual_is_pushdown_safe(Query *subquery, Index rti, Node *qual, - bool *differentTypes); + bool *unsafeColumns); static void subquery_push_qual(Query *subquery, RangeTblEntry *rte, Index rti, Node *qual); static void recurse_push_qual(Node *setOp, Query *topquery, @@ -269,8 +274,9 @@ set_rel_size(PlannerInfo *root, RelOptInfo *rel, case RTE_SUBQUERY: /* - * Subqueries don't support parameterized paths, so just go - * ahead and build their paths immediately. + * Subqueries don't support making a choice between + * parameterized and unparameterized paths, so just go ahead + * and build their paths immediately. */ set_subquery_pathlist(root, rel, rti, rte); break; @@ -283,8 +289,9 @@ set_rel_size(PlannerInfo *root, RelOptInfo *rel, case RTE_CTE: /* - * CTEs don't support parameterized paths, so just go ahead - * and build their paths immediately. + * CTEs don't support making a choice between parameterized + * and unparameterized paths, so just go ahead and build their + * paths immediately. */ if (rte->self_reference) set_worktable_pathlist(root, rel, rte); @@ -371,17 +378,6 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) /* Mark rel with estimated output rows, width, etc */ set_baserel_size_estimates(root, rel); - - /* - * Check to see if we can extract any restriction conditions from join - * quals that are OR-of-AND structures. If so, add them to the rel's - * restriction list, and redo the above steps. - */ - if (create_or_index_quals(root, rel)) - { - check_partial_indexes(root, rel); - set_baserel_size_estimates(root, rel); - } } @@ -392,6 +388,8 @@ set_plain_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) static void set_plain_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) { + Relids required_outer; + #ifdef PGXC #ifndef XCP /* @@ -410,8 +408,15 @@ set_plain_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) #endif /* XCP */ #endif /* PGXC */ + /* + * We don't support pushing join clauses into the quals of a seqscan, but + * it could still have required parameterization due to LATERAL refs in + * its tlist. + */ + required_outer = rel->lateral_relids; + /* Consider sequential scan */ - add_path(rel, create_seqscan_path(root, rel, NULL)); + add_path(rel, create_seqscan_path(root, rel, required_outer)); /* Consider index scans */ create_index_paths(root, rel); @@ -438,9 +443,6 @@ set_foreign_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) /* Mark rel with estimated output rows, width, etc */ set_foreign_size_estimates(root, rel); - /* Get FDW routine pointers for the rel */ - rel->fdwroutine = GetFdwRoutineByRelId(rte->relid); - /* Let FDW adjust the size estimates, if it can */ rel->fdwroutine->GetForeignRelSize(root, rel, rte->relid); } @@ -463,7 +465,7 @@ set_foreign_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) * set_append_rel_size * Set size estimates for an "append relation" * - * The passed-in rel and RTE represent the entire append relation. The + * The passed-in rel and RTE represent the entire append relation. The * relation's contents are computed by appending together the output of * the individual member relations. Note that in the inheritance case, * the first member relation is actually the same table as is mentioned in @@ -527,7 +529,7 @@ set_append_rel_size(PlannerInfo *root, RelOptInfo *rel, /* * We have to copy the parent's targetlist and quals to the child, - * with appropriate substitution of variables. However, only the + * with appropriate substitution of variables. However, only the * baserestrictinfo quals are needed before we can check for * constraint exclusion; so do that first and then check to see if we * can disregard this child. @@ -575,10 +577,10 @@ set_append_rel_size(PlannerInfo *root, RelOptInfo *rel, * CE failed, so finish copying/modifying targetlist and join quals. * * Note: the resulting childrel->reltargetlist may contain arbitrary - * expressions, which normally would not occur in a reltargetlist. - * That is okay because nothing outside of this routine will look at - * the child rel's reltargetlist. We do have to cope with the case - * while constructing attr_widths estimates below, though. + * expressions, which otherwise would not occur in a reltargetlist. + * Code that might be looking at an appendrel child must cope with + * such. (Normally, a reltargetlist would only include Vars and + * PlaceHolderVars.) */ childrel->joininfo = (List *) adjust_appendrel_attrs(root, @@ -591,7 +593,7 @@ set_append_rel_size(PlannerInfo *root, RelOptInfo *rel, /* * We have to make child entries in the EquivalenceClass data - * structures as well. This is needed either if the parent + * structures as well. This is needed either if the parent * participates in some eclass joins (because we will want to consider * inner-indexscan joins on the individual children) or if the parent * has useful pathkeys (because we should try to build MergeAppend @@ -632,7 +634,7 @@ set_append_rel_size(PlannerInfo *root, RelOptInfo *rel, /* * Accumulate per-column estimates too. We need not do anything - * for PlaceHolderVars in the parent list. If child expression + * for PlaceHolderVars in the parent list. If child expression * isn't a Var, or we didn't record a width estimate for it, we * have to fall back on a datatype-based estimate. * @@ -649,7 +651,8 @@ set_append_rel_size(PlannerInfo *root, RelOptInfo *rel, int pndx = parentvar->varattno - rel->min_attr; int32 child_width = 0; - if (IsA(childvar, Var)) + if (IsA(childvar, Var) && + ((Var *) childvar)->varno == childrel->relid) { int cndx = ((Var *) childvar)->varattno - childrel->min_attr; @@ -700,13 +703,14 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, int parentRTindex = rti; List *live_childrels = NIL; List *subpaths = NIL; + bool subpaths_valid = true; List *all_child_pathkeys = NIL; List *all_child_outers = NIL; ListCell *l; /* * Generate access paths for each member relation, and remember the - * cheapest path for each one. Also, identify all pathkeys (orderings) + * cheapest path for each one. Also, identify all pathkeys (orderings) * and parameterizations (required_outer sets) available for the member * relations. */ @@ -739,18 +743,24 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, continue; /* - * Child is live, so add its cheapest access path to the Append path - * we are constructing for the parent. + * Child is live, so add it to the live_childrels list for use below. */ - subpaths = accumulate_append_subpath(subpaths, - childrel->cheapest_total_path); - - /* Remember which childrels are live, for logic below */ live_childrels = lappend(live_childrels, childrel); /* + * If child has an unparameterized cheapest-total path, add that to + * the unparameterized Append path we are constructing for the parent. + * If not, there's no workable unparameterized path. + */ + if (childrel->cheapest_total_path->param_info == NULL) + subpaths = accumulate_append_subpath(subpaths, + childrel->cheapest_total_path); + else + subpaths_valid = false; + + /* * Collect lists of all the available path orderings and - * parameterizations for all the children. We use these as a + * parameterizations for all the children. We use these as a * heuristic to indicate which sort orderings and parameterizations we * should build Append and MergeAppend paths for. */ @@ -814,17 +824,20 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, } /* - * Next, build an unordered, unparameterized Append path for the rel. - * (Note: this is correct even if we have zero or one live subpath due to - * constraint exclusion.) + * If we found unparameterized paths for all children, build an unordered, + * unparameterized Append path for the rel. (Note: this is correct even + * if we have zero or one live subpath due to constraint exclusion.) */ - add_path(rel, (Path *) create_append_path(rel, subpaths, NULL)); + if (subpaths_valid) + add_path(rel, (Path *) create_append_path(rel, subpaths, NULL)); /* - * Build unparameterized MergeAppend paths based on the collected list of - * child pathkeys. + * Also build unparameterized MergeAppend paths based on the collected + * list of child pathkeys. */ - generate_mergeappend_paths(root, rel, live_childrels, all_child_pathkeys); + if (subpaths_valid) + generate_mergeappend_paths(root, rel, live_childrels, + all_child_pathkeys); /* * Build Append paths for each parameterization seen among the child rels. @@ -833,7 +846,7 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, * so that not that many cases actually get considered here.) * * The Append node itself cannot enforce quals, so all qual checking must - * be done in the child paths. This means that to have a parameterized + * be done in the child paths. This means that to have a parameterized * Append path, we must have the exact same parameterization for each * child path; otherwise some children might be failing to check the * moved-down quals. To make them match up, we can try to increase the @@ -842,39 +855,29 @@ set_append_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, foreach(l, all_child_outers) { Relids required_outer = (Relids) lfirst(l); - bool ok = true; ListCell *lcr; /* Select the child paths for an Append with this parameterization */ subpaths = NIL; + subpaths_valid = true; foreach(lcr, live_childrels) { RelOptInfo *childrel = (RelOptInfo *) lfirst(lcr); - Path *cheapest_total; - - cheapest_total = - get_cheapest_path_for_pathkeys(childrel->pathlist, - NIL, - required_outer, - TOTAL_COST); - Assert(cheapest_total != NULL); + Path *subpath; - /* Children must have exactly the desired parameterization */ - if (!bms_equal(PATH_REQ_OUTER(cheapest_total), required_outer)) + subpath = get_cheapest_parameterized_child_path(root, + childrel, + required_outer); + if (subpath == NULL) { - cheapest_total = reparameterize_path(root, cheapest_total, - required_outer, 1.0); - if (cheapest_total == NULL) - { - ok = false; - break; - } + /* failed to make a suitable path for this child */ + subpaths_valid = false; + break; } - - subpaths = accumulate_append_subpath(subpaths, cheapest_total); + subpaths = accumulate_append_subpath(subpaths, subpath); } - if (ok) + if (subpaths_valid) add_path(rel, (Path *) create_append_path(rel, subpaths, required_outer)); } @@ -948,7 +951,8 @@ generate_mergeappend_paths(PlannerInfo *root, RelOptInfo *rel, { cheapest_startup = cheapest_total = childrel->cheapest_total_path; - Assert(cheapest_total != NULL); + /* Assert we do have an unparameterized path for this child */ + Assert(cheapest_total->param_info == NULL); } /* @@ -981,13 +985,91 @@ generate_mergeappend_paths(PlannerInfo *root, RelOptInfo *rel, } /* + * get_cheapest_parameterized_child_path + * Get cheapest path for this relation that has exactly the requested + * parameterization. + * + * Returns NULL if unable to create such a path. + */ +static Path * +get_cheapest_parameterized_child_path(PlannerInfo *root, RelOptInfo *rel, + Relids required_outer) +{ + Path *cheapest; + ListCell *lc; + + /* + * Look up the cheapest existing path with no more than the needed + * parameterization. If it has exactly the needed parameterization, we're + * done. + */ + cheapest = get_cheapest_path_for_pathkeys(rel->pathlist, + NIL, + required_outer, + TOTAL_COST); + Assert(cheapest != NULL); + if (bms_equal(PATH_REQ_OUTER(cheapest), required_outer)) + return cheapest; + + /* + * Otherwise, we can "reparameterize" an existing path to match the given + * parameterization, which effectively means pushing down additional + * joinquals to be checked within the path's scan. However, some existing + * paths might check the available joinquals already while others don't; + * therefore, it's not clear which existing path will be cheapest after + * reparameterization. We have to go through them all and find out. + */ + cheapest = NULL; + foreach(lc, rel->pathlist) + { + Path *path = (Path *) lfirst(lc); + + /* Can't use it if it needs more than requested parameterization */ + if (!bms_is_subset(PATH_REQ_OUTER(path), required_outer)) + continue; + + /* + * Reparameterization can only increase the path's cost, so if it's + * already more expensive than the current cheapest, forget it. + */ + if (cheapest != NULL && + compare_path_costs(cheapest, path, TOTAL_COST) <= 0) + continue; + + /* Reparameterize if needed, then recheck cost */ + if (!bms_equal(PATH_REQ_OUTER(path), required_outer)) + { + path = reparameterize_path(root, path, required_outer, 1.0); + if (path == NULL) + continue; /* failed to reparameterize this one */ + Assert(bms_equal(PATH_REQ_OUTER(path), required_outer)); + + if (cheapest != NULL && + compare_path_costs(cheapest, path, TOTAL_COST) <= 0) + continue; + } + + /* We have a new best path */ + cheapest = path; + } + + /* Return the best path, or NULL if we found no suitable candidate */ + return cheapest; +} + +/* * accumulate_append_subpath * Add a subpath to the list being built for an Append or MergeAppend * - * It's possible that the child is itself an Append path, in which case - * we can "cut out the middleman" and just add its child paths to our - * own list. (We don't try to do this earlier because we need to - * apply both levels of transformation to the quals.) + * It's possible that the child is itself an Append or MergeAppend path, in + * which case we can "cut out the middleman" and just add its child paths to + * our own list. (We don't try to do this earlier because we need to apply + * both levels of transformation to the quals.) + * + * Note that if we omit a child MergeAppend in this way, we are effectively + * omitting a sort step, which seems fine: if the parent is to be an Append, + * its result would be unsorted anyway, while if the parent is to be a + * MergeAppend, there's no point in a separate sort on a child. */ static List * accumulate_append_subpath(List *subpaths, Path *path) @@ -999,6 +1081,13 @@ accumulate_append_subpath(List *subpaths, Path *path) /* list_copy is important here to avoid sharing list substructure */ return list_concat(subpaths, list_copy(apath->subpaths)); } + else if (IsA(path, MergeAppendPath)) + { + MergeAppendPath *mpath = (MergeAppendPath *) path; + + /* list_copy is important here to avoid sharing list substructure */ + return list_concat(subpaths, list_copy(mpath->subpaths)); + } else return lappend(subpaths, path); } @@ -1052,8 +1141,13 @@ has_multiple_baserels(PlannerInfo *root) * set_subquery_pathlist * Build the (single) access path for a subquery RTE * - * There's no need for a separate set_subquery_size phase, since we don't - * support parameterized paths for subqueries. + * We don't currently support generating parameterized paths for subqueries + * by pushing join clauses down into them; it seems too expensive to re-plan + * the subquery multiple times to consider different alternatives. So the + * subquery will have exactly one path. (The path will be parameterized + * if the subquery contains LATERAL references, otherwise not.) Since there's + * no freedom of action here, there's no need for a separate set_subquery_size + * phase: we just make the path right away. */ static void set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, @@ -1061,7 +1155,8 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, { Query *parse = root->parse; Query *subquery = rte->subquery; - bool *differentTypes; + Relids required_outer; + bool *unsafeColumns; double tuple_fraction; PlannerInfo *subroot; List *pathkeys; @@ -1076,8 +1171,19 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, */ subquery = copyObject(subquery); - /* We need a workspace for keeping track of set-op type coercions */ - differentTypes = (bool *) + /* + * If it's a LATERAL subquery, it might contain some Vars of the current + * query level, requiring it to be treated as parameterized, even though + * we don't support pushing down join quals into subqueries. + */ + required_outer = rel->lateral_relids; + + /* + * We need a workspace for keeping track of unsafe-to-reference columns. + * unsafeColumns[i] is set TRUE if we've found that output column i of the + * subquery is unsafe to use in a pushed-down qual. + */ + unsafeColumns = (bool *) palloc0((list_length(subquery->targetList) + 1) * sizeof(bool)); /* @@ -1094,10 +1200,9 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, * pseudoconstant clauses; better to have the gating node above the * subquery. * - * Also, if the sub-query has "security_barrier" flag, it means the + * Also, if the sub-query has the "security_barrier" flag, it means the * sub-query originated from a view that must enforce row-level security. - * We must not push down quals in order to avoid information leaks, either - * via side-effects or error output. + * Then we must not push down quals that contain leaky functions. * * Non-pushed-down clauses will get evaluated as qpquals of the * SubqueryScan node. @@ -1106,7 +1211,7 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, * push down a pushable qual, because it'd result in a worse plan? */ if (rel->baserestrictinfo != NIL && - subquery_is_pushdown_safe(subquery, subquery, differentTypes)) + subquery_is_pushdown_safe(subquery, subquery, unsafeColumns)) { /* OK to consider pushing down individual quals */ List *upperrestrictlist = NIL; @@ -1120,7 +1225,7 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, if (!rinfo->pseudoconstant && (!rte->security_barrier || !contain_leaky_functions(clause)) && - qual_is_pushdown_safe(subquery, rti, clause, differentTypes)) + qual_is_pushdown_safe(subquery, rti, clause, unsafeColumns)) { /* Push it down */ subquery_push_qual(subquery, rte, rti, clause); @@ -1134,7 +1239,7 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, rel->baserestrictinfo = upperrestrictlist; } - pfree(differentTypes); + pfree(unsafeColumns); /* * We can safely pass the outer tuple_fraction down to the subquery if the @@ -1152,6 +1257,9 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, else tuple_fraction = root->tuple_fraction; + /* plan_params should not be in use in current query level */ + Assert(root->plan_params == NIL); + /* Generate the plan for the subquery */ rel->subplan = subquery_planner(root->glob, subquery, root, @@ -1159,10 +1267,14 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, &subroot); rel->subroot = subroot; + /* Isolate the params needed by this specific subplan */ + rel->subplan_params = root->plan_params; + root->plan_params = NIL; + /* * It's possible that constraint exclusion proved the subquery empty. If * so, it's convenient to turn it back into a dummy path so that we will - * recognize appropriate optimizations at this level. + * recognize appropriate optimizations at this query level. */ if (is_dummy_plan(rel->subplan)) { @@ -1222,7 +1334,7 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, && root->parse->hasWindowFuncs))))) elog(ERROR, "Postgres-XL does not currently support ORDER BY in subqueries"); #else - add_path(rel, create_subqueryscan_path(root, rel, pathkeys, NULL)); + add_path(rel, create_subqueryscan_path(root, rel, pathkeys, required_outer)); #endif /* Select cheapest path (pretty easy in this case...) */ @@ -1236,8 +1348,65 @@ set_subquery_pathlist(PlannerInfo *root, RelOptInfo *rel, static void set_function_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) { + Relids required_outer; + List *pathkeys = NIL; + + /* + * We don't support pushing join clauses into the quals of a function + * scan, but it could still have required parameterization due to LATERAL + * refs in the function expression. + */ + required_outer = rel->lateral_relids; + + /* + * The result is considered unordered unless ORDINALITY was used, in which + * case it is ordered by the ordinal column (the last one). See if we + * care, by checking for uses of that Var in equivalence classes. + */ + if (rte->funcordinality) + { + AttrNumber ordattno = rel->max_attr; + Var *var = NULL; + ListCell *lc; + + /* + * Is there a Var for it in reltargetlist? If not, the query did not + * reference the ordinality column, or at least not in any way that + * would be interesting for sorting. + */ + foreach(lc, rel->reltargetlist) + { + Var *node = (Var *) lfirst(lc); + + /* checking varno/varlevelsup is just paranoia */ + if (IsA(node, Var) && + node->varattno == ordattno && + node->varno == rel->relid && + node->varlevelsup == 0) + { + var = node; + break; + } + } + + /* + * Try to build pathkeys for this Var with int8 sorting. We tell + * build_expression_pathkey not to build any new equivalence class; if + * the Var isn't already mentioned in some EC, it means that nothing + * cares about the ordering. + */ + if (var) + pathkeys = build_expression_pathkey(root, + (Expr *) var, + NULL, /* below outer joins */ + Int8LessOperator, + rel->relids, + false); + } + /* Generate appropriate path */ - add_path(rel, create_functionscan_path(root, rel)); + add_path(rel, create_functionscan_path(root, rel, + pathkeys, required_outer)); /* Select cheapest path (pretty easy in this case...) */ set_cheapest(rel); @@ -1250,8 +1419,17 @@ set_function_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) static void set_values_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) { + Relids required_outer; + + /* + * We don't support pushing join clauses into the quals of a values scan, + * but it could still have required parameterization due to LATERAL refs + * in the values expressions. + */ + required_outer = rel->lateral_relids; + /* Generate appropriate path */ - add_path(rel, create_valuesscan_path(root, rel)); + add_path(rel, create_valuesscan_path(root, rel, required_outer)); /* Select cheapest path (pretty easy in this case...) */ set_cheapest(rel); @@ -1262,7 +1440,7 @@ set_values_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) * Build the (single) access path for a non-self-reference CTE RTE * * There's no need for a separate set_cte_size phase, since we don't - * support parameterized paths for CTEs. + * support join-qual-parameterized paths for CTEs. */ static void set_cte_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) @@ -1273,6 +1451,7 @@ set_cte_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) int ndx; ListCell *lc; int plan_id; + Relids required_outer; /* * Find the referenced CTE, and locate the plan previously made for it. @@ -1311,8 +1490,15 @@ set_cte_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) /* Mark rel with estimated output rows, width, etc */ set_cte_size_estimates(root, rel, cteplan); + /* + * We don't support pushing join clauses into the quals of a CTE scan, but + * it could still have required parameterization due to LATERAL refs in + * its tlist. + */ + required_outer = rel->lateral_relids; + /* Generate appropriate path */ - add_path(rel, create_ctescan_path(root, rel)); + add_path(rel, create_ctescan_path(root, rel, required_outer)); /* Select cheapest path (pretty easy in this case...) */ set_cheapest(rel); @@ -1323,7 +1509,7 @@ set_cte_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) * Build the (single) access path for a self-reference CTE RTE * * There's no need for a separate set_worktable_size phase, since we don't - * support parameterized paths for CTEs. + * support join-qual-parameterized paths for CTEs. */ static void set_worktable_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) @@ -1331,6 +1517,7 @@ set_worktable_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) Plan *cteplan; PlannerInfo *cteroot; Index levelsup; + Relids required_outer; /* * We need to find the non-recursive term's plan, which is in the plan @@ -1355,8 +1542,16 @@ set_worktable_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) /* Mark rel with estimated output rows, width, etc */ set_cte_size_estimates(root, rel, cteplan); + /* + * We don't support pushing join clauses into the quals of a worktable + * scan, but it could still have required parameterization due to LATERAL + * refs in its tlist. (I'm not sure this is actually possible given the + * restrictions on recursive references, but it's easy enough to support.) + */ + required_outer = rel->lateral_relids; + /* Generate appropriate path */ - add_path(rel, create_worktablescan_path(root, rel)); + add_path(rel, create_worktablescan_path(root, rel, required_outer)); /* Select cheapest path (pretty easy in this case...) */ set_cheapest(rel); @@ -1454,7 +1649,7 @@ make_rel_from_joinlist(PlannerInfo *root, List *joinlist) * independent jointree items in the query. This is > 1. * * 'initial_rels' is a list of RelOptInfo nodes for each independent - * jointree item. These are the components to be joined together. + * jointree item. These are the components to be joined together. * Note that levels_needed == list_length(initial_rels). * * Returns the final level of join relations, i.e., the relation that is @@ -1470,7 +1665,7 @@ make_rel_from_joinlist(PlannerInfo *root, List *joinlist) * needed for these paths need have been instantiated. * * Note to plugin authors: the functions invoked during standard_join_search() - * modify root->join_rel_list and root->join_rel_hash. If you want to do more + * modify root->join_rel_list and root->join_rel_hash. If you want to do more * than one join-order search, you'll probably need to save and restore the * original states of those data structures. See geqo_eval() for an example. */ @@ -1564,17 +1759,19 @@ standard_join_search(PlannerInfo *root, int levels_needed, List *initial_rels) * 3. If the subquery contains EXCEPT or EXCEPT ALL set ops we cannot push * quals into it, because that could change the results. * - * 4. For subqueries using UNION/UNION ALL/INTERSECT/INTERSECT ALL, we can - * push quals into each component query, but the quals can only reference - * subquery columns that suffer no type coercions in the set operation. - * Otherwise there are possible semantic gotchas. So, we check the - * component queries to see if any of them have different output types; - * differentTypes[k] is set true if column k has different type in any - * component. + * In addition, we make several checks on the subquery's output columns + * to see if it is safe to reference them in pushed-down quals. If output + * column k is found to be unsafe to reference, we set unsafeColumns[k] to + * TRUE, but we don't reject the subquery overall since column k might + * not be referenced by some/all quals. The unsafeColumns[] array will be + * consulted later by qual_is_pushdown_safe(). It's better to do it this + * way than to make the checks directly in qual_is_pushdown_safe(), because + * when the subquery involves set operations we have to check the output + * expressions in each arm of the set op. */ static bool subquery_is_pushdown_safe(Query *subquery, Query *topquery, - bool *differentTypes) + bool *unsafeColumns) { SetOperationStmt *topop; @@ -1586,13 +1783,22 @@ subquery_is_pushdown_safe(Query *subquery, Query *topquery, if (subquery->hasWindowFuncs) return false; + /* + * If we're at a leaf query, check for unsafe expressions in its target + * list, and mark any unsafe ones in unsafeColumns[]. (Non-leaf nodes in + * setop trees have only simple Vars in their tlists, so no need to check + * them.) + */ + if (subquery->setOperations == NULL) + check_output_expressions(subquery, unsafeColumns); + /* Are we at top level, or looking at a setop component? */ if (subquery == topquery) { /* Top level, so check any component queries */ if (subquery->setOperations != NULL) if (!recurse_pushdown_safe(subquery->setOperations, topquery, - differentTypes)) + unsafeColumns)) return false; } else @@ -1605,7 +1811,7 @@ subquery_is_pushdown_safe(Query *subquery, Query *topquery, Assert(topop && IsA(topop, SetOperationStmt)); compare_tlist_datatypes(subquery->targetList, topop->colTypes, - differentTypes); + unsafeColumns); } return true; } @@ -1615,7 +1821,7 @@ subquery_is_pushdown_safe(Query *subquery, Query *topquery, */ static bool recurse_pushdown_safe(Node *setOp, Query *topquery, - bool *differentTypes) + bool *unsafeColumns) { if (IsA(setOp, RangeTblRef)) { @@ -1624,19 +1830,19 @@ recurse_pushdown_safe(Node *setOp, Query *topquery, Query *subquery = rte->subquery; Assert(subquery != NULL); - return subquery_is_pushdown_safe(subquery, topquery, differentTypes); + return subquery_is_pushdown_safe(subquery, topquery, unsafeColumns); } else if (IsA(setOp, SetOperationStmt)) { SetOperationStmt *op = (SetOperationStmt *) setOp; - /* EXCEPT is no good */ + /* EXCEPT is no good (point 3 for subquery_is_pushdown_safe) */ if (op->op == SETOP_EXCEPT) return false; /* Else recurse */ - if (!recurse_pushdown_safe(op->larg, topquery, differentTypes)) + if (!recurse_pushdown_safe(op->larg, topquery, unsafeColumns)) return false; - if (!recurse_pushdown_safe(op->rarg, topquery, differentTypes)) + if (!recurse_pushdown_safe(op->rarg, topquery, unsafeColumns)) return false; } else @@ -1648,17 +1854,92 @@ recurse_pushdown_safe(Node *setOp, Query *topquery, } /* - * Compare tlist's datatypes against the list of set-operation result types. - * For any items that are different, mark the appropriate element of - * differentTypes[] to show that this column will have type conversions. + * check_output_expressions - check subquery's output expressions for safety + * + * There are several cases in which it's unsafe to push down an upper-level + * qual if it references a particular output column of a subquery. We check + * each output column of the subquery and set unsafeColumns[k] to TRUE if + * that column is unsafe for a pushed-down qual to reference. The conditions + * checked here are: + * + * 1. We must not push down any quals that refer to subselect outputs that + * return sets, else we'd introduce functions-returning-sets into the + * subquery's WHERE/HAVING quals. + * + * 2. We must not push down any quals that refer to subselect outputs that + * contain volatile functions, for fear of introducing strange results due + * to multiple evaluation of a volatile function. + * + * 3. If the subquery uses DISTINCT ON, we must not push down any quals that + * refer to non-DISTINCT output columns, because that could change the set + * of rows returned. (This condition is vacuous for DISTINCT, because then + * there are no non-DISTINCT output columns, so we needn't check. But note + * we are assuming that the qual can't distinguish values that the DISTINCT + * operator sees as equal. This is a bit shaky but we have no way to test + * for the case, and it's unlikely enough that we shouldn't refuse the + * optimization just because it could theoretically happen.) + */ +static void +check_output_expressions(Query *subquery, bool *unsafeColumns) +{ + ListCell *lc; + + foreach(lc, subquery->targetList) + { + TargetEntry *tle = (TargetEntry *) lfirst(lc); + + if (tle->resjunk) + continue; /* ignore resjunk columns */ + + /* We need not check further if output col is already known unsafe */ + if (unsafeColumns[tle->resno]) + continue; + + /* Functions returning sets are unsafe (point 1) */ + if (expression_returns_set((Node *) tle->expr)) + { + unsafeColumns[tle->resno] = true; + continue; + } + + /* Volatile functions are unsafe (point 2) */ + if (contain_volatile_functions((Node *) tle->expr)) + { + unsafeColumns[tle->resno] = true; + continue; + } + + /* If subquery uses DISTINCT ON, check point 3 */ + if (subquery->hasDistinctOn && + !targetIsInSortList(tle, InvalidOid, subquery->distinctClause)) + { + /* non-DISTINCT column, so mark it unsafe */ + unsafeColumns[tle->resno] = true; + continue; + } + } +} + +/* + * For subqueries using UNION/UNION ALL/INTERSECT/INTERSECT ALL, we can + * push quals into each component query, but the quals can only reference + * subquery columns that suffer no type coercions in the set operation. + * Otherwise there are possible semantic gotchas. So, we check the + * component queries to see if any of them have output types different from + * the top-level setop outputs. unsafeColumns[k] is set true if column k + * has different type in any component. * * We don't have to care about typmods here: the only allowed difference * between set-op input and output typmods is input is a specific typmod * and output is -1, and that does not require a coercion. + * + * tlist is a subquery tlist. + * colTypes is an OID list of the top-level setop's output column types. + * unsafeColumns[] is the result array. */ static void compare_tlist_datatypes(List *tlist, List *colTypes, - bool *differentTypes) + bool *unsafeColumns) { ListCell *l; ListCell *colType = list_head(colTypes); @@ -1672,7 +1953,7 @@ compare_tlist_datatypes(List *tlist, List *colTypes, if (colType == NULL) elog(ERROR, "wrong number of tlist entries"); if (exprType((Node *) tle->expr) != lfirst_oid(colType)) - differentTypes[tle->resno] = true; + unsafeColumns[tle->resno] = true; colType = lnext(colType); } if (colType != NULL) @@ -1695,34 +1976,15 @@ compare_tlist_datatypes(List *tlist, List *colTypes, * (since there is no easy way to name that within the subquery itself). * * 3. The qual must not refer to any subquery output columns that were - * found to have inconsistent types across a set operation tree by - * subquery_is_pushdown_safe(). - * - * 4. If the subquery uses DISTINCT ON, we must not push down any quals that - * refer to non-DISTINCT output columns, because that could change the set - * of rows returned. (This condition is vacuous for DISTINCT, because then - * there are no non-DISTINCT output columns, so we needn't check. But note - * we are assuming that the qual can't distinguish values that the DISTINCT - * operator sees as equal. This is a bit shaky but we have no way to test - * for the case, and it's unlikely enough that we shouldn't refuse the - * optimization just because it could theoretically happen.) - * - * 5. We must not push down any quals that refer to subselect outputs that - * return sets, else we'd introduce functions-returning-sets into the - * subquery's WHERE/HAVING quals. - * - * 6. We must not push down any quals that refer to subselect outputs that - * contain volatile functions, for fear of introducing strange results due - * to multiple evaluation of a volatile function. + * found to be unsafe to reference by subquery_is_pushdown_safe(). */ static bool qual_is_pushdown_safe(Query *subquery, Index rti, Node *qual, - bool *differentTypes) + bool *unsafeColumns) { bool safe = true; List *vars; ListCell *vl; - Bitmapset *tested = NULL; /* Refuse subselects (point 1) */ if (contain_subplans(qual)) @@ -1730,7 +1992,7 @@ qual_is_pushdown_safe(Query *subquery, Index rti, Node *qual, /* * It would be unsafe to push down window function calls, but at least for - * the moment we could never see any in a qual anyhow. (The same applies + * the moment we could never see any in a qual anyhow. (The same applies * to aggregates, which we check for in pull_var_clause below.) */ Assert(!contain_window_function(qual)); @@ -1745,7 +2007,6 @@ qual_is_pushdown_safe(Query *subquery, Index rti, Node *qual, foreach(vl, vars) { Var *var = (Var *) lfirst(vl); - TargetEntry *tle; /* * XXX Punt if we find any PlaceHolderVars in the restriction clause. @@ -1761,6 +2022,7 @@ qual_is_pushdown_safe(Query *subquery, Index rti, Node *qual, } Assert(var->varno == rti); + Assert(var->varattno >= 0); /* Check point 2 */ if (var->varattno == 0) @@ -1769,45 +2031,8 @@ qual_is_pushdown_safe(Query *subquery, Index rti, Node *qual, break; } - /* - * We use a bitmapset to avoid testing the same attno more than once. - * (NB: this only works because subquery outputs can't have negative - * attnos.) - */ - if (bms_is_member(var->varattno, tested)) - continue; - tested = bms_add_member(tested, var->varattno); - /* Check point 3 */ - if (differentTypes[var->varattno]) - { - safe = false; - break; - } - - /* Must find the tlist element referenced by the Var */ - tle = get_tle_by_resno(subquery->targetList, var->varattno); - Assert(tle != NULL); - Assert(!tle->resjunk); - - /* If subquery uses DISTINCT ON, check point 4 */ - if (subquery->hasDistinctOn && - !targetIsInSortList(tle, InvalidOid, subquery->distinctClause)) - { - /* non-DISTINCT column, so fail */ - safe = false; - break; - } - - /* Refuse functions returning sets (point 5) */ - if (expression_returns_set((Node *) tle->expr)) - { - safe = false; - break; - } - - /* Refuse volatile functions (point 6) */ - if (contain_volatile_functions((Node *) tle->expr)) + if (unsafeColumns[var->varattno]) { safe = false; break; @@ -1815,7 +2040,6 @@ qual_is_pushdown_safe(Query *subquery, Index rti, Node *qual, } list_free(vars); - bms_free(tested); return safe; } @@ -1843,10 +2067,10 @@ subquery_push_qual(Query *subquery, RangeTblEntry *rte, Index rti, Node *qual) * This step also ensures that when we are pushing into a setop tree, * each component query gets its own copy of the qual. */ - qual = ResolveNew(qual, rti, 0, rte, - subquery->targetList, - CMD_SELECT, 0, - &subquery->hasSubLinks); + qual = ReplaceVarsFromTargetList(qual, rti, 0, rte, + subquery->targetList, + REPLACEVARS_REPORT_ERROR, 0, + &subquery->hasSubLinks); /* * Now attach the qual to the proper place: normally WHERE, but if the @@ -2077,10 +2301,16 @@ debug_print_rel(PlannerInfo *root, RelOptInfo *rel) printf("\tpath list:\n"); foreach(l, rel->pathlist) print_path(root, lfirst(l), 1); - printf("\n\tcheapest startup path:\n"); - print_path(root, rel->cheapest_startup_path, 1); - printf("\n\tcheapest total path:\n"); - print_path(root, rel->cheapest_total_path, 1); + if (rel->cheapest_startup_path) + { + printf("\n\tcheapest startup path:\n"); + print_path(root, rel->cheapest_startup_path, 1); + } + if (rel->cheapest_total_path) + { + printf("\n\tcheapest total path:\n"); + print_path(root, rel->cheapest_total_path, 1); + } printf("\n"); fflush(stdout); } diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c index bb148d7b07..9b657fb21f 100644 --- a/src/backend/optimizer/path/clausesel.c +++ b/src/backend/optimizer/path/clausesel.c @@ -3,7 +3,7 @@ * clausesel.c * Routines to compute clause selectivities * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -58,7 +58,7 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause, * See clause_selectivity() for the meaning of the additional parameters. * * Our basic approach is to take the product of the selectivities of the - * subclauses. However, that's only right if the subclauses have independent + * subclauses. However, that's only right if the subclauses have independent * probabilities, and in reality they are often NOT independent. So, * we want to be smarter where we can. @@ -75,12 +75,12 @@ static void addRangeClause(RangeQueryClause **rqlist, Node *clause, * see that hisel is the fraction of the range below the high bound, while * losel is the fraction above the low bound; so hisel can be interpreted * directly as a 0..1 value but we need to convert losel to 1-losel before - * interpreting it as a value. Then the available range is 1-losel to hisel. + * interpreting it as a value. Then the available range is 1-losel to hisel. * However, this calculation double-excludes nulls, so really we need * hisel + losel + null_frac - 1.) * * If either selectivity is exactly DEFAULT_INEQ_SEL, we forget this equation - * and instead use DEFAULT_RANGE_INEQ_SEL. The same applies if the equation + * and instead use DEFAULT_RANGE_INEQ_SEL. The same applies if the equation * yields an impossible (negative) result. * * A free side-effect is that we can recognize redundant inequalities such @@ -174,7 +174,7 @@ clauselist_selectivity(PlannerInfo *root, { /* * If it's not a "<" or ">" operator, just merge the - * selectivity in generically. But if it's the right oprrest, + * selectivity in generically. But if it's the right oprrest, * add the clause to rqlist for later processing. */ switch (get_oprrest(expr->opno)) @@ -459,14 +459,14 @@ treat_as_join_clause(Node *clause, RestrictInfo *rinfo, * nestloop join's inner relation --- varRelid should then be the ID of the * inner relation. * - * When varRelid is 0, all variables are treated as variables. This + * When varRelid is 0, all variables are treated as variables. This * is appropriate for ordinary join clauses and restriction clauses. * * jointype is the join type, if the clause is a join clause. Pass JOIN_INNER * if the clause isn't a join clause. * * sjinfo is NULL for a non-join clause, otherwise it provides additional - * context information about the join being performed. There are some + * context information about the join being performed. There are some * special cases: * 1. For a special (not INNER) join, sjinfo is always a member of * root->join_info_list. @@ -501,7 +501,7 @@ clause_selectivity(PlannerInfo *root, /* * If the clause is marked pseudoconstant, then it will be used as a * gating qual and should not affect selectivity estimates; hence - * return 1.0. The only exception is that a constant FALSE may be + * return 1.0. The only exception is that a constant FALSE may be * taken as having selectivity 0.0, since it will surely mean no rows * out of the plan. This case is simple enough that we need not * bother caching the result. @@ -520,11 +520,11 @@ clause_selectivity(PlannerInfo *root, /* * If possible, cache the result of the selectivity calculation for - * the clause. We can cache if varRelid is zero or the clause + * the clause. We can cache if varRelid is zero or the clause * contains only vars of that relid --- otherwise varRelid will affect * the result, so mustn't cache. Outer join quals might be examined * with either their join's actual jointype or JOIN_INNER, so we need - * two cache variables to remember both cases. Note: we assume the + * two cache variables to remember both cases. Note: we assume the * result won't change if we are switching the input relations or * considering a unique-ified case, so we only need one cache variable * for all non-JOIN_INNER cases. @@ -578,6 +578,7 @@ clause_selectivity(PlannerInfo *root, list_make2(var, makeBoolConst(true, false)), + InvalidOid, varRelid); } } @@ -649,13 +650,15 @@ clause_selectivity(PlannerInfo *root, } else if (is_opclause(clause) || IsA(clause, DistinctExpr)) { - Oid opno = ((OpExpr *) clause)->opno; + OpExpr *opclause = (OpExpr *) clause; + Oid opno = opclause->opno; if (treat_as_join_clause(clause, rinfo, varRelid, sjinfo)) { /* Estimate selectivity for a join clause. */ s1 = join_selectivity(root, opno, - ((OpExpr *) clause)->args, + opclause->args, + opclause->inputcollid, jointype, sjinfo); } @@ -663,7 +666,8 @@ clause_selectivity(PlannerInfo *root, { /* Estimate selectivity for a restriction clause. */ s1 = restriction_selectivity(root, opno, - ((OpExpr *) clause)->args, + opclause->args, + opclause->inputcollid, varRelid); } @@ -681,7 +685,7 @@ clause_selectivity(PlannerInfo *root, /* * This is not an operator, so we guess at the selectivity. THIS IS A * HACK TO GET V4 OUT THE DOOR. FUNCS SHOULD BE ABLE TO HAVE - * SELECTIVITIES THEMSELVES. -- JMH 7/9/92 + * SELECTIVITIES THEMSELVES. -- JMH 7/9/92 */ s1 = (Selectivity) 0.3333333; } diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index ba71c15594..b0a5cb4c36 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -24,7 +24,7 @@ * * Obviously, taking constants for these values is an oversimplification, * but it's tough enough to get any useful estimates even at this level of - * detail. Note that all of these parameters are user-settable, in case + * detail. Note that all of these parameters are user-settable, in case * the default values are drastically off for a particular platform. * * seq_page_cost and random_page_cost can also be overridden for an individual @@ -62,7 +62,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * Portions Copyright (c) 2012-2014, TransLattice, Inc. - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * IDENTIFICATION @@ -73,8 +73,12 @@ #include "postgres.h" +#ifdef _MSC_VER +#include <float.h> /* for _isnan */ +#endif #include <math.h> +#include "access/htup_details.h" #include "executor/executor.h" #include "executor/nodeHash.h" #include "miscadmin.h" @@ -500,7 +504,7 @@ cost_index(IndexPath *path, PlannerInfo *root, double loop_count) * computed for us by query_planner. * * Caller is expected to have ensured that tuples_fetched is greater than zero - * and rounded to integer (see clamp_row_est). The result will likewise be + * and rounded to integer (see clamp_row_est). The result will likewise be * greater than zero and integral. */ double @@ -701,7 +705,7 @@ cost_bitmap_heap_scan(Path *path, PlannerInfo *root, RelOptInfo *baserel, /* * For small numbers of pages we should charge spc_random_page_cost * apiece, while if nearly all the table's pages are being read, it's more - * appropriate to charge spc_seq_page_cost apiece. The effect is + * appropriate to charge spc_seq_page_cost apiece. The effect is * nonlinear, too. For lack of a better idea, interpolate like this to * determine the cost per page. */ @@ -776,7 +780,7 @@ cost_bitmap_tree_node(Path *path, Cost *cost, Selectivity *selec) * Estimate the cost of a BitmapAnd node * * Note that this considers only the costs of index scanning and bitmap - * creation, not the eventual heap access. In that sense the object isn't + * creation, not the eventual heap access. In that sense the object isn't * truly a Path, but it has enough path-like properties (costs in particular) * to warrant treating it as one. We don't bother to set the path rows field, * however. @@ -835,7 +839,7 @@ cost_bitmap_or_node(BitmapOrPath *path, PlannerInfo *root) /* * We estimate OR selectivity on the assumption that the inputs are * non-overlapping, since that's often the case in "x IN (list)" type - * situations. Of course, we clamp to 1.0 at the end. + * situations. Of course, we clamp to 1.0 at the end. * * The runtime cost of the BitmapOr itself is estimated at 100x * cpu_operator_cost for each tbm_union needed. Probably too small, @@ -868,14 +872,19 @@ cost_bitmap_or_node(BitmapOrPath *path, PlannerInfo *root) /* * cost_tidscan * Determines and returns the cost of scanning a relation using TIDs. + * + * 'baserel' is the relation to be scanned + * 'tidquals' is the list of TID-checkable quals + * 'param_info' is the ParamPathInfo if this is a parameterized path, else NULL */ void cost_tidscan(Path *path, PlannerInfo *root, - RelOptInfo *baserel, List *tidquals) + RelOptInfo *baserel, List *tidquals, ParamPathInfo *param_info) { Cost startup_cost = 0; Cost run_cost = 0; bool isCurrentOf = false; + QualCost qpqual_cost; Cost cpu_per_tuple; QualCost tid_qual_cost; int ntuples; @@ -886,8 +895,11 @@ cost_tidscan(Path *path, PlannerInfo *root, Assert(baserel->relid > 0); Assert(baserel->rtekind == RTE_RELATION); - /* For now, tidscans are never parameterized */ - path->rows = baserel->rows; + /* Mark the path with the correct row estimate */ + if (param_info) + path->rows = param_info->ppi_rows; + else + path->rows = baserel->rows; /* Count how many tuples we expect to retrieve */ ntuples = 0; @@ -916,7 +928,7 @@ cost_tidscan(Path *path, PlannerInfo *root, /* * We must force TID scan for WHERE CURRENT OF, because only nodeTidscan.c - * understands how to do it correctly. Therefore, honor enable_tidscan + * understands how to do it correctly. Therefore, honor enable_tidscan * only when CURRENT OF isn't present. Also note that cost_qual_eval * counts a CurrentOfExpr as having startup cost disable_cost, which we * subtract off here; that's to prevent other plan types such as seqscan @@ -944,10 +956,12 @@ cost_tidscan(Path *path, PlannerInfo *root, /* disk costs --- assume each tuple on a different page */ run_cost += spc_random_page_cost * ntuples; - /* CPU costs */ - startup_cost += baserel->baserestrictcost.startup + - tid_qual_cost.per_tuple; - cpu_per_tuple = cpu_tuple_cost + baserel->baserestrictcost.per_tuple - + /* Add scanning CPU costs */ + get_restriction_qual_cost(root, baserel, param_info, &qpqual_cost); + + /* XXX currently we assume TID quals are a subset of qpquals */ + startup_cost += qpqual_cost.startup + tid_qual_cost.per_tuple; + cpu_per_tuple = cpu_tuple_cost + qpqual_cost.per_tuple - tid_qual_cost.per_tuple; run_cost += cpu_per_tuple * ntuples; @@ -1002,12 +1016,17 @@ cost_subqueryscan(Path *path, PlannerInfo *root, /* * cost_functionscan * Determines and returns the cost of scanning a function RTE. + * + * 'baserel' is the relation to be scanned + * 'param_info' is the ParamPathInfo if this is a parameterized path, else NULL */ void -cost_functionscan(Path *path, PlannerInfo *root, RelOptInfo *baserel) +cost_functionscan(Path *path, PlannerInfo *root, + RelOptInfo *baserel, ParamPathInfo *param_info) { Cost startup_cost = 0; Cost run_cost = 0; + QualCost qpqual_cost; Cost cpu_per_tuple; RangeTblEntry *rte; QualCost exprcost; @@ -1017,15 +1036,18 @@ cost_functionscan(Path *path, PlannerInfo *root, RelOptInfo *baserel) rte = planner_rt_fetch(baserel->relid, root); Assert(rte->rtekind == RTE_FUNCTION); - /* functionscans are never parameterized */ - path->rows = baserel->rows; + /* Mark the path with the correct row estimate */ + if (param_info) + path->rows = param_info->ppi_rows; + else + path->rows = baserel->rows; /* - * Estimate costs of executing the function expression. + * Estimate costs of executing the function expression(s). * - * Currently, nodeFunctionscan.c always executes the function to + * Currently, nodeFunctionscan.c always executes the functions to * completion before returning any rows, and caches the results in a - * tuplestore. So the function eval cost is all startup cost, and per-row + * tuplestore. So the function eval cost is all startup cost, and per-row * costs are minimal. * * XXX in principle we ought to charge tuplestore spill costs if the @@ -1033,13 +1055,15 @@ cost_functionscan(Path *path, PlannerInfo *root, RelOptInfo *baserel) * estimates for functions tend to be, there's not a lot of point in that * refinement right now. */ - cost_qual_eval_node(&exprcost, rte->funcexpr, root); + cost_qual_eval_node(&exprcost, (Node *) rte->functions, root); startup_cost += exprcost.startup + exprcost.per_tuple; /* Add scanning CPU costs */ - startup_cost += baserel->baserestrictcost.startup; - cpu_per_tuple = cpu_tuple_cost + baserel->baserestrictcost.per_tuple; + get_restriction_qual_cost(root, baserel, param_info, &qpqual_cost); + + startup_cost += qpqual_cost.startup; + cpu_per_tuple = cpu_tuple_cost + qpqual_cost.per_tuple; run_cost += cpu_per_tuple * baserel->tuples; path->startup_cost = startup_cost; @@ -1049,20 +1073,28 @@ cost_functionscan(Path *path, PlannerInfo *root, RelOptInfo *baserel) /* * cost_valuesscan * Determines and returns the cost of scanning a VALUES RTE. + * + * 'baserel' is the relation to be scanned + * 'param_info' is the ParamPathInfo if this is a parameterized path, else NULL */ void -cost_valuesscan(Path *path, PlannerInfo *root, RelOptInfo *baserel) +cost_valuesscan(Path *path, PlannerInfo *root, + RelOptInfo *baserel, ParamPathInfo *param_info) { Cost startup_cost = 0; Cost run_cost = 0; + QualCost qpqual_cost; Cost cpu_per_tuple; /* Should only be applied to base relations that are values lists */ Assert(baserel->relid > 0); Assert(baserel->rtekind == RTE_VALUES); - /* valuesscans are never parameterized */ - path->rows = baserel->rows; + /* Mark the path with the correct row estimate */ + if (param_info) + path->rows = param_info->ppi_rows; + else + path->rows = baserel->rows; /* * For now, estimate list evaluation cost at one operator eval per list @@ -1071,8 +1103,10 @@ cost_valuesscan(Path *path, PlannerInfo *root, RelOptInfo *baserel) cpu_per_tuple = cpu_operator_cost; /* Add scanning CPU costs */ - startup_cost += baserel->baserestrictcost.startup; - cpu_per_tuple += cpu_tuple_cost + baserel->baserestrictcost.per_tuple; + get_restriction_qual_cost(root, baserel, param_info, &qpqual_cost); + + startup_cost += qpqual_cost.startup; + cpu_per_tuple += cpu_tuple_cost + qpqual_cost.per_tuple; run_cost += cpu_per_tuple * baserel->tuples; path->startup_cost = startup_cost; @@ -1085,30 +1119,37 @@ cost_valuesscan(Path *path, PlannerInfo *root, RelOptInfo *baserel) * * Note: this is used for both self-reference and regular CTEs; the * possible cost differences are below the threshold of what we could - * estimate accurately anyway. Note that the costs of evaluating the + * estimate accurately anyway. Note that the costs of evaluating the * referenced CTE query are added into the final plan as initplan costs, * and should NOT be counted here. */ void -cost_ctescan(Path *path, PlannerInfo *root, RelOptInfo *baserel) +cost_ctescan(Path *path, PlannerInfo *root, + RelOptInfo *baserel, ParamPathInfo *param_info) { Cost startup_cost = 0; Cost run_cost = 0; + QualCost qpqual_cost; Cost cpu_per_tuple; /* Should only be applied to base relations that are CTEs */ Assert(baserel->relid > 0); Assert(baserel->rtekind == RTE_CTE); - /* ctescans are never parameterized */ - path->rows = baserel->rows; + /* Mark the path with the correct row estimate */ + if (param_info) + path->rows = param_info->ppi_rows; + else + path->rows = baserel->rows; /* Charge one CPU tuple cost per row for tuplestore manipulation */ cpu_per_tuple = cpu_tuple_cost; /* Add scanning CPU costs */ - startup_cost += baserel->baserestrictcost.startup; - cpu_per_tuple += cpu_tuple_cost + baserel->baserestrictcost.per_tuple; + get_restriction_qual_cost(root, baserel, param_info, &qpqual_cost); + + startup_cost += qpqual_cost.startup; + cpu_per_tuple += cpu_tuple_cost + qpqual_cost.per_tuple; run_cost += cpu_per_tuple * baserel->tuples; path->startup_cost = startup_cost; @@ -1172,7 +1213,7 @@ cost_recursive_union(Plan *runion, Plan *nrterm, Plan *rterm) * If the total volume exceeds sort_mem, we switch to a tape-style merge * algorithm. There will still be about t*log2(t) tuple comparisons in * total, but we will also need to write and read each tuple once per - * merge pass. We expect about ceil(logM(r)) merge passes where r is the + * merge pass. We expect about ceil(logM(r)) merge passes where r is the * number of initial runs formed and M is the merge order used by tuplesort.c. * Since the average initial run should be about twice sort_mem, we have * disk traffic = 2 * relsize * ceil(logM(p / (2*sort_mem))) @@ -1186,7 +1227,7 @@ cost_recursive_union(Plan *runion, Plan *nrterm, Plan *rterm) * accesses (XXX can't we refine that guess?) * * By default, we charge two operator evals per tuple comparison, which should - * be in the right ballpark in most cases. The caller can tweak this by + * be in the right ballpark in most cases. The caller can tweak this by * specifying nonzero comparison_cost; typically that's used for any extra * work that has to be done to prepare the inputs to the comparison operators. * @@ -1310,7 +1351,7 @@ cost_sort(Path *path, PlannerInfo *root, * Determines and returns the cost of a MergeAppend node. * * MergeAppend merges several pre-sorted input streams, using a heap that - * at any given instant holds the next tuple from each stream. If there + * at any given instant holds the next tuple from each stream. If there * are N streams, we need about N*log2(N) tuple comparisons to construct * the heap at startup, and then for each output tuple, about log2(N) * comparisons to delete the top heap entry and another log2(N) comparisons @@ -1469,7 +1510,7 @@ cost_agg(Path *path, PlannerInfo *root, * group otherwise. We charge cpu_tuple_cost for each output tuple. * * Note: in this cost model, AGG_SORTED and AGG_HASHED have exactly the - * same total CPU cost, but AGG_SORTED has lower startup cost. If the + * same total CPU cost, but AGG_SORTED has lower startup cost. If the * input path is already sorted appropriately, AGG_SORTED should be * preferred (since it has no risk of memory overflow). This will happen * as long as the computed total costs are indeed exactly equal --- but if @@ -1562,6 +1603,14 @@ cost_windowagg(Path *path, PlannerInfo *root, startup_cost += argcosts.startup; wfunccost += argcosts.per_tuple; + /* + * Add the filter's cost to per-input-row costs. XXX We should reduce + * input expression costs according to filter selectivity. + */ + cost_qual_eval_node(&argcosts, (Node *) wfunc->aggfilter, root); + startup_cost += argcosts.startup; + wfunccost += argcosts.per_tuple; + total_cost += wfunccost * input_tuples; } @@ -2069,10 +2118,10 @@ initial_cost_mergejoin(PlannerInfo *root, JoinCostWorkspace *workspace, * Unlike other costsize functions, this routine makes one actual decision: * whether we should materialize the inner path. We do that either because * the inner path can't support mark/restore, or because it's cheaper to - * use an interposed Material node to handle mark/restore. When the decision + * use an interposed Material node to handle mark/restore. When the decision * is cost-based it would be logically cleaner to build and cost two separate * paths with and without that flag set; but that would require repeating most - * of the cost calculations, which are not all that cheap. Since the choice + * of the cost calculations, which are not all that cheap. Since the choice * will not affect output pathkeys or startup cost, only total cost, there is * no possibility of wanting to keep both paths. So it seems best to make * the decision here and record it in the path's materialize_inner field. @@ -2136,7 +2185,7 @@ final_cost_mergejoin(PlannerInfo *root, MergePath *path, qp_qual_cost.per_tuple -= merge_qual_cost.per_tuple; /* - * Get approx # tuples passing the mergequals. We use approx_tuple_count + * Get approx # tuples passing the mergequals. We use approx_tuple_count * here because we need an estimate done with JOIN_INNER semantics. */ mergejointuples = approx_tuple_count(root, &path->jpath, mergeclauses); @@ -2150,7 +2199,7 @@ final_cost_mergejoin(PlannerInfo *root, MergePath *path, * estimated approximately as size of merge join output minus size of * inner relation. Assume that the distinct key values are 1, 2, ..., and * denote the number of values of each key in the outer relation as m1, - * m2, ...; in the inner relation, n1, n2, ... Then we have + * m2, ...; in the inner relation, n1, n2, ... Then we have * * size of join = m1 * n1 + m2 * n2 + ... * @@ -2161,7 +2210,7 @@ final_cost_mergejoin(PlannerInfo *root, MergePath *path, * This equation works correctly for outer tuples having no inner match * (nk = 0), but not for inner tuples having no outer match (mk = 0); we * are effectively subtracting those from the number of rescanned tuples, - * when we should not. Can we do better without expensive selectivity + * when we should not. Can we do better without expensive selectivity * computations? * * The whole issue is moot if we are working from a unique-ified outer @@ -2181,7 +2230,7 @@ final_cost_mergejoin(PlannerInfo *root, MergePath *path, /* * Decide whether we want to materialize the inner input to shield it from - * mark/restore and performing re-fetches. Our cost model for regular + * mark/restore and performing re-fetches. Our cost model for regular * re-fetches is that a re-fetch costs the same as an original fetch, * which is probably an overestimate; but on the other hand we ignore the * bookkeeping costs of mark/restore. Not clear if it's worth developing @@ -2283,7 +2332,7 @@ final_cost_mergejoin(PlannerInfo *root, MergePath *path, /* * For each tuple that gets through the mergejoin proper, we charge * cpu_tuple_cost plus the cost of evaluating additional restriction - * clauses that are to be applied at the join. (This is pessimistic since + * clauses that are to be applied at the join. (This is pessimistic since * not all of the quals may get evaluated at each tuple.) * * Note: we could adjust for SEMI/ANTI joins skipping some qual @@ -2435,7 +2484,7 @@ initial_cost_hashjoin(PlannerInfo *root, JoinCostWorkspace *workspace, * If inner relation is too big then we will need to "batch" the join, * which implies writing and reading most of the tuples to disk an extra * time. Charge seq_page_cost per page, since the I/O should be nice and - * sequential. Writing the inner rel counts as startup cost, all the rest + * sequential. Writing the inner rel counts as startup cost, all the rest * as run cost. */ if (numbatches > 1) @@ -2666,7 +2715,7 @@ final_cost_hashjoin(PlannerInfo *root, HashPath *path, /* * For each tuple that gets through the hashjoin proper, we charge * cpu_tuple_cost plus the cost of evaluating additional restriction - * clauses that are to be applied at the join. (This is pessimistic since + * clauses that are to be applied at the join. (This is pessimistic since * not all of the quals may get evaluated at each tuple.) */ startup_cost += qp_qual_cost.startup; @@ -2719,7 +2768,7 @@ cost_subplan(PlannerInfo *root, SubPlan *subplan, Plan *plan) { /* * Otherwise we will be rescanning the subplan output on each - * evaluation. We need to estimate how much of the output we will + * evaluation. We need to estimate how much of the output we will * actually need to scan. NOTE: this logic should agree with the * tuple_fraction estimates used by make_subplan() in * plan/subselect.c. @@ -2767,10 +2816,10 @@ cost_subplan(PlannerInfo *root, SubPlan *subplan, Plan *plan) /* * cost_rescan * Given a finished Path, estimate the costs of rescanning it after - * having done so the first time. For some Path types a rescan is + * having done so the first time. For some Path types a rescan is * cheaper than an original scan (if no parameters change), and this * function embodies knowledge about that. The default is to return - * the same costs stored in the Path. (Note that the cost estimates + * the same costs stored in the Path. (Note that the cost estimates * actually stored in Paths are always for first scans.) * * This function is not currently intended to model effects such as rescans @@ -2811,7 +2860,7 @@ cost_rescan(PlannerInfo *root, Path *path, { /* * These plan types materialize their final result in a - * tuplestore or tuplesort object. So the rescan cost is only + * tuplestore or tuplesort object. So the rescan cost is only * cpu_tuple_cost per tuple, unless the result is large enough * to spill to disk. */ @@ -2836,8 +2885,8 @@ cost_rescan(PlannerInfo *root, Path *path, { /* * These plan types not only materialize their results, but do - * not implement qual filtering or projection. So they are - * even cheaper to rescan than the ones above. We charge only + * not implement qual filtering or projection. So they are + * even cheaper to rescan than the ones above. We charge only * cpu_operator_cost per tuple. (Note: keep that in sync with * the run_cost charge in cost_sort, and also see comments in * cost_material before you change it.) @@ -2978,8 +3027,15 @@ cost_qual_eval_walker(Node *node, cost_qual_eval_context *context) * evaluation of AND/OR? Probably *not*, because that would make the * results depend on the clause ordering, and we are not in any position * to expect that the current ordering of the clauses is the one that's - * going to end up being used. The above per-RestrictInfo caching would + * going to end up being used. The above per-RestrictInfo caching would * not mix well with trying to re-order clauses anyway. + * + * Another issue that is entirely ignored here is that if a set-returning + * function is below top level in the tree, the functions/operators above + * it will need to be evaluated multiple times. In practical use, such + * cases arise so seldom as to not be worth the added complexity needed; + * moreover, since our rowcount estimates for functions tend to be pretty + * phony, the results would also be pretty phony. */ if (IsA(node, FuncExpr)) { @@ -3093,7 +3149,7 @@ cost_qual_eval_walker(Node *node, cost_qual_eval_context *context) else if (IsA(node, AlternativeSubPlan)) { /* - * Arbitrarily use the first alternative plan for costing. (We should + * Arbitrarily use the first alternative plan for costing. (We should * certainly only include one alternative, and we don't yet have * enough information to know which one the executor is most likely to * use.) @@ -3237,13 +3293,13 @@ compute_semi_anti_join_factors(PlannerInfo *root, /* * jselec can be interpreted as the fraction of outer-rel rows that have * any matches (this is true for both SEMI and ANTI cases). And nselec is - * the fraction of the Cartesian product that matches. So, the average + * the fraction of the Cartesian product that matches. So, the average * number of matches for each outer-rel row that has at least one match is * nselec * inner_rows / jselec. * * Note: it is correct to use the inner rel's "rows" count here, even * though we might later be considering a parameterized inner path with - * fewer rows. This is because we have included all the join clauses in + * fewer rows. This is because we have included all the join clauses in * the selectivity estimate. */ if (jselec > 0) /* protect against zero divide */ @@ -3571,7 +3627,7 @@ calc_joinrel_size_estimate(PlannerInfo *root, double nrows; /* - * Compute joinclause selectivity. Note that we are only considering + * Compute joinclause selectivity. Note that we are only considering * clauses that become restriction clauses at this join level; we are not * double-counting them because they were not considered in estimating the * sizes of the component rels. @@ -3629,7 +3685,7 @@ calc_joinrel_size_estimate(PlannerInfo *root, * * If we are doing an outer join, take that into account: the joinqual * selectivity has to be clamped using the knowledge that the output must - * be at least as large as the non-nullable input. However, any + * be at least as large as the non-nullable input. However, any * pushed-down quals are applied after the outer join, so their * selectivity applies fully. * @@ -3700,7 +3756,7 @@ set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel) /* * Compute per-output-column width estimates by examining the subquery's - * targetlist. For any output that is a plain Var, get the width estimate + * targetlist. For any output that is a plain Var, get the width estimate * that was made while planning the subquery. Otherwise, we leave it to * set_rel_width to fill in a datatype-based default estimate. */ @@ -3716,6 +3772,15 @@ set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel) continue; /* + * The subquery could be an expansion of a view that's had columns + * added to it since the current query was parsed, so that there are + * non-junk tlist columns in it that don't correspond to any column + * visible at our query level. Ignore such columns. + */ + if (te->resno < rel->min_attr || te->resno > rel->max_attr) + continue; + + /* * XXX This currently doesn't work for subqueries containing set * operations, because the Vars in their tlists are bogus references * to the first leaf subquery, which wouldn't give the right answer @@ -3736,7 +3801,6 @@ set_subquery_size_estimates(PlannerInfo *root, RelOptInfo *rel) item_width = subrel->attr_widths[var->varattno - subrel->min_attr]; } - Assert(te->resno >= rel->min_attr && te->resno <= rel->max_attr); rel->attr_widths[te->resno - rel->min_attr] = item_width; } @@ -3757,14 +3821,26 @@ void set_function_size_estimates(PlannerInfo *root, RelOptInfo *rel) { RangeTblEntry *rte; + ListCell *lc; /* Should only be applied to base relations that are functions */ Assert(rel->relid > 0); rte = planner_rt_fetch(rel->relid, root); Assert(rte->rtekind == RTE_FUNCTION); - /* Estimate number of rows the function itself will return */ - rel->tuples = clamp_row_est(expression_returns_set_rows(rte->funcexpr)); + /* + * Estimate number of rows the functions will return. The rowcount of the + * node is that of the largest function result. + */ + rel->tuples = 0; + foreach(lc, rte->functions) + { + RangeTblFunction *rtfunc = (RangeTblFunction *) lfirst(lc); + double ntup = expression_returns_set_rows(rtfunc->funcexpr); + + if (ntup > rel->tuples) + rel->tuples = ntup; + } /* Now estimate number of output rows, etc */ set_baserel_size_estimates(root, rel); @@ -3848,7 +3924,7 @@ set_cte_size_estimates(PlannerInfo *root, RelOptInfo *rel, Plan *cteplan) * of estimating baserestrictcost, so we set that, and we also set up width * using what will be purely datatype-driven estimates from the targetlist. * There is no way to do anything sane with the rows value, so we just put - * a default estimate and hope that the wrapper can improve on it. The + * a default estimate and hope that the wrapper can improve on it. The * wrapper's GetForeignRelSize function will be called momentarily. * * The rel's targetlist and restrictinfo list must have been constructed @@ -3899,13 +3975,19 @@ set_rel_width(PlannerInfo *root, RelOptInfo *rel) { Node *node = (Node *) lfirst(lc); - if (IsA(node, Var)) + /* + * Ordinarily, a Var in a rel's reltargetlist must belong to that rel; + * but there are corner cases involving LATERAL references where that + * isn't so. If the Var has the wrong varno, fall through to the + * generic case (it doesn't seem worth the trouble to be any smarter). + */ + if (IsA(node, Var) && + ((Var *) node)->varno == rel->relid) { Var *var = (Var *) node; int ndx; int32 item_width; - Assert(var->varno == rel->relid); Assert(var->varattno >= rel->min_attr); Assert(var->varattno <= rel->max_attr); @@ -3963,7 +4045,7 @@ set_rel_width(PlannerInfo *root, RelOptInfo *rel) { /* * We could be looking at an expression pulled up from a subquery, - * or a ROW() representing a whole-row child Var, etc. Do what we + * or a ROW() representing a whole-row child Var, etc. Do what we * can using the expression type information. */ int32 item_width; diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c index e34b9553bd..b7aff3775e 100644 --- a/src/backend/optimizer/path/equivclass.c +++ b/src/backend/optimizer/path/equivclass.c @@ -6,7 +6,7 @@ * See src/backend/optimizer/README for discussion of EquivalenceClasses. * * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * IDENTIFICATION @@ -30,7 +30,7 @@ static EquivalenceMember *add_eq_member(EquivalenceClass *ec, - Expr *expr, Relids relids, + Expr *expr, Relids relids, Relids nullable_relids, bool is_child, Oid datatype); static void generate_base_implied_equalities_const(PlannerInfo *root, EquivalenceClass *ec); @@ -74,7 +74,7 @@ static bool reconsider_full_join_clause(PlannerInfo *root, * * If below_outer_join is true, then the clause was found below the nullable * side of an outer join, so its sides might validly be both NULL rather than - * strictly equal. We can still deduce equalities in such cases, but we take + * strictly equal. We can still deduce equalities in such cases, but we take * care to mark an EquivalenceClass if it came from any such clauses. Also, * we have to check that both sides are either pseudo-constants or strict * functions of Vars, else they might not both go to NULL above the outer @@ -106,7 +106,9 @@ process_equivalence(PlannerInfo *root, RestrictInfo *restrictinfo, Expr *item1; Expr *item2; Relids item1_relids, - item2_relids; + item2_relids, + item1_nullable_relids, + item2_nullable_relids; List *opfamilies; EquivalenceClass *ec1, *ec2; @@ -139,9 +141,9 @@ process_equivalence(PlannerInfo *root, RestrictInfo *restrictinfo, collation); /* - * Reject clauses of the form X=X. These are not as redundant as they + * Reject clauses of the form X=X. These are not as redundant as they * might seem at first glance: assuming the operator is strict, this is - * really an expensive way to write X IS NOT NULL. So we must not risk + * really an expensive way to write X IS NOT NULL. So we must not risk * just losing the clause, which would be possible if there is already a * single-element EquivalenceClass containing X. The case is not common * enough to be worth contorting the EC machinery for, so just reject the @@ -163,6 +165,12 @@ process_equivalence(PlannerInfo *root, RestrictInfo *restrictinfo, return false; /* RHS is non-strict but not constant */ } + /* Calculate nullable-relid sets for each side of the clause */ + item1_nullable_relids = bms_intersect(item1_relids, + restrictinfo->nullable_relids); + item2_nullable_relids = bms_intersect(item2_relids, + restrictinfo->nullable_relids); + /* * We use the declared input types of the operator, not exprType() of the * inputs, as the nominal datatypes for opfamily lookup. This presumes @@ -179,14 +187,14 @@ process_equivalence(PlannerInfo *root, RestrictInfo *restrictinfo, * Sweep through the existing EquivalenceClasses looking for matches to * item1 and item2. These are the possible outcomes: * - * 1. We find both in the same EC. The equivalence is already known, so + * 1. We find both in the same EC. The equivalence is already known, so * there's nothing to do. * * 2. We find both in different ECs. Merge the two ECs together. * * 3. We find just one. Add the other to its EC. * - * 4. We find neither. Make a new, two-entry EC. + * 4. We find neither. Make a new, two-entry EC. * * Note: since all ECs are built through this process or the similar * search in get_eclass_for_sort_expr(), it's impossible that we'd match @@ -277,11 +285,19 @@ process_equivalence(PlannerInfo *root, RestrictInfo *restrictinfo, } /* - * Case 2: need to merge ec1 and ec2. We add ec2's items to ec1, then - * set ec2's ec_merged link to point to ec1 and remove ec2 from the - * eq_classes list. We cannot simply delete ec2 because that could - * leave dangling pointers in existing PathKeys. We leave it behind - * with a link so that the merged EC can be found. + * Case 2: need to merge ec1 and ec2. This should never happen after + * we've built any canonical pathkeys; if it did, those pathkeys might + * be rendered non-canonical by the merge. + */ + if (root->canon_pathkeys != NIL) + elog(ERROR, "too late to merge equivalence classes"); + + /* + * We add ec2's items to ec1, then set ec2's ec_merged link to point + * to ec1 and remove ec2 from the eq_classes list. We cannot simply + * delete ec2 because that could leave dangling pointers in existing + * PathKeys. We leave it behind with a link so that the merged EC can + * be found. */ ec1->ec_members = list_concat(ec1->ec_members, ec2->ec_members); ec1->ec_sources = list_concat(ec1->ec_sources, ec2->ec_sources); @@ -309,7 +325,8 @@ process_equivalence(PlannerInfo *root, RestrictInfo *restrictinfo, else if (ec1) { /* Case 3: add item2 to ec1 */ - em2 = add_eq_member(ec1, item2, item2_relids, false, item2_type); + em2 = add_eq_member(ec1, item2, item2_relids, item2_nullable_relids, + false, item2_type); ec1->ec_sources = lappend(ec1->ec_sources, restrictinfo); ec1->ec_below_outer_join |= below_outer_join; /* mark the RI as associated with this eclass */ @@ -322,7 +339,8 @@ process_equivalence(PlannerInfo *root, RestrictInfo *restrictinfo, else if (ec2) { /* Case 3: add item1 to ec2 */ - em1 = add_eq_member(ec2, item1, item1_relids, false, item1_type); + em1 = add_eq_member(ec2, item1, item1_relids, item1_nullable_relids, + false, item1_type); ec2->ec_sources = lappend(ec2->ec_sources, restrictinfo); ec2->ec_below_outer_join |= below_outer_join; /* mark the RI as associated with this eclass */ @@ -349,8 +367,10 @@ process_equivalence(PlannerInfo *root, RestrictInfo *restrictinfo, ec->ec_broken = false; ec->ec_sortref = 0; ec->ec_merged = NULL; - em1 = add_eq_member(ec, item1, item1_relids, false, item1_type); - em2 = add_eq_member(ec, item2, item2_relids, false, item2_type); + em1 = add_eq_member(ec, item1, item1_relids, item1_nullable_relids, + false, item1_type); + em2 = add_eq_member(ec, item2, item2_relids, item2_nullable_relids, + false, item2_type); root->eq_classes = lappend(root->eq_classes, ec); @@ -386,7 +406,7 @@ process_equivalence(PlannerInfo *root, RestrictInfo *restrictinfo, * Also, the expression's exposed collation must match the EC's collation. * This is important because in comparisons like "foo < bar COLLATE baz", * only one of the expressions has the correct exposed collation as we receive - * it from the parser. Forcing both of them to have it ensures that all + * it from the parser. Forcing both of them to have it ensures that all * variant spellings of such a construct behave the same. Again, we can * stick on a RelabelType to force the right exposed collation. (It might * work to not label the collation at all in EC members, but this is risky @@ -431,13 +451,13 @@ canonicalize_ec_expression(Expr *expr, Oid req_type, Oid req_collation) req_type, -1, req_collation, - COERCE_DONTCARE); + COERCE_IMPLICIT_CAST); else if (exprCollation((Node *) expr) != req_collation) expr = (Expr *) makeRelabelType(expr, req_type, exprTypmod((Node *) expr), req_collation, - COERCE_DONTCARE); + COERCE_IMPLICIT_CAST); } return expr; @@ -448,12 +468,13 @@ canonicalize_ec_expression(Expr *expr, Oid req_type, Oid req_collation) */ static EquivalenceMember * add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, - bool is_child, Oid datatype) + Relids nullable_relids, bool is_child, Oid datatype) { EquivalenceMember *em = makeNode(EquivalenceMember); em->em_expr = expr; em->em_relids = relids; + em->em_nullable_relids = nullable_relids; em->em_is_const = false; em->em_is_child = is_child; em->em_datatype = datatype; @@ -489,17 +510,24 @@ add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, * equivalence class it is a member of; if none, optionally build a new * single-member EquivalenceClass for it. * + * expr is the expression, and nullable_relids is the set of base relids + * that are potentially nullable below it. We actually only care about + * the set of such relids that are used in the expression; but for caller + * convenience, we perform that intersection step here. The caller need + * only be sure that nullable_relids doesn't omit any nullable rels that + * might appear in the expr. + * * sortref is the SortGroupRef of the originating SortGroupClause, if any, - * or zero if not. (It should never be zero if the expression is volatile!) + * or zero if not. (It should never be zero if the expression is volatile!) * * If rel is not NULL, it identifies a specific relation we're considering * a path for, and indicates that child EC members for that relation can be - * considered. Otherwise child members are ignored. (Note: since child EC + * considered. Otherwise child members are ignored. (Note: since child EC * members aren't guaranteed unique, a non-NULL value means that there could * be more than one EC that matches the expression; if so it's order-dependent * which one you get. This is annoying but it only happens in corner cases, - * so for now we live with just reporting the first match. See also - * generate_implied_equalities_for_indexcol and match_pathkeys_to_index.) + * so for now we live with just reporting the first match. See also + * generate_implied_equalities_for_column and match_pathkeys_to_index.) * * If create_it is TRUE, we'll build a new EquivalenceClass when there is no * match. If create_it is FALSE, we just return NULL when no match. @@ -517,6 +545,7 @@ add_eq_member(EquivalenceClass *ec, Expr *expr, Relids relids, EquivalenceClass * get_eclass_for_sort_expr(PlannerInfo *root, Expr *expr, + Relids nullable_relids, List *opfamilies, Oid opcintype, Oid collation, @@ -524,6 +553,7 @@ get_eclass_for_sort_expr(PlannerInfo *root, Relids rel, bool create_it) { + Relids expr_relids; EquivalenceClass *newec; EquivalenceMember *newem; ListCell *lc1; @@ -535,6 +565,12 @@ get_eclass_for_sort_expr(PlannerInfo *root, expr = canonicalize_ec_expression(expr, opcintype, collation); /* + * Get the precise set of nullable relids appearing in the expression. + */ + expr_relids = pull_varnos((Node *) expr); + nullable_relids = bms_intersect(nullable_relids, expr_relids); + + /* * Scan through the existing EquivalenceClasses for a match */ foreach(lc1, root->eq_classes) @@ -608,8 +644,8 @@ get_eclass_for_sort_expr(PlannerInfo *root, if (newec->ec_has_volatile && sortref == 0) /* should not happen */ elog(ERROR, "volatile EquivalenceClass has no sortref"); - newem = add_eq_member(newec, copyObject(expr), pull_varnos((Node *) expr), - false, opcintype); + newem = add_eq_member(newec, copyObject(expr), expr_relids, + nullable_relids, false, opcintype); /* * add_eq_member doesn't check for volatile functions, set-returning @@ -644,7 +680,7 @@ get_eclass_for_sort_expr(PlannerInfo *root, * * When an EC contains pseudoconstants, our strategy is to generate * "member = const1" clauses where const1 is the first constant member, for - * every other member (including other constants). If we are able to do this + * every other member (including other constants). If we are able to do this * then we don't need any "var = var" comparisons because we've successfully * constrained all the vars at their points of creation. If we fail to * generate any of these clauses due to lack of cross-type operators, we fall @@ -669,7 +705,7 @@ get_eclass_for_sort_expr(PlannerInfo *root, * "WHERE a.x = b.y AND b.y = a.z", the scheme breaks down if we cannot * generate "a.x = a.z" as a restriction clause for A.) In this case we mark * the EC "ec_broken" and fall back to regurgitating its original source - * RestrictInfos at appropriate times. We do not try to retract any derived + * RestrictInfos at appropriate times. We do not try to retract any derived * clauses already generated from the broken EC, so the resulting plan could * be poor due to bad selectivity estimates caused by redundant clauses. But * the correct solution to that is to fix the opfamilies ... @@ -756,7 +792,12 @@ generate_base_implied_equalities_const(PlannerInfo *root, } } - /* Find the constant member to use */ + /* + * Find the constant member to use. We prefer an actual constant to + * pseudo-constants (such as Params), because the constraint exclusion + * machinery might be able to exclude relations on the basis of generated + * "var = const" equalities, but "var = param" won't work for that. + */ foreach(lc, ec->ec_members) { EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc); @@ -764,7 +805,8 @@ generate_base_implied_equalities_const(PlannerInfo *root, if (cur_em->em_is_const) { const_em = cur_em; - break; + if (IsA(cur_em->em_expr, Const)) + break; } } Assert(const_em != NULL); @@ -789,7 +831,9 @@ generate_base_implied_equalities_const(PlannerInfo *root, } process_implied_equality(root, eq_op, ec->ec_collation, cur_em->em_expr, const_em->em_expr, - ec->ec_relids, + bms_copy(ec->ec_relids), + bms_union(cur_em->em_nullable_relids, + const_em->em_nullable_relids), ec->ec_below_outer_join, cur_em->em_is_const); } @@ -844,7 +888,9 @@ generate_base_implied_equalities_no_const(PlannerInfo *root, } process_implied_equality(root, eq_op, ec->ec_collation, prev_em->em_expr, cur_em->em_expr, - ec->ec_relids, + bms_copy(ec->ec_relids), + bms_union(prev_em->em_nullable_relids, + cur_em->em_nullable_relids), ec->ec_below_outer_join, false); } @@ -922,8 +968,8 @@ generate_base_implied_equalities_broken(PlannerInfo *root, * built any join RelOptInfos. * * An annoying special case for parameterized scans is that the inner rel can - * be an appendrel child (an "other rel"). In this case we must generate - * appropriate clauses using child EC members. add_child_rel_equivalences + * be an appendrel child (an "other rel"). In this case we must generate + * appropriate clauses using child EC members. add_child_rel_equivalences * must already have been done for the child rel. * * The results are sufficient for use in merge, hash, and plain nestloop join @@ -937,7 +983,7 @@ generate_base_implied_equalities_broken(PlannerInfo *root, * we consider different join paths, we avoid generating multiple copies: * whenever we select a particular pair of EquivalenceMembers to join, * we check to see if the pair matches any original clause (in ec_sources) - * or previously-built clause (in ec_derives). This saves memory and allows + * or previously-built clause (in ec_derives). This saves memory and allows * re-use of information cached in RestrictInfos. * * join_relids should always equal bms_union(outer_relids, inner_rel->relids). @@ -1033,7 +1079,7 @@ generate_join_implied_equalities_normal(PlannerInfo *root, * First, scan the EC to identify member values that are computable at the * outer rel, at the inner rel, or at this relation but not in either * input rel. The outer-rel members should already be enforced equal, - * likewise for the inner-rel members. We'll need to create clauses to + * likewise for the inner-rel members. We'll need to create clauses to * enforce that any newly computable members are all equal to each other * as well as to at least one input member, plus enforce at least one * outer-rel member equal to at least one inner-rel member. @@ -1059,7 +1105,7 @@ generate_join_implied_equalities_normal(PlannerInfo *root, } /* - * First, select the joinclause if needed. We can equate any one outer + * First, select the joinclause if needed. We can equate any one outer * member to any one inner member, but we have to find a datatype * combination for which an opfamily member operator exists. If we have * choices, we prefer simple Var members (possibly with RelabelType) since @@ -1277,8 +1323,8 @@ create_join_clause(PlannerInfo *root, /* * Search to see if we already built a RestrictInfo for this pair of - * EquivalenceMembers. We can use either original source clauses or - * previously-derived clauses. The check on opno is probably redundant, + * EquivalenceMembers. We can use either original source clauses or + * previously-derived clauses. The check on opno is probably redundant, * but be safe ... */ foreach(lc, ec->ec_sources) @@ -1312,7 +1358,9 @@ create_join_clause(PlannerInfo *root, leftem->em_expr, rightem->em_expr, bms_union(leftem->em_relids, - rightem->em_relids)); + rightem->em_relids), + bms_union(leftem->em_nullable_relids, + rightem->em_nullable_relids)); /* Mark the clause as redundant, or not */ rinfo->parent_ec = parent_ec; @@ -1407,7 +1455,7 @@ create_join_clause(PlannerInfo *root, * * Outer join clauses that are marked outerjoin_delayed are special: this * condition means that one or both VARs might go to null due to a lower - * outer join. We can still push a constant through the clause, but only + * outer join. We can still push a constant through the clause, but only * if its operator is strict; and we *have to* throw the clause back into * regular joinclause processing. By keeping the strict join clause, * we ensure that any null-extended rows that are mistakenly generated due @@ -1534,7 +1582,8 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, left_type, right_type, inner_datatype; - Relids inner_relids; + Relids inner_relids, + inner_nullable_relids; ListCell *lc1; Assert(is_opclause(rinfo->clause)); @@ -1561,6 +1610,8 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, inner_datatype = left_type; inner_relids = rinfo->left_relids; } + inner_nullable_relids = bms_intersect(inner_relids, + rinfo->nullable_relids); /* Scan EquivalenceClasses for a match to outervar */ foreach(lc1, root->eq_classes) @@ -1598,7 +1649,7 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, /* * Yes it does! Try to generate a clause INNERVAR = CONSTANT for each - * CONSTANT in the EC. Note that we must succeed with at least one + * CONSTANT in the EC. Note that we must succeed with at least one * constant before we can decide to throw away the outer-join clause. */ match = false; @@ -1619,7 +1670,8 @@ reconsider_outer_join_clause(PlannerInfo *root, RestrictInfo *rinfo, cur_ec->ec_collation, innervar, cur_em->em_expr, - inner_relids); + bms_copy(inner_relids), + bms_copy(inner_nullable_relids)); if (process_equivalence(root, newrinfo, true)) match = true; } @@ -1653,7 +1705,9 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) left_type, right_type; Relids left_relids, - right_relids; + right_relids, + left_nullable_relids, + right_nullable_relids; ListCell *lc1; /* Can't use an outerjoin_delayed clause here */ @@ -1669,6 +1723,10 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) rightvar = (Expr *) get_rightop(rinfo->clause); left_relids = rinfo->left_relids; right_relids = rinfo->right_relids; + left_nullable_relids = bms_intersect(left_relids, + rinfo->nullable_relids); + right_nullable_relids = bms_intersect(right_relids, + rinfo->nullable_relids); foreach(lc1, root->eq_classes) { @@ -1754,7 +1812,8 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) cur_ec->ec_collation, leftvar, cur_em->em_expr, - left_relids); + bms_copy(left_relids), + bms_copy(left_nullable_relids)); if (process_equivalence(root, newrinfo, true)) matchleft = true; } @@ -1767,7 +1826,8 @@ reconsider_full_join_clause(PlannerInfo *root, RestrictInfo *rinfo) cur_ec->ec_collation, rightvar, cur_em->em_expr, - right_relids); + bms_copy(right_relids), + bms_copy(right_nullable_relids)); if (process_equivalence(root, newrinfo, true)) matchright = true; } @@ -1877,16 +1937,20 @@ add_child_rel_equivalences(PlannerInfo *root, if (cur_ec->ec_has_volatile) continue; - /* No point in searching if parent rel not mentioned in eclass */ - if (!bms_is_subset(parent_rel->relids, cur_ec->ec_relids)) + /* + * No point in searching if parent rel not mentioned in eclass; but we + * can't tell that for sure if parent rel is itself a child. + */ + if (parent_rel->reloptkind == RELOPT_BASEREL && + !bms_is_subset(parent_rel->relids, cur_ec->ec_relids)) continue; foreach(lc2, cur_ec->ec_members) { EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); - if (cur_em->em_is_const || cur_em->em_is_child) - continue; /* ignore consts and children here */ + if (cur_em->em_is_const) + continue; /* ignore consts here */ /* Does it reference parent_rel? */ if (bms_overlap(cur_em->em_relids, parent_rel->relids)) @@ -1894,6 +1958,7 @@ add_child_rel_equivalences(PlannerInfo *root, /* Yes, generate transformed child version */ Expr *child_expr; Relids new_relids; + Relids new_nullable_relids; child_expr = (Expr *) adjust_appendrel_attrs(root, @@ -1910,7 +1975,21 @@ add_child_rel_equivalences(PlannerInfo *root, parent_rel->relids); new_relids = bms_add_members(new_relids, child_rel->relids); - (void) add_eq_member(cur_ec, child_expr, new_relids, + /* + * And likewise for nullable_relids. Note this code assumes + * parent and child relids are singletons. + */ + new_nullable_relids = cur_em->em_nullable_relids; + if (bms_overlap(new_nullable_relids, parent_rel->relids)) + { + new_nullable_relids = bms_difference(new_nullable_relids, + parent_rel->relids); + new_nullable_relids = bms_add_members(new_nullable_relids, + child_rel->relids); + } + + (void) add_eq_member(cur_ec, child_expr, + new_relids, new_nullable_relids, true, cur_em->em_datatype); } } @@ -1921,12 +2000,12 @@ add_child_rel_equivalences(PlannerInfo *root, /* * mutate_eclass_expressions * Apply an expression tree mutator to all expressions stored in - * equivalence classes. + * equivalence classes (but ignore child exprs unless include_child_exprs). * * This is a bit of a hack ... it's currently needed only by planagg.c, * which needs to do a global search-and-replace of MIN/MAX Aggrefs * after eclasses are already set up. Without changing the eclasses too, - * subsequent matching of ORDER BY clauses would fail. + * subsequent matching of ORDER BY and DISTINCT clauses would fail. * * Note that we assume the mutation won't affect relation membership or any * other properties we keep track of (which is a bit bogus, but by the time @@ -1936,7 +2015,8 @@ add_child_rel_equivalences(PlannerInfo *root, void mutate_eclass_expressions(PlannerInfo *root, Node *(*mutator) (), - void *context) + void *context, + bool include_child_exprs) { ListCell *lc1; @@ -1949,6 +2029,9 @@ mutate_eclass_expressions(PlannerInfo *root, { EquivalenceMember *cur_em = (EquivalenceMember *) lfirst(lc2); + if (cur_em->em_is_child && !include_child_exprs) + continue; /* ignore children unless requested */ + cur_em->em_expr = (Expr *) mutator((Node *) cur_em->em_expr, context); } @@ -1957,26 +2040,36 @@ mutate_eclass_expressions(PlannerInfo *root, /* - * generate_implied_equalities_for_indexcol - * Create EC-derived joinclauses usable with a specific index column. + * generate_implied_equalities_for_column + * Create EC-derived joinclauses usable with a specific column. + * + * This is used by indxpath.c to extract potentially indexable joinclauses + * from ECs, and can be used by foreign data wrappers for similar purposes. + * We assume that only expressions in Vars of a single table are of interest, + * but the caller provides a callback function to identify exactly which + * such expressions it would like to know about. * - * We assume that any given index column could appear in only one EC. + * We assume that any given table/index column could appear in only one EC. * (This should be true in all but the most pathological cases, and if it * isn't, we stop on the first match anyway.) Therefore, what we return - * is a redundant list of clauses equating the index column to each of + * is a redundant list of clauses equating the table/index column to each of * the other-relation values it is known to be equal to. Any one of - * these clauses can be used to create a parameterized indexscan, and there - * is no value in using more than one. (But it *is* worthwhile to create + * these clauses can be used to create a parameterized path, and there + * is no value in using more than one. (But it *is* worthwhile to create * a separate parameterized path for each one, since that leads to different * join orders.) + * + * The caller can pass a Relids set of rels we aren't interested in joining + * to, so as to save the work of creating useless clauses. */ List * -generate_implied_equalities_for_indexcol(PlannerInfo *root, - IndexOptInfo *index, - int indexcol) +generate_implied_equalities_for_column(PlannerInfo *root, + RelOptInfo *rel, + ec_matches_callback_type callback, + void *callback_arg, + Relids prohibited_rels) { List *result = NIL; - RelOptInfo *rel = index->rel; bool is_child_rel = (rel->reloptkind == RELOPT_OTHER_MEMBER_REL); Index parent_relid; ListCell *lc1; @@ -2009,12 +2102,12 @@ generate_implied_equalities_for_indexcol(PlannerInfo *root, continue; /* - * Scan members, looking for a match to the indexable column. Note - * that child EC members are considered, but only when they belong to - * the target relation. (Unlike regular members, the same expression + * Scan members, looking for a match to the target column. Note that + * child EC members are considered, but only when they belong to the + * target relation. (Unlike regular members, the same expression * could be a child member of more than one EC. Therefore, it's - * potentially order-dependent which EC a child relation's index - * column gets matched to. This is annoying but it only happens in + * potentially order-dependent which EC a child relation's target + * column gets matched to. This is annoying but it only happens in * corner cases, so for now we live with just reporting the first * match. See also get_eclass_for_sort_expr.) */ @@ -2023,8 +2116,7 @@ generate_implied_equalities_for_indexcol(PlannerInfo *root, { cur_em = (EquivalenceMember *) lfirst(lc2); if (bms_equal(cur_em->em_relids, rel->relids) && - eclass_member_matches_indexcol(cur_ec, cur_em, - index, indexcol)) + callback(root, rel, cur_ec, cur_em, callback_arg)) break; cur_em = NULL; } @@ -2050,6 +2142,10 @@ generate_implied_equalities_for_indexcol(PlannerInfo *root, bms_overlap(other_em->em_relids, rel->relids)) continue; + /* Forget it if caller doesn't want joins to this rel */ + if (bms_overlap(other_em->em_relids, prohibited_rels)) + continue; + /* * Also, if this is a child rel, avoid generating a useless join * to its parent rel. @@ -2090,7 +2186,7 @@ generate_implied_equalities_for_indexcol(PlannerInfo *root, * a joinclause involving the two given relations. * * This is essentially a very cut-down version of - * generate_join_implied_equalities(). Note it's OK to occasionally say "yes" + * generate_join_implied_equalities(). Note it's OK to occasionally say "yes" * incorrectly. Hence we don't bother with details like whether the lack of a * cross-type operator might prevent the clause from actually being generated. */ @@ -2126,7 +2222,7 @@ have_relevant_eclass_joinclause(PlannerInfo *root, * OK as a possibly-overoptimistic heuristic. * * We don't test ec_has_const either, even though a const eclass won't - * generate real join clauses. This is because if we had "WHERE a.x = + * generate real join clauses. This is because if we had "WHERE a.x = * b.y and a.x = 42", it is worth considering a join between a and b, * since the join result is likely to be small even though it'll end * up being an unqualified nestloop. @@ -2183,7 +2279,7 @@ has_relevant_eclass_joinclause(PlannerInfo *root, RelOptInfo *rel1) * against the specified relation. * * This is just a heuristic test and doesn't have to be exact; it's better - * to say "yes" incorrectly than "no". Hence we don't bother with details + * to say "yes" incorrectly than "no". Hence we don't bother with details * like whether the lack of a cross-type operator might prevent the clause * from actually being generated. */ @@ -2204,7 +2300,7 @@ eclass_useful_for_merging(EquivalenceClass *eclass, /* * Note we don't test ec_broken; if we did, we'd need a separate code path - * to look through ec_sources. Checking the members anyway is OK as a + * to look through ec_sources. Checking the members anyway is OK as a * possibly-overoptimistic heuristic. */ diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c index 2e8ccd0578..42dcb111ae 100644 --- a/src/backend/optimizer/path/indxpath.c +++ b/src/backend/optimizer/path/indxpath.c @@ -4,7 +4,7 @@ * Routines to determine which indexes are usable for scanning a * given relation, and create Paths accordingly. * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -78,6 +78,13 @@ typedef struct Bitmapset *clauseids; /* quals+preds represented as a bitmapset */ } PathClauseUsage; +/* Callback argument for ec_member_matches_indexcol */ +typedef struct +{ + IndexOptInfo *index; /* index we're considering */ + int indexcol; /* index column we want to match to */ +} ec_member_matches_arg; + static void consider_index_join_clauses(PlannerInfo *root, RelOptInfo *rel, IndexOptInfo *index, @@ -85,13 +92,26 @@ static void consider_index_join_clauses(PlannerInfo *root, RelOptInfo *rel, IndexClauseSet *jclauseset, IndexClauseSet *eclauseset, List **bitindexpaths); -static void expand_eclass_clause_combinations(PlannerInfo *root, - RelOptInfo *rel, - IndexOptInfo *index, - int thiscol, int lastcol, - IndexClauseSet *clauseset, - IndexClauseSet *eclauseset, - List **bitindexpaths); +static void consider_index_join_outer_rels(PlannerInfo *root, RelOptInfo *rel, + IndexOptInfo *index, + IndexClauseSet *rclauseset, + IndexClauseSet *jclauseset, + IndexClauseSet *eclauseset, + List **bitindexpaths, + List *indexjoinclauses, + int considered_clauses, + List **considered_relids); +static void get_join_index_paths(PlannerInfo *root, RelOptInfo *rel, + IndexOptInfo *index, + IndexClauseSet *rclauseset, + IndexClauseSet *jclauseset, + IndexClauseSet *eclauseset, + List **bitindexpaths, + Relids relids, + List **considered_relids); +static bool eclass_already_used(EquivalenceClass *parent_ec, Relids oldrelids, + List *indexjoinclauses); +static bool bms_equal_any(Relids relids, List *relids_list); static void get_index_paths(PlannerInfo *root, RelOptInfo *rel, IndexOptInfo *index, IndexClauseSet *clauses, List **bitindexpaths); @@ -101,7 +121,8 @@ static List *build_index_paths(PlannerInfo *root, RelOptInfo *rel, SaOpControl saop_control, ScanTypeControl scantype); static List *build_paths_for_OR(PlannerInfo *root, RelOptInfo *rel, List *clauses, List *other_clauses); -static List *drop_indexable_join_clauses(RelOptInfo *rel, List *clauses); +static List *generate_bitmap_or_paths(PlannerInfo *root, RelOptInfo *rel, + List *clauses, List *other_clauses); static Path *choose_bitmap_and(PlannerInfo *root, RelOptInfo *rel, List *paths); static int path_usage_comparator(const void *a, const void *b); @@ -147,6 +168,9 @@ static void match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, List **clause_columns_p); static Expr *match_clause_to_ordering_op(IndexOptInfo *index, int indexcol, Expr *clause, Oid pk_opfamily); +static bool ec_member_matches_indexcol(PlannerInfo *root, RelOptInfo *rel, + EquivalenceClass *ec, EquivalenceMember *em, + void *arg); static bool match_boolean_index_clause(Node *clause, int indexcol, IndexOptInfo *index); static bool match_special_index_operator(Expr *clause, @@ -194,6 +218,15 @@ static Const *string_to_const(const char *str, Oid datatype); * 'rel' is the relation for which we want to generate index paths * * Note: check_partial_indexes() must have been run previously for this rel. + * + * Note: in cases involving LATERAL references in the relation's tlist, it's + * possible that rel->lateral_relids is nonempty. Currently, we include + * lateral_relids into the parameterization reported for each path, but don't + * take it into account otherwise. The fact that any such rels *must* be + * available as parameter sources perhaps should influence our choices of + * index quals ... but for now, it doesn't seem worth troubling over. + * In particular, comments below about "unparameterized" paths should be read + * as meaning "unparameterized so far as the indexquals are concerned". */ void create_index_paths(PlannerInfo *root, RelOptInfo *rel) @@ -205,7 +238,7 @@ create_index_paths(PlannerInfo *root, RelOptInfo *rel) IndexClauseSet rclauseset; IndexClauseSet jclauseset; IndexClauseSet eclauseset; - ListCell *ilist; + ListCell *lc; /* Skip the whole mess if no indexes */ if (rel->indexlist == NIL) @@ -215,9 +248,9 @@ create_index_paths(PlannerInfo *root, RelOptInfo *rel) bitindexpaths = bitjoinpaths = joinorclauses = NIL; /* Examine each index in turn */ - foreach(ilist, rel->indexlist) + foreach(lc, rel->indexlist) { - IndexOptInfo *index = (IndexOptInfo *) lfirst(ilist); + IndexOptInfo *index = (IndexOptInfo *) lfirst(lc); /* Protect limited-size array in IndexClauseSets */ Assert(index->ncolumns <= INDEX_MAX_KEYS); @@ -237,7 +270,7 @@ create_index_paths(PlannerInfo *root, RelOptInfo *rel) match_restriction_clauses_to_index(rel, index, &rclauseset); /* - * Build index paths from the restriction clauses. These will be + * Build index paths from the restriction clauses. These will be * non-parameterized paths. Plain paths go directly to add_path(), * bitmap paths are added to bitindexpaths to be handled below. */ @@ -245,10 +278,10 @@ create_index_paths(PlannerInfo *root, RelOptInfo *rel) &bitindexpaths); /* - * Identify the join clauses that can match the index. For the moment - * we keep them separate from the restriction clauses. Note that this - * finds only "loose" join clauses that have not been merged into - * EquivalenceClasses. Also, collect join OR clauses for later. + * Identify the join clauses that can match the index. For the moment + * we keep them separate from the restriction clauses. Note that this + * step finds only "loose" join clauses that have not been merged into + * EquivalenceClasses. Also, collect join OR clauses for later. */ MemSet(&jclauseset, 0, sizeof(jclauseset)); match_join_clauses_to_index(root, rel, index, @@ -259,11 +292,12 @@ create_index_paths(PlannerInfo *root, RelOptInfo *rel) * the index. */ MemSet(&eclauseset, 0, sizeof(eclauseset)); - match_eclass_clauses_to_index(root, index, &eclauseset); + match_eclass_clauses_to_index(root, index, + &eclauseset); /* - * If we found any plain or eclass join clauses, decide what to do - * with 'em. + * If we found any plain or eclass join clauses, build parameterized + * index paths using them. */ if (jclauseset.nonempty || eclauseset.nonempty) consider_index_join_clauses(root, rel, index, @@ -278,8 +312,7 @@ create_index_paths(PlannerInfo *root, RelOptInfo *rel) * restriction list. Add these to bitindexpaths. */ indexpaths = generate_bitmap_or_paths(root, rel, - rel->baserestrictinfo, NIL, - false); + rel->baserestrictinfo, NIL); bitindexpaths = list_concat(bitindexpaths, indexpaths); /* @@ -287,8 +320,7 @@ create_index_paths(PlannerInfo *root, RelOptInfo *rel) * the joinclause list. Add these to bitjoinpaths. */ indexpaths = generate_bitmap_or_paths(root, rel, - joinorclauses, rel->baserestrictinfo, - false); + joinorclauses, rel->baserestrictinfo); bitjoinpaths = list_concat(bitjoinpaths, indexpaths); /* @@ -304,31 +336,83 @@ create_index_paths(PlannerInfo *root, RelOptInfo *rel) BitmapHeapPath *bpath; bitmapqual = choose_bitmap_and(root, rel, bitindexpaths); - bpath = create_bitmap_heap_path(root, rel, bitmapqual, NULL, 1.0); + bpath = create_bitmap_heap_path(root, rel, bitmapqual, + rel->lateral_relids, 1.0); add_path(rel, (Path *) bpath); } /* - * Likewise, if we found anything usable, generate a BitmapHeapPath for - * the most promising combination of join bitmap index paths. Note there - * will be only one such path no matter how many join clauses are - * available. (XXX is that good enough, or do we need to consider even - * more paths for different subsets of possible join partners? Also, - * should we add in restriction bitmap paths as well?) + * Likewise, if we found anything usable, generate BitmapHeapPaths for the + * most promising combinations of join bitmap index paths. Our strategy + * is to generate one such path for each distinct parameterization seen + * among the available bitmap index paths. This may look pretty + * expensive, but usually there won't be very many distinct + * parameterizations. (This logic is quite similar to that in + * consider_index_join_clauses, but we're working with whole paths not + * individual clauses.) */ if (bitjoinpaths != NIL) { - Path *bitmapqual; - Relids required_outer; - double loop_count; - BitmapHeapPath *bpath; + List *path_outer; + List *all_path_outers; + ListCell *lc; - bitmapqual = choose_bitmap_and(root, rel, bitjoinpaths); - required_outer = get_bitmap_tree_required_outer(bitmapqual); - loop_count = get_loop_count(root, required_outer); - bpath = create_bitmap_heap_path(root, rel, bitmapqual, - required_outer, loop_count); - add_path(rel, (Path *) bpath); + /* + * path_outer holds the parameterization of each path in bitjoinpaths + * (to save recalculating that several times), while all_path_outers + * holds all distinct parameterization sets. + */ + path_outer = all_path_outers = NIL; + foreach(lc, bitjoinpaths) + { + Path *path = (Path *) lfirst(lc); + Relids required_outer; + + required_outer = get_bitmap_tree_required_outer(path); + path_outer = lappend(path_outer, required_outer); + if (!bms_equal_any(required_outer, all_path_outers)) + all_path_outers = lappend(all_path_outers, required_outer); + } + + /* Now, for each distinct parameterization set ... */ + foreach(lc, all_path_outers) + { + Relids max_outers = (Relids) lfirst(lc); + List *this_path_set; + Path *bitmapqual; + Relids required_outer; + double loop_count; + BitmapHeapPath *bpath; + ListCell *lcp; + ListCell *lco; + + /* Identify all the bitmap join paths needing no more than that */ + this_path_set = NIL; + forboth(lcp, bitjoinpaths, lco, path_outer) + { + Path *path = (Path *) lfirst(lcp); + Relids p_outers = (Relids) lfirst(lco); + + if (bms_is_subset(p_outers, max_outers)) + this_path_set = lappend(this_path_set, path); + } + + /* + * Add in restriction bitmap paths, since they can be used + * together with any join paths. + */ + this_path_set = list_concat(this_path_set, bitindexpaths); + + /* Select best AND combination for this parameterization */ + bitmapqual = choose_bitmap_and(root, rel, this_path_set); + + /* And push that path into the mix */ + required_outer = get_bitmap_tree_required_outer(bitmapqual); + loop_count = get_loop_count(root, required_outer); + bpath = create_bitmap_heap_path(root, rel, bitmapqual, + required_outer, loop_count); + add_path(rel, (Path *) bpath); + } } } @@ -346,9 +430,6 @@ create_index_paths(PlannerInfo *root, RelOptInfo *rel) * 'jclauseset' is the collection of indexable simple join clauses * 'eclauseset' is the collection of indexable clauses from EquivalenceClasses * '*bitindexpaths' is the list to add bitmap paths to - * - * Note: this changes the clause lists contained in the passed clausesets, - * but we don't care since the caller is done with them. */ static void consider_index_join_clauses(PlannerInfo *root, RelOptInfo *rel, @@ -358,156 +439,278 @@ consider_index_join_clauses(PlannerInfo *root, RelOptInfo *rel, IndexClauseSet *eclauseset, List **bitindexpaths) { - IndexClauseSet clauseset; - int last_eclass_col; + int considered_clauses = 0; + List *considered_relids = NIL; int indexcol; /* - * We can always include any restriction clauses in the index clauses. - * However, it's not obvious which subsets of the join clauses are worth - * generating paths from, and it's unlikely that considering every - * possible subset is worth the cycles. Our current heuristic is based on - * the index columns, with the idea that later index columns are less - * useful than earlier ones; therefore it's unlikely to be worth trying - * combinations that would remove a clause from an earlier index column - * while adding one to a later column. Also, we know that all the eclass - * clauses for a particular column are redundant, so we should use only - * one of them. However, eclass clauses will always represent equality - * which is the strongest type of index constraint, so those are - * high-value and we should try every available combination when we have - * eclass clauses for more than one column. Furthermore, it's unlikely to - * be useful to combine an eclass clause with non-eclass clauses for the - * same index column. These considerations lead to the following - * heuristics: - * - * First, start with the restriction clauses, and add on all simple join - * clauses for column 1. If there are any such join clauses, generate - * paths with this collection of clauses. Then, if there are eclass - * clauses for column 1, generate paths with each one of them replacing - * any other clauses we have for column 1. + * The strategy here is to identify every potentially useful set of outer + * rels that can provide indexable join clauses. For each such set, + * select all the join clauses available from those outer rels, add on all + * the indexable restriction clauses, and generate plain and/or bitmap + * index paths for that set of clauses. This is based on the assumption + * that it's always better to apply a clause as an indexqual than as a + * filter (qpqual); which is where an available clause would end up being + * applied if we omit it from the indexquals. * - * Next, add on all simple join clauses for column 2. If there are any - * such join clauses, generate paths with this collection. If there are - * eclass clauses for columns 1 or 2, generate paths with each such clause - * replacing other clauses for its index column, including cases where we - * use restriction or simple join clauses for one column and an eclass - * clause for the other. + * This looks expensive, but in most practical cases there won't be very + * many distinct sets of outer rels to consider. As a safety valve when + * that's not true, we use a heuristic: limit the number of outer rel sets + * considered to a multiple of the number of clauses considered. (We'll + * always consider using each individual join clause, though.) * - * Repeat for each additional index column. + * For simplicity in selecting relevant clauses, we represent each set of + * outer rels as a maximum set of clause_relids --- that is, the indexed + * relation itself is also included in the relids set. considered_relids + * lists all relids sets we've already tried. */ + for (indexcol = 0; indexcol < index->ncolumns; indexcol++) + { + /* Consider each applicable simple join clause */ + considered_clauses += list_length(jclauseset->indexclauses[indexcol]); + consider_index_join_outer_rels(root, rel, index, + rclauseset, jclauseset, eclauseset, + bitindexpaths, + jclauseset->indexclauses[indexcol], + considered_clauses, + &considered_relids); + /* Consider each applicable eclass join clause */ + considered_clauses += list_length(eclauseset->indexclauses[indexcol]); + consider_index_join_outer_rels(root, rel, index, + rclauseset, jclauseset, eclauseset, + bitindexpaths, + eclauseset->indexclauses[indexcol], + considered_clauses, + &considered_relids); + } +} - /* Set up working set with just the restriction clauses */ - memcpy(&clauseset, rclauseset, sizeof(clauseset)); - /* Even if it's empty right now, it won't be by the time we use it */ - clauseset.nonempty = true; +/* + * consider_index_join_outer_rels + * Generate parameterized paths based on clause relids in the clause list. + * + * Workhorse for consider_index_join_clauses; see notes therein for rationale. + * + * 'rel', 'index', 'rclauseset', 'jclauseset', 'eclauseset', and + * 'bitindexpaths' as above + * 'indexjoinclauses' is a list of RestrictInfos for join clauses + * 'considered_clauses' is the total number of clauses considered (so far) + * '*considered_relids' is a list of all relids sets already considered + */ +static void +consider_index_join_outer_rels(PlannerInfo *root, RelOptInfo *rel, + IndexOptInfo *index, + IndexClauseSet *rclauseset, + IndexClauseSet *jclauseset, + IndexClauseSet *eclauseset, + List **bitindexpaths, + List *indexjoinclauses, + int considered_clauses, + List **considered_relids) +{ + ListCell *lc; - last_eclass_col = -1; - for (indexcol = 0; indexcol < index->ncolumns; indexcol++) + /* Examine relids of each joinclause in the given list */ + foreach(lc, indexjoinclauses) { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + Relids clause_relids = rinfo->clause_relids; + ListCell *lc2; + + /* If we already tried its relids set, no need to do so again */ + if (bms_equal_any(clause_relids, *considered_relids)) + continue; + /* - * If we don't have either simple join clauses or eclass clauses for - * this column, no new paths can be created in this iteration. + * Generate the union of this clause's relids set with each + * previously-tried set. This ensures we try this clause along with + * every interesting subset of previous clauses. However, to avoid + * exponential growth of planning time when there are many clauses, + * limit the number of relid sets accepted to 10 * considered_clauses. + * + * Note: get_join_index_paths adds entries to *considered_relids, but + * it prepends them to the list, so that we won't visit new entries + * during the inner foreach loop. No real harm would be done if we + * did, since the subset check would reject them; but it would waste + * some cycles. */ - if (jclauseset->indexclauses[indexcol] == NIL && - eclauseset->indexclauses[indexcol] == NIL) - continue; + foreach(lc2, *considered_relids) + { + Relids oldrelids = (Relids) lfirst(lc2); - /* Add any simple join clauses to the working set */ - clauseset.indexclauses[indexcol] = - list_concat(clauseset.indexclauses[indexcol], - jclauseset->indexclauses[indexcol]); + /* + * If either is a subset of the other, no new set is possible. + * This isn't a complete test for redundancy, but it's easy and + * cheap. get_join_index_paths will check more carefully if we + * already generated the same relids set. + */ + if (bms_subset_compare(clause_relids, oldrelids) != BMS_DIFFERENT) + continue; - /* Set recursion depth to reach last col with eclass clauses */ - if (eclauseset->indexclauses[indexcol] != NIL) - last_eclass_col = indexcol; + /* + * If this clause was derived from an equivalence class, the + * clause list may contain other clauses derived from the same + * eclass. We should not consider that combining this clause with + * one of those clauses generates a usefully different + * parameterization; so skip if any clause derived from the same + * eclass would already have been included when using oldrelids. + */ + if (rinfo->parent_ec && + eclass_already_used(rinfo->parent_ec, oldrelids, + indexjoinclauses)) + continue; - /* Do we have eclass clauses for any column now under consideration? */ - if (last_eclass_col >= 0) - { - /* Yes, so recursively generate all eclass clause combinations */ - expand_eclass_clause_combinations(root, rel, index, - 0, last_eclass_col, - &clauseset, eclauseset, - bitindexpaths); - } - else - { - /* No, consider the newly-enlarged set of simple join clauses */ - get_index_paths(root, rel, index, &clauseset, bitindexpaths); + /* + * If the number of relid sets considered exceeds our heuristic + * limit, stop considering combinations of clauses. We'll still + * consider the current clause alone, though (below this loop). + */ + if (list_length(*considered_relids) >= 10 * considered_clauses) + break; + + /* OK, try the union set */ + get_join_index_paths(root, rel, index, + rclauseset, jclauseset, eclauseset, + bitindexpaths, + bms_union(clause_relids, oldrelids), + considered_relids); } + + /* Also try this set of relids by itself */ + get_join_index_paths(root, rel, index, + rclauseset, jclauseset, eclauseset, + bitindexpaths, + clause_relids, + considered_relids); } } /* - * expand_eclass_clause_combinations - * Generate all combinations of eclass join clauses for first N columns, - * and construct parameterized index paths for each combination. + * get_join_index_paths + * Generate index paths using clauses from the specified outer relations. + * In addition to generating paths, relids is added to *considered_relids + * if not already present. * * Workhorse for consider_index_join_clauses; see notes therein for rationale. - * It's convenient to use recursion to implement the enumeration, since we - * can have at most INDEX_MAX_KEYS recursion levels. * - * 'rel', 'index', 'eclauseset', 'bitindexpaths' as above - * 'thiscol' is the current index column number/recursion level - * 'lastcol' is the last index column we should consider eclass clauses for - * 'clauseset' is the current collection of indexable clauses + * 'rel', 'index', 'rclauseset', 'jclauseset', 'eclauseset', + * 'bitindexpaths', 'considered_relids' as above + * 'relids' is the current set of relids to consider (the target rel plus + * one or more outer rels) */ static void -expand_eclass_clause_combinations(PlannerInfo *root, RelOptInfo *rel, - IndexOptInfo *index, - int thiscol, int lastcol, - IndexClauseSet *clauseset, - IndexClauseSet *eclauseset, - List **bitindexpaths) +get_join_index_paths(PlannerInfo *root, RelOptInfo *rel, + IndexOptInfo *index, + IndexClauseSet *rclauseset, + IndexClauseSet *jclauseset, + IndexClauseSet *eclauseset, + List **bitindexpaths, + Relids relids, + List **considered_relids) { - List *save_clauses; - ListCell *lc; + IndexClauseSet clauseset; + int indexcol; - /* If past last eclass column, end the recursion and generate paths */ - if (thiscol > lastcol) - { - get_index_paths(root, rel, index, clauseset, bitindexpaths); + /* If we already considered this relids set, don't repeat the work */ + if (bms_equal_any(relids, *considered_relids)) return; - } - /* If no eclass clauses to consider for this column, just recurse */ - if (eclauseset->indexclauses[thiscol] == NIL) + /* Identify indexclauses usable with this relids set */ + MemSet(&clauseset, 0, sizeof(clauseset)); + + for (indexcol = 0; indexcol < index->ncolumns; indexcol++) { - Assert(thiscol < lastcol); - expand_eclass_clause_combinations(root, rel, index, - thiscol + 1, lastcol, - clauseset, eclauseset, - bitindexpaths); - return; + ListCell *lc; + + /* First find applicable simple join clauses */ + foreach(lc, jclauseset->indexclauses[indexcol]) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (bms_is_subset(rinfo->clause_relids, relids)) + clauseset.indexclauses[indexcol] = + lappend(clauseset.indexclauses[indexcol], rinfo); + } + + /* + * Add applicable eclass join clauses. The clauses generated for each + * column are redundant (cf generate_implied_equalities_for_column), + * so we need at most one. This is the only exception to the general + * rule of using all available index clauses. + */ + foreach(lc, eclauseset->indexclauses[indexcol]) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (bms_is_subset(rinfo->clause_relids, relids)) + { + clauseset.indexclauses[indexcol] = + lappend(clauseset.indexclauses[indexcol], rinfo); + break; + } + } + + /* Add restriction clauses (this is nondestructive to rclauseset) */ + clauseset.indexclauses[indexcol] = + list_concat(clauseset.indexclauses[indexcol], + rclauseset->indexclauses[indexcol]); + + if (clauseset.indexclauses[indexcol] != NIL) + clauseset.nonempty = true; } - /* We'll momentarily save and restore the list of non-eclass clauses */ - save_clauses = clauseset->indexclauses[thiscol]; + /* We should have found something, else caller passed silly relids */ + Assert(clauseset.nonempty); - /* If we have non-eclass clauses for this column, first try with those */ - if (save_clauses) - expand_eclass_clause_combinations(root, rel, index, - thiscol + 1, lastcol, - clauseset, eclauseset, - bitindexpaths); + /* Build index path(s) using the collected set of clauses */ + get_index_paths(root, rel, index, &clauseset, bitindexpaths); - /* For each eclass clause alternative ... */ - foreach(lc, eclauseset->indexclauses[thiscol]) + /* + * Remember we considered paths for this set of relids. We use lcons not + * lappend to avoid confusing the loop in consider_index_join_outer_rels. + */ + *considered_relids = lcons(relids, *considered_relids); +} + +/* + * eclass_already_used + * True if any join clause usable with oldrelids was generated from + * the specified equivalence class. + */ +static bool +eclass_already_used(EquivalenceClass *parent_ec, Relids oldrelids, + List *indexjoinclauses) +{ + ListCell *lc; + + foreach(lc, indexjoinclauses) { RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); - /* Replace any existing clauses with the eclass clause */ - clauseset->indexclauses[thiscol] = list_make1(rinfo); - - /* Recurse to advance to next column */ - expand_eclass_clause_combinations(root, rel, index, - thiscol + 1, lastcol, - clauseset, eclauseset, - bitindexpaths); + if (rinfo->parent_ec == parent_ec && + bms_is_subset(rinfo->clause_relids, oldrelids)) + return true; } + return false; +} + +/* + * bms_equal_any + * True if relids is bms_equal to any member of relids_list + * + * Perhaps this should be in bitmapset.c someday. + */ +static bool +bms_equal_any(Relids relids, List *relids_list) +{ + ListCell *lc; - /* Restore previous list contents */ - clauseset->indexclauses[thiscol] = save_clauses; + foreach(lc, relids_list) + { + if (bms_equal(relids, (Relids) lfirst(lc))) + return true; + } + return false; } @@ -519,7 +722,7 @@ expand_eclass_clause_combinations(PlannerInfo *root, RelOptInfo *rel, * bitmap indexpaths are added to *bitindexpaths for later processing. * * This is a fairly simple frontend to build_index_paths(). Its reason for - * existence is mainly to handle ScalarArrayOpExpr quals properly. If the + * existence is mainly to handle ScalarArrayOpExpr quals properly. If the * index AM supports them natively, we should just include them in simple * index paths. If not, we should exclude them while building simple index * paths, and then make a separate attempt to include them in bitmap paths. @@ -533,7 +736,7 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel, ListCell *lc; /* - * Build simple index paths using the clauses. Allow ScalarArrayOpExpr + * Build simple index paths using the clauses. Allow ScalarArrayOpExpr * clauses only if the index AM supports them natively. */ indexpaths = build_index_paths(root, rel, @@ -545,7 +748,7 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel, * Submit all the ones that can form plain IndexScan plans to add_path. (A * plain IndexPath can represent either a plain IndexScan or an * IndexOnlyScan, but for our purposes here that distinction does not - * matter. However, some of the indexes might support only bitmap scans, + * matter. However, some of the indexes might support only bitmap scans, * and those we mustn't submit to add_path here.) * * Also, pick out the ones that are usable as bitmap scans. For that, we @@ -589,7 +792,7 @@ get_index_paths(PlannerInfo *root, RelOptInfo *rel, * We return a list of paths because (1) this routine checks some cases * that should cause us to not generate any IndexPath, and (2) in some * cases we want to consider both a forward and a backward scan, so as - * to obtain both sort orders. Note that the paths are just returned + * to obtain both sort orders. Note that the paths are just returned * to the caller and not immediately fed to add_path(). * * At top level, useful_predicate should be exactly the index's predOK flag @@ -631,6 +834,7 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel, List *index_pathkeys; List *useful_pathkeys; bool found_clause; + bool found_lower_saop_clause; bool pathkeys_possibly_useful; bool index_is_ordered; bool index_only_scan; @@ -668,13 +872,22 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel, * if saop_control is SAOP_REQUIRE, it has to be a ScalarArrayOpExpr * clause. * + * found_lower_saop_clause is set true if there's a ScalarArrayOpExpr + * index clause for a non-first index column. This prevents us from + * assuming that the scan result is ordered. (Actually, the result is + * still ordered if there are equality constraints for all earlier + * columns, but it seems too expensive and non-modular for this code to be + * aware of that refinement.) + * * We also build a Relids set showing which outer rels are required by the - * selected clauses. + * selected clauses. Any lateral_relids are included in that, but not + * otherwise accounted for. */ index_clauses = NIL; clause_columns = NIL; found_clause = false; - outer_relids = NULL; + found_lower_saop_clause = false; + outer_relids = bms_copy(rel->lateral_relids); for (indexcol = 0; indexcol < index->ncolumns; indexcol++) { ListCell *lc; @@ -689,6 +902,8 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel, if (saop_control == SAOP_PER_AM && !index->amsearcharray) continue; found_clause = true; + if (indexcol > 0) + found_lower_saop_clause = true; } else { @@ -725,9 +940,11 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel, /* * 2. Compute pathkeys describing index's ordering, if any, then see how * many of them are actually useful for this query. This is not relevant - * if we are only trying to build bitmap indexscans. + * if we are only trying to build bitmap indexscans, nor if we have to + * assume the scan is unordered. */ pathkeys_possibly_useful = (scantype != ST_BITMAPSCAN && + !found_lower_saop_clause && has_useful_pathkeys(root, rel)); index_is_ordered = (index->sortopfamily != NULL); if (index_is_ordered && pathkeys_possibly_useful) @@ -758,7 +975,7 @@ build_index_paths(PlannerInfo *root, RelOptInfo *rel, } /* - * 3. Check if an index-only scan is possible. If we're not building + * 3. Check if an index-only scan is possible. If we're not building * plain indexscans, this isn't relevant since bitmap scans don't support * index data retrieval anyway. */ @@ -863,13 +1080,13 @@ build_paths_for_OR(PlannerInfo *root, RelOptInfo *rel, continue; /* - * Ignore partial indexes that do not match the query. If a partial + * Ignore partial indexes that do not match the query. If a partial * index is marked predOK then we know it's OK. Otherwise, we have to * test whether the added clauses are sufficient to imply the * predicate. If so, we can use the index in the current context. * * We set useful_predicate to true iff the predicate was proven using - * the current set of clauses. This is needed to prevent matching a + * the current set of clauses. This is needed to prevent matching a * predOK index to an arm of an OR, which would be a legal but * pointlessly inefficient plan. (A better plan will be generated by * just scanning the predOK index alone, no OR.) @@ -936,16 +1153,10 @@ build_paths_for_OR(PlannerInfo *root, RelOptInfo *rel, * other_clauses is a list of additional clauses that can be assumed true * for the purpose of generating indexquals, but are not to be searched for * ORs. (See build_paths_for_OR() for motivation.) - * - * If restriction_only is true, ignore OR elements that are join clauses. - * When using this feature it is caller's responsibility that neither clauses - * nor other_clauses contain any join clauses that are not ORs, as we do not - * re-filter those lists. */ -List * +static List * generate_bitmap_or_paths(PlannerInfo *root, RelOptInfo *rel, - List *clauses, List *other_clauses, - bool restriction_only) + List *clauses, List *other_clauses) { List *result = NIL; List *all_clauses; @@ -984,9 +1195,6 @@ generate_bitmap_or_paths(PlannerInfo *root, RelOptInfo *rel, { List *andargs = ((BoolExpr *) orarg)->args; - if (restriction_only) - andargs = drop_indexable_join_clauses(rel, andargs); - indlist = build_paths_for_OR(root, rel, andargs, all_clauses); @@ -995,8 +1203,7 @@ generate_bitmap_or_paths(PlannerInfo *root, RelOptInfo *rel, indlist = list_concat(indlist, generate_bitmap_or_paths(root, rel, andargs, - all_clauses, - restriction_only)); + all_clauses)); } else { @@ -1006,9 +1213,6 @@ generate_bitmap_or_paths(PlannerInfo *root, RelOptInfo *rel, Assert(!restriction_is_or_clause((RestrictInfo *) orarg)); orargs = list_make1(orarg); - if (restriction_only) - orargs = drop_indexable_join_clauses(rel, orargs); - indlist = build_paths_for_OR(root, rel, orargs, all_clauses); @@ -1046,41 +1250,13 @@ generate_bitmap_or_paths(PlannerInfo *root, RelOptInfo *rel, return result; } -/* - * drop_indexable_join_clauses - * Remove any indexable join clauses from the list. - * - * This is a helper for generate_bitmap_or_paths(). We leave OR clauses - * in the list whether they are joins or not, since we might be able to - * extract a restriction item from an OR list. It's safe to leave such - * clauses in the list because match_clauses_to_index() will ignore them, - * so there's no harm in passing such clauses to build_paths_for_OR(). - */ -static List * -drop_indexable_join_clauses(RelOptInfo *rel, List *clauses) -{ - List *result = NIL; - ListCell *lc; - - foreach(lc, clauses) - { - RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); - - Assert(IsA(rinfo, RestrictInfo)); - if (restriction_is_or_clause(rinfo) || - bms_is_subset(rinfo->clause_relids, rel->relids)) - result = lappend(result, rinfo); - } - return result; -} - /* * choose_bitmap_and * Given a nonempty list of bitmap paths, AND them into one path. * * This is a nontrivial decision since we can legally use any subset of the - * given path set. We want to choose a good tradeoff between selectivity + * given path set. We want to choose a good tradeoff between selectivity * and cost of computing the bitmap. * * The result is either a single one of the inputs, or a BitmapAndPath @@ -1107,12 +1283,12 @@ choose_bitmap_and(PlannerInfo *root, RelOptInfo *rel, List *paths) * In theory we should consider every nonempty subset of the given paths. * In practice that seems like overkill, given the crude nature of the * estimates, not to mention the possible effects of higher-level AND and - * OR clauses. Moreover, it's completely impractical if there are a large + * OR clauses. Moreover, it's completely impractical if there are a large * number of paths, since the work would grow as O(2^N). * * As a heuristic, we first check for paths using exactly the same sets of * WHERE clauses + index predicate conditions, and reject all but the - * cheapest-to-scan in any such group. This primarily gets rid of indexes + * cheapest-to-scan in any such group. This primarily gets rid of indexes * that include the interesting columns but also irrelevant columns. (In * situations where the DBA has gone overboard on creating variant * indexes, this can make for a very large reduction in the number of @@ -1132,14 +1308,15 @@ choose_bitmap_and(PlannerInfo *root, RelOptInfo *rel, List *paths) * costsize.c and clausesel.c aren't very smart about redundant clauses. * They will usually double-count the redundant clauses, producing a * too-small selectivity that makes a redundant AND step look like it - * reduces the total cost. Perhaps someday that code will be smarter and + * reduces the total cost. Perhaps someday that code will be smarter and * we can remove this limitation. (But note that this also defends * against flat-out duplicate input paths, which can happen because * match_join_clauses_to_index will find the same OR join clauses that - * create_or_index_quals has pulled OR restriction clauses out of.) + * extract_restriction_or_clauses has pulled OR restriction clauses out + * of.) * * For the same reason, we reject AND combinations in which an index - * predicate clause duplicates another clause. Here we find it necessary + * predicate clause duplicates another clause. Here we find it necessary * to be even stricter: we'll reject a partial index if any of its * predicate clauses are implied by the set of WHERE clauses and predicate * clauses used so far. This covers cases such as a condition "x = 42" @@ -1202,7 +1379,7 @@ choose_bitmap_and(PlannerInfo *root, RelOptInfo *rel, List *paths) /* * For each surviving index, consider it as an "AND group leader", and see * whether adding on any of the later indexes results in an AND path with - * cheaper total cost than before. Then take the cheapest AND group. + * cheaper total cost than before. Then take the cheapest AND group. */ for (i = 0; i < npaths; i++) { @@ -1489,8 +1666,7 @@ get_bitmap_tree_required_outer(Path *bitmapqual) * These are appended to the initial contents of *quals and *preds (hence * caller should initialize those to NIL). * - * This is sort of a simplified version of make_restrictinfo_from_bitmapqual; - * here, we are not trying to produce an accurate representation of the AND/OR + * Note we are not trying to produce an accurate representation of the AND/OR * semantics of the Path, but just find out all the base conditions used. * * The result lists contain pointers to the expressions used in the Path, @@ -1535,7 +1711,7 @@ find_indexpath_quals(Path *bitmapqual, List **quals, List **preds) /* * find_list_position * Return the given node's position (counting from 0) in the given - * list of nodes. If it's not equal() to any existing list member, + * list of nodes. If it's not equal() to any existing list member, * add it at the end, and return that position. */ static int @@ -1641,7 +1817,7 @@ check_index_only(RelOptInfo *rel, IndexOptInfo *index) * Since we produce parameterized paths before we've begun to generate join * relations, it's impossible to predict exactly how many times a parameterized * path will be iterated; we don't know the size of the relation that will be - * on the outside of the nestloop. However, we should try to account for + * on the outside of the nestloop. However, we should try to account for * multiple iterations somehow in costing the path. The heuristic embodied * here is to use the rowcount of the smallest other base relation needed in * the join clauses used by the path. (We could alternatively consider the @@ -1730,7 +1906,7 @@ match_join_clauses_to_index(PlannerInfo *root, RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); /* Check if clause can be moved to this rel */ - if (!join_clause_is_movable_to(rinfo, rel->relid)) + if (!join_clause_is_movable_to(rinfo, rel)) continue; /* Potentially usable, so see if it matches the index or is an OR */ @@ -1758,16 +1934,22 @@ match_eclass_clauses_to_index(PlannerInfo *root, IndexOptInfo *index, for (indexcol = 0; indexcol < index->ncolumns; indexcol++) { + ec_member_matches_arg arg; List *clauses; - clauses = generate_implied_equalities_for_indexcol(root, - index, - indexcol); + /* Generate clauses, skipping any that join to lateral_referencers */ + arg.index = index; + arg.indexcol = indexcol; + clauses = generate_implied_equalities_for_column(root, + index->rel, + ec_member_matches_indexcol, + (void *) &arg, + index->rel->lateral_referencers); /* * We have to check whether the results actually do match the index, * since for non-btree indexes the EC's equality operators might not - * be in the index opclass (cf eclass_member_matches_indexcol). + * be in the index opclass (cf ec_member_matches_indexcol). */ match_clauses_to_index(index, clauses, clauseset); } @@ -1850,7 +2032,7 @@ match_clause_to_index(IndexOptInfo *index, * doesn't involve a volatile function or a Var of the index's relation. * In particular, Vars belonging to other relations of the query are * accepted here, since a clause of that form can be used in a - * parameterized indexscan. It's the responsibility of higher code levels + * parameterized indexscan. It's the responsibility of higher code levels * to manage restriction and join clauses appropriately. * * Note: we do need to check for Vars of the index's relation on the @@ -1874,7 +2056,7 @@ match_clause_to_index(IndexOptInfo *index, * It is also possible to match RowCompareExpr clauses to indexes (but * currently, only btree indexes handle this). In this routine we will * report a match if the first column of the row comparison matches the - * target index column. This is sufficient to guarantee that some index + * target index column. This is sufficient to guarantee that some index * condition can be constructed from the RowCompareExpr --- whether the * remaining columns match the index too is considered in * adjust_rowcompare_for_index(). @@ -1912,7 +2094,7 @@ match_clause_to_indexcol(IndexOptInfo *index, bool plain_op; /* - * Never match pseudoconstants to indexes. (Normally this could not + * Never match pseudoconstants to indexes. (Normally this could not * happen anyway, since a pseudoconstant clause couldn't contain a Var, * but what if someone builds an expression index on a constant? It's not * totally unreasonable to do so with a partial index, either.) @@ -2196,7 +2378,7 @@ match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, * We allow any column of the index to match each pathkey; they * don't have to match left-to-right as you might expect. This is * correct for GiST, which is the sole existing AM supporting - * amcanorderbyop. We might need different logic in future for + * amcanorderbyop. We might need different logic in future for * other implementations. */ for (indexcol = 0; indexcol < index->ncolumns; indexcol++) @@ -2247,7 +2429,7 @@ match_pathkeys_to_index(IndexOptInfo *index, List *pathkeys, * Note that we currently do not consider the collation of the ordering * operator's result. In practical cases the result type will be numeric * and thus have no collation, and it's not very clear what to match to - * if it did have a collation. The index's collation should match the + * if it did have a collation. The index's collation should match the * ordering operator's input collation, not its result. * * If successful, return 'clause' as-is if the indexkey is on the left, @@ -2353,12 +2535,83 @@ match_clause_to_ordering_op(IndexOptInfo *index, void check_partial_indexes(PlannerInfo *root, RelOptInfo *rel) { - List *restrictinfo_list = rel->baserestrictinfo; - ListCell *ilist; + List *clauselist; + bool have_partial; + Relids otherrels; + ListCell *lc; + + /* + * Frequently, there will be no partial indexes, so first check to make + * sure there's something useful to do here. + */ + have_partial = false; + foreach(lc, rel->indexlist) + { + IndexOptInfo *index = (IndexOptInfo *) lfirst(lc); + + if (index->indpred == NIL) + continue; /* ignore non-partial indexes */ + + if (index->predOK) + continue; /* don't repeat work if already proven OK */ + + have_partial = true; + break; + } + if (!have_partial) + return; - foreach(ilist, rel->indexlist) + /* + * Construct a list of clauses that we can assume true for the purpose of + * proving the index(es) usable. Restriction clauses for the rel are + * always usable, and so are any join clauses that are "movable to" this + * rel. Also, we can consider any EC-derivable join clauses (which must + * be "movable to" this rel, by definition). + */ + clauselist = list_copy(rel->baserestrictinfo); + + /* Scan the rel's join clauses */ + foreach(lc, rel->joininfo) { - IndexOptInfo *index = (IndexOptInfo *) lfirst(ilist); + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + /* Check if clause can be moved to this rel */ + if (!join_clause_is_movable_to(rinfo, rel)) + continue; + + clauselist = lappend(clauselist, rinfo); + } + + /* + * Add on any equivalence-derivable join clauses. Computing the correct + * relid sets for generate_join_implied_equalities is slightly tricky + * because the rel could be a child rel rather than a true baserel, and in + * that case we must remove its parent's relid from all_baserels. + */ + if (rel->reloptkind == RELOPT_OTHER_MEMBER_REL) + { + /* Lookup parent->child translation data */ + AppendRelInfo *appinfo = find_childrel_appendrelinfo(root, rel); + + otherrels = bms_difference(root->all_baserels, + bms_make_singleton(appinfo->parent_relid)); + } + else + otherrels = bms_difference(root->all_baserels, rel->relids); + + if (!bms_is_empty(otherrels)) + clauselist = + list_concat(clauselist, + generate_join_implied_equalities(root, + bms_union(rel->relids, + otherrels), + otherrels, + rel)); + + /* Now try to prove each index predicate true */ + foreach(lc, rel->indexlist) + { + IndexOptInfo *index = (IndexOptInfo *) lfirst(lc); if (index->indpred == NIL) continue; /* ignore non-partial indexes */ @@ -2366,8 +2619,7 @@ check_partial_indexes(PlannerInfo *root, RelOptInfo *rel) if (index->predOK) continue; /* don't repeat work if already proven OK */ - index->predOK = predicate_implied_by(index->indpred, - restrictinfo_list); + index->predOK = predicate_implied_by(index->indpred, clauselist); } } @@ -2376,15 +2628,18 @@ check_partial_indexes(PlannerInfo *root, RelOptInfo *rel) ****************************************************************************/ /* - * eclass_member_matches_indexcol + * ec_member_matches_indexcol * Test whether an EquivalenceClass member matches an index column. * - * This is exported for use by generate_implied_equalities_for_indexcol. + * This is a callback for use by generate_implied_equalities_for_column. */ -bool -eclass_member_matches_indexcol(EquivalenceClass *ec, EquivalenceMember *em, - IndexOptInfo *index, int indexcol) +static bool +ec_member_matches_indexcol(PlannerInfo *root, RelOptInfo *rel, + EquivalenceClass *ec, EquivalenceMember *em, + void *arg) { + IndexOptInfo *index = ((ec_member_matches_arg *) arg)->index; + int indexcol = ((ec_member_matches_arg *) arg)->indexcol; Oid curFamily = index->opfamily[indexcol]; Oid curCollation = index->indexcollations[indexcol]; @@ -2395,7 +2650,7 @@ eclass_member_matches_indexcol(EquivalenceClass *ec, EquivalenceMember *em, * whether clauses generated from the EC could be used with the index, so * don't check the opfamily. This might mean we return "true" for a * useless EC, so we have to recheck the results of - * generate_implied_equalities_for_indexcol; see + * generate_implied_equalities_for_column; see * match_eclass_clauses_to_index. */ if (index->relam == BTREE_AM_OID && @@ -2424,7 +2679,7 @@ eclass_member_matches_indexcol(EquivalenceClass *ec, EquivalenceMember *em, * if it is true. * 2. A list of expressions in this relation, and a corresponding list of * equality operators. The caller must have already checked that the operators - * represent equality. (Note: the operators could be cross-type; the + * represent equality. (Note: the operators could be cross-type; the * expressions should correspond to their RHS inputs.) * * The caller need only supply equality conditions arising from joins; @@ -2613,7 +2868,7 @@ match_index_to_operand(Node *operand, int indkey; /* - * Ignore any RelabelType node above the operand. This is needed to be + * Ignore any RelabelType node above the operand. This is needed to be * able to apply indexscanning in binary-compatible-operator cases. Note: * we can assume there is at most one RelabelType node; * eval_const_expressions() will have simplified if more than one. @@ -2680,10 +2935,10 @@ match_index_to_operand(Node *operand, * indexscan machinery. The key idea is that these operators allow us * to derive approximate indexscan qual clauses, such that any tuples * that pass the operator clause itself must also satisfy the simpler - * indexscan condition(s). Then we can use the indexscan machinery + * indexscan condition(s). Then we can use the indexscan machinery * to avoid scanning as much of the table as we'd otherwise have to, * while applying the original operator as a qpqual condition to ensure - * we deliver only the tuples we want. (In essence, we're using a regular + * we deliver only the tuples we want. (In essence, we're using a regular * index as if it were a lossy index.) * * An example of what we're doing is @@ -2697,7 +2952,7 @@ match_index_to_operand(Node *operand, * * Another thing that we do with this machinery is to provide special * smarts for "boolean" indexes (that is, indexes on boolean columns - * that support boolean equality). We can transform a plain reference + * that support boolean equality). We can transform a plain reference * to the indexkey into "indexkey = true", or "NOT indexkey" into * "indexkey = false", so as to make the expression indexable using the * regular index operators. (As of Postgres 8.1, we must do this here @@ -2785,7 +3040,6 @@ match_special_index_operator(Expr *clause, Oid opfamily, Oid idxcollation, Oid expr_coll; Const *patt; Const *prefix = NULL; - Const *rest = NULL; Pattern_Prefix_Status pstatus = Pattern_Prefix_None; /* @@ -2814,13 +3068,13 @@ match_special_index_operator(Expr *clause, Oid opfamily, Oid idxcollation, case OID_NAME_LIKE_OP: /* the right-hand const is type text for all of these */ pstatus = pattern_fixed_prefix(patt, Pattern_Type_Like, expr_coll, - &prefix, &rest); + &prefix, NULL); isIndexable = (pstatus != Pattern_Prefix_None); break; case OID_BYTEA_LIKE_OP: pstatus = pattern_fixed_prefix(patt, Pattern_Type_Like, expr_coll, - &prefix, &rest); + &prefix, NULL); isIndexable = (pstatus != Pattern_Prefix_None); break; @@ -2829,7 +3083,7 @@ match_special_index_operator(Expr *clause, Oid opfamily, Oid idxcollation, case OID_NAME_ICLIKE_OP: /* the right-hand const is type text for all of these */ pstatus = pattern_fixed_prefix(patt, Pattern_Type_Like_IC, expr_coll, - &prefix, &rest); + &prefix, NULL); isIndexable = (pstatus != Pattern_Prefix_None); break; @@ -2838,7 +3092,7 @@ match_special_index_operator(Expr *clause, Oid opfamily, Oid idxcollation, case OID_NAME_REGEXEQ_OP: /* the right-hand const is type text for all of these */ pstatus = pattern_fixed_prefix(patt, Pattern_Type_Regex, expr_coll, - &prefix, &rest); + &prefix, NULL); isIndexable = (pstatus != Pattern_Prefix_None); break; @@ -2847,7 +3101,7 @@ match_special_index_operator(Expr *clause, Oid opfamily, Oid idxcollation, case OID_NAME_ICREGEXEQ_OP: /* the right-hand const is type text for all of these */ pstatus = pattern_fixed_prefix(patt, Pattern_Type_Regex_IC, expr_coll, - &prefix, &rest); + &prefix, NULL); isIndexable = (pstatus != Pattern_Prefix_None); break; @@ -3115,13 +3369,12 @@ expand_indexqual_opclause(RestrictInfo *rinfo, Oid opfamily, Oid idxcollation) Oid expr_coll = ((OpExpr *) clause)->inputcollid; Const *patt = (Const *) rightop; Const *prefix = NULL; - Const *rest = NULL; Pattern_Prefix_Status pstatus; /* * LIKE and regex operators are not members of any btree index opfamily, * but they can be members of opfamilies for more exotic index types such - * as GIN. Therefore, we should only do expansion if the operator is + * as GIN. Therefore, we should only do expansion if the operator is * actually not in the opfamily. But checking that requires a syscache * lookup, so it's best to first see if the operator is one we are * interested in. @@ -3135,7 +3388,7 @@ expand_indexqual_opclause(RestrictInfo *rinfo, Oid opfamily, Oid idxcollation) if (!op_in_opfamily(expr_op, opfamily)) { pstatus = pattern_fixed_prefix(patt, Pattern_Type_Like, expr_coll, - &prefix, &rest); + &prefix, NULL); return prefix_quals(leftop, opfamily, idxcollation, prefix, pstatus); } break; @@ -3147,7 +3400,7 @@ expand_indexqual_opclause(RestrictInfo *rinfo, Oid opfamily, Oid idxcollation) { /* the right-hand const is type text for all of these */ pstatus = pattern_fixed_prefix(patt, Pattern_Type_Like_IC, expr_coll, - &prefix, &rest); + &prefix, NULL); return prefix_quals(leftop, opfamily, idxcollation, prefix, pstatus); } break; @@ -3159,7 +3412,7 @@ expand_indexqual_opclause(RestrictInfo *rinfo, Oid opfamily, Oid idxcollation) { /* the right-hand const is type text for all of these */ pstatus = pattern_fixed_prefix(patt, Pattern_Type_Regex, expr_coll, - &prefix, &rest); + &prefix, NULL); return prefix_quals(leftop, opfamily, idxcollation, prefix, pstatus); } break; @@ -3171,7 +3424,7 @@ expand_indexqual_opclause(RestrictInfo *rinfo, Oid opfamily, Oid idxcollation) { /* the right-hand const is type text for all of these */ pstatus = pattern_fixed_prefix(patt, Pattern_Type_Regex_IC, expr_coll, - &prefix, &rest); + &prefix, NULL); return prefix_quals(leftop, opfamily, idxcollation, prefix, pstatus); } break; @@ -3239,7 +3492,7 @@ expand_indexqual_rowcompare(RestrictInfo *rinfo, * column matches) or a simple OpExpr (if the first-column match is all * there is). In these cases the modified clause is always "<=" or ">=" * even when the original was "<" or ">" --- this is necessary to match all - * the rows that could match the original. (We are essentially building a + * the rows that could match the original. (We are essentially building a * lossy version of the row comparison when we do this.) * * *indexcolnos receives an integer list of the index column numbers (zero diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 65f86194e1..be54f3de0b 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -3,7 +3,7 @@ * joinpath.c * Routines to find all possible paths for processing a set of joins * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -22,23 +22,26 @@ #include "optimizer/paths.h" +#define PATH_PARAM_BY_REL(path, rel) \ + ((path)->param_info && bms_overlap(PATH_REQ_OUTER(path), (rel)->relids)) + static void sort_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, List *restrictlist, List *mergeclause_list, JoinType jointype, SpecialJoinInfo *sjinfo, - Relids param_source_rels); + Relids param_source_rels, Relids extra_lateral_rels); static void match_unsorted_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, List *restrictlist, List *mergeclause_list, JoinType jointype, SpecialJoinInfo *sjinfo, SemiAntiJoinFactors *semifactors, - Relids param_source_rels); + Relids param_source_rels, Relids extra_lateral_rels); static void hash_inner_and_outer(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, List *restrictlist, JoinType jointype, SpecialJoinInfo *sjinfo, SemiAntiJoinFactors *semifactors, - Relids param_source_rels); + Relids param_source_rels, Relids extra_lateral_rels); static List *select_mergejoin_clauses(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, @@ -84,6 +87,7 @@ add_paths_to_joinrel(PlannerInfo *root, bool mergejoin_allowed = true; SemiAntiJoinFactors semifactors; Relids param_source_rels = NULL; + Relids extra_lateral_rels = NULL; ListCell *lc; /* @@ -103,7 +107,7 @@ add_paths_to_joinrel(PlannerInfo *root, /* * If it's SEMI or ANTI join, compute correction factors for cost - * estimation. These will be the same for all paths. + * estimation. These will be the same for all paths. */ if (jointype == JOIN_SEMI || jointype == JOIN_ANTI) compute_semi_anti_join_factors(root, outerrel, innerrel, @@ -118,7 +122,7 @@ add_paths_to_joinrel(PlannerInfo *root, * to the parameter source rel instead of joining to the other input rel. * This restriction reduces the number of parameterized paths we have to * deal with at higher join levels, without compromising the quality of - * the resulting plan. We express the restriction as a Relids set that + * the resulting plan. We express the restriction as a Relids set that * must overlap the parameterization of any proposed join path. */ foreach(lc, root->join_info_list) @@ -148,13 +152,69 @@ add_paths_to_joinrel(PlannerInfo *root, } /* + * However, when a LATERAL subquery is involved, we have to be a bit + * laxer, because there will simply not be any paths for the joinrel that + * aren't parameterized by whatever the subquery is parameterized by, + * unless its parameterization is resolved within the joinrel. Hence, add + * to param_source_rels anything that is laterally referenced in either + * input and is not in the join already. + */ + foreach(lc, root->lateral_info_list) + { + LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(lc); + + if (bms_is_subset(ljinfo->lateral_rhs, joinrel->relids)) + param_source_rels = bms_join(param_source_rels, + bms_difference(ljinfo->lateral_lhs, + joinrel->relids)); + } + + /* + * Another issue created by LATERAL references is that PlaceHolderVars + * that need to be computed at this join level might contain lateral + * references to rels not in the join, meaning that the paths for the join + * would need to be marked as parameterized by those rels, independently + * of all other considerations. Set extra_lateral_rels to the set of such + * rels. This will not affect our decisions as to which paths to + * generate; we merely add these rels to their required_outer sets. + */ + foreach(lc, root->placeholder_list) + { + PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(lc); + + /* PHVs without lateral refs can be skipped over quickly */ + if (phinfo->ph_lateral == NULL) + continue; + /* Is it due to be evaluated at this join, and not in either input? */ + if (bms_is_subset(phinfo->ph_eval_at, joinrel->relids) && + !bms_is_subset(phinfo->ph_eval_at, outerrel->relids) && + !bms_is_subset(phinfo->ph_eval_at, innerrel->relids)) + { + /* Yes, remember its lateral rels */ + extra_lateral_rels = bms_add_members(extra_lateral_rels, + phinfo->ph_lateral); + } + } + + /* + * Make sure extra_lateral_rels doesn't list anything within the join, and + * that it's NULL if empty. (This allows us to use bms_add_members to add + * it to required_outer below, while preserving the property that + * required_outer is exactly NULL if empty.) + */ + extra_lateral_rels = bms_del_members(extra_lateral_rels, joinrel->relids); + if (bms_is_empty(extra_lateral_rels)) + extra_lateral_rels = NULL; + + /* * 1. Consider mergejoin paths where both relations must be explicitly - * sorted. Skip this if we can't mergejoin. + * sorted. Skip this if we can't mergejoin. */ if (mergejoin_allowed) sort_inner_and_outer(root, joinrel, outerrel, innerrel, restrictlist, mergeclause_list, jointype, - sjinfo, param_source_rels); + sjinfo, + param_source_rels, extra_lateral_rels); /* * 2. Consider paths where the outer relation need not be explicitly @@ -166,13 +226,14 @@ add_paths_to_joinrel(PlannerInfo *root, if (mergejoin_allowed) match_unsorted_outer(root, joinrel, outerrel, innerrel, restrictlist, mergeclause_list, jointype, - sjinfo, &semifactors, param_source_rels); + sjinfo, &semifactors, + param_source_rels, extra_lateral_rels); #ifdef NOT_USED /* * 3. Consider paths where the inner relation need not be explicitly - * sorted. This includes mergejoins only (nestloops were already built in + * sorted. This includes mergejoins only (nestloops were already built in * match_unsorted_outer). * * Diked out as redundant 2/13/2000 -- tgl. There isn't any really @@ -184,7 +245,8 @@ add_paths_to_joinrel(PlannerInfo *root, if (mergejoin_allowed) match_unsorted_inner(root, joinrel, outerrel, innerrel, restrictlist, mergeclause_list, jointype, - sjinfo, &semifactors, param_source_rels); + sjinfo, &semifactors, + param_source_rels, extra_lateral_rels); #endif /* @@ -195,7 +257,8 @@ add_paths_to_joinrel(PlannerInfo *root, if (enable_hashjoin || jointype == JOIN_FULL) hash_inner_and_outer(root, joinrel, outerrel, innerrel, restrictlist, jointype, - sjinfo, &semifactors, param_source_rels); + sjinfo, &semifactors, + param_source_rels, extra_lateral_rels); } /* @@ -210,6 +273,7 @@ try_nestloop_path(PlannerInfo *root, SpecialJoinInfo *sjinfo, SemiAntiJoinFactors *semifactors, Relids param_source_rels, + Relids extra_lateral_rels, Path *outer_path, Path *inner_path, List *restrict_clauses, @@ -233,6 +297,12 @@ try_nestloop_path(PlannerInfo *root, } /* + * Independently of that, add parameterization needed for any + * PlaceHolderVars that need to be computed at the join. + */ + required_outer = bms_add_members(required_outer, extra_lateral_rels); + + /* * Do a precheck to quickly eliminate obviously-inferior paths. We * calculate a cheap lower bound on the path's cost and then use * add_path_precheck() to see if the path is clearly going to be dominated @@ -280,6 +350,7 @@ try_mergejoin_path(PlannerInfo *root, JoinType jointype, SpecialJoinInfo *sjinfo, Relids param_source_rels, + Relids extra_lateral_rels, Path *outer_path, Path *inner_path, List *restrict_clauses, @@ -306,6 +377,12 @@ try_mergejoin_path(PlannerInfo *root, } /* + * Independently of that, add parameterization needed for any + * PlaceHolderVars that need to be computed at the join. + */ + required_outer = bms_add_members(required_outer, extra_lateral_rels); + + /* * If the given paths are already well enough ordered, we can skip doing * an explicit sort. */ @@ -362,6 +439,7 @@ try_hashjoin_path(PlannerInfo *root, SpecialJoinInfo *sjinfo, SemiAntiJoinFactors *semifactors, Relids param_source_rels, + Relids extra_lateral_rels, Path *outer_path, Path *inner_path, List *restrict_clauses, @@ -385,6 +463,12 @@ try_hashjoin_path(PlannerInfo *root, } /* + * Independently of that, add parameterization needed for any + * PlaceHolderVars that need to be computed at the join. + */ + required_outer = bms_add_members(required_outer, extra_lateral_rels); + + /* * See comments in try_nestloop_path(). Also note that hashjoin paths * never have any output pathkeys, per comments in create_hashjoin_path. */ @@ -423,7 +507,7 @@ try_hashjoin_path(PlannerInfo *root, * We already know that the clause is a binary opclause referencing only the * rels in the current join. The point here is to check whether it has the * form "outerrel_expr op innerrel_expr" or "innerrel_expr op outerrel_expr", - * rather than mixing outer and inner vars on either side. If it matches, + * rather than mixing outer and inner vars on either side. If it matches, * we set the transient flag outer_is_left to identify which side is which. */ static inline bool @@ -462,6 +546,7 @@ clause_sides_match_join(RestrictInfo *rinfo, RelOptInfo *outerrel, * 'jointype' is the type of join to do * 'sjinfo' is extra info about the join for selectivity estimation * 'param_source_rels' are OK targets for parameterization of result paths + * 'extra_lateral_rels' are additional parameterization for result paths */ static void sort_inner_and_outer(PlannerInfo *root, @@ -472,7 +557,8 @@ sort_inner_and_outer(PlannerInfo *root, List *mergeclause_list, JoinType jointype, SpecialJoinInfo *sjinfo, - Relids param_source_rels) + Relids param_source_rels, + Relids extra_lateral_rels) { Path *outer_path; Path *inner_path; @@ -485,18 +571,30 @@ sort_inner_and_outer(PlannerInfo *root, * cheapest-startup-cost input paths later, and only if they don't need a * sort. * - * This function intentionally does not consider parameterized input paths - * (implicit in the fact that it only looks at cheapest_total_path, which - * is always unparameterized). If we did so, we'd have a combinatorial - * explosion of mergejoin paths of dubious value. This interacts with - * decisions elsewhere that also discriminate against mergejoins with - * parameterized inputs; see comments in src/backend/optimizer/README. - * - * If unique-ification is requested, do it and then handle as a plain - * inner join. + * This function intentionally does not consider parameterized input + * paths, except when the cheapest-total is parameterized. If we did so, + * we'd have a combinatorial explosion of mergejoin paths of dubious + * value. This interacts with decisions elsewhere that also discriminate + * against mergejoins with parameterized inputs; see comments in + * src/backend/optimizer/README. */ outer_path = outerrel->cheapest_total_path; inner_path = innerrel->cheapest_total_path; + + /* + * If either cheapest-total path is parameterized by the other rel, we + * can't use a mergejoin. (There's no use looking for alternative input + * paths, since these should already be the least-parameterized available + * paths.) + */ + if (PATH_PARAM_BY_REL(outer_path, innerrel) || + PATH_PARAM_BY_REL(inner_path, outerrel)) + return; + + /* + * If unique-ification is requested, do it and then handle as a plain + * inner join. + */ if (jointype == JOIN_UNIQUE_OUTER) { outer_path = (Path *) create_unique_path(root, outerrel, @@ -521,7 +619,7 @@ sort_inner_and_outer(PlannerInfo *root, * * Actually, it's not quite true that every mergeclause ordering will * generate a different path order, because some of the clauses may be - * partially redundant (refer to the same EquivalenceClasses). Therefore, + * partially redundant (refer to the same EquivalenceClasses). Therefore, * what we do is convert the mergeclause list to a list of canonical * pathkeys, and then consider different orderings of the pathkeys. * @@ -590,6 +688,7 @@ sort_inner_and_outer(PlannerInfo *root, jointype, sjinfo, param_source_rels, + extra_lateral_rels, outer_path, inner_path, restrictlist, @@ -614,7 +713,7 @@ sort_inner_and_outer(PlannerInfo *root, * cheapest-total inner-indexscan path (if any), and one on the * cheapest-startup inner-indexscan path (if different). * - * We also consider mergejoins if mergejoin clauses are available. We have + * We also consider mergejoins if mergejoin clauses are available. We have * two ways to generate the inner path for a mergejoin: sort the cheapest * inner path, or use an inner path that is already suitably ordered for the * merge. If we have several mergeclauses, it could be that there is no inner @@ -635,6 +734,7 @@ sort_inner_and_outer(PlannerInfo *root, * 'sjinfo' is extra info about the join for selectivity estimation * 'semifactors' contains valid data if jointype is SEMI or ANTI * 'param_source_rels' are OK targets for parameterization of result paths + * 'extra_lateral_rels' are additional parameterization for result paths */ static void match_unsorted_outer(PlannerInfo *root, @@ -646,7 +746,8 @@ match_unsorted_outer(PlannerInfo *root, JoinType jointype, SpecialJoinInfo *sjinfo, SemiAntiJoinFactors *semifactors, - Relids param_source_rels) + Relids param_source_rels, + Relids extra_lateral_rels) { JoinType save_jointype = jointype; bool nestjoinOK; @@ -691,11 +792,22 @@ match_unsorted_outer(PlannerInfo *root, } /* + * If inner_cheapest_total is parameterized by the outer rel, ignore it; + * we will consider it below as a member of cheapest_parameterized_paths, + * but the other possibilities considered in this routine aren't usable. + */ + if (PATH_PARAM_BY_REL(inner_cheapest_total, outerrel)) + inner_cheapest_total = NULL; + + /* * If we need to unique-ify the inner path, we will consider only the * cheapest-total inner. */ if (save_jointype == JOIN_UNIQUE_INNER) { + /* No way to do this with an inner path parameterized by outer rel */ + if (inner_cheapest_total == NULL) + return; inner_cheapest_total = (Path *) create_unique_path(root, innerrel, inner_cheapest_total, sjinfo); Assert(inner_cheapest_total); @@ -707,7 +819,7 @@ match_unsorted_outer(PlannerInfo *root, * enable_material is off or the path in question materializes its * output anyway. */ - if (enable_material && + if (enable_material && inner_cheapest_total != NULL && !ExecMaterializesOutput(inner_cheapest_total->pathtype)) matpath = (Path *) create_material_path(innerrel, inner_cheapest_total); @@ -728,13 +840,13 @@ match_unsorted_outer(PlannerInfo *root, /* * We cannot use an outer path that is parameterized by the inner rel. */ - if (bms_overlap(PATH_REQ_OUTER(outerpath), innerrel->relids)) + if (PATH_PARAM_BY_REL(outerpath, innerrel)) continue; /* * If we need to unique-ify the outer path, it's pointless to consider - * any but the cheapest outer. (XXX we don't consider parameterized - * outers, nor inners, for unique-ified cases. Should we?) + * any but the cheapest outer. (XXX we don't consider parameterized + * outers, nor inners, for unique-ified cases. Should we?) */ if (save_jointype == JOIN_UNIQUE_OUTER) { @@ -765,6 +877,7 @@ match_unsorted_outer(PlannerInfo *root, sjinfo, semifactors, param_source_rels, + extra_lateral_rels, outerpath, inner_cheapest_total, restrictlist, @@ -774,7 +887,7 @@ match_unsorted_outer(PlannerInfo *root, { /* * Consider nestloop joins using this outer path and various - * available paths for the inner relation. We consider the + * available paths for the inner relation. We consider the * cheapest-total paths for each available parameterization of the * inner relation, including the unparameterized case. */ @@ -790,6 +903,7 @@ match_unsorted_outer(PlannerInfo *root, sjinfo, semifactors, param_source_rels, + extra_lateral_rels, outerpath, innerpath, restrictlist, @@ -804,6 +918,7 @@ match_unsorted_outer(PlannerInfo *root, sjinfo, semifactors, param_source_rels, + extra_lateral_rels, outerpath, matpath, restrictlist, @@ -814,6 +929,10 @@ match_unsorted_outer(PlannerInfo *root, if (save_jointype == JOIN_UNIQUE_OUTER) continue; + /* Can't do anything else if inner rel is parameterized by outer */ + if (inner_cheapest_total == NULL) + continue; + /* Look for useful mergeclauses (if any) */ mergeclauses = find_mergeclauses_for_pathkeys(root, outerpath->pathkeys, @@ -855,6 +974,7 @@ match_unsorted_outer(PlannerInfo *root, jointype, sjinfo, param_source_rels, + extra_lateral_rels, outerpath, inner_cheapest_total, restrictlist, @@ -922,7 +1042,7 @@ match_unsorted_outer(PlannerInfo *root, /* * Look for an inner path ordered well enough for the first - * 'sortkeycnt' innersortkeys. NB: trialsortkeys list is modified + * 'sortkeycnt' innersortkeys. NB: trialsortkeys list is modified * destructively, which is why we made a copy... */ trialsortkeys = list_truncate(trialsortkeys, sortkeycnt); @@ -953,6 +1073,7 @@ match_unsorted_outer(PlannerInfo *root, jointype, sjinfo, param_source_rels, + extra_lateral_rels, outerpath, innerpath, restrictlist, @@ -998,6 +1119,7 @@ match_unsorted_outer(PlannerInfo *root, jointype, sjinfo, param_source_rels, + extra_lateral_rels, outerpath, innerpath, restrictlist, @@ -1032,6 +1154,7 @@ match_unsorted_outer(PlannerInfo *root, * 'sjinfo' is extra info about the join for selectivity estimation * 'semifactors' contains valid data if jointype is SEMI or ANTI * 'param_source_rels' are OK targets for parameterization of result paths + * 'extra_lateral_rels' are additional parameterization for result paths */ static void hash_inner_and_outer(PlannerInfo *root, @@ -1042,7 +1165,8 @@ hash_inner_and_outer(PlannerInfo *root, JoinType jointype, SpecialJoinInfo *sjinfo, SemiAntiJoinFactors *semifactors, - Relids param_source_rels) + Relids param_source_rels, + Relids extra_lateral_rels) { bool isouterjoin = IS_OUTER_JOIN(jointype); List *hashclauses; @@ -1092,6 +1216,16 @@ hash_inner_and_outer(PlannerInfo *root, Path *cheapest_total_outer = outerrel->cheapest_total_path; Path *cheapest_total_inner = innerrel->cheapest_total_path; + /* + * If either cheapest-total path is parameterized by the other rel, we + * can't use a hashjoin. (There's no use looking for alternative + * input paths, since these should already be the least-parameterized + * available paths.) + */ + if (PATH_PARAM_BY_REL(cheapest_total_outer, innerrel) || + PATH_PARAM_BY_REL(cheapest_total_inner, outerrel)) + return; + /* Unique-ify if need be; we ignore parameterized possibilities */ if (jointype == JOIN_UNIQUE_OUTER) { @@ -1106,6 +1240,7 @@ hash_inner_and_outer(PlannerInfo *root, sjinfo, semifactors, param_source_rels, + extra_lateral_rels, cheapest_total_outer, cheapest_total_inner, restrictlist, @@ -1125,17 +1260,20 @@ hash_inner_and_outer(PlannerInfo *root, sjinfo, semifactors, param_source_rels, + extra_lateral_rels, cheapest_total_outer, cheapest_total_inner, restrictlist, hashclauses); - if (cheapest_startup_outer != cheapest_total_outer) + if (cheapest_startup_outer != NULL && + cheapest_startup_outer != cheapest_total_outer) try_hashjoin_path(root, joinrel, jointype, sjinfo, semifactors, param_source_rels, + extra_lateral_rels, cheapest_startup_outer, cheapest_total_inner, restrictlist, @@ -1153,16 +1291,18 @@ hash_inner_and_outer(PlannerInfo *root, ListCell *lc1; ListCell *lc2; - try_hashjoin_path(root, - joinrel, - jointype, - sjinfo, - semifactors, - param_source_rels, - cheapest_startup_outer, - cheapest_total_inner, - restrictlist, - hashclauses); + if (cheapest_startup_outer != NULL) + try_hashjoin_path(root, + joinrel, + jointype, + sjinfo, + semifactors, + param_source_rels, + extra_lateral_rels, + cheapest_startup_outer, + cheapest_total_inner, + restrictlist, + hashclauses); foreach(lc1, outerrel->cheapest_parameterized_paths) { @@ -1172,7 +1312,7 @@ hash_inner_and_outer(PlannerInfo *root, * We cannot use an outer path that is parameterized by the * inner rel. */ - if (bms_overlap(PATH_REQ_OUTER(outerpath), innerrel->relids)) + if (PATH_PARAM_BY_REL(outerpath, innerrel)) continue; foreach(lc2, innerrel->cheapest_parameterized_paths) @@ -1183,8 +1323,7 @@ hash_inner_and_outer(PlannerInfo *root, * We cannot use an inner path that is parameterized by * the outer rel, either. */ - if (bms_overlap(PATH_REQ_OUTER(innerpath), - outerrel->relids)) + if (PATH_PARAM_BY_REL(innerpath, outerrel)) continue; if (outerpath == cheapest_startup_outer && @@ -1197,6 +1336,7 @@ hash_inner_and_outer(PlannerInfo *root, sjinfo, semifactors, param_source_rels, + extra_lateral_rels, outerpath, innerpath, restrictlist, diff --git a/src/backend/optimizer/path/joinrels.c b/src/backend/optimizer/path/joinrels.c index e6a0f8dab6..610892890f 100644 --- a/src/backend/optimizer/path/joinrels.c +++ b/src/backend/optimizer/path/joinrels.c @@ -3,7 +3,7 @@ * joinrels.c * Routines to determine which relations should be joined * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -178,25 +178,62 @@ join_search_one_level(PlannerInfo *root, int level) } /*---------- - * Normally, we should always have made at least one join of the current - * level. However, when special joins are involved, there may be no legal - * way to make an N-way join for some values of N. For example consider + * Last-ditch effort: if we failed to find any usable joins so far, force + * a set of cartesian-product joins to be generated. This handles the + * special case where all the available rels have join clauses but we + * cannot use any of those clauses yet. This can only happen when we are + * considering a join sub-problem (a sub-joinlist) and all the rels in the + * sub-problem have only join clauses with rels outside the sub-problem. + * An example is * - * SELECT ... FROM t1 WHERE - * x IN (SELECT ... FROM t2,t3 WHERE ...) AND - * y IN (SELECT ... FROM t4,t5 WHERE ...) + * SELECT ... FROM a INNER JOIN b ON TRUE, c, d, ... + * WHERE a.w = c.x and b.y = d.z; * - * We will flatten this query to a 5-way join problem, but there are - * no 4-way joins that join_is_legal() will consider legal. We have - * to accept failure at level 4 and go on to discover a workable - * bushy plan at level 5. - * - * However, if there are no special joins then join_is_legal() should - * never fail, and so the following sanity check is useful. + * If the "a INNER JOIN b" sub-problem does not get flattened into the + * upper level, we must be willing to make a cartesian join of a and b; + * but the code above will not have done so, because it thought that both + * a and b have joinclauses. We consider only left-sided and right-sided + * cartesian joins in this case (no bushy). *---------- */ - if (joinrels[level] == NIL && root->join_info_list == NIL) - elog(ERROR, "failed to build any %d-way joins", level); + if (joinrels[level] == NIL) + { + /* + * This loop is just like the first one, except we always call + * make_rels_by_clauseless_joins(). + */ + foreach(r, joinrels[level - 1]) + { + RelOptInfo *old_rel = (RelOptInfo *) lfirst(r); + + make_rels_by_clauseless_joins(root, + old_rel, + list_head(joinrels[1])); + } + + /*---------- + * When special joins are involved, there may be no legal way + * to make an N-way join for some values of N. For example consider + * + * SELECT ... FROM t1 WHERE + * x IN (SELECT ... FROM t2,t3 WHERE ...) AND + * y IN (SELECT ... FROM t4,t5 WHERE ...) + * + * We will flatten this query to a 5-way join problem, but there are + * no 4-way joins that join_is_legal() will consider legal. We have + * to accept failure at level 4 and go on to discover a workable + * bushy plan at level 5. + * + * However, if there are no special joins and no lateral references + * then join_is_legal() should never fail, and so the following sanity + * check is useful. + *---------- + */ + if (joinrels[level] == NIL && + root->join_info_list == NIL && + root->lateral_info_list == NIL) + elog(ERROR, "failed to build any %d-way joins", level); + } } /* @@ -295,10 +332,12 @@ join_is_legal(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2, bool reversed; bool unique_ified; bool is_valid_inner; + bool lateral_fwd; + bool lateral_rev; ListCell *l; /* - * Ensure output params are set on failure return. This is just to + * Ensure output params are set on failure return. This is just to * suppress uninitialized-variable warnings from overly anal compilers. */ *sjinfo_p = NULL; @@ -306,7 +345,7 @@ join_is_legal(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2, /* * If we have any special joins, the proposed join might be illegal; and - * in any case we have to determine its join type. Scan the join info + * in any case we have to determine its join type. Scan the join info * list for conflicts. */ match_sjinfo = NULL; @@ -474,6 +513,47 @@ join_is_legal(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2, (match_sjinfo == NULL || unique_ified)) return false; /* invalid join path */ + /* + * We also have to check for constraints imposed by LATERAL references. + * The proposed rels could each contain lateral references to the other, + * in which case the join is impossible. If there are lateral references + * in just one direction, then the join has to be done with a nestloop + * with the lateral referencer on the inside. If the join matches an SJ + * that cannot be implemented by such a nestloop, the join is impossible. + */ + lateral_fwd = lateral_rev = false; + foreach(l, root->lateral_info_list) + { + LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l); + + if (bms_is_subset(ljinfo->lateral_rhs, rel2->relids) && + bms_overlap(ljinfo->lateral_lhs, rel1->relids)) + { + /* has to be implemented as nestloop with rel1 on left */ + if (lateral_rev) + return false; /* have lateral refs in both directions */ + lateral_fwd = true; + if (!bms_is_subset(ljinfo->lateral_lhs, rel1->relids)) + return false; /* rel1 can't compute the required parameter */ + if (match_sjinfo && + (reversed || match_sjinfo->jointype == JOIN_FULL)) + return false; /* not implementable as nestloop */ + } + if (bms_is_subset(ljinfo->lateral_rhs, rel1->relids) && + bms_overlap(ljinfo->lateral_lhs, rel2->relids)) + { + /* has to be implemented as nestloop with rel2 on left */ + if (lateral_fwd) + return false; /* have lateral refs in both directions */ + lateral_rev = true; + if (!bms_is_subset(ljinfo->lateral_lhs, rel2->relids)) + return false; /* rel2 can't compute the required parameter */ + if (match_sjinfo && + (!reversed || match_sjinfo->jointype == JOIN_FULL)) + return false; /* not implementable as nestloop */ + } + } + /* Otherwise, it's a valid join */ *sjinfo_p = match_sjinfo; *reversed_p = reversed; @@ -529,7 +609,7 @@ make_join_rel(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2) /* * If it's a plain inner join, then we won't have found anything in - * join_info_list. Make up a SpecialJoinInfo so that selectivity + * join_info_list. Make up a SpecialJoinInfo so that selectivity * estimation functions will know what's being joined. */ if (sjinfo == NULL) @@ -718,12 +798,21 @@ make_join_rel(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2) /* * have_join_order_restriction * Detect whether the two relations should be joined to satisfy - * a join-order restriction arising from special joins. + * a join-order restriction arising from special or lateral joins. * * In practice this is always used with have_relevant_joinclause(), and so * could be merged with that function, but it seems clearer to separate the * two concerns. We need this test because there are degenerate cases where * a clauseless join must be performed to satisfy join-order restrictions. + * Also, if one rel has a lateral reference to the other, we should consider + * joining them even if the join would be clauseless. + * + * Note: this is only a problem if one side of a degenerate outer join + * contains multiple rels, or a clauseless join is required within an + * IN/EXISTS RHS; else we will find a join path via the "last ditch" case in + * join_search_one_level(). We could dispense with this test if we were + * willing to try bushy plans in the "last ditch" case, but that seems much + * less efficient. */ bool have_join_order_restriction(PlannerInfo *root, @@ -733,6 +822,22 @@ have_join_order_restriction(PlannerInfo *root, ListCell *l; /* + * If either side has a lateral reference to the other, attempt the join + * regardless of outer-join considerations. + */ + foreach(l, root->lateral_info_list) + { + LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l); + + if (bms_is_subset(ljinfo->lateral_rhs, rel2->relids) && + bms_overlap(ljinfo->lateral_lhs, rel1->relids)) + return true; + if (bms_is_subset(ljinfo->lateral_rhs, rel1->relids) && + bms_overlap(ljinfo->lateral_lhs, rel2->relids)) + return true; + } + + /* * It's possible that the rels correspond to the left and right sides of a * degenerate outer join, that is, one with no joinclause mentioning the * non-nullable side; in which case we should force the join to occur. @@ -805,12 +910,13 @@ have_join_order_restriction(PlannerInfo *root, /* * has_join_restriction - * Detect whether the specified relation has join-order restrictions - * due to being inside an outer join or an IN (sub-SELECT). + * Detect whether the specified relation has join-order restrictions, + * due to being inside an outer join or an IN (sub-SELECT), + * or participating in any LATERAL references. * * Essentially, this tests whether have_join_order_restriction() could * succeed with this rel and some other one. It's OK if we sometimes - * say "true" incorrectly. (Therefore, we don't bother with the relatively + * say "true" incorrectly. (Therefore, we don't bother with the relatively * expensive has_legal_joinclause test.) */ static bool @@ -818,6 +924,15 @@ has_join_restriction(PlannerInfo *root, RelOptInfo *rel) { ListCell *l; + foreach(l, root->lateral_info_list) + { + LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l); + + if (bms_is_subset(ljinfo->lateral_rhs, rel->relids) || + bms_overlap(ljinfo->lateral_lhs, rel->relids)) + return true; + } + foreach(l, root->join_info_list) { SpecialJoinInfo *sjinfo = (SpecialJoinInfo *) lfirst(l); @@ -912,7 +1027,7 @@ is_dummy_rel(RelOptInfo *rel) * dummy. * * Also, when called during GEQO join planning, we are in a short-lived - * memory context. We must make sure that the dummy path attached to a + * memory context. We must make sure that the dummy path attached to a * baserel survives the GEQO cycle, else the baserel is trashed for future * GEQO cycles. On the other hand, when we are marking a joinrel during GEQO, * we don't want the dummy path to clutter the main planning context. Upshot diff --git a/src/backend/optimizer/path/orindxpath.c b/src/backend/optimizer/path/orindxpath.c deleted file mode 100644 index c918c4e8da..0000000000 --- a/src/backend/optimizer/path/orindxpath.c +++ /dev/null @@ -1,187 +0,0 @@ -/*------------------------------------------------------------------------- - * - * orindxpath.c - * Routines to find index paths that match a set of OR clauses - * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group - * Portions Copyright (c) 1994, Regents of the University of California - * - * - * IDENTIFICATION - * src/backend/optimizer/path/orindxpath.c - * - *------------------------------------------------------------------------- - */ - -#include "postgres.h" - -#include "optimizer/cost.h" -#include "optimizer/paths.h" -#include "optimizer/restrictinfo.h" - - -/*---------- - * create_or_index_quals - * Examine join OR-of-AND quals to see if any useful restriction OR - * clauses can be extracted. If so, add them to the query. - * - * Although a join clause must reference other relations overall, - * an OR of ANDs clause might contain sub-clauses that reference just this - * relation and can be used to build a restriction clause. - * For example consider - * WHERE ((a.x = 42 AND b.y = 43) OR (a.x = 44 AND b.z = 45)); - * We can transform this into - * WHERE ((a.x = 42 AND b.y = 43) OR (a.x = 44 AND b.z = 45)) - * AND (a.x = 42 OR a.x = 44) - * AND (b.y = 43 OR b.z = 45); - * which opens the potential to build OR indexscans on a and b. In essence - * this is a partial transformation to CNF (AND of ORs format). It is not - * complete, however, because we do not unravel the original OR --- doing so - * would usually bloat the qualification expression to little gain. - * - * The added quals are partially redundant with the original OR, and therefore - * will cause the size of the joinrel to be underestimated when it is finally - * formed. (This would be true of a full transformation to CNF as well; the - * fault is not really in the transformation, but in clauselist_selectivity's - * inability to recognize redundant conditions.) To minimize the collateral - * damage, we want to minimize the number of quals added. Therefore we do - * not add every possible extracted restriction condition to the query. - * Instead, we search for the single restriction condition that generates - * the most useful (cheapest) OR indexscan, and add only that condition. - * This is a pretty ad-hoc heuristic, but quite useful. - * - * We can then compensate for the redundancy of the added qual by poking - * the recorded selectivity of the original OR clause, thereby ensuring - * the added qual doesn't change the estimated size of the joinrel when - * it is finally formed. This is a MAJOR HACK: it depends on the fact - * that clause selectivities are cached and on the fact that the same - * RestrictInfo node will appear in every joininfo list that might be used - * when the joinrel is formed. And it probably isn't right in cases where - * the size estimation is nonlinear (i.e., outer and IN joins). But it - * beats not doing anything. - * - * NOTE: one might think this messiness could be worked around by generating - * the indexscan path with a small path->rows value, and not touching the - * rel's baserestrictinfo or rel->rows. However, that does not work. - * The optimizer's fundamental design assumes that every general-purpose - * Path for a given relation generates the same number of rows. Without - * this assumption we'd not be able to optimize solely on the cost of Paths, - * but would have to take number of output rows into account as well. - * (The parameterized-paths stuff almost fixes this, but not quite...) - * - * 'rel' is the relation entry for which quals are to be created - * - * If successful, adds qual(s) to rel->baserestrictinfo and returns TRUE. - * If no quals available, returns FALSE and doesn't change rel. - * - * Note: check_partial_indexes() must have been run previously. - *---------- - */ -bool -create_or_index_quals(PlannerInfo *root, RelOptInfo *rel) -{ - BitmapOrPath *bestpath = NULL; - RestrictInfo *bestrinfo = NULL; - List *newrinfos; - RestrictInfo *or_rinfo; - Selectivity or_selec, - orig_selec; - ListCell *i; - - /* Skip the whole mess if no indexes */ - if (rel->indexlist == NIL) - return false; - - /* - * Find potentially interesting OR joinclauses. We can use any joinclause - * that is considered safe to move to this rel by the parameterized-path - * machinery, even though what we are going to do with it is not exactly a - * parameterized path. - */ - foreach(i, rel->joininfo) - { - RestrictInfo *rinfo = (RestrictInfo *) lfirst(i); - - if (restriction_is_or_clause(rinfo) && - join_clause_is_movable_to(rinfo, rel->relid)) - { - /* - * Use the generate_bitmap_or_paths() machinery to estimate the - * value of each OR clause. We can use regular restriction - * clauses along with the OR clause contents to generate - * indexquals. We pass restriction_only = true so that any - * sub-clauses that are actually joins will be ignored. - */ - List *orpaths; - ListCell *k; - - orpaths = generate_bitmap_or_paths(root, rel, - list_make1(rinfo), - rel->baserestrictinfo, - true); - - /* Locate the cheapest OR path */ - foreach(k, orpaths) - { - BitmapOrPath *path = (BitmapOrPath *) lfirst(k); - - Assert(IsA(path, BitmapOrPath)); - if (bestpath == NULL || - path->path.total_cost < bestpath->path.total_cost) - { - bestpath = path; - bestrinfo = rinfo; - } - } - } - } - - /* Fail if no suitable clauses found */ - if (bestpath == NULL) - return false; - - /* - * Convert the path's indexclauses structure to a RestrictInfo tree. We - * include any partial-index predicates so as to get a reasonable - * representation of what the path is actually scanning. - */ - newrinfos = make_restrictinfo_from_bitmapqual((Path *) bestpath, - true, true); - - /* It's possible we get back something other than a single OR clause */ - if (list_length(newrinfos) != 1) - return false; - or_rinfo = (RestrictInfo *) linitial(newrinfos); - Assert(IsA(or_rinfo, RestrictInfo)); - if (!restriction_is_or_clause(or_rinfo)) - return false; - - /* - * OK, add it to the rel's restriction list. - */ - rel->baserestrictinfo = list_concat(rel->baserestrictinfo, newrinfos); - - /* - * Adjust the original OR clause's cached selectivity to compensate for - * the selectivity of the added (but redundant) lower-level qual. This - * should result in the join rel getting approximately the same rows - * estimate as it would have gotten without all these shenanigans. (XXX - * major hack alert ... this depends on the assumption that the - * selectivity will stay cached ...) - */ - or_selec = clause_selectivity(root, (Node *) or_rinfo, - 0, JOIN_INNER, NULL); - if (or_selec > 0 && or_selec < 1) - { - orig_selec = clause_selectivity(root, (Node *) bestrinfo, - 0, JOIN_INNER, NULL); - bestrinfo->norm_selec = orig_selec / or_selec; - /* clamp result to sane range */ - if (bestrinfo->norm_selec > 1) - bestrinfo->norm_selec = 1; - /* It isn't an outer join clause, so no need to adjust outer_selec */ - } - - /* Tell caller to recompute partial index status and rowcount estimate */ - return true; -} diff --git a/src/backend/optimizer/path/pathkeys.c b/src/backend/optimizer/path/pathkeys.c index 20a5644edd..5d953dfb45 100644 --- a/src/backend/optimizer/path/pathkeys.c +++ b/src/backend/optimizer/path/pathkeys.c @@ -7,7 +7,7 @@ * the nature and use of path keys. * * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * IDENTIFICATION @@ -28,8 +28,6 @@ #include "utils/lsyscache.h" -static PathKey *makePathKey(EquivalenceClass *eclass, Oid opfamily, - int strategy, bool nulls_first); static PathKey *make_canonical_pathkey(PlannerInfo *root, EquivalenceClass *eclass, Oid opfamily, int strategy, bool nulls_first); @@ -42,34 +40,15 @@ static bool right_merge_direction(PlannerInfo *root, PathKey *pathkey); ****************************************************************************/ /* - * makePathKey - * create a PathKey node - * - * This does not promise to create a canonical PathKey, it's merely a - * convenience routine to build the specified node. - */ -static PathKey * -makePathKey(EquivalenceClass *eclass, Oid opfamily, - int strategy, bool nulls_first) -{ - PathKey *pk = makeNode(PathKey); - - pk->pk_eclass = eclass; - pk->pk_opfamily = opfamily; - pk->pk_strategy = strategy; - pk->pk_nulls_first = nulls_first; - - return pk; -} - -/* * make_canonical_pathkey * Given the parameters for a PathKey, find any pre-existing matching * pathkey in the query's list of "canonical" pathkeys. Make a new * entry if there's not one already. * * Note that this function must not be used until after we have completed - * merging EquivalenceClasses. + * merging EquivalenceClasses. (We don't try to enforce that here; instead, + * equivclass.c will complain if a merge occurs after root->canon_pathkeys + * has become nonempty.) */ static PathKey * make_canonical_pathkey(PlannerInfo *root, @@ -100,7 +79,12 @@ make_canonical_pathkey(PlannerInfo *root, */ oldcontext = MemoryContextSwitchTo(root->planner_cxt); - pk = makePathKey(eclass, opfamily, strategy, nulls_first); + pk = makeNode(PathKey); + pk->pk_eclass = eclass; + pk->pk_opfamily = opfamily; + pk->pk_strategy = strategy; + pk->pk_nulls_first = nulls_first; + root->canon_pathkeys = lappend(root->canon_pathkeys, pk); MemoryContextSwitchTo(oldcontext); @@ -112,8 +96,7 @@ make_canonical_pathkey(PlannerInfo *root, * pathkey_is_redundant * Is a pathkey redundant with one already in the given list? * - * Both the given pathkey and the list members must be canonical for this - * to work properly. We detect two cases: + * We detect two cases: * * 1. If the new pathkey's equivalence class contains a constant, and isn't * below an outer join, then we can disregard it as a sort key. An example: @@ -135,6 +118,12 @@ make_canonical_pathkey(PlannerInfo *root, * Note in particular that we need not compare opfamily (all the opfamilies * of the EC have the same notion of equality) nor sort direction. * + * Both the given pathkey and the list members must be canonical for this + * to work properly, but that's okay since we no longer ever construct any + * non-canonical pathkeys. (Note: the notion of a pathkey *list* being + * canonical includes the additional requirement of no redundant entries, + * which is exactly what we are checking for here.) + * * Because the equivclass.c machinery forms only one copy of any EC per query, * pointer comparison is enough to decide whether canonical ECs are the same. */ @@ -144,9 +133,6 @@ pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys) EquivalenceClass *new_ec = new_pathkey->pk_eclass; ListCell *lc; - /* Assert we've been given canonical pathkeys */ - Assert(!new_ec->ec_merged); - /* Check for EC containing a constant --- unconditionally redundant */ if (EC_MUST_BE_REDUNDANT(new_ec)) return true; @@ -156,9 +142,6 @@ pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys) { PathKey *old_pathkey = (PathKey *) lfirst(lc); - /* Assert we've been given canonical pathkeys */ - Assert(!old_pathkey->pk_eclass->ec_merged); - if (new_ec == old_pathkey->pk_eclass) return true; } @@ -167,75 +150,29 @@ pathkey_is_redundant(PathKey *new_pathkey, List *pathkeys) } /* - * canonicalize_pathkeys - * Convert a not-necessarily-canonical pathkeys list to canonical form. - * - * Note that this function must not be used until after we have completed - * merging EquivalenceClasses. - */ -List * -canonicalize_pathkeys(PlannerInfo *root, List *pathkeys) -{ - List *new_pathkeys = NIL; - ListCell *l; - - foreach(l, pathkeys) - { - PathKey *pathkey = (PathKey *) lfirst(l); - EquivalenceClass *eclass; - PathKey *cpathkey; - - /* Find the canonical (merged) EquivalenceClass */ - eclass = pathkey->pk_eclass; - while (eclass->ec_merged) - eclass = eclass->ec_merged; - - /* - * If we can tell it's redundant just from the EC, skip. - * pathkey_is_redundant would notice that, but we needn't even bother - * constructing the node... - */ - if (EC_MUST_BE_REDUNDANT(eclass)) - continue; - - /* OK, build a canonicalized PathKey struct */ - cpathkey = make_canonical_pathkey(root, - eclass, - pathkey->pk_opfamily, - pathkey->pk_strategy, - pathkey->pk_nulls_first); - - /* Add to list unless redundant */ - if (!pathkey_is_redundant(cpathkey, new_pathkeys)) - new_pathkeys = lappend(new_pathkeys, cpathkey); - } - return new_pathkeys; -} - -/* * make_pathkey_from_sortinfo * Given an expression and sort-order information, create a PathKey. - * If canonicalize = true, the result is a "canonical" PathKey, - * otherwise not. (But note it might be redundant anyway.) + * The result is always a "canonical" PathKey, but it might be redundant. + * + * expr is the expression, and nullable_relids is the set of base relids + * that are potentially nullable below it. * * If the PathKey is being generated from a SortGroupClause, sortref should be * the SortGroupClause's SortGroupRef; otherwise zero. * * If rel is not NULL, it identifies a specific relation we're considering * a path for, and indicates that child EC members for that relation can be - * considered. Otherwise child members are ignored. (See the comments for + * considered. Otherwise child members are ignored. (See the comments for * get_eclass_for_sort_expr.) * * create_it is TRUE if we should create any missing EquivalenceClass * needed to represent the sort key. If it's FALSE, we return NULL if the * sort key isn't already present in any EquivalenceClass. - * - * canonicalize should always be TRUE after EquivalenceClass merging has - * been performed, but FALSE if we haven't done EquivalenceClass merging yet. */ static PathKey * make_pathkey_from_sortinfo(PlannerInfo *root, Expr *expr, + Relids nullable_relids, Oid opfamily, Oid opcintype, Oid collation, @@ -243,8 +180,7 @@ make_pathkey_from_sortinfo(PlannerInfo *root, bool nulls_first, Index sortref, Relids rel, - bool create_it, - bool canonicalize) + bool create_it) { int16 strategy; Oid equality_op; @@ -256,7 +192,7 @@ make_pathkey_from_sortinfo(PlannerInfo *root, /* * EquivalenceClasses need to contain opfamily lists based on the family * membership of mergejoinable equality operators, which could belong to - * more than one opfamily. So we have to look up the opfamily's equality + * more than one opfamily. So we have to look up the opfamily's equality * operator and get its membership. */ equality_op = get_opfamily_member(opfamily, @@ -272,8 +208,8 @@ make_pathkey_from_sortinfo(PlannerInfo *root, equality_op); /* Now find or (optionally) create a matching EquivalenceClass */ - eclass = get_eclass_for_sort_expr(root, expr, opfamilies, - opcintype, collation, + eclass = get_eclass_for_sort_expr(root, expr, nullable_relids, + opfamilies, opcintype, collation, sortref, rel, create_it); /* Fail if no EC and !create_it */ @@ -281,11 +217,8 @@ make_pathkey_from_sortinfo(PlannerInfo *root, return NULL; /* And finally we can find or create a PathKey node */ - if (canonicalize) - return make_canonical_pathkey(root, eclass, opfamily, - strategy, nulls_first); - else - return makePathKey(eclass, opfamily, strategy, nulls_first); + return make_canonical_pathkey(root, eclass, opfamily, + strategy, nulls_first); } /* @@ -298,11 +231,11 @@ make_pathkey_from_sortinfo(PlannerInfo *root, static PathKey * make_pathkey_from_sortop(PlannerInfo *root, Expr *expr, + Relids nullable_relids, Oid ordering_op, bool nulls_first, Index sortref, - bool create_it, - bool canonicalize) + bool create_it) { Oid opfamily, opcintype, @@ -320,6 +253,7 @@ make_pathkey_from_sortop(PlannerInfo *root, return make_pathkey_from_sortinfo(root, expr, + nullable_relids, opfamily, opcintype, collation, @@ -327,8 +261,7 @@ make_pathkey_from_sortop(PlannerInfo *root, nulls_first, sortref, NULL, - create_it, - canonicalize); + create_it); } @@ -341,9 +274,8 @@ make_pathkey_from_sortop(PlannerInfo *root, * Compare two pathkeys to see if they are equivalent, and if not whether * one is "better" than the other. * - * This function may only be applied to canonicalized pathkey lists. - * In the canonical representation, pathkeys can be checked for equality - * by simple pointer comparison. + * We assume the pathkeys are canonical, and so they can be checked for + * equality by simple pointer comparison. */ PathKeysComparison compare_pathkeys(List *keys1, List *keys2) @@ -364,15 +296,6 @@ compare_pathkeys(List *keys1, List *keys2) PathKey *pathkey1 = (PathKey *) lfirst(key1); PathKey *pathkey2 = (PathKey *) lfirst(key2); - /* - * XXX would like to check that we've been given canonicalized input, - * but PlannerInfo not accessible here... - */ -#ifdef NOT_USED - Assert(list_member_ptr(root->canon_pathkeys, pathkey1)); - Assert(list_member_ptr(root->canon_pathkeys, pathkey2)); -#endif - if (pathkey1 != pathkey2) return PATHKEYS_DIFFERENT; /* no need to keep looking */ } @@ -414,7 +337,7 @@ pathkeys_contained_in(List *keys1, List *keys2) * Return NULL if no such path. * * 'paths' is a list of possible paths that all generate the same relation - * 'pathkeys' represents a required ordering (already canonicalized!) + * 'pathkeys' represents a required ordering (in canonical form!) * 'required_outer' denotes allowable outer relations for parameterized paths * 'cost_criterion' is STARTUP_COST or TOTAL_COST */ @@ -432,7 +355,7 @@ get_cheapest_path_for_pathkeys(List *paths, List *pathkeys, /* * Since cost comparison is a lot cheaper than pathkey comparison, do - * that first. (XXX is that still true?) + * that first. (XXX is that still true?) */ if (matched_path != NULL && compare_path_costs(matched_path, path, cost_criterion) <= 0) @@ -455,7 +378,7 @@ get_cheapest_path_for_pathkeys(List *paths, List *pathkeys, * parameter. * * 'paths' is a list of possible paths that all generate the same relation - * 'pathkeys' represents a required ordering (already canonicalized!) + * 'pathkeys' represents a required ordering (in canonical form!) * 'required_outer' denotes allowable outer relations for parameterized paths * 'fraction' is the fraction of the total tuples expected to be retrieved */ @@ -474,7 +397,7 @@ get_cheapest_fractional_path_for_pathkeys(List *paths, /* * Since cost comparison is a lot cheaper than pathkey comparison, do - * that first. (XXX is that still true?) + * that first. (XXX is that still true?) */ if (matched_path != NULL && compare_fractional_path_costs(matched_path, path, fraction) <= 0) @@ -544,9 +467,13 @@ build_index_pathkeys(PlannerInfo *root, nulls_first = index->nulls_first[i]; } - /* OK, try to make a canonical pathkey for this sort key */ + /* + * OK, try to make a canonical pathkey for this sort key. Note we're + * underneath any outer joins, so nullable_relids should be NULL. + */ cpathkey = make_pathkey_from_sortinfo(root, indexkey, + NULL, index->sortopfamily[i], index->opcintype[i], index->indexcollations[i], @@ -554,8 +481,7 @@ build_index_pathkeys(PlannerInfo *root, nulls_first, 0, index->rel->relids, - false, - true); + false); /* * If the sort key isn't already present in any EquivalenceClass, then @@ -576,9 +502,60 @@ build_index_pathkeys(PlannerInfo *root, } /* + * build_expression_pathkey + * Build a pathkeys list that describes an ordering by a single expression + * using the given sort operator. + * + * expr, nullable_relids, and rel are as for make_pathkey_from_sortinfo. + * We induce the other arguments assuming default sort order for the operator. + * + * Similarly to make_pathkey_from_sortinfo, the result is NIL if create_it + * is false and the expression isn't already in some EquivalenceClass. + */ +List * +build_expression_pathkey(PlannerInfo *root, + Expr *expr, + Relids nullable_relids, + Oid opno, + Relids rel, + bool create_it) +{ + List *pathkeys; + Oid opfamily, + opcintype; + int16 strategy; + PathKey *cpathkey; + + /* Find the operator in pg_amop --- failure shouldn't happen */ + if (!get_ordering_op_properties(opno, + &opfamily, &opcintype, &strategy)) + elog(ERROR, "operator %u is not a valid ordering operator", + opno); + + cpathkey = make_pathkey_from_sortinfo(root, + expr, + nullable_relids, + opfamily, + opcintype, + exprCollation((Node *) expr), + (strategy == BTGreaterStrategyNumber), + (strategy == BTGreaterStrategyNumber), + 0, + rel, + create_it); + + if (cpathkey) + pathkeys = list_make1(cpathkey); + else + pathkeys = NIL; + + return pathkeys; +} + +/* * convert_subquery_pathkeys * Build a pathkeys list that describes the ordering of a subquery's - * result, in the terms of the outer query. This is essentially a + * result, in the terms of the outer query. This is essentially a * task of conversion. * * 'rel': outer query's RelOptInfo for the subquery relation. @@ -631,15 +608,18 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, /* * Note: it might look funny to be setting sortref = 0 for a - * reference to a volatile sub_eclass. However, the + * reference to a volatile sub_eclass. However, the * expression is *not* volatile in the outer query: it's just * a Var referencing whatever the subquery emitted. (IOW, the * outer query isn't going to re-execute the volatile - * expression itself.) So this is okay. + * expression itself.) So this is okay. Likewise, it's + * correct to pass nullable_relids = NULL, because we're + * underneath any outer joins appearing in the outer query. */ outer_ec = get_eclass_for_sort_expr(root, outer_expr, + NULL, sub_eclass->ec_opfamilies, sub_member->em_datatype, sub_eclass->ec_collation, @@ -665,7 +645,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, /* * Otherwise, the sub_pathkey's EquivalenceClass could contain * multiple elements (representing knowledge that multiple items - * are effectively equal). Each element might match none, one, or + * are effectively equal). Each element might match none, one, or * more of the output columns that are visible to the outer query. * This means we may have multiple possible representations of the * sub_pathkey in the context of the outer query. Ideally we @@ -727,6 +707,7 @@ convert_subquery_pathkeys(PlannerInfo *root, RelOptInfo *rel, /* See if we have a matching EC for that */ outer_ec = get_eclass_for_sort_expr(root, outer_expr, + NULL, sub_eclass->ec_opfamilies, sub_expr_type, sub_expr_coll, @@ -829,12 +810,15 @@ build_join_pathkeys(PlannerInfo *root, * Generate a pathkeys list that represents the sort order specified * by a list of SortGroupClauses * - * If canonicalize is TRUE, the resulting PathKeys are all in canonical form; - * otherwise not. canonicalize should always be TRUE after EquivalenceClass - * merging has been performed, but FALSE if we haven't done EquivalenceClass - * merging yet. (We provide this option because grouping_planner() needs to - * be able to represent requested pathkeys before the equivalence classes have - * been created for the query.) + * The resulting PathKeys are always in canonical form. (Actually, there + * is no longer any code anywhere that creates non-canonical PathKeys.) + * + * We assume that root->nullable_baserels is the set of base relids that could + * have gone to NULL below the SortGroupClause expressions. This is okay if + * the expressions came from the query's top level (ORDER BY, DISTINCT, etc) + * and if this function is only invoked after deconstruct_jointree. In the + * future we might have to make callers pass in the appropriate + * nullable-relids set, but for now it seems unnecessary. * * 'sortclauses' is a list of SortGroupClause nodes * 'tlist' is the targetlist to find the referenced tlist entries in @@ -842,8 +826,7 @@ build_join_pathkeys(PlannerInfo *root, List * make_pathkeys_for_sortclauses(PlannerInfo *root, List *sortclauses, - List *tlist, - bool canonicalize) + List *tlist) { List *pathkeys = NIL; ListCell *l; @@ -858,19 +841,14 @@ make_pathkeys_for_sortclauses(PlannerInfo *root, Assert(OidIsValid(sortcl->sortop)); pathkey = make_pathkey_from_sortop(root, sortkey, + root->nullable_baserels, sortcl->sortop, sortcl->nulls_first, sortcl->tleSortGroupRef, - true, - canonicalize); + true); /* Canonical form eliminates redundant ordering keys */ - if (canonicalize) - { - if (!pathkey_is_redundant(pathkey, pathkeys)) - pathkeys = lappend(pathkeys, pathkey); - } - else + if (!pathkey_is_redundant(pathkey, pathkeys)) pathkeys = lappend(pathkeys, pathkey); } return pathkeys; @@ -895,7 +873,7 @@ make_pathkeys_for_sortclauses(PlannerInfo *root, * right sides. * * Note this is called before EC merging is complete, so the links won't - * necessarily point to canonical ECs. Before they are actually used for + * necessarily point to canonical ECs. Before they are actually used for * anything, update_mergeclause_eclasses must be called to ensure that * they've been updated to point to canonical ECs. */ @@ -919,6 +897,7 @@ initialize_mergeclause_eclasses(PlannerInfo *root, RestrictInfo *restrictinfo) restrictinfo->left_ec = get_eclass_for_sort_expr(root, (Expr *) get_leftop(clause), + restrictinfo->nullable_relids, restrictinfo->mergeopfamilies, lefttype, ((OpExpr *) clause)->inputcollid, @@ -928,6 +907,7 @@ initialize_mergeclause_eclasses(PlannerInfo *root, RestrictInfo *restrictinfo) restrictinfo->right_ec = get_eclass_for_sort_expr(root, (Expr *) get_rightop(clause), + restrictinfo->nullable_relids, restrictinfo->mergeopfamilies, righttype, ((OpExpr *) clause)->inputcollid, @@ -1027,7 +1007,7 @@ find_mergeclauses_for_pathkeys(PlannerInfo *root, * It's possible that multiple matching clauses might have different * ECs on the other side, in which case the order we put them into our * result makes a difference in the pathkeys required for the other - * input path. However this routine hasn't got any info about which + * input path. However this routine hasn't got any info about which * order would be best, so we don't worry about that. * * It's also possible that the selected mergejoin clauses produce @@ -1058,7 +1038,7 @@ find_mergeclauses_for_pathkeys(PlannerInfo *root, /* * If we didn't find a mergeclause, we're done --- any additional - * sort-key positions in the pathkeys are useless. (But we can still + * sort-key positions in the pathkeys are useless. (But we can still * mergejoin if we found at least one mergeclause.) */ if (matched_restrictinfos == NIL) @@ -1090,7 +1070,7 @@ find_mergeclauses_for_pathkeys(PlannerInfo *root, * Returns a pathkeys list that can be applied to the outer relation. * * Since we assume here that a sort is required, there is no particular use - * in matching any available ordering of the outerrel. (joinpath.c has an + * in matching any available ordering of the outerrel. (joinpath.c has an * entirely separate code path for considering sort-free mergejoins.) Rather, * it's interesting to try to match the requested query_pathkeys so that a * second output sort may be avoided; and failing that, we try to list "more @@ -1421,7 +1401,7 @@ pathkeys_useful_for_merging(PlannerInfo *root, RelOptInfo *rel, List *pathkeys) /* * If we didn't find a mergeclause, we're done --- any additional - * sort-key positions in the pathkeys are useless. (But we can still + * sort-key positions in the pathkeys are useless. (But we can still * mergejoin if we found at least one mergeclause.) */ if (matched) @@ -1451,7 +1431,7 @@ right_merge_direction(PlannerInfo *root, PathKey *pathkey) pathkey->pk_opfamily == query_pathkey->pk_opfamily) { /* - * Found a matching query sort column. Prefer this pathkey's + * Found a matching query sort column. Prefer this pathkey's * direction iff it matches. Note that we ignore pk_nulls_first, * which means that a sort might be needed anyway ... but we still * want to prefer only one of the two possible directions, and we @@ -1527,13 +1507,13 @@ truncate_useless_pathkeys(PlannerInfo *root, * useful according to truncate_useless_pathkeys(). * * This is a cheap test that lets us skip building pathkeys at all in very - * simple queries. It's OK to err in the direction of returning "true" when + * simple queries. It's OK to err in the direction of returning "true" when * there really aren't any usable pathkeys, but erring in the other direction * is bad --- so keep this in sync with the routines above! * * We could make the test more complex, for example checking to see if any of * the joinclauses are really mergejoinable, but that likely wouldn't win - * often enough to repay the extra cycles. Queries with neither a join nor + * often enough to repay the extra cycles. Queries with neither a join nor * a sort are reasonably common, though, so this much work seems worthwhile. */ bool diff --git a/src/backend/optimizer/path/tidpath.c b/src/backend/optimizer/path/tidpath.c index 35702c2705..a31d67493b 100644 --- a/src/backend/optimizer/path/tidpath.c +++ b/src/backend/optimizer/path/tidpath.c @@ -19,13 +19,13 @@ * representation all the way through to execution. * * There is currently no special support for joins involving CTID; in - * particular nothing corresponding to best_inner_indexscan(). Since it's + * particular nothing corresponding to best_inner_indexscan(). Since it's * not very useful to store TIDs of one table in another table, there * doesn't seem to be enough use-case to justify adding a lot of code * for that. * * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -57,7 +57,7 @@ static List *TidQualFromRestrictinfo(List *restrictinfo, int varno); * or * pseudoconstant = CTID * - * We check that the CTID Var belongs to relation "varno". That is probably + * We check that the CTID Var belongs to relation "varno". That is probably * redundant considering this is only applied to restriction clauses, but * let's be safe. */ @@ -249,10 +249,19 @@ TidQualFromRestrictinfo(List *restrictinfo, int varno) void create_tidscan_paths(PlannerInfo *root, RelOptInfo *rel) { + Relids required_outer; List *tidquals; + /* + * We don't support pushing join clauses into the quals of a tidscan, but + * it could still have required parameterization due to LATERAL refs in + * its tlist. + */ + required_outer = rel->lateral_relids; + tidquals = TidQualFromRestrictinfo(rel->baserestrictinfo, rel->relid); if (tidquals) - add_path(rel, (Path *) create_tidscan_path(root, rel, tidquals)); + add_path(rel, (Path *) create_tidscan_path(root, rel, tidquals, + required_outer)); } diff --git a/src/backend/optimizer/plan/analyzejoins.c b/src/backend/optimizer/plan/analyzejoins.c index 5abb114ba8..129fc3dfae 100644 --- a/src/backend/optimizer/plan/analyzejoins.c +++ b/src/backend/optimizer/plan/analyzejoins.c @@ -11,7 +11,7 @@ * is that we have to work harder to clean up after ourselves when we modify * the query, since the derived data structures have to be updated too. * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -40,7 +40,7 @@ static List *remove_rel_from_joinlist(List *joinlist, int relid, int *nremoved); * Check for relations that don't actually need to be joined at all, * and remove them from the query. * - * We are passed the current joinlist and return the updated list. Other + * We are passed the current joinlist and return the updated list. Other * data structures that have to be updated are accessible via "root". */ List * @@ -90,7 +90,7 @@ restart: * Restart the scan. This is necessary to ensure we find all * removable joins independently of ordering of the join_info_list * (note that removal of attr_needed bits may make a join appear - * removable that did not before). Also, since we just deleted the + * removable that did not before). Also, since we just deleted the * current list cell, we'd have to have some kluge to continue the * list scan anyway. */ @@ -107,7 +107,7 @@ restart: * We already know that the clause is a binary opclause referencing only the * rels in the current join. The point here is to check whether it has the * form "outerrel_expr op innerrel_expr" or "innerrel_expr op outerrel_expr", - * rather than mixing outer and inner vars on either side. If it matches, + * rather than mixing outer and inner vars on either side. If it matches, * we set the transient flag outer_is_left to identify which side is which. */ static inline bool @@ -154,7 +154,7 @@ join_is_removable(PlannerInfo *root, SpecialJoinInfo *sjinfo) /* * Currently, we only know how to remove left joins to a baserel with - * unique indexes. We can check most of these criteria pretty trivially + * unique indexes. We can check most of these criteria pretty trivially * to avoid doing useless extra work. But checking whether any of the * indexes are unique would require iterating over the indexlist, so for * now we just make sure there are indexes of some sort or other. If none @@ -202,7 +202,9 @@ join_is_removable(PlannerInfo *root, SpecialJoinInfo *sjinfo) * that will be used above the join. We only need to fail if such a PHV * actually references some inner-rel attributes; but the correct check * for that is relatively expensive, so we first check against ph_eval_at, - * which must mention the inner rel if the PHV uses any inner-rel attrs. + * which must mention the inner rel if the PHV uses any inner-rel attrs as + * non-lateral references. Note that if the PHV's syntactic scope is just + * the inner rel, we can't drop the rel even if the PHV is variable-free. */ foreach(l, root->placeholder_list) { @@ -210,9 +212,13 @@ join_is_removable(PlannerInfo *root, SpecialJoinInfo *sjinfo) if (bms_is_subset(phinfo->ph_needed, joinrelids)) continue; /* PHV is not used above the join */ + if (bms_overlap(phinfo->ph_lateral, innerrel->relids)) + return false; /* it references innerrel laterally */ if (!bms_overlap(phinfo->ph_eval_at, innerrel->relids)) continue; /* it definitely doesn't reference innerrel */ - if (bms_overlap(pull_varnos((Node *) phinfo->ph_var), + if (bms_is_subset(phinfo->ph_eval_at, innerrel->relids)) + return false; /* there isn't any other place to eval PHV */ + if (bms_overlap(pull_varnos((Node *) phinfo->ph_var->phexpr), innerrel->relids)) return false; /* it does reference innerrel */ } @@ -298,6 +304,7 @@ remove_rel_from_query(PlannerInfo *root, int relid, Relids joinrelids) List *joininfos; Index rti; ListCell *l; + ListCell *nextl; /* * Mark the rel as "dead" to show it is no longer part of the join tree. @@ -351,24 +358,40 @@ remove_rel_from_query(PlannerInfo *root, int relid, Relids joinrelids) } /* - * Likewise remove references from PlaceHolderVar data structures. + * Likewise remove references from LateralJoinInfo data structures. * - * Here we have a special case: if a PHV's eval_at set is just the target - * relid, we want to leave it that way instead of reducing it to the empty - * set. An empty eval_at set would confuse later processing since it - * would match every possible eval placement. + * If we are deleting a LATERAL subquery, we can forget its + * LateralJoinInfos altogether. Otherwise, make sure the target is not + * included in any lateral_lhs set. (It probably can't be, since that + * should have precluded deciding to remove it; but let's cope anyway.) + */ + for (l = list_head(root->lateral_info_list); l != NULL; l = nextl) + { + LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(l); + + nextl = lnext(l); + ljinfo->lateral_rhs = bms_del_member(ljinfo->lateral_rhs, relid); + if (bms_is_empty(ljinfo->lateral_rhs)) + root->lateral_info_list = list_delete_ptr(root->lateral_info_list, + ljinfo); + else + { + ljinfo->lateral_lhs = bms_del_member(ljinfo->lateral_lhs, relid); + Assert(!bms_is_empty(ljinfo->lateral_lhs)); + } + } + + /* + * Likewise remove references from PlaceHolderVar data structures. */ foreach(l, root->placeholder_list) { PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(l); phinfo->ph_eval_at = bms_del_member(phinfo->ph_eval_at, relid); - if (bms_is_empty(phinfo->ph_eval_at)) /* oops, belay that */ - phinfo->ph_eval_at = bms_add_member(phinfo->ph_eval_at, relid); - + Assert(!bms_is_empty(phinfo->ph_eval_at)); + Assert(!bms_is_member(relid, phinfo->ph_lateral)); phinfo->ph_needed = bms_del_member(phinfo->ph_needed, relid); - /* ph_may_need probably isn't used after this, but fix it anyway */ - phinfo->ph_may_need = bms_del_member(phinfo->ph_may_need, relid); } /* diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index 3ace15943b..88afebb99b 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -10,7 +10,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * Portions Copyright (c) 2012-2014, TransLattice, Inc. - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -25,6 +25,7 @@ #include <math.h> #include "access/skey.h" +#include "catalog/pg_class.h" #include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/makefuncs.h" @@ -36,6 +37,7 @@ #include "optimizer/placeholder.h" #include "optimizer/plancat.h" #include "optimizer/planmain.h" +#include "optimizer/planner.h" #include "optimizer/predtest.h" #include "optimizer/restrictinfo.h" #include "optimizer/subselect.h" @@ -44,6 +46,7 @@ #include "parser/parse_clause.h" #include "parser/parsetree.h" #ifdef PGXC +#include "access/htup_details.h" #include "access/gtm.h" #include "parser/parse_coerce.h" #include "pgxc/pgxc.h" @@ -71,9 +74,9 @@ static Plan *create_plan_recurse(PlannerInfo *root, Path *best_path); static Plan *create_scan_plan(PlannerInfo *root, Path *best_path); -static List *build_relation_tlist(RelOptInfo *rel); +static List *build_path_tlist(PlannerInfo *root, Path *path); static bool use_physical_tlist(PlannerInfo *root, RelOptInfo *rel); -static void disuse_physical_tlist(Plan *plan, Path *path); +static void disuse_physical_tlist(PlannerInfo *root, Plan *plan, Path *path); static Plan *create_gating_plan(PlannerInfo *root, Plan *plan, List *quals); static Plan *create_join_plan(PlannerInfo *root, JoinPath *best_path); static Plan *create_append_plan(PlannerInfo *root, AppendPath *best_path); @@ -145,6 +148,8 @@ static HashJoin *create_hashjoin_plan(PlannerInfo *root, HashPath *best_path, Plan *outer_plan, Plan *inner_plan); static Node *replace_nestloop_params(PlannerInfo *root, Node *expr); static Node *replace_nestloop_params_mutator(Node *node, PlannerInfo *root); +static void process_subquery_nestloop_params(PlannerInfo *root, + List *subplan_params); static List *fix_indexqual_references(PlannerInfo *root, IndexPath *index_path); static List *fix_indexorderby_references(PlannerInfo *root, IndexPath *index_path); static Node *fix_indexqual_operand(Node *node, IndexOptInfo *index, int indexcol); @@ -173,9 +178,7 @@ static BitmapHeapScan *make_bitmap_heapscan(List *qptlist, static TidScan *make_tidscan(List *qptlist, List *qpqual, Index scanrelid, List *tidquals); static FunctionScan *make_functionscan(List *qptlist, List *qpqual, - Index scanrelid, Node *funcexpr, List *funccolnames, - List *funccoltypes, List *funccoltypmods, - List *funccolcollations); + Index scanrelid, List *functions, bool funcordinality); static ValuesScan *make_valuesscan(List *qptlist, List *qpqual, Index scanrelid, List *values_lists); static CteScan *make_ctescan(List *qptlist, List *qpqual, @@ -252,7 +255,7 @@ static int add_sort_column(AttrNumber colIdx, Oid sortOp, Oid coll, /* * create_plan * Creates the access plan for a query by recursively processing the - * desired tree of pathnodes, starting at the node 'best_path'. For + * desired tree of pathnodes, starting at the node 'best_path'. For * every pathnode found, we create a corresponding plan node containing * appropriate id, target list, and qualification information. * @@ -269,6 +272,9 @@ create_plan(PlannerInfo *root, Path *best_path) { Plan *plan; + /* plan_params should not be in use in current query level */ + Assert(root->plan_params == NIL); + /* Initialize this module's private workspace in PlannerInfo */ root->curOuterRels = NULL; root->curOuterParams = NIL; @@ -285,6 +291,12 @@ create_plan(PlannerInfo *root, Path *best_path) if (root->curOuterParams != NIL) elog(ERROR, "failed to assign all NestLoopParams to plan nodes"); + /* + * Reset plan_params to ensure param IDs used for nestloop params are not + * re-used later + */ + root->plan_params = NIL; + return plan; } @@ -374,7 +386,7 @@ create_scan_plan(PlannerInfo *root, Path *best_path) /* * For table scans, rather than using the relation targetlist (which is * only those Vars actually needed by the query), we prefer to generate a - * tlist containing all Vars in order. This will allow the executor to + * tlist containing all Vars in order. This will allow the executor to * optimize away projection of the table tuples, if possible. (Note that * planner.c may replace the tlist we generate here, forcing projection to * occur.) @@ -391,11 +403,13 @@ create_scan_plan(PlannerInfo *root, Path *best_path) tlist = build_physical_tlist(root, rel); /* if fail because of dropped cols, use regular method */ if (tlist == NIL) - tlist = build_relation_tlist(rel); + tlist = build_path_tlist(root, best_path); } } else - tlist = build_relation_tlist(rel); + { + tlist = build_path_tlist(root, best_path); + } /* * Extract the relevant restriction clauses from the parent relation. The @@ -527,11 +541,12 @@ create_scan_plan(PlannerInfo *root, Path *best_path) } /* - * Build a target list (ie, a list of TargetEntry) for a relation. + * Build a target list (ie, a list of TargetEntry) for the Path's output. */ static List * -build_relation_tlist(RelOptInfo *rel) +build_path_tlist(PlannerInfo *root, Path *path) { + RelOptInfo *rel = path->parent; List *tlist = NIL; int resno = 1; ListCell *v; @@ -541,6 +556,15 @@ build_relation_tlist(RelOptInfo *rel) /* Do we really need to copy here? Not sure */ Node *node = (Node *) copyObject(lfirst(v)); + /* + * If it's a parameterized path, there might be lateral references in + * the tlist, which need to be replaced with Params. There's no need + * to remake the TargetEntry nodes, so apply this to each list item + * separately. + */ + if (path->param_info) + node = replace_nestloop_params(root, node); + tlist = lappend(tlist, makeTargetEntry((Expr *) node, resno, NULL, @@ -612,11 +636,11 @@ use_physical_tlist(PlannerInfo *root, RelOptInfo *rel) * * If the plan node immediately above a scan would prefer to get only * needed Vars and not a physical tlist, it must call this routine to - * undo the decision made by use_physical_tlist(). Currently, Hash, Sort, + * undo the decision made by use_physical_tlist(). Currently, Hash, Sort, * and Material nodes want this, so they don't have to store useless columns. */ static void -disuse_physical_tlist(Plan *plan, Path *path) +disuse_physical_tlist(PlannerInfo *root, Plan *plan, Path *path) { /* Only need to undo it for path types handled by create_scan_plan() */ switch (path->pathtype) @@ -632,7 +656,7 @@ disuse_physical_tlist(Plan *plan, Path *path) case T_CteScan: case T_WorkTableScan: case T_ForeignScan: - plan->targetlist = build_relation_tlist(path->parent); + plan->targetlist = build_path_tlist(root, path); break; default: break; @@ -743,7 +767,7 @@ create_join_plan(PlannerInfo *root, JoinPath *best_path) /* * * Expensive function pullups may have pulled local predicates * into - * this path node. Put them in the qpqual of the plan node. * JMH, + * this path node. Put them in the qpqual of the plan node. * JMH, * 6/15/92 */ if (get_loc_restrictinfo(best_path) != NIL) @@ -1398,18 +1422,18 @@ static Plan * create_append_plan(PlannerInfo *root, AppendPath *best_path) { Append *plan; - List *tlist = build_relation_tlist(best_path->path.parent); + List *tlist = build_path_tlist(root, &best_path->path); List *subplans = NIL; ListCell *subpaths; /* - * It is possible for the subplans list to contain only one entry, or even - * no entries. Handle these cases specially. + * The subpaths list could be empty, if every child was proven empty by + * constraint exclusion. In that case generate a dummy plan that returns + * no rows. * - * XXX ideally, if there's just one entry, we'd not bother to generate an - * Append node but just return the single child. At the moment this does - * not work because the varno of the child scan plan won't match the - * parent-rel Vars it'll be asked to emit. + * Note that an AppendPath with no members is also generated in certain + * cases where there was no appending construct at all, but we know the + * relation is empty (see set_dummy_rel_pathlist). */ if (best_path->subpaths == NIL) { @@ -1421,7 +1445,7 @@ create_append_plan(PlannerInfo *root, AppendPath *best_path) NULL); } - /* Normal case with multiple subpaths */ + /* Build the plan for each child */ foreach(subpaths, best_path->subpaths) { Path *subpath = (Path *) lfirst(subpaths); @@ -1429,6 +1453,13 @@ create_append_plan(PlannerInfo *root, AppendPath *best_path) subplans = lappend(subplans, create_plan_recurse(root, subpath)); } + /* + * XXX ideally, if there's just one child, we'd not bother to generate an + * Append node but just return the single child. At the moment this does + * not work because the varno of the child scan plan won't match the + * parent-rel Vars it'll be asked to emit. + */ + plan = make_append(subplans, tlist); return (Plan *) plan; @@ -1446,7 +1477,7 @@ create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path) { MergeAppend *node = makeNode(MergeAppend); Plan *plan = &node->plan; - List *tlist = build_relation_tlist(best_path->path.parent); + List *tlist = build_path_tlist(root, &best_path->path); List *pathkeys = best_path->path.pathkeys; List *subplans = NIL; ListCell *subpaths; @@ -1465,7 +1496,7 @@ create_merge_append_plan(PlannerInfo *root, MergeAppendPath *best_path) /* Compute sort column info, and adjust MergeAppend's tlist as needed */ (void) prepare_sort_from_pathkeys(root, plan, pathkeys, - NULL, + best_path->path.parent->relids, NULL, true, &node->numCols, @@ -1575,7 +1606,7 @@ create_material_plan(PlannerInfo *root, MaterialPath *best_path) subplan = create_plan_recurse(root, best_path->subpath); /* We don't want any excess columns in the materialized tuples */ - disuse_physical_tlist(subplan, best_path->subpath); + disuse_physical_tlist(root, subplan, best_path->subpath); plan = make_material(subplan); @@ -1624,7 +1655,7 @@ create_unique_plan(PlannerInfo *root, UniquePath *best_path) * should be left as-is if we don't need to add any expressions; but if we * do have to add expressions, then a projection step will be needed at * runtime anyway, so we may as well remove unneeded items. Therefore - * newtlist starts from build_relation_tlist() not just a copy of the + * newtlist starts from build_path_tlist() not just a copy of the * subplan's tlist; and we don't install it into the subplan unless we are * sorting or stuff has to be added. */ @@ -1632,7 +1663,7 @@ create_unique_plan(PlannerInfo *root, UniquePath *best_path) uniq_exprs = best_path->uniq_exprs; /* initialize modified subplan tlist as just the "required" vars */ - newtlist = build_relation_tlist(best_path->path.parent); + newtlist = build_path_tlist(root, &best_path->path); nextresno = list_length(newtlist) + 1; newitems = false; @@ -1657,10 +1688,12 @@ create_unique_plan(PlannerInfo *root, UniquePath *best_path) if (newitems || best_path->umethod == UNIQUE_PATH_SORT) { /* - * If the top plan node can't do projections, we need to add a Result - * node to help it along. + * If the top plan node can't do projections and its existing target + * list isn't already what we need, we need to add a Result node to + * help it along. */ - if (!is_projection_capable_plan(subplan)) + if (!is_projection_capable_plan(subplan) && + !tlist_same_exprs(newtlist, subplan->targetlist)) subplan = (Plan *) make_result(root, newtlist, NULL, subplan); else subplan->targetlist = newtlist; @@ -1728,7 +1761,7 @@ create_unique_plan(PlannerInfo *root, UniquePath *best_path) * subplan tlist. */ plan = (Plan *) make_agg(root, - build_relation_tlist(best_path->path.parent), + build_path_tlist(root, &best_path->path), NIL, AGG_HASHED, NULL, @@ -1893,7 +1926,7 @@ create_remotescan_plan(PlannerInfo *root, subplan = create_plan_recurse(root, best_path->subpath); /* We don't want any excess columns in the remote tuples */ - disuse_physical_tlist(subplan, best_path->subpath); + disuse_physical_tlist(root, subplan, best_path->subpath); plan = make_remotesubplan(root, subplan, best_path->path.distribution, @@ -1935,7 +1968,7 @@ find_push_down_plan_int(PlannerInfo *root, Plan *plan, bool force, Plan **parent if (parent && IsA(plan, SubqueryScan)) { Plan *subplan = ((SubqueryScan *)plan)->subplan; - Plan *remote_plan = find_push_down_plan_int(root, ((SubqueryScan *)plan)->subplan, force, + RemoteSubplan *remote_plan = find_push_down_plan_int(root, ((SubqueryScan *)plan)->subplan, force, &((SubqueryScan *)plan)->subplan); /* @@ -1943,7 +1976,7 @@ find_push_down_plan_int(PlannerInfo *root, Plan *plan, bool force, Plan **parent * subquery plan, then we must also update the link stored in the * RelOptInfo corresponding to this subquery */ - if ((remote_plan == subplan) && parent) + if ((((Plan *)remote_plan) == subplan) && parent) { Assert(root); RelOptInfo *rel = find_base_rel(root, ((SubqueryScan *)plan)->scan.scanrelid); @@ -2068,10 +2101,10 @@ create_indexscan_plan(PlannerInfo *root, /* * The qpqual list must contain all restrictions not automatically handled * by the index, other than pseudoconstant clauses which will be handled - * by a separate gating plan node. All the predicates in the indexquals + * by a separate gating plan node. All the predicates in the indexquals * will be checked (either by the index itself, or by nodeIndexscan.c), * but if there are any "special" operators involved then they must be - * included in qpqual. The upshot is that qpqual must contain + * included in qpqual. The upshot is that qpqual must contain * scan_clauses minus whatever appears in indexquals. * * In normal cases simple pointer equality checks will be enough to spot @@ -2208,15 +2241,15 @@ create_bitmap_scan_plan(PlannerInfo *root, /* * The qpqual list must contain all restrictions not automatically handled * by the index, other than pseudoconstant clauses which will be handled - * by a separate gating plan node. All the predicates in the indexquals + * by a separate gating plan node. All the predicates in the indexquals * will be checked (either by the index itself, or by * nodeBitmapHeapscan.c), but if there are any "special" operators - * involved then they must be added to qpqual. The upshot is that qpqual + * involved then they must be added to qpqual. The upshot is that qpqual * must contain scan_clauses minus whatever appears in indexquals. * * This loop is similar to the comparable code in create_indexscan_plan(), * but with some differences because it has to compare the scan clauses to - * stripped (no RestrictInfos) indexquals. See comments there for more + * stripped (no RestrictInfos) indexquals. See comments there for more * info. * * In normal cases simple equal() checks will be enough to spot duplicate @@ -2261,7 +2294,7 @@ create_bitmap_scan_plan(PlannerInfo *root, /* * When dealing with special operators, we will at this point have - * duplicate clauses in qpqual and bitmapqualorig. We may as well drop + * duplicate clauses in qpqual and bitmapqualorig. We may as well drop * 'em from bitmapqualorig, since there's no point in making the tests * twice. */ @@ -2311,9 +2344,6 @@ create_bitmap_scan_plan(PlannerInfo *root, * OR subtrees. This could be done in a less hacky way if we returned the * indexquals in RestrictInfo form, but that would be slower and still pretty * messy, since we'd have to build new RestrictInfos in many cases.) - * - * Note: if you find yourself changing this, you probably need to change - * make_restrictinfo_from_bitmapqual too. */ static Plan * create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, @@ -2376,7 +2406,7 @@ create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, /* * Here, we only detect qual-free subplans. A qual-free subplan would * cause us to generate "... OR true ..." which we may as well reduce - * to just "true". We do not try to eliminate redundant subclauses + * to just "true". We do not try to eliminate redundant subclauses * because (a) it's not as likely as in the AND case, and (b) we might * well be working with hundreds or even thousands of OR conditions, * perhaps from a long IN list. The performance of list_append_unique @@ -2472,7 +2502,7 @@ create_bitmap_subplan(PlannerInfo *root, Path *bitmapqual, /* * We know that the index predicate must have been implied by the * query condition as a whole, but it may or may not be implied by - * the conditions that got pushed into the bitmapqual. Avoid + * the conditions that got pushed into the bitmapqual. Avoid * generating redundant conditions. */ if (!predicate_implied_by(list_make1(pred), ipath->indexclauses)) @@ -2511,6 +2541,7 @@ create_tidscan_plan(PlannerInfo *root, TidPath *best_path, { TidScan *scan_plan; Index scan_relid = best_path->path.parent->relid; + List *tidquals = best_path->tidquals; List *ortidquals; /* it should be a base rel... */ @@ -2523,11 +2554,20 @@ create_tidscan_plan(PlannerInfo *root, TidPath *best_path, /* Reduce RestrictInfo list to bare expressions; ignore pseudoconstants */ scan_clauses = extract_actual_clauses(scan_clauses, false); + /* Replace any outer-relation variables with nestloop params */ + if (best_path->path.param_info) + { + tidquals = (List *) + replace_nestloop_params(root, (Node *) tidquals); + scan_clauses = (List *) + replace_nestloop_params(root, (Node *) scan_clauses); + } + /* * Remove any clauses that are TID quals. This is a bit tricky since the * tidquals list has implicit OR semantics. */ - ortidquals = best_path->tidquals; + ortidquals = tidquals; if (list_length(ortidquals) > 1) ortidquals = list_make1(make_orclause(ortidquals)); scan_clauses = list_difference(scan_clauses, ortidquals); @@ -2535,7 +2575,7 @@ create_tidscan_plan(PlannerInfo *root, TidPath *best_path, scan_plan = make_tidscan(tlist, scan_clauses, scan_relid, - best_path->tidquals); + tidquals); copy_path_costsize(&scan_plan->scan.plan, &best_path->path); @@ -2569,6 +2609,8 @@ create_subqueryscan_plan(PlannerInfo *root, Path *best_path, { scan_clauses = (List *) replace_nestloop_params(root, (Node *) scan_clauses); + process_subquery_nestloop_params(root, + best_path->parent->subplan_params); } scan_plan = make_subqueryscan(tlist, @@ -2593,11 +2635,13 @@ create_functionscan_plan(PlannerInfo *root, Path *best_path, FunctionScan *scan_plan; Index scan_relid = best_path->parent->relid; RangeTblEntry *rte; + List *functions; /* it should be a function base rel... */ Assert(scan_relid > 0); rte = planner_rt_fetch(scan_relid, root); Assert(rte->rtekind == RTE_FUNCTION); + functions = rte->functions; /* Sort clauses into best execution order */ scan_clauses = order_qual_clauses(root, scan_clauses); @@ -2605,12 +2649,17 @@ create_functionscan_plan(PlannerInfo *root, Path *best_path, /* Reduce RestrictInfo list to bare expressions; ignore pseudoconstants */ scan_clauses = extract_actual_clauses(scan_clauses, false); + /* Replace any outer-relation variables with nestloop params */ + if (best_path->param_info) + { + scan_clauses = (List *) + replace_nestloop_params(root, (Node *) scan_clauses); + /* The function expressions could contain nestloop params, too */ + functions = (List *) replace_nestloop_params(root, (Node *) functions); + } + scan_plan = make_functionscan(tlist, scan_clauses, scan_relid, - rte->funcexpr, - rte->eref->colnames, - rte->funccoltypes, - rte->funccoltypmods, - rte->funccolcollations); + functions, rte->funcordinality); copy_path_costsize(&scan_plan->scan.plan, best_path); @@ -2629,11 +2678,13 @@ create_valuesscan_plan(PlannerInfo *root, Path *best_path, ValuesScan *scan_plan; Index scan_relid = best_path->parent->relid; RangeTblEntry *rte; + List *values_lists; /* it should be a values base rel... */ Assert(scan_relid > 0); rte = planner_rt_fetch(scan_relid, root); Assert(rte->rtekind == RTE_VALUES); + values_lists = rte->values_lists; /* Sort clauses into best execution order */ scan_clauses = order_qual_clauses(root, scan_clauses); @@ -2641,8 +2692,18 @@ create_valuesscan_plan(PlannerInfo *root, Path *best_path, /* Reduce RestrictInfo list to bare expressions; ignore pseudoconstants */ scan_clauses = extract_actual_clauses(scan_clauses, false); + /* Replace any outer-relation variables with nestloop params */ + if (best_path->param_info) + { + scan_clauses = (List *) + replace_nestloop_params(root, (Node *) scan_clauses); + /* The values lists could contain nestloop params, too */ + values_lists = (List *) + replace_nestloop_params(root, (Node *) values_lists); + } + scan_plan = make_valuesscan(tlist, scan_clauses, scan_relid, - rte->values_lists); + values_lists); copy_path_costsize(&scan_plan->scan.plan, best_path); @@ -2727,6 +2788,13 @@ create_ctescan_plan(PlannerInfo *root, Path *best_path, /* Reduce RestrictInfo list to bare expressions; ignore pseudoconstants */ scan_clauses = extract_actual_clauses(scan_clauses, false); + /* Replace any outer-relation variables with nestloop params */ + if (best_path->param_info) + { + scan_clauses = (List *) + replace_nestloop_params(root, (Node *) scan_clauses); + } + scan_plan = make_ctescan(tlist, scan_clauses, scan_relid, plan_id, cte_param_id); @@ -2780,6 +2848,13 @@ create_worktablescan_plan(PlannerInfo *root, Path *best_path, /* Reduce RestrictInfo list to bare expressions; ignore pseudoconstants */ scan_clauses = extract_actual_clauses(scan_clauses, false); + /* Replace any outer-relation variables with nestloop params */ + if (best_path->param_info) + { + scan_clauses = (List *) + replace_nestloop_params(root, (Node *) scan_clauses); + } + scan_plan = make_worktablescan(tlist, scan_clauses, scan_relid, cteroot->wt_param_id); @@ -3139,14 +3214,14 @@ create_foreignscan_plan(PlannerInfo *root, ForeignPath *best_path, Assert(rte->rtekind == RTE_RELATION); /* - * Sort clauses into best execution order. We do this first since the FDW + * Sort clauses into best execution order. We do this first since the FDW * might have more info than we do and wish to adjust the ordering. */ scan_clauses = order_qual_clauses(root, scan_clauses); /* * Let the FDW perform its processing on the restriction clauses and - * generate the plan node. Note that the FDW might remove restriction + * generate the plan node. Note that the FDW might remove restriction * clauses that it intends to execute remotely, or even add more (if it * has selected some join clauses for remote use but also wants them * rechecked locally). @@ -3203,16 +3278,16 @@ create_nestloop_plan(PlannerInfo *root, Plan *outer_plan, Plan *inner_plan) { - NestLoop *join_plan; - List *tlist = build_relation_tlist(best_path->path.parent); - List *joinrestrictclauses = best_path->joinrestrictinfo; - List *joinclauses; - List *otherclauses; - Relids outerrelids; - List *nestParams; - ListCell *cell; - ListCell *prev; - ListCell *next; + NestLoop *join_plan; + List *tlist = build_path_tlist(root, &best_path->path); + List *joinrestrictclauses = best_path->joinrestrictinfo; + List *joinclauses; + List *otherclauses; + Relids outerrelids; + List *nestParams; + ListCell *cell; + ListCell *prev; + ListCell *next; /* Sort join qual clauses into best execution order */ joinrestrictclauses = order_qual_clauses(root, joinrestrictclauses); @@ -3315,7 +3390,7 @@ create_mergejoin_plan(PlannerInfo *root, Plan *outer_plan, Plan *inner_plan) { - List *tlist = build_relation_tlist(best_path->jpath.path.parent); + List *tlist = build_path_tlist(root, &best_path->jpath.path); List *joinclauses; List *otherclauses; List *mergeclauses; @@ -3383,7 +3458,7 @@ create_mergejoin_plan(PlannerInfo *root, */ if (best_path->outersortkeys) { - disuse_physical_tlist(outer_plan, best_path->jpath.outerjoinpath); + disuse_physical_tlist(root, outer_plan, best_path->jpath.outerjoinpath); outer_plan = (Plan *) make_sort_from_pathkeys(root, outer_plan, @@ -3396,7 +3471,7 @@ create_mergejoin_plan(PlannerInfo *root, if (best_path->innersortkeys) { - disuse_physical_tlist(inner_plan, best_path->jpath.innerjoinpath); + disuse_physical_tlist(root, inner_plan, best_path->jpath.innerjoinpath); inner_plan = (Plan *) make_sort_from_pathkeys(root, inner_plan, @@ -3610,7 +3685,7 @@ create_hashjoin_plan(PlannerInfo *root, Plan *outer_plan, Plan *inner_plan) { - List *tlist = build_relation_tlist(best_path->jpath.path.parent); + List *tlist = build_path_tlist(root, &best_path->jpath.path); List *joinclauses; List *otherclauses; List *hashclauses; @@ -3667,11 +3742,11 @@ create_hashjoin_plan(PlannerInfo *root, best_path->jpath.outerjoinpath->parent->relids); /* We don't want any excess columns in the hashed tuples */ - disuse_physical_tlist(inner_plan, best_path->jpath.innerjoinpath); + disuse_physical_tlist(root, inner_plan, best_path->jpath.innerjoinpath); /* If we expect batching, suppress excess columns in outer tuples too */ if (best_path->num_batches > 1) - disuse_physical_tlist(outer_plan, best_path->jpath.outerjoinpath); + disuse_physical_tlist(root, outer_plan, best_path->jpath.outerjoinpath); /* * If there is a single join clause and we can identify the outer variable @@ -3859,6 +3934,92 @@ replace_nestloop_params_mutator(Node *node, PlannerInfo *root) } /* + * process_subquery_nestloop_params + * Handle params of a parameterized subquery that need to be fed + * from an outer nestloop. + * + * Currently, that would be *all* params that a subquery in FROM has demanded + * from the current query level, since they must be LATERAL references. + * + * The subplan's references to the outer variables are already represented + * as PARAM_EXEC Params, so we need not modify the subplan here. What we + * do need to do is add entries to root->curOuterParams to signal the parent + * nestloop plan node that it must provide these values. + */ +static void +process_subquery_nestloop_params(PlannerInfo *root, List *subplan_params) +{ + ListCell *ppl; + + foreach(ppl, subplan_params) + { + PlannerParamItem *pitem = (PlannerParamItem *) lfirst(ppl); + + if (IsA(pitem->item, Var)) + { + Var *var = (Var *) pitem->item; + NestLoopParam *nlp; + ListCell *lc; + + /* If not from a nestloop outer rel, complain */ + if (!bms_is_member(var->varno, root->curOuterRels)) + elog(ERROR, "non-LATERAL parameter required by subquery"); + /* Is this param already listed in root->curOuterParams? */ + foreach(lc, root->curOuterParams) + { + nlp = (NestLoopParam *) lfirst(lc); + if (nlp->paramno == pitem->paramId) + { + Assert(equal(var, nlp->paramval)); + /* Present, so nothing to do */ + break; + } + } + if (lc == NULL) + { + /* No, so add it */ + nlp = makeNode(NestLoopParam); + nlp->paramno = pitem->paramId; + nlp->paramval = copyObject(var); + root->curOuterParams = lappend(root->curOuterParams, nlp); + } + } + else if (IsA(pitem->item, PlaceHolderVar)) + { + PlaceHolderVar *phv = (PlaceHolderVar *) pitem->item; + NestLoopParam *nlp; + ListCell *lc; + + /* If not from a nestloop outer rel, complain */ + if (!bms_is_subset(find_placeholder_info(root, phv, false)->ph_eval_at, + root->curOuterRels)) + elog(ERROR, "non-LATERAL parameter required by subquery"); + /* Is this param already listed in root->curOuterParams? */ + foreach(lc, root->curOuterParams) + { + nlp = (NestLoopParam *) lfirst(lc); + if (nlp->paramno == pitem->paramId) + { + Assert(equal(phv, nlp->paramval)); + /* Present, so nothing to do */ + break; + } + } + if (lc == NULL) + { + /* No, so add it */ + nlp = makeNode(NestLoopParam); + nlp->paramno = pitem->paramId; + nlp->paramval = copyObject(phv); + root->curOuterParams = lappend(root->curOuterParams, nlp); + } + } + else + elog(ERROR, "unexpected type of subquery parameter"); + } +} + +/* * fix_indexqual_references * Adjust indexqual clauses to the form the executor's indexqual * machinery needs. @@ -3912,7 +4073,7 @@ fix_indexqual_references(PlannerInfo *root, IndexPath *index_path) /* * Check to see if the indexkey is on the right; if so, commute - * the clause. The indexkey should be the side that refers to + * the clause. The indexkey should be the side that refers to * (only) the base relation. */ if (!bms_equal(rinfo->left_relids, index->rel->relids)) @@ -4006,7 +4167,7 @@ fix_indexqual_references(PlannerInfo *root, IndexPath *index_path) * * This is a simplified version of fix_indexqual_references. The input does * not have RestrictInfo nodes, and we assume that indxpath.c already - * commuted the clauses to put the index keys on the left. Also, we don't + * commuted the clauses to put the index keys on the left. Also, we don't * bother to support any cases except simple OpExprs, since nothing else * is allowed for ordering operators. */ @@ -4245,7 +4406,7 @@ order_qual_clauses(PlannerInfo *root, List *clauses) /* * Sort. We don't use qsort() because it's not guaranteed stable for - * equal keys. The expected number of entries is small enough that a + * equal keys. The expected number of entries is small enough that a * simple insertion sort should be good enough. */ for (i = 1; i < nitems; i++) @@ -4497,11 +4658,8 @@ static FunctionScan * make_functionscan(List *qptlist, List *qpqual, Index scanrelid, - Node *funcexpr, - List *funccolnames, - List *funccoltypes, - List *funccoltypmods, - List *funccolcollations) + List *functions, + bool funcordinality) { FunctionScan *node = makeNode(FunctionScan); Plan *plan = &node->scan.plan; @@ -4512,11 +4670,8 @@ make_functionscan(List *qptlist, plan->lefttree = NULL; plan->righttree = NULL; node->scan.scanrelid = scanrelid; - node->funcexpr = funcexpr; - node->funccolnames = funccolnames; - node->funccoltypes = funccoltypes; - node->funccoltypmods = funccoltypmods; - node->funccolcollations = funccolcollations; + node->functions = functions; + node->funcordinality = funcordinality; return node; } @@ -5454,7 +5609,7 @@ add_sort_column(AttrNumber colIdx, Oid sortOp, Oid coll, bool nulls_first, * prepare_sort_from_pathkeys * Prepare to sort according to given pathkeys * - * This is used to set up for both Sort and MergeAppend nodes. It calculates + * This is used to set up for both Sort and MergeAppend nodes. It calculates * the executor's representation of the sort key information, and adjusts the * plan targetlist if needed to add resjunk sort columns. * @@ -5467,7 +5622,7 @@ add_sort_column(AttrNumber colIdx, Oid sortOp, Oid coll, bool nulls_first, * * We must convert the pathkey information into arrays of sort key column * numbers, sort operator OIDs, collation OIDs, and nulls-first flags, - * which is the representation the executor wants. These are returned into + * which is the representation the executor wants. These are returned into * the output parameters *p_numsortkeys etc. * * When looking for matches to an EquivalenceClass's members, we will only @@ -5886,6 +6041,9 @@ make_sort_from_groupcols(PlannerInfo *root, SortGroupClause *grpcl = (SortGroupClause *) lfirst(l); TargetEntry *tle = get_tle_by_resno(sub_tlist, grpColIdx[numsortkeys]); + if (!tle) + elog(ERROR, "could not retrive tle for sort-from-groupcols"); + sortColIdx[numsortkeys] = tle->resno; sortOperators[numsortkeys] = grpcl->sortop; collations[numsortkeys] = exprCollation((Node *) tle->expr); @@ -5917,7 +6075,7 @@ make_material(Plan *lefttree) * materialize_finished_plan: stick a Material node atop a completed plan * * There are a couple of places where we want to attach a Material node - * after completion of subquery_planner(). This currently requires hackery. + * after completion of subquery_planner(). This currently requires hackery. * Since subquery_planner has already run SS_finalize_plan on the subplan * tree, we have to kluge up parameter lists for the Material node. * Possibly this could be fixed by postponing SS_finalize_plan processing @@ -6126,8 +6284,8 @@ make_agg(PlannerInfo *root, List *tlist, List *qual, * anything for Aggref nodes; this is okay since they are really * comparable to Vars. * - * See notes in grouping_planner about why only make_agg, make_windowagg - * and make_group worry about tlist eval cost. + * See notes in add_tlist_costs_to_plan about why only make_agg, + * make_windowagg and make_group worry about tlist eval cost. */ if (qual) { @@ -6136,10 +6294,7 @@ make_agg(PlannerInfo *root, List *tlist, List *qual, plan->total_cost += qual_cost.startup; plan->total_cost += qual_cost.per_tuple * plan->plan_rows; } - cost_qual_eval(&qual_cost, tlist, root); - plan->startup_cost += qual_cost.startup; - plan->total_cost += qual_cost.startup; - plan->total_cost += qual_cost.per_tuple * plan->plan_rows; + add_tlist_costs_to_plan(root, plan, tlist); plan->qual = qual; plan->targetlist = tlist; @@ -6295,7 +6450,6 @@ make_windowagg(PlannerInfo *root, List *tlist, WindowAgg *node = makeNode(WindowAgg); Plan *plan = &node->plan; Path windowagg_path; /* dummy for result of cost_windowagg */ - QualCost qual_cost; node->winref = winref; node->partNumCols = partNumCols; @@ -6320,13 +6474,10 @@ make_windowagg(PlannerInfo *root, List *tlist, /* * We also need to account for the cost of evaluation of the tlist. * - * See notes in grouping_planner about why only make_agg, make_windowagg - * and make_group worry about tlist eval cost. + * See notes in add_tlist_costs_to_plan about why only make_agg, + * make_windowagg and make_group worry about tlist eval cost. */ - cost_qual_eval(&qual_cost, tlist, root); - plan->startup_cost += qual_cost.startup; - plan->total_cost += qual_cost.startup; - plan->total_cost += qual_cost.per_tuple * plan->plan_rows; + add_tlist_costs_to_plan(root, plan, tlist); plan->targetlist = tlist; plan->lefttree = lefttree; @@ -6377,8 +6528,8 @@ make_group(PlannerInfo *root, * lower plan level and will only be copied by the Group node. Worth * fixing? * - * See notes in grouping_planner about why only make_agg, make_windowagg - * and make_group worry about tlist eval cost. + * See notes in add_tlist_costs_to_plan about why only make_agg, + * make_windowagg and make_group worry about tlist eval cost. */ if (qual) { @@ -6387,10 +6538,7 @@ make_group(PlannerInfo *root, plan->total_cost += qual_cost.startup; plan->total_cost += qual_cost.per_tuple * plan->plan_rows; } - cost_qual_eval(&qual_cost, tlist, root); - plan->startup_cost += qual_cost.startup; - plan->total_cost += qual_cost.startup; - plan->total_cost += qual_cost.per_tuple * plan->plan_rows; + add_tlist_costs_to_plan(root, plan, tlist); plan->qual = qual; plan->targetlist = tlist; @@ -6402,7 +6550,7 @@ make_group(PlannerInfo *root, /* * distinctList is a list of SortGroupClauses, identifying the targetlist items - * that should be considered by the Unique filter. The input path must + * that should be considered by the Unique filter. The input path must * already be sorted accordingly. */ Unique * @@ -6423,7 +6571,7 @@ make_unique(Plan *lefttree, List *distinctList) /* * Charge one cpu_operator_cost per comparison per input tuple. We assume - * all columns get compared at most of the tuples. (XXX probably this is + * all columns get compared at most of the tuples. (XXX probably this is * an overestimate.) */ plan->total_cost += cpu_operator_cost * plan->plan_rows * numCols; @@ -6818,23 +6966,29 @@ make_result(PlannerInfo *root, * Build a ModifyTable plan node * * Currently, we don't charge anything extra for the actual table modification - * work, nor for the RETURNING expressions if any. It would only be window - * dressing, since these are always top-level nodes and there is no way for - * the costs to change any higher-level planning choices. But we might want - * to make it look better sometime. + * work, nor for the WITH CHECK OPTIONS or RETURNING expressions if any. It + * would only be window dressing, since these are always top-level nodes and + * there is no way for the costs to change any higher-level planning choices. + * But we might want to make it look better sometime. */ ModifyTable * -make_modifytable(CmdType operation, bool canSetTag, - List *resultRelations, - List *subplans, List *returningLists, +make_modifytable(PlannerInfo *root, + CmdType operation, bool canSetTag, + List *resultRelations, List *subplans, + List *withCheckOptionLists, List *returningLists, List *rowMarks, int epqParam) { ModifyTable *node = makeNode(ModifyTable); Plan *plan = &node->plan; double total_size; + List *fdw_private_list; ListCell *subnode; + ListCell *lc; + int i; Assert(list_length(resultRelations) == list_length(subplans)); + Assert(withCheckOptionLists == NIL || + list_length(resultRelations) == list_length(withCheckOptionLists)); Assert(returningLists == NIL || list_length(resultRelations) == list_length(returningLists)); @@ -6871,10 +7025,58 @@ make_modifytable(CmdType operation, bool canSetTag, node->resultRelations = resultRelations; node->resultRelIndex = -1; /* will be set correctly in setrefs.c */ node->plans = subplans; + node->withCheckOptionLists = withCheckOptionLists; node->returningLists = returningLists; node->rowMarks = rowMarks; node->epqParam = epqParam; + /* + * For each result relation that is a foreign table, allow the FDW to + * construct private plan data, and accumulate it all into a list. + */ + fdw_private_list = NIL; + i = 0; + foreach(lc, resultRelations) + { + Index rti = lfirst_int(lc); + FdwRoutine *fdwroutine; + List *fdw_private; + + /* + * If possible, we want to get the FdwRoutine from our RelOptInfo for + * the table. But sometimes we don't have a RelOptInfo and must get + * it the hard way. (In INSERT, the target relation is not scanned, + * so it's not a baserel; and there are also corner cases for + * updatable views where the target rel isn't a baserel.) + */ + if (rti < root->simple_rel_array_size && + root->simple_rel_array[rti] != NULL) + { + RelOptInfo *resultRel = root->simple_rel_array[rti]; + + fdwroutine = resultRel->fdwroutine; + } + else + { + RangeTblEntry *rte = planner_rt_fetch(rti, root); + + Assert(rte->rtekind == RTE_RELATION); + if (rte->relkind == RELKIND_FOREIGN_TABLE) + fdwroutine = GetFdwRoutineByRelId(rte->relid); + else + fdwroutine = NULL; + } + + if (fdwroutine != NULL && + fdwroutine->PlanForeignModify != NULL) + fdw_private = fdwroutine->PlanForeignModify(root, node, rti, i); + else + fdw_private = NIL; + fdw_private_list = lappend(fdw_private_list, fdw_private); + i++; + } + node->fdwPrivLists = fdw_private_list; + return node; } diff --git a/src/backend/optimizer/plan/initsplan.c b/src/backend/optimizer/plan/initsplan.c index 3c7fa632b8..f88e493edb 100644 --- a/src/backend/optimizer/plan/initsplan.c +++ b/src/backend/optimizer/plan/initsplan.c @@ -3,7 +3,7 @@ * initsplan.c * Target list, qualification, joininfo initialization routines * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -22,9 +22,12 @@ #include "optimizer/paths.h" #include "optimizer/placeholder.h" #include "optimizer/planmain.h" +#include "optimizer/planner.h" #include "optimizer/prep.h" #include "optimizer/restrictinfo.h" #include "optimizer/var.h" +#include "parser/analyze.h" +#include "rewrite/rewriteManip.h" #include "utils/lsyscache.h" @@ -33,9 +36,21 @@ int from_collapse_limit; int join_collapse_limit; +/* Elements of the postponed_qual_list used during deconstruct_recurse */ +typedef struct PostponedQual +{ + Node *qual; /* a qual clause waiting to be processed */ + Relids relids; /* the set of baserels it references */ +} PostponedQual; + + +static void extract_lateral_references(PlannerInfo *root, RelOptInfo *brel, + Index rtindex); +static void add_lateral_info(PlannerInfo *root, Relids lhs, Relids rhs); static List *deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, - Relids *qualscope, Relids *inner_join_rels); + Relids *qualscope, Relids *inner_join_rels, + List **postponed_qual_list); static SpecialJoinInfo *make_outerjoininfo(PlannerInfo *root, Relids left_rels, Relids right_rels, Relids inner_join_rels, @@ -46,9 +61,13 @@ static void distribute_qual_to_rels(PlannerInfo *root, Node *clause, JoinType jointype, Relids qualscope, Relids ojscope, - Relids outerjoin_nonnullable); + Relids outerjoin_nonnullable, + Relids deduced_nullable_relids, + List **postponed_qual_list); static bool check_outerjoin_delay(PlannerInfo *root, Relids *relids_p, Relids *nullable_relids_p, bool is_pushed_down); +static bool check_equivalence_delay(PlannerInfo *root, + RestrictInfo *restrictinfo); static bool check_redundant_nullability_qual(PlannerInfo *root, Node *clause); static void check_mergejoinable(RestrictInfo *restrictinfo); static void check_hashjoinable(RestrictInfo *restrictinfo); @@ -68,12 +87,12 @@ static void check_hashjoinable(RestrictInfo *restrictinfo); * appearing in the jointree. * * The initial invocation must pass root->parse->jointree as the value of - * jtnode. Internally, the function recurses through the jointree. + * jtnode. Internally, the function recurses through the jointree. * * At the end of this process, there should be one baserel RelOptInfo for * every non-join RTE that is used in the query. Therefore, this routine * is the only place that should call build_simple_rel with reloptkind - * RELOPT_BASEREL. (Note: build_simple_rel recurses internally to build + * RELOPT_BASEREL. (Note: build_simple_rel recurses internally to build * "other rel" RelOptInfos for the members of any appendrels we find here.) */ void @@ -146,10 +165,9 @@ build_base_rel_tlists(PlannerInfo *root, List *final_tlist) * The list may also contain PlaceHolderVars. These don't necessarily * have a single owning relation; we keep their attr_needed info in * root->placeholder_list instead. If create_new_ph is true, it's OK - * to create new PlaceHolderInfos, and we also have to update ph_may_need; - * otherwise, the PlaceHolderInfos must already exist, and we should only - * update their ph_needed. (It should be true before deconstruct_jointree - * begins, and false after that.) + * to create new PlaceHolderInfos; otherwise, the PlaceHolderInfos must + * already exist, and we should only update their ph_needed. (This should + * be true before deconstruct_jointree begins, and false after that.) */ void add_vars_to_targetlist(PlannerInfo *root, List *vars, @@ -169,6 +187,8 @@ add_vars_to_targetlist(PlannerInfo *root, List *vars, RelOptInfo *rel = find_base_rel(root, var->varno); int attno = var->varattno; + if (bms_is_subset(where_needed, rel->relids)) + continue; Assert(attno >= rel->min_attr && attno <= rel->max_attr); attno -= rel->min_attr; if (rel->attr_needed[attno] == NULL) @@ -187,21 +207,401 @@ add_vars_to_targetlist(PlannerInfo *root, List *vars, PlaceHolderInfo *phinfo = find_placeholder_info(root, phv, create_new_ph); - /* Always adjust ph_needed */ phinfo->ph_needed = bms_add_members(phinfo->ph_needed, where_needed); + } + else + elog(ERROR, "unrecognized node type: %d", (int) nodeTag(node)); + } +} + + +/***************************************************************************** + * + * LATERAL REFERENCES + * + *****************************************************************************/ + +/* + * find_lateral_references + * For each LATERAL subquery, extract all its references to Vars and + * PlaceHolderVars of the current query level, and make sure those values + * will be available for evaluation of the subquery. + * + * While later planning steps ensure that the Var/PHV source rels are on the + * outside of nestloops relative to the LATERAL subquery, we also need to + * ensure that the Vars/PHVs propagate up to the nestloop join level; this + * means setting suitable where_needed values for them. + * + * Note that this only deals with lateral references in unflattened LATERAL + * subqueries. When we flatten a LATERAL subquery, its lateral references + * become plain Vars in the parent query, but they may have to be wrapped in + * PlaceHolderVars if they need to be forced NULL by outer joins that don't + * also null the LATERAL subquery. That's all handled elsewhere. + * + * This has to run before deconstruct_jointree, since it might result in + * creation of PlaceHolderInfos. + */ +void +find_lateral_references(PlannerInfo *root) +{ + Index rti; + + /* We need do nothing if the query contains no LATERAL RTEs */ + if (!root->hasLateralRTEs) + return; + + /* + * Examine all baserels (the rel array has been set up by now). + */ + for (rti = 1; rti < root->simple_rel_array_size; rti++) + { + RelOptInfo *brel = root->simple_rel_array[rti]; + + /* there may be empty slots corresponding to non-baserel RTEs */ + if (brel == NULL) + continue; + + Assert(brel->relid == rti); /* sanity check on array */ + + /* + * This bit is less obvious than it might look. We ignore appendrel + * otherrels and consider only their parent baserels. In a case where + * a LATERAL-containing UNION ALL subquery was pulled up, it is the + * otherrel that is actually going to be in the plan. However, we + * want to mark all its lateral references as needed by the parent, + * because it is the parent's relid that will be used for join + * planning purposes. And the parent's RTE will contain all the + * lateral references we need to know, since the pulled-up member is + * nothing but a copy of parts of the original RTE's subquery. We + * could visit the parent's children instead and transform their + * references back to the parent's relid, but it would be much more + * complicated for no real gain. (Important here is that the child + * members have not yet received any processing beyond being pulled + * up.) Similarly, in appendrels created by inheritance expansion, + * it's sufficient to look at the parent relation. + */ + + /* ignore RTEs that are "other rels" */ + if (brel->reloptkind != RELOPT_BASEREL) + continue; + + extract_lateral_references(root, brel, rti); + } +} + +static void +extract_lateral_references(PlannerInfo *root, RelOptInfo *brel, Index rtindex) +{ + RangeTblEntry *rte = root->simple_rte_array[rtindex]; + List *vars; + List *newvars; + Relids where_needed; + ListCell *lc; + + /* No cross-references are possible if it's not LATERAL */ + if (!rte->lateral) + return; + + /* Fetch the appropriate variables */ + if (rte->rtekind == RTE_SUBQUERY) + vars = pull_vars_of_level((Node *) rte->subquery, 1); + else if (rte->rtekind == RTE_FUNCTION) + vars = pull_vars_of_level((Node *) rte->functions, 0); + else if (rte->rtekind == RTE_VALUES) + vars = pull_vars_of_level((Node *) rte->values_lists, 0); + else + { + Assert(false); + return; /* keep compiler quiet */ + } + + if (vars == NIL) + return; /* nothing to do */ + + /* Copy each Var (or PlaceHolderVar) and adjust it to match our level */ + newvars = NIL; + foreach(lc, vars) + { + Node *node = (Node *) lfirst(lc); + + node = copyObject(node); + if (IsA(node, Var)) + { + Var *var = (Var *) node; + + /* Adjustment is easy since it's just one node */ + var->varlevelsup = 0; + } + else if (IsA(node, PlaceHolderVar)) + { + PlaceHolderVar *phv = (PlaceHolderVar *) node; + int levelsup = phv->phlevelsup; + + /* Have to work harder to adjust the contained expression too */ + if (levelsup != 0) + IncrementVarSublevelsUp(node, -levelsup, 0); /* - * If we are creating PlaceHolderInfos, mark them with the correct - * maybe-needed locations. Otherwise, it's too late to change - * that. + * If we pulled the PHV out of a subquery RTE, its expression + * needs to be preprocessed. subquery_planner() already did this + * for level-zero PHVs in function and values RTEs, though. */ - if (create_new_ph) - mark_placeholder_maybe_needed(root, phinfo, where_needed); + if (levelsup > 0) + phv->phexpr = preprocess_phv_expression(root, phv->phexpr); } else - elog(ERROR, "unrecognized node type: %d", (int) nodeTag(node)); + Assert(false); + newvars = lappend(newvars, node); + } + + list_free(vars); + + /* + * We mark the Vars as being "needed" at the LATERAL RTE. This is a bit + * of a cheat: a more formal approach would be to mark each one as needed + * at the join of the LATERAL RTE with its source RTE. But it will work, + * and it's much less tedious than computing a separate where_needed for + * each Var. + */ + where_needed = bms_make_singleton(rtindex); + + /* + * Push Vars into their source relations' targetlists, and PHVs into + * root->placeholder_list. + */ + add_vars_to_targetlist(root, newvars, where_needed, true); + + /* Remember the lateral references for create_lateral_join_info */ + brel->lateral_vars = newvars; +} + +/* + * create_lateral_join_info + * For each unflattened LATERAL subquery, create LateralJoinInfo(s) and add + * them to root->lateral_info_list, and fill in the per-rel lateral_relids + * and lateral_referencers sets. Also generate LateralJoinInfo(s) to + * represent any lateral references within PlaceHolderVars (this part deals + * with the effects of flattened LATERAL subqueries). + * + * This has to run after deconstruct_jointree, because we need to know the + * final ph_eval_at values for PlaceHolderVars. + */ +void +create_lateral_join_info(PlannerInfo *root) +{ + Index rti; + ListCell *lc; + + /* We need do nothing if the query contains no LATERAL RTEs */ + if (!root->hasLateralRTEs) + return; + + /* + * Examine all baserels (the rel array has been set up by now). + */ + for (rti = 1; rti < root->simple_rel_array_size; rti++) + { + RelOptInfo *brel = root->simple_rel_array[rti]; + Relids lateral_relids; + + /* there may be empty slots corresponding to non-baserel RTEs */ + if (brel == NULL) + continue; + + Assert(brel->relid == rti); /* sanity check on array */ + + /* ignore RTEs that are "other rels" */ + if (brel->reloptkind != RELOPT_BASEREL) + continue; + + lateral_relids = NULL; + + /* consider each laterally-referenced Var or PHV */ + foreach(lc, brel->lateral_vars) + { + Node *node = (Node *) lfirst(lc); + + if (IsA(node, Var)) + { + Var *var = (Var *) node; + + add_lateral_info(root, bms_make_singleton(var->varno), + brel->relids); + lateral_relids = bms_add_member(lateral_relids, + var->varno); + } + else if (IsA(node, PlaceHolderVar)) + { + PlaceHolderVar *phv = (PlaceHolderVar *) node; + PlaceHolderInfo *phinfo = find_placeholder_info(root, phv, + false); + + add_lateral_info(root, phinfo->ph_eval_at, brel->relids); + lateral_relids = bms_add_members(lateral_relids, + phinfo->ph_eval_at); + } + else + Assert(false); + } + + /* We now know all the relids needed for lateral refs in this rel */ + if (bms_is_empty(lateral_relids)) + continue; /* ensure lateral_relids is NULL if empty */ + brel->lateral_relids = lateral_relids; + } + + /* + * Now check for lateral references within PlaceHolderVars, and make + * LateralJoinInfos describing each such reference. Unlike references in + * unflattened LATERAL RTEs, the referencing location could be a join. + */ + foreach(lc, root->placeholder_list) + { + PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(lc); + Relids eval_at = phinfo->ph_eval_at; + + if (phinfo->ph_lateral != NULL) + { + List *vars = pull_var_clause((Node *) phinfo->ph_var->phexpr, + PVC_RECURSE_AGGREGATES, + PVC_INCLUDE_PLACEHOLDERS); + ListCell *lc2; + + foreach(lc2, vars) + { + Node *node = (Node *) lfirst(lc2); + + if (IsA(node, Var)) + { + Var *var = (Var *) node; + + if (!bms_is_member(var->varno, eval_at)) + add_lateral_info(root, + bms_make_singleton(var->varno), + eval_at); + } + else if (IsA(node, PlaceHolderVar)) + { + PlaceHolderVar *other_phv = (PlaceHolderVar *) node; + PlaceHolderInfo *other_phi; + + other_phi = find_placeholder_info(root, other_phv, + false); + if (!bms_is_subset(other_phi->ph_eval_at, eval_at)) + add_lateral_info(root, other_phi->ph_eval_at, eval_at); + } + else + Assert(false); + } + + list_free(vars); + } + } + + /* If we found no lateral references, we're done. */ + if (root->lateral_info_list == NIL) + return; + + /* + * Now that we've identified all lateral references, make a second pass in + * which we mark each baserel with the set of relids of rels that + * reference it laterally (essentially, the inverse mapping of + * lateral_relids). We'll need this for join_clause_is_movable_to(). + * + * Also, propagate lateral_relids and lateral_referencers from appendrel + * parent rels to their child rels. We intentionally give each child rel + * the same minimum parameterization, even though it's quite possible that + * some don't reference all the lateral rels. This is because any append + * path for the parent will have to have the same parameterization for + * every child anyway, and there's no value in forcing extra + * reparameterize_path() calls. Similarly, a lateral reference to the + * parent prevents use of otherwise-movable join rels for each child. + */ + for (rti = 1; rti < root->simple_rel_array_size; rti++) + { + RelOptInfo *brel = root->simple_rel_array[rti]; + Relids lateral_referencers; + + if (brel == NULL) + continue; + if (brel->reloptkind != RELOPT_BASEREL) + continue; + + /* Compute lateral_referencers using the finished lateral_info_list */ + lateral_referencers = NULL; + foreach(lc, root->lateral_info_list) + { + LateralJoinInfo *ljinfo = (LateralJoinInfo *) lfirst(lc); + + if (bms_is_member(brel->relid, ljinfo->lateral_lhs)) + lateral_referencers = bms_add_members(lateral_referencers, + ljinfo->lateral_rhs); + } + brel->lateral_referencers = lateral_referencers; + + /* + * If it's an appendrel parent, copy its lateral_relids and + * lateral_referencers to each child rel. + */ + if (root->simple_rte_array[rti]->inh) + { + foreach(lc, root->append_rel_list) + { + AppendRelInfo *appinfo = (AppendRelInfo *) lfirst(lc); + RelOptInfo *childrel; + + if (appinfo->parent_relid != rti) + continue; + childrel = root->simple_rel_array[appinfo->child_relid]; + Assert(childrel->reloptkind == RELOPT_OTHER_MEMBER_REL); + Assert(childrel->lateral_relids == NULL); + childrel->lateral_relids = brel->lateral_relids; + Assert(childrel->lateral_referencers == NULL); + childrel->lateral_referencers = brel->lateral_referencers; + } + } + } +} + +/* + * add_lateral_info + * Add a LateralJoinInfo to root->lateral_info_list, if needed + * + * We suppress redundant list entries. The passed Relids are copied if saved. + */ +static void +add_lateral_info(PlannerInfo *root, Relids lhs, Relids rhs) +{ + LateralJoinInfo *ljinfo; + ListCell *lc; + + /* Sanity-check the input */ + Assert(!bms_is_empty(lhs)); + Assert(!bms_is_empty(rhs)); + Assert(!bms_overlap(lhs, rhs)); + + /* + * The input is redundant if it has the same RHS and an LHS that is a + * subset of an existing entry's. If an existing entry has the same RHS + * and an LHS that is a subset of the new one, it's redundant, but we + * don't trouble to get rid of it. The only case that is really worth + * worrying about is identical entries, and we handle that well enough + * with this simple logic. + */ + foreach(lc, root->lateral_info_list) + { + ljinfo = (LateralJoinInfo *) lfirst(lc); + if (bms_equal(rhs, ljinfo->lateral_rhs) && + bms_is_subset(lhs, ljinfo->lateral_lhs)) + return; } + + /* Not there, so make a new entry */ + ljinfo = makeNode(LateralJoinInfo); + ljinfo->lateral_lhs = bms_copy(lhs); + ljinfo->lateral_rhs = bms_copy(rhs); + root->lateral_info_list = lappend(root->lateral_info_list, ljinfo); } @@ -215,7 +615,7 @@ add_vars_to_targetlist(PlannerInfo *root, List *vars, * deconstruct_jointree * Recursively scan the query's join tree for WHERE and JOIN/ON qual * clauses, and add these to the appropriate restrictinfo and joininfo - * lists belonging to base RelOptInfos. Also, add SpecialJoinInfo nodes + * lists belonging to base RelOptInfos. Also, add SpecialJoinInfo nodes * to root->join_info_list for any outer joins appearing in the query tree. * Return a "joinlist" data structure showing the join order decisions * that need to be made by make_one_rel(). @@ -232,23 +632,34 @@ add_vars_to_targetlist(PlannerInfo *root, List *vars, * be evaluated at the lowest level where all the variables it mentions are * available. However, we cannot push a qual down into the nullable side(s) * of an outer join since the qual might eliminate matching rows and cause a - * NULL row to be incorrectly emitted by the join. Therefore, we artificially + * NULL row to be incorrectly emitted by the join. Therefore, we artificially * OR the minimum-relids of such an outer join into the required_relids of - * clauses appearing above it. This forces those clauses to be delayed until + * clauses appearing above it. This forces those clauses to be delayed until * application of the outer join (or maybe even higher in the join tree). */ List * deconstruct_jointree(PlannerInfo *root) { + List *result; Relids qualscope; Relids inner_join_rels; + List *postponed_qual_list = NIL; /* Start recursion at top of jointree */ Assert(root->parse->jointree != NULL && IsA(root->parse->jointree, FromExpr)); - return deconstruct_recurse(root, (Node *) root->parse->jointree, false, - &qualscope, &inner_join_rels); + /* this is filled as we scan the jointree */ + root->nullable_baserels = NULL; + + result = deconstruct_recurse(root, (Node *) root->parse->jointree, false, + &qualscope, &inner_join_rels, + &postponed_qual_list); + + /* Shouldn't be any leftover quals */ + Assert(postponed_qual_list == NIL); + + return result; } /* @@ -266,13 +677,16 @@ deconstruct_jointree(PlannerInfo *root) * *inner_join_rels gets the set of base Relids syntactically included in * inner joins appearing at or below this jointree node (do not modify * or free this, either) + * *postponed_qual_list is a list of PostponedQual structs, which we can + * add quals to if they turn out to belong to a higher join level * Return value is the appropriate joinlist for this jointree node * * In addition, entries will be added to root->join_info_list for outer joins. */ static List * deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, - Relids *qualscope, Relids *inner_join_rels) + Relids *qualscope, Relids *inner_join_rels, + List **postponed_qual_list) { List *joinlist; @@ -295,6 +709,7 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, else if (IsA(jtnode, FromExpr)) { FromExpr *f = (FromExpr *) jtnode; + List *child_postponed_quals = NIL; int remaining; ListCell *l; @@ -317,7 +732,8 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, sub_joinlist = deconstruct_recurse(root, lfirst(l), below_outer_join, &sub_qualscope, - inner_join_rels); + inner_join_rels, + &child_postponed_quals); *qualscope = bms_add_members(*qualscope, sub_qualscope); sub_members = list_length(sub_joinlist); remaining--; @@ -339,6 +755,23 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, *inner_join_rels = *qualscope; /* + * Try to process any quals postponed by children. If they need + * further postponement, add them to my output postponed_qual_list. + */ + foreach(l, child_postponed_quals) + { + PostponedQual *pq = (PostponedQual *) lfirst(l); + + if (bms_is_subset(pq->relids, *qualscope)) + distribute_qual_to_rels(root, pq->qual, + false, below_outer_join, JOIN_INNER, + *qualscope, NULL, NULL, NULL, + NULL); + else + *postponed_qual_list = lappend(*postponed_qual_list, pq); + } + + /* * Now process the top-level quals. */ foreach(l, (List *) f->quals) @@ -347,20 +780,24 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, distribute_qual_to_rels(root, qual, false, below_outer_join, JOIN_INNER, - *qualscope, NULL, NULL); + *qualscope, NULL, NULL, NULL, + postponed_qual_list); } } else if (IsA(jtnode, JoinExpr)) { JoinExpr *j = (JoinExpr *) jtnode; + List *child_postponed_quals = NIL; Relids leftids, rightids, left_inners, right_inners, nonnullable_rels, + nullable_rels, ojscope; List *leftjoinlist, *rightjoinlist; + List *my_quals; SpecialJoinInfo *sjinfo; ListCell *l; @@ -370,7 +807,7 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, * regard for whether this level is an outer join, which is correct. * Then we place our own join quals, which are restricted by lower * outer joins in any case, and are forced to this level if this is an - * outer join and they mention the outer side. Finally, if this is an + * outer join and they mention the outer side. Finally, if this is an * outer join, we create a join_info_list entry for the join. This * will prevent quals above us in the join tree that use those rels * from being pushed down below this level. (It's okay for upper @@ -381,60 +818,110 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, case JOIN_INNER: leftjoinlist = deconstruct_recurse(root, j->larg, below_outer_join, - &leftids, &left_inners); + &leftids, &left_inners, + &child_postponed_quals); rightjoinlist = deconstruct_recurse(root, j->rarg, below_outer_join, - &rightids, &right_inners); + &rightids, &right_inners, + &child_postponed_quals); *qualscope = bms_union(leftids, rightids); *inner_join_rels = *qualscope; /* Inner join adds no restrictions for quals */ nonnullable_rels = NULL; + /* and it doesn't force anything to null, either */ + nullable_rels = NULL; break; case JOIN_LEFT: case JOIN_ANTI: leftjoinlist = deconstruct_recurse(root, j->larg, below_outer_join, - &leftids, &left_inners); + &leftids, &left_inners, + &child_postponed_quals); rightjoinlist = deconstruct_recurse(root, j->rarg, true, - &rightids, &right_inners); + &rightids, &right_inners, + &child_postponed_quals); *qualscope = bms_union(leftids, rightids); *inner_join_rels = bms_union(left_inners, right_inners); nonnullable_rels = leftids; + nullable_rels = rightids; break; case JOIN_SEMI: leftjoinlist = deconstruct_recurse(root, j->larg, below_outer_join, - &leftids, &left_inners); + &leftids, &left_inners, + &child_postponed_quals); rightjoinlist = deconstruct_recurse(root, j->rarg, below_outer_join, - &rightids, &right_inners); + &rightids, &right_inners, + &child_postponed_quals); *qualscope = bms_union(leftids, rightids); *inner_join_rels = bms_union(left_inners, right_inners); /* Semi join adds no restrictions for quals */ nonnullable_rels = NULL; + + /* + * Theoretically, a semijoin would null the RHS; but since the + * RHS can't be accessed above the join, this is immaterial + * and we needn't account for it. + */ + nullable_rels = NULL; break; case JOIN_FULL: leftjoinlist = deconstruct_recurse(root, j->larg, true, - &leftids, &left_inners); + &leftids, &left_inners, + &child_postponed_quals); rightjoinlist = deconstruct_recurse(root, j->rarg, true, - &rightids, &right_inners); + &rightids, &right_inners, + &child_postponed_quals); *qualscope = bms_union(leftids, rightids); *inner_join_rels = bms_union(left_inners, right_inners); /* each side is both outer and inner */ nonnullable_rels = *qualscope; + nullable_rels = *qualscope; break; default: /* JOIN_RIGHT was eliminated during reduce_outer_joins() */ elog(ERROR, "unrecognized join type: %d", (int) j->jointype); nonnullable_rels = NULL; /* keep compiler quiet */ + nullable_rels = NULL; leftjoinlist = rightjoinlist = NIL; break; } + /* Report all rels that will be nulled anywhere in the jointree */ + root->nullable_baserels = bms_add_members(root->nullable_baserels, + nullable_rels); + + /* + * Try to process any quals postponed by children. If they need + * further postponement, add them to my output postponed_qual_list. + * Quals that can be processed now must be included in my_quals, so + * that they'll be handled properly in make_outerjoininfo. + */ + my_quals = NIL; + foreach(l, child_postponed_quals) + { + PostponedQual *pq = (PostponedQual *) lfirst(l); + + if (bms_is_subset(pq->relids, *qualscope)) + my_quals = lappend(my_quals, pq->qual); + else + { + /* + * We should not be postponing any quals past an outer join. + * If this Assert fires, pull_up_subqueries() messed up. + */ + Assert(j->jointype == JOIN_INNER); + *postponed_qual_list = lappend(*postponed_qual_list, pq); + } + } + /* list_concat is nondestructive of its second argument */ + my_quals = list_concat(my_quals, (List *) j->quals); + /* * For an OJ, form the SpecialJoinInfo now, because we need the OJ's * semantic scope (ojscope) to pass to distribute_qual_to_rels. But @@ -450,7 +937,7 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, leftids, rightids, *inner_join_rels, j->jointype, - (List *) j->quals); + my_quals); if (j->jointype == JOIN_SEMI) ojscope = NULL; else @@ -463,15 +950,16 @@ deconstruct_recurse(PlannerInfo *root, Node *jtnode, bool below_outer_join, ojscope = NULL; } - /* Process the qual clauses */ - foreach(l, (List *) j->quals) + /* Process the JOIN's qual clauses */ + foreach(l, my_quals) { Node *qual = (Node *) lfirst(l); distribute_qual_to_rels(root, qual, false, below_outer_join, j->jointype, *qualscope, - ojscope, nonnullable_rels); + ojscope, nonnullable_rels, NULL, + postponed_qual_list); } /* Now we can add the SpecialJoinInfo to join_info_list */ @@ -564,14 +1052,14 @@ make_outerjoininfo(PlannerInfo *root, Assert(jointype != JOIN_RIGHT); /* - * Presently the executor cannot support FOR UPDATE/SHARE marking of rels - * appearing on the nullable side of an outer join. (It's somewhat unclear - * what that would mean, anyway: what should we mark when a result row is - * generated from no element of the nullable relation?) So, complain if - * any nullable rel is FOR UPDATE/SHARE. + * Presently the executor cannot support FOR [KEY] UPDATE/SHARE marking of + * rels appearing on the nullable side of an outer join. (It's somewhat + * unclear what that would mean, anyway: what should we mark when a result + * row is generated from no element of the nullable relation?) So, + * complain if any nullable rel is FOR [KEY] UPDATE/SHARE. * * You might be wondering why this test isn't made far upstream in the - * parser. It's because the parser hasn't got enough info --- consider + * parser. It's because the parser hasn't got enough info --- consider * FOR UPDATE applied to a view. Only after rewriting and flattening do * we know whether the view contains an outer join. * @@ -586,7 +1074,10 @@ make_outerjoininfo(PlannerInfo *root, (jointype == JOIN_FULL && bms_is_member(rc->rti, left_rels))) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("SELECT FOR UPDATE/SHARE cannot be applied to the nullable side of an outer join"))); + /*------ + translator: %s is a SQL row locking clause such as FOR UPDATE */ + errmsg("%s cannot be applied to the nullable side of an outer join", + LCS_asString(rc->strength)))); } sjinfo->syn_lefthand = left_rels; @@ -626,7 +1117,7 @@ make_outerjoininfo(PlannerInfo *root, min_lefthand = bms_intersect(clause_relids, left_rels); /* - * Similarly for required RHS. But here, we must also include any lower + * Similarly for required RHS. But here, we must also include any lower * inner joins, to ensure we don't try to commute with any of them. */ min_righthand = bms_int_members(bms_union(clause_relids, inner_join_rels), @@ -678,7 +1169,7 @@ make_outerjoininfo(PlannerInfo *root, * Here, we have to consider that "our join condition" includes any * clauses that syntactically appeared above the lower OJ and below * ours; those are equivalent to degenerate clauses in our OJ and must - * be treated as such. Such clauses obviously can't reference our + * be treated as such. Such clauses obviously can't reference our * LHS, and they must be non-strict for the lower OJ's RHS (else * reduce_outer_joins would have reduced the lower OJ to a plain * join). Hence the other ways in which we handle clauses within our @@ -704,11 +1195,11 @@ make_outerjoininfo(PlannerInfo *root, /* * Examine PlaceHolderVars. If a PHV is supposed to be evaluated within - * this join's nullable side, and it may get used above this join, then - * ensure that min_righthand contains the full eval_at set of the PHV. - * This ensures that the PHV actually can be evaluated within the RHS. - * Note that this works only because we should already have determined the - * final eval_at level for any PHV syntactically within this join. + * this join's nullable side, then ensure that min_righthand contains the + * full eval_at set of the PHV. This ensures that the PHV actually can be + * evaluated within the RHS. Note that this works only because we should + * already have determined the final eval_at level for any PHV + * syntactically within this join. */ foreach(l, root->placeholder_list) { @@ -719,11 +1210,6 @@ make_outerjoininfo(PlannerInfo *root, if (!bms_is_subset(ph_syn_level, right_rels)) continue; - /* We can also ignore it if it's certainly not used above this join */ - /* XXX this test is probably overly conservative */ - if (bms_is_subset(phinfo->ph_may_need, min_righthand)) - continue; - /* Else, prevent join from being formed before we eval the PHV */ min_righthand = bms_add_members(min_righthand, phinfo->ph_eval_at); } @@ -762,11 +1248,12 @@ make_outerjoininfo(PlannerInfo *root, * distribute_qual_to_rels * Add clause information to either the baserestrictinfo or joininfo list * (depending on whether the clause is a join) of each base relation - * mentioned in the clause. A RestrictInfo node is created and added to + * mentioned in the clause. A RestrictInfo node is created and added to * the appropriate list for each rel. Alternatively, if the clause uses a * mergejoinable operator and is not delayed by outer-join rules, enter * the left- and right-side expressions into the query's list of - * EquivalenceClasses. + * EquivalenceClasses. Alternatively, if the clause needs to be treated + * as belonging to a higher join level, just add it to postponed_qual_list. * * 'clause': the qual clause to be distributed * 'is_deduced': TRUE if the qual came from implied-equality deduction @@ -780,13 +1267,22 @@ make_outerjoininfo(PlannerInfo *root, * baserels appearing on the outer (nonnullable) side of the join * (for FULL JOIN this includes both sides of the join, and must in fact * equal qualscope) + * 'deduced_nullable_relids': if is_deduced is TRUE, the nullable relids to + * impute to the clause; otherwise NULL + * 'postponed_qual_list': list of PostponedQual structs, which we can add + * this qual to if it turns out to belong to a higher join level. + * Can be NULL if caller knows postponement is impossible. * * 'qualscope' identifies what level of JOIN the qual came from syntactically. * 'ojscope' is needed if we decide to force the qual up to the outer-join * level, which will be ojscope not necessarily qualscope. * - * At the time this is called, root->join_info_list must contain entries for - * all and only those special joins that are syntactically below this qual. + * In normal use (when is_deduced is FALSE), at the time this is called, + * root->join_info_list must contain entries for all and only those special + * joins that are syntactically below this qual. But when is_deduced is TRUE, + * we are adding new deduced clauses after completion of deconstruct_jointree, + * so it cannot be assumed that root->join_info_list has anything to do with + * qual placement. */ static void distribute_qual_to_rels(PlannerInfo *root, Node *clause, @@ -795,7 +1291,9 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, JoinType jointype, Relids qualscope, Relids ojscope, - Relids outerjoin_nonnullable) + Relids outerjoin_nonnullable, + Relids deduced_nullable_relids, + List **postponed_qual_list) { Relids relids; bool is_pushed_down; @@ -812,11 +1310,33 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, relids = pull_varnos(clause); /* - * Cross-check: clause should contain no relids not within its scope. - * Otherwise the parser messed up. + * In ordinary SQL, a WHERE or JOIN/ON clause can't reference any rels + * that aren't within its syntactic scope; however, if we pulled up a + * LATERAL subquery then we might find such references in quals that have + * been pulled up. We need to treat such quals as belonging to the join + * level that includes every rel they reference. Although we could make + * pull_up_subqueries() place such quals correctly to begin with, it's + * easier to handle it here. When we find a clause that contains Vars + * outside its syntactic scope, we add it to the postponed-quals list, and + * process it once we've recursed back up to the appropriate join level. */ if (!bms_is_subset(relids, qualscope)) - elog(ERROR, "JOIN qualification cannot refer to other relations"); + { + PostponedQual *pq = (PostponedQual *) palloc(sizeof(PostponedQual)); + + Assert(root->hasLateralRTEs); /* shouldn't happen otherwise */ + Assert(jointype == JOIN_INNER); /* mustn't postpone past outer join */ + Assert(!is_deduced); /* shouldn't be deduced, either */ + pq->qual = clause; + pq->relids = relids; + *postponed_qual_list = lappend(*postponed_qual_list, pq); + return; + } + + /* + * If it's an outer-join clause, also check that relids is a subset of + * ojscope. (This should not fail if the syntactic scope check passed.) + */ if (ojscope && !bms_is_subset(relids, ojscope)) elog(ERROR, "JOIN qualification cannot refer to other relations"); @@ -837,10 +1357,10 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, * gating Result plan node. We put such a clause into the regular * RestrictInfo lists for the moment, but eventually createplan.c will * pull it out and make a gating Result node immediately above whatever - * plan node the pseudoconstant clause is assigned to. It's usually best + * plan node the pseudoconstant clause is assigned to. It's usually best * to put a gating node as high in the plan tree as possible. If we are * not below an outer join, we can actually push the pseudoconstant qual - * all the way to the top of the tree. If we are below an outer join, we + * all the way to the top of the tree. If we are below an outer join, we * leave the qual at its original syntactic level (we could push it up to * just below the outer join, but that seems more complex than it's * worth). @@ -894,7 +1414,7 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, * Note: it is not immediately obvious that a simple boolean is enough * for this: if for some reason we were to attach a degenerate qual to * its original join level, it would need to be treated as an outer join - * qual there. However, this cannot happen, because all the rels the + * qual there. However, this cannot happen, because all the rels the * clause mentions must be in the outer join's min_righthand, therefore * the join it needs must be formed before the outer join; and we always * attach quals to the lowest level where they can be evaluated. But @@ -908,12 +1428,13 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, * If the qual came from implied-equality deduction, it should not be * outerjoin-delayed, else deducer blew it. But we can't check this * because the join_info_list may now contain OJs above where the qual - * belongs. + * belongs. For the same reason, we must rely on caller to supply the + * correct nullable_relids set. */ Assert(!ojscope); is_pushed_down = true; outerjoin_delayed = false; - nullable_relids = NULL; + nullable_relids = deduced_nullable_relids; /* Don't feed it back for more deductions */ maybe_equivalence = false; maybe_outer_join = false; @@ -927,7 +1448,7 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, * We can't use such a clause to deduce equivalence (the left and * right sides might be unequal above the join because one of them has * gone to NULL) ... but we might be able to use it for more limited - * deductions, if it is mergejoinable. So consider adding it to the + * deductions, if it is mergejoinable. So consider adding it to the * lists of set-aside outer-join clauses. */ is_pushed_down = false; @@ -957,7 +1478,7 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, else { /* - * Normal qual clause or degenerate outer-join clause. Either way, we + * Normal qual clause or degenerate outer-join clause. Either way, we * can mark it as pushed-down. */ is_pushed_down = true; @@ -971,7 +1492,8 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, if (outerjoin_delayed) { /* Should still be a subset of current scope ... */ - Assert(bms_is_subset(relids, qualscope)); + Assert(root->hasLateralRTEs || bms_is_subset(relids, qualscope)); + Assert(ojscope == NULL || bms_is_subset(relids, ojscope)); /* * Because application of the qual will be delayed by outer join, @@ -1076,7 +1598,7 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, * * In all cases, it's important to initialize the left_ec and right_ec * fields of a mergejoinable clause, so that all possibly mergejoinable - * expressions have representations in EquivalenceClasses. If + * expressions have representations in EquivalenceClasses. If * process_equivalence is successful, it will take care of that; * otherwise, we have to call initialize_mergeclause_eclasses to do it. */ @@ -1084,7 +1606,8 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, { if (maybe_equivalence) { - if (process_equivalence(root, restrictinfo, below_outer_join)) + if (check_equivalence_delay(root, restrictinfo) && + process_equivalence(root, restrictinfo, below_outer_join)) return; /* EC rejected it, so set left_ec/right_ec the hard way ... */ initialize_mergeclause_eclasses(root, restrictinfo); @@ -1151,7 +1674,7 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, * For an is_pushed_down qual, we can evaluate the qual as soon as (1) we have * all the rels it mentions, and (2) we are at or above any outer joins that * can null any of these rels and are below the syntactic location of the - * given qual. We must enforce (2) because pushing down such a clause below + * given qual. We must enforce (2) because pushing down such a clause below * the OJ might cause the OJ to emit null-extended rows that should not have * been formed, or that should have been rejected by the clause. (This is * only an issue for non-strict quals, since if we can prove a qual mentioning @@ -1177,7 +1700,7 @@ distribute_qual_to_rels(PlannerInfo *root, Node *clause, * required relids overlap the LHS too) causes that OJ's delay_upper_joins * flag to be set TRUE. This will prevent any higher-level OJs from * being interchanged with that OJ, which would result in not having any - * correct place to evaluate the qual. (The case we care about here is a + * correct place to evaluate the qual. (The case we care about here is a * sub-select WHERE clause within the RHS of some outer join. The WHERE * clause must effectively be treated as a degenerate clause of that outer * join's condition. Rather than trying to match such clauses with joins @@ -1257,6 +1780,44 @@ check_outerjoin_delay(PlannerInfo *root, } /* + * check_equivalence_delay + * Detect whether a potential equivalence clause is rendered unsafe + * by outer-join-delay considerations. Return TRUE if it's safe. + * + * The initial tests in distribute_qual_to_rels will consider a mergejoinable + * clause to be a potential equivalence clause if it is not outerjoin_delayed. + * But since the point of equivalence processing is that we will recombine the + * two sides of the clause with others, we have to check that each side + * satisfies the not-outerjoin_delayed condition on its own; otherwise it might + * not be safe to evaluate everywhere we could place a derived equivalence + * condition. + */ +static bool +check_equivalence_delay(PlannerInfo *root, + RestrictInfo *restrictinfo) +{ + Relids relids; + Relids nullable_relids; + + /* fast path if no special joins */ + if (root->join_info_list == NIL) + return true; + + /* must copy restrictinfo's relids to avoid changing it */ + relids = bms_copy(restrictinfo->left_relids); + /* check left side does not need delay */ + if (check_outerjoin_delay(root, &relids, &nullable_relids, true)) + return false; + + /* and similarly for the right side */ + relids = bms_copy(restrictinfo->right_relids); + if (check_outerjoin_delay(root, &relids, &nullable_relids, true)) + return false; + + return true; +} + +/* * check_redundant_nullability_qual * Check to see if the qual is an IS NULL qual that is redundant with * a lower JOIN_ANTI join. @@ -1366,11 +1927,20 @@ distribute_restrictinfo_to_rels(PlannerInfo *root, * variable-free. Otherwise the qual is applied at the lowest join level * that provides all its variables. * + * "nullable_relids" is the set of relids used in the expressions that are + * potentially nullable below the expressions. (This has to be supplied by + * caller because this function is used after deconstruct_jointree, so we + * don't have knowledge of where the clause items came from.) + * * "both_const" indicates whether both items are known pseudo-constant; * in this case it is worth applying eval_const_expressions() in case we * can produce constant TRUE or constant FALSE. (Otherwise it's not, * because the expressions went through eval_const_expressions already.) * + * Note: this function will copy item1 and item2, but it is caller's + * responsibility to make sure that the Relids parameters are fresh copies + * not shared with other uses. + * * This is currently used only when an EquivalenceClass is found to * contain pseudoconstants. See path/pathkeys.c for more details. */ @@ -1381,6 +1951,7 @@ process_implied_equality(PlannerInfo *root, Expr *item1, Expr *item2, Relids qualscope, + Relids nullable_relids, bool below_outer_join, bool both_const) { @@ -1414,15 +1985,13 @@ process_implied_equality(PlannerInfo *root, } } - /* Make a copy of qualscope to avoid problems if source EC changes */ - qualscope = bms_copy(qualscope); - /* * Push the new clause into all the appropriate restrictinfo lists. */ distribute_qual_to_rels(root, (Node *) clause, true, below_outer_join, JOIN_INNER, - qualscope, NULL, NULL); + qualscope, NULL, NULL, nullable_relids, + NULL); } /* @@ -1431,6 +2000,10 @@ process_implied_equality(PlannerInfo *root, * This overlaps the functionality of process_implied_equality(), but we * must return the RestrictInfo, not push it into the joininfo tree. * + * Note: this function will copy item1 and item2, but it is caller's + * responsibility to make sure that the Relids parameters are fresh copies + * not shared with other uses. + * * Note: we do not do initialize_mergeclause_eclasses() here. It is * caller's responsibility that left_ec/right_ec be set as necessary. */ @@ -1439,7 +2012,8 @@ build_implied_join_equality(Oid opno, Oid collation, Expr *item1, Expr *item2, - Relids qualscope) + Relids qualscope, + Relids nullable_relids) { RestrictInfo *restrictinfo; Expr *clause; @@ -1456,9 +2030,6 @@ build_implied_join_equality(Oid opno, InvalidOid, collation); - /* Make a copy of qualscope to avoid problems if source EC changes */ - qualscope = bms_copy(qualscope); - /* * Build the RestrictInfo node itself. */ @@ -1468,7 +2039,7 @@ build_implied_join_equality(Oid opno, false, /* pseudoconstant */ qualscope, /* required_relids */ NULL, /* outer_relids */ - NULL); /* nullable_relids */ + nullable_relids); /* nullable_relids */ /* Set mergejoinability/hashjoinability flags */ check_mergejoinable(restrictinfo); @@ -1527,7 +2098,7 @@ check_mergejoinable(RestrictInfo *restrictinfo) * info fields in the restrictinfo. * * Currently, we support hashjoin for binary opclauses where - * the operator is a hashjoinable operator. The arguments can be + * the operator is a hashjoinable operator. The arguments can be * anything --- as long as there are no volatile functions in them. */ static void diff --git a/src/backend/optimizer/plan/planagg.c b/src/backend/optimizer/plan/planagg.c index c0394f787c..cd5264bcff 100644 --- a/src/backend/optimizer/plan/planagg.c +++ b/src/backend/optimizer/plan/planagg.c @@ -10,9 +10,9 @@ * ORDER BY col ASC/DESC * LIMIT 1) * Given a suitable index on tab.col, this can be much faster than the - * generic scan-all-the-rows aggregation plan. We can handle multiple + * generic scan-all-the-rows aggregation plan. We can handle multiple * MIN/MAX aggregates by generating multiple subqueries, and their - * orderings can be different. However, if the query contains any + * orderings can be different. However, if the query contains any * non-optimizable aggregates, there's no point since we'll have to * scan all the rows anyway. * @@ -22,7 +22,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * Portions Copyright (c) 2012-2014, TransLattice, Inc. - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -33,6 +33,7 @@ */ #include "postgres.h" +#include "access/htup_details.h" #include "catalog/pg_aggregate.h" #include "catalog/pg_type.h" #include "nodes/makefuncs.h" @@ -41,7 +42,9 @@ #include "optimizer/cost.h" #include "optimizer/paths.h" #include "optimizer/planmain.h" +#include "optimizer/planner.h" #include "optimizer/subselect.h" +#include "optimizer/tlist.h" #include "parser/parsetree.h" #include "parser/parse_clause.h" #include "utils/lsyscache.h" @@ -51,6 +54,7 @@ static bool find_minmax_aggs_walker(Node *node, List **context); static bool build_minmax_path(PlannerInfo *root, MinMaxAggInfo *mminfo, Oid eqop, Oid sortop, bool nulls_first); +static void minmax_qp_callback(PlannerInfo *root, void *extra); static void make_agg_subplan(PlannerInfo *root, MinMaxAggInfo *mminfo); static Node *replace_aggs_with_params_mutator(Node *node, PlannerInfo *root); static Oid fetch_agg_sort_op(Oid aggfnoid); @@ -129,7 +133,7 @@ preprocess_minmax_aggregates(PlannerInfo *root, List *tlist) /* * Scan the tlist and HAVING qual to find all the aggregates and verify - * all are MIN/MAX aggregates. Stop as soon as we find one that isn't. + * all are MIN/MAX aggregates. Stop as soon as we find one that isn't. */ aggs_list = NIL; if (find_minmax_aggs_walker((Node *) tlist, &aggs_list)) @@ -164,7 +168,7 @@ preprocess_minmax_aggregates(PlannerInfo *root, List *tlist) * We can use either an ordering that gives NULLS FIRST or one that * gives NULLS LAST; furthermore there's unlikely to be much * performance difference between them, so it doesn't seem worth - * costing out both ways if we get a hit on the first one. NULLS + * costing out both ways if we get a hit on the first one. NULLS * FIRST is more likely to be available if the operator is a * reverse-sort operator, so try that first if reverse. */ @@ -210,7 +214,6 @@ optimize_minmax_aggregates(PlannerInfo *root, List *tlist, Path agg_p; Plan *plan; Node *hqual; - QualCost tlist_cost; ListCell *lc; /* Nothing to do if preprocess_minmax_aggs rejected the query */ @@ -261,7 +264,10 @@ optimize_minmax_aggregates(PlannerInfo *root, List *tlist, /* * We have to replace Aggrefs with Params in equivalence classes too, else - * ORDER BY or DISTINCT on an optimized aggregate will fail. + * ORDER BY or DISTINCT on an optimized aggregate will fail. We don't + * need to process child eclass members though, since they aren't of + * interest anymore --- and replace_aggs_with_params_mutator isn't able to + * handle Aggrefs containing translated child Vars, anyway. * * Note: at some point it might become necessary to mutate other data * structures too, such as the query's sortClause or distinctClause. Right @@ -269,7 +275,8 @@ optimize_minmax_aggregates(PlannerInfo *root, List *tlist, */ mutate_eclass_expressions(root, replace_aggs_with_params_mutator, - (void *) root); + (void *) root, + false); /* * Generate the output plan --- basically just a Result @@ -277,9 +284,7 @@ optimize_minmax_aggregates(PlannerInfo *root, List *tlist, plan = (Plan *) make_result(root, tlist, hqual, NULL); /* Account for evaluation cost of the tlist (make_result did the rest) */ - cost_qual_eval(&tlist_cost, tlist, root); - plan->startup_cost += tlist_cost.startup; - plan->total_cost += tlist_cost.startup + tlist_cost.per_tuple; + add_tlist_costs_to_plan(root, plan, tlist); return plan; } @@ -315,15 +320,40 @@ find_minmax_aggs_walker(Node *node, List **context) ListCell *l; Assert(aggref->agglevelsup == 0); - if (list_length(aggref->args) != 1 || aggref->aggorder != NIL) + if (list_length(aggref->args) != 1) return true; /* it couldn't be MIN/MAX */ + + /* + * ORDER BY is usually irrelevant for MIN/MAX, but it can change the + * outcome if the aggsortop's operator class recognizes non-identical + * values as equal. For example, 4.0 and 4.00 are equal according to + * numeric_ops, yet distinguishable. If MIN() receives more than one + * value equal to 4.0 and no value less than 4.0, it is unspecified + * which of those equal values MIN() returns. An ORDER BY expression + * that differs for each of those equal values of the argument + * expression makes the result predictable once again. This is a + * niche requirement, and we do not implement it with subquery paths. + * In any case, this test lets us reject ordered-set aggregates + * quickly. + */ + if (aggref->aggorder != NIL) + return true; /* note: we do not care if DISTINCT is mentioned ... */ - curTarget = (TargetEntry *) linitial(aggref->args); + + /* + * We might implement the optimization when a FILTER clause is present + * by adding the filter to the quals of the generated subquery. For + * now, just punt. + */ + if (aggref->aggfilter != NULL) + return true; aggsortop = fetch_agg_sort_op(aggref->aggfnoid); if (!OidIsValid(aggsortop)) return true; /* not a MIN/MAX aggregate */ + curTarget = (TargetEntry *) linitial(aggref->args); + if (contain_mutable_functions((Node *) curTarget->expr)) return true; /* not potentially indexable */ @@ -380,9 +410,8 @@ build_minmax_path(PlannerInfo *root, MinMaxAggInfo *mminfo, TargetEntry *tle; NullTest *ntest; SortGroupClause *sortcl; - Path *cheapest_path; + RelOptInfo *final_rel; Path *sorted_path; - double dNumGroups; Cost path_cost; double path_fraction; @@ -399,8 +428,9 @@ build_minmax_path(PlannerInfo *root, MinMaxAggInfo *mminfo, subroot->parse = parse = (Query *) copyObject(root->parse); /* make sure subroot planning won't change root->init_plans contents */ subroot->init_plans = list_copy(root->init_plans); - /* There shouldn't be any OJ info to translate, as yet */ + /* There shouldn't be any OJ or LATERAL info to translate, as yet */ Assert(subroot->join_info_list == NIL); + Assert(subroot->lateral_info_list == NIL); /* and we haven't created PlaceHolderInfos, either */ Assert(subroot->placeholder_list == NIL); @@ -447,42 +477,31 @@ build_minmax_path(PlannerInfo *root, MinMaxAggInfo *mminfo, FLOAT8PASSBYVAL); /* - * Set up requested pathkeys. - */ - subroot->group_pathkeys = NIL; - subroot->window_pathkeys = NIL; - subroot->distinct_pathkeys = NIL; - - subroot->sort_pathkeys = - make_pathkeys_for_sortclauses(subroot, - parse->sortClause, - parse->targetList, - false); - - subroot->query_pathkeys = subroot->sort_pathkeys; - - /* * Generate the best paths for this query, telling query_planner that we * have LIMIT 1. */ - query_planner(subroot, parse->targetList, 1.0, 1.0, - &cheapest_path, &sorted_path, &dNumGroups); + subroot->tuple_fraction = 1.0; + subroot->limit_tuples = 1.0; + + final_rel = query_planner(subroot, parse->targetList, + minmax_qp_callback, NULL); /* - * Fail if no presorted path. However, if query_planner determines that - * the presorted path is also the cheapest, it will set sorted_path to - * NULL ... don't be fooled. (This is kind of a pain here, but it - * simplifies life for grouping_planner, so leave it be.) + * Get the best presorted path, that being the one that's cheapest for + * fetching just one row. If there's no such path, fail. */ + if (final_rel->rows > 1.0) + path_fraction = 1.0 / final_rel->rows; + else + path_fraction = 1.0; + + sorted_path = + get_cheapest_fractional_path_for_pathkeys(final_rel->pathlist, + subroot->query_pathkeys, + NULL, + path_fraction); if (!sorted_path) - { - if (cheapest_path && - pathkeys_contained_in(subroot->sort_pathkeys, - cheapest_path->pathkeys)) - sorted_path = cheapest_path; - else - return false; - } + return false; /* * Determine cost to get just the first row of the presorted path. @@ -490,11 +509,6 @@ build_minmax_path(PlannerInfo *root, MinMaxAggInfo *mminfo, * Note: cost calculation here should match * compare_fractional_path_costs(). */ - if (sorted_path->parent->rows > 1.0) - path_fraction = 1.0 / sorted_path->parent->rows; - else - path_fraction = 1.0; - path_cost = sorted_path->startup_cost + path_fraction * (sorted_path->total_cost - sorted_path->startup_cost); @@ -507,6 +521,24 @@ build_minmax_path(PlannerInfo *root, MinMaxAggInfo *mminfo, } /* + * Compute query_pathkeys and other pathkeys during plan generation + */ +static void +minmax_qp_callback(PlannerInfo *root, void *extra) +{ + root->group_pathkeys = NIL; + root->window_pathkeys = NIL; + root->distinct_pathkeys = NIL; + + root->sort_pathkeys = + make_pathkeys_for_sortclauses(root, + root->parse->sortClause, + root->parse->targetList); + + root->query_pathkeys = root->sort_pathkeys; +} + +/* * Construct a suitable plan for a converted aggregate query */ static void @@ -522,7 +554,27 @@ make_agg_subplan(PlannerInfo *root, MinMaxAggInfo *mminfo) */ plan = create_plan(subroot, mminfo->path); - plan->targetlist = subparse->targetList; + /* + * If the top-level plan node is one that cannot do expression evaluation + * and its existing target list isn't already what we need, we must insert + * a Result node to project the desired tlist. + */ + if (!is_projection_capable_plan(plan) && + !tlist_same_exprs(subparse->targetList, plan->targetlist)) + { + plan = (Plan *) make_result(subroot, + subparse->targetList, + NULL, + plan); + } + else + { + /* + * Otherwise, just replace the subplan's flat tlist with the desired + * tlist. + */ + plan->targetlist = subparse->targetList; + } #ifdef XCP /* Set plan distribution */ diff --git a/src/backend/optimizer/plan/planmain.c b/src/backend/optimizer/plan/planmain.c index 9838dc45d5..93484a0cd5 100644 --- a/src/backend/optimizer/plan/planmain.c +++ b/src/backend/optimizer/plan/planmain.c @@ -9,7 +9,7 @@ * shorn of features like subselects, inheritance, aggregates, grouping, * and so on. (Those are the things planner.c deals with.) * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -20,18 +20,11 @@ */ #include "postgres.h" -#include "miscadmin.h" -#include "optimizer/cost.h" +#include "optimizer/orclauses.h" #include "optimizer/pathnode.h" #include "optimizer/paths.h" #include "optimizer/placeholder.h" #include "optimizer/planmain.h" -#include "optimizer/tlist.h" -#include "utils/selfuncs.h" - - -/* Local functions */ -static void canonicalize_all_pathkeys(PlannerInfo *root); /* @@ -40,90 +33,58 @@ static void canonicalize_all_pathkeys(PlannerInfo *root); * which may involve joins but not any fancier features. * * Since query_planner does not handle the toplevel processing (grouping, - * sorting, etc) it cannot select the best path by itself. It selects - * two paths: the cheapest path that produces all the required tuples, - * independent of any ordering considerations, and the cheapest path that - * produces the expected fraction of the required tuples in the required - * ordering, if there is a path that is cheaper for this than just sorting - * the output of the cheapest overall path. The caller (grouping_planner) - * will make the final decision about which to use. + * sorting, etc) it cannot select the best path by itself. Instead, it + * returns the RelOptInfo for the top level of joining, and the caller + * (grouping_planner) can choose one of the surviving paths for the rel. + * Normally it would choose either the rel's cheapest path, or the cheapest + * path for the desired sort order. * - * Input parameters: * root describes the query to plan * tlist is the target list the query should produce * (this is NOT necessarily root->parse->targetList!) - * tuple_fraction is the fraction of tuples we expect will be retrieved - * limit_tuples is a hard limit on number of tuples to retrieve, - * or -1 if no limit - * - * Output parameters: - * *cheapest_path receives the overall-cheapest path for the query - * *sorted_path receives the cheapest presorted path for the query, - * if any (NULL if there is no useful presorted path) - * *num_groups receives the estimated number of groups, or 1 if query - * does not use grouping - * - * Note: the PlannerInfo node also includes a query_pathkeys field, which is - * both an input and an output of query_planner(). The input value signals - * query_planner that the indicated sort order is wanted in the final output - * plan. But this value has not yet been "canonicalized", since the needed - * info does not get computed until we scan the qual clauses. We canonicalize - * it as soon as that task is done. (The main reason query_pathkeys is a - * PlannerInfo field and not a passed parameter is that the low-level routines - * in indxpath.c need to see it.) - * - * Note: the PlannerInfo node includes other pathkeys fields besides - * query_pathkeys, all of which need to be canonicalized once the info is - * available. See canonicalize_all_pathkeys. + * qp_callback is a function to compute query_pathkeys once it's safe to do so + * qp_extra is optional extra data to pass to qp_callback * - * tuple_fraction is interpreted as follows: - * 0: expect all tuples to be retrieved (normal case) - * 0 < tuple_fraction < 1: expect the given fraction of tuples available - * from the plan to be retrieved - * tuple_fraction >= 1: tuple_fraction is the absolute number of tuples - * expected to be retrieved (ie, a LIMIT specification) - * Note that a nonzero tuple_fraction could come from outer context; it is - * therefore not redundant with limit_tuples. We use limit_tuples to determine - * whether a bounded sort can be used at runtime. + * Note: the PlannerInfo node also includes a query_pathkeys field, which + * tells query_planner the sort order that is desired in the final output + * plan. This value is *not* available at call time, but is computed by + * qp_callback once we have completed merging the query's equivalence classes. + * (We cannot construct canonical pathkeys until that's done.) */ -void +RelOptInfo * query_planner(PlannerInfo *root, List *tlist, - double tuple_fraction, double limit_tuples, - Path **cheapest_path, Path **sorted_path, - double *num_groups) + query_pathkeys_callback qp_callback, void *qp_extra) { Query *parse = root->parse; List *joinlist; RelOptInfo *final_rel; - Path *cheapestpath; - Path *sortedpath; Index rti; double total_pages; - /* Make tuple_fraction, limit_tuples accessible to lower-level routines */ - root->tuple_fraction = tuple_fraction; - root->limit_tuples = limit_tuples; - - *num_groups = 1; /* default result */ - /* * If the query has an empty join tree, then it's something easy like - * "SELECT 2+2;" or "INSERT ... VALUES()". Fall through quickly. + * "SELECT 2+2;" or "INSERT ... VALUES()". Fall through quickly. */ if (parse->jointree->fromlist == NIL) { - /* We need a trivial path result */ - *cheapest_path = (Path *) - create_result_path((List *) parse->jointree->quals); - *sorted_path = NULL; + /* We need a dummy joinrel to describe the empty set of baserels */ + final_rel = build_empty_join_rel(root); + + /* The only path for it is a trivial Result path */ + add_path(final_rel, (Path *) + create_result_path((List *) parse->jointree->quals)); + + /* Select cheapest path (pretty easy in this case...) */ + set_cheapest(final_rel); /* - * We still are required to canonicalize any pathkeys, in case it's - * something like "SELECT 2+2 ORDER BY 1". + * We still are required to call qp_callback, in case it's something + * like "SELECT 2+2 ORDER BY 1". */ root->canon_pathkeys = NIL; - canonicalize_all_pathkeys(root); - return; + (*qp_callback) (root, qp_extra); + + return final_rel; } /* @@ -141,6 +102,7 @@ query_planner(PlannerInfo *root, List *tlist, root->right_join_clauses = NIL; root->full_join_clauses = NIL; root->join_info_list = NIL; + root->lateral_info_list = NIL; root->placeholder_list = NIL; root->initial_rels = NIL; @@ -167,7 +129,7 @@ query_planner(PlannerInfo *root, List *tlist, /* * Examine the targetlist and join tree, adding entries to baserel * targetlists for all referenced Vars, and generating PlaceHolderInfo - * entries for all referenced PlaceHolderVars. Restrict and join clauses + * entries for all referenced PlaceHolderVars. Restrict and join clauses * are added to appropriate lists belonging to the mentioned relations. We * also build EquivalenceClasses for provably equivalent expressions. The * SpecialJoinInfo list is also built to hold information about join order @@ -178,6 +140,8 @@ query_planner(PlannerInfo *root, List *tlist, find_placeholders_in_jointree(root); + find_lateral_references(root); + joinlist = deconstruct_jointree(root); /* @@ -189,29 +153,29 @@ query_planner(PlannerInfo *root, List *tlist, /* * If we formed any equivalence classes, generate additional restriction - * clauses as appropriate. (Implied join clauses are formed on-the-fly + * clauses as appropriate. (Implied join clauses are formed on-the-fly * later.) */ generate_base_implied_equalities(root); /* * We have completed merging equivalence sets, so it's now possible to - * convert previously generated pathkeys (in particular, the requested - * query_pathkeys) to canonical form. + * generate pathkeys in canonical form; so compute query_pathkeys and + * other pathkeys fields in PlannerInfo. */ - canonicalize_all_pathkeys(root); + (*qp_callback) (root, qp_extra); /* * Examine any "placeholder" expressions generated during subquery pullup. * Make sure that the Vars they need are marked as needed at the relevant - * join level. This must be done before join removal because it might + * join level. This must be done before join removal because it might * cause Vars or placeholders to be needed above a join when they weren't * so marked before. */ fix_placeholder_input_needed_levels(root); /* - * Remove any useless outer joins. Ideally this would be done during + * Remove any useless outer joins. Ideally this would be done during * jointree preprocessing, but the necessary information isn't available * until we've built baserel data structures and classified qual clauses. */ @@ -225,6 +189,19 @@ query_planner(PlannerInfo *root, List *tlist, add_placeholders_to_base_rels(root); /* + * Create the LateralJoinInfo list now that we have finalized + * PlaceHolderVar eval levels and made any necessary additions to the + * lateral_vars lists for lateral references within PlaceHolderVars. + */ + create_lateral_join_info(root); + + /* + * Look for join OR clauses that we can extract single-relation + * restriction OR clauses from. + */ + extract_restriction_or_clauses(root); + + /* * We should now have size estimates for every actual table involved in * the query, and we also know which if any have been deleted from the * query by join removal; so we can compute total_table_pages. @@ -258,180 +235,10 @@ query_planner(PlannerInfo *root, List *tlist, */ final_rel = make_one_rel(root, joinlist); - if (!final_rel || !final_rel->cheapest_total_path) + /* Check that we got at least one usable path */ + if (!final_rel || !final_rel->cheapest_total_path || + final_rel->cheapest_total_path->param_info != NULL) elog(ERROR, "failed to construct the join relation"); - /* - * If there's grouping going on, estimate the number of result groups. We - * couldn't do this any earlier because it depends on relation size - * estimates that were set up above. - * - * Then convert tuple_fraction to fractional form if it is absolute, and - * adjust it based on the knowledge that grouping_planner will be doing - * grouping or aggregation work with our result. - * - * This introduces some undesirable coupling between this code and - * grouping_planner, but the alternatives seem even uglier; we couldn't - * pass back completed paths without making these decisions here. - */ - if (parse->groupClause) - { - List *groupExprs; - - groupExprs = get_sortgrouplist_exprs(parse->groupClause, - parse->targetList); - *num_groups = estimate_num_groups(root, - groupExprs, - final_rel->rows); - - /* - * In GROUP BY mode, an absolute LIMIT is relative to the number of - * groups not the number of tuples. If the caller gave us a fraction, - * keep it as-is. (In both cases, we are effectively assuming that - * all the groups are about the same size.) - */ - if (tuple_fraction >= 1.0) - tuple_fraction /= *num_groups; - - /* - * If both GROUP BY and ORDER BY are specified, we will need two - * levels of sort --- and, therefore, certainly need to read all the - * tuples --- unless ORDER BY is a subset of GROUP BY. Likewise if we - * have both DISTINCT and GROUP BY, or if we have a window - * specification not compatible with the GROUP BY. - */ - if (!pathkeys_contained_in(root->sort_pathkeys, root->group_pathkeys) || - !pathkeys_contained_in(root->distinct_pathkeys, root->group_pathkeys) || - !pathkeys_contained_in(root->window_pathkeys, root->group_pathkeys)) - tuple_fraction = 0.0; - - /* In any case, limit_tuples shouldn't be specified here */ - Assert(limit_tuples < 0); - } - else if (parse->hasAggs || root->hasHavingQual) - { - /* - * Ungrouped aggregate will certainly want to read all the tuples, and - * it will deliver a single result row (so leave *num_groups 1). - */ - tuple_fraction = 0.0; - - /* limit_tuples shouldn't be specified here */ - Assert(limit_tuples < 0); - } - else if (parse->distinctClause) - { - /* - * Since there was no grouping or aggregation, it's reasonable to - * assume the UNIQUE filter has effects comparable to GROUP BY. Return - * the estimated number of output rows for use by caller. (If DISTINCT - * is used with grouping, we ignore its effects for rowcount - * estimation purposes; this amounts to assuming the grouped rows are - * distinct already.) - */ - List *distinctExprs; - - distinctExprs = get_sortgrouplist_exprs(parse->distinctClause, - parse->targetList); - *num_groups = estimate_num_groups(root, - distinctExprs, - final_rel->rows); - - /* - * Adjust tuple_fraction the same way as for GROUP BY, too. - */ - if (tuple_fraction >= 1.0) - tuple_fraction /= *num_groups; - - /* limit_tuples shouldn't be specified here */ - Assert(limit_tuples < 0); - } - else - { - /* - * Plain non-grouped, non-aggregated query: an absolute tuple fraction - * can be divided by the number of tuples. - */ - if (tuple_fraction >= 1.0) - tuple_fraction /= final_rel->rows; - } - - /* - * Pick out the cheapest-total path and the cheapest presorted path for - * the requested pathkeys (if there is one). We should take the tuple - * fraction into account when selecting the cheapest presorted path, but - * not when selecting the cheapest-total path, since if we have to sort - * then we'll have to fetch all the tuples. (But there's a special case: - * if query_pathkeys is NIL, meaning order doesn't matter, then the - * "cheapest presorted" path will be the cheapest overall for the tuple - * fraction.) - * - * The cheapest-total path is also the one to use if grouping_planner - * decides to use hashed aggregation, so we return it separately even if - * this routine thinks the presorted path is the winner. - */ - cheapestpath = final_rel->cheapest_total_path; - - sortedpath = - get_cheapest_fractional_path_for_pathkeys(final_rel->pathlist, - root->query_pathkeys, - NULL, - tuple_fraction); - - /* Don't return same path in both guises; just wastes effort */ - if (sortedpath == cheapestpath) - sortedpath = NULL; - - /* - * Forget about the presorted path if it would be cheaper to sort the - * cheapest-total path. Here we need consider only the behavior at the - * tuple fraction point. - */ - if (sortedpath) - { - Path sort_path; /* dummy for result of cost_sort */ - - if (root->query_pathkeys == NIL || - pathkeys_contained_in(root->query_pathkeys, - cheapestpath->pathkeys)) - { - /* No sort needed for cheapest path */ - sort_path.startup_cost = cheapestpath->startup_cost; - sort_path.total_cost = cheapestpath->total_cost; - } - else - { - /* Figure cost for sorting */ - cost_sort(&sort_path, root, root->query_pathkeys, - cheapestpath->total_cost, - final_rel->rows, final_rel->width, - 0.0, work_mem, limit_tuples); - } - - if (compare_fractional_path_costs(sortedpath, &sort_path, - tuple_fraction) > 0) - { - /* Presorted path is a loser */ - sortedpath = NULL; - } - } - - *cheapest_path = cheapestpath; - *sorted_path = sortedpath; -} - - -/* - * canonicalize_all_pathkeys - * Canonicalize all pathkeys that were generated before entering - * query_planner and then stashed in PlannerInfo. - */ -static void -canonicalize_all_pathkeys(PlannerInfo *root) -{ - root->query_pathkeys = canonicalize_pathkeys(root, root->query_pathkeys); - root->group_pathkeys = canonicalize_pathkeys(root, root->group_pathkeys); - root->window_pathkeys = canonicalize_pathkeys(root, root->window_pathkeys); - root->distinct_pathkeys = canonicalize_pathkeys(root, root->distinct_pathkeys); - root->sort_pathkeys = canonicalize_pathkeys(root, root->sort_pathkeys); + return final_rel; } diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 9dca3a25cc..a72e1e2c78 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -8,7 +8,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * Portions Copyright (c) 2012-2014, TransLattice, Inc. - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -22,6 +22,7 @@ #include <limits.h> +#include "access/htup_details.h" #include "executor/executor.h" #include "executor/nodeAgg.h" #include "miscadmin.h" @@ -48,6 +49,7 @@ #include "pgxc/pgxc.h" #include "pgxc/planner.h" #endif +#include "utils/selfuncs.h" /* GUC parameter */ @@ -58,14 +60,24 @@ planner_hook_type planner_hook = NULL; /* Expression kind codes for preprocess_expression */ -#define EXPRKIND_QUAL 0 -#define EXPRKIND_TARGET 1 -#define EXPRKIND_RTFUNC 2 -#define EXPRKIND_VALUES 3 -#define EXPRKIND_LIMIT 4 -#define EXPRKIND_APPINFO 5 - +#define EXPRKIND_QUAL 0 +#define EXPRKIND_TARGET 1 +#define EXPRKIND_RTFUNC 2 +#define EXPRKIND_RTFUNC_LATERAL 3 +#define EXPRKIND_VALUES 4 +#define EXPRKIND_VALUES_LATERAL 5 +#define EXPRKIND_LIMIT 6 +#define EXPRKIND_APPINFO 7 +#define EXPRKIND_PHV 8 + +/* Passthrough data for standard_qp_callback */ +typedef struct +{ + List *tlist; /* preprocessed query targetlist */ + List *activeWindows; /* active windows, if any */ +} standard_qp_extra; +/* Local functions */ static Node *preprocess_expression(PlannerInfo *root, Node *expr, int kind); static void preprocess_qual_conditions(PlannerInfo *root, Node *jtnode); static Plan *inheritance_planner(PlannerInfo *root); @@ -74,7 +86,9 @@ static void preprocess_rowmarks(PlannerInfo *root); static double preprocess_limit(PlannerInfo *root, double tuple_fraction, int64 *offset_est, int64 *count_est); +static bool limit_needed(Query *parse); static void preprocess_groupclause(PlannerInfo *root); +static void standard_qp_callback(PlannerInfo *root, void *extra); static bool choose_hashed_grouping(PlannerInfo *root, double tuple_fraction, double limit_tuples, double path_rows, int path_width, @@ -96,10 +110,10 @@ static void locate_grouping_columns(PlannerInfo *root, AttrNumber *groupColIdx); static List *postprocess_setop_tlist(List *new_tlist, List *orig_tlist); static List *select_active_windows(PlannerInfo *root, WindowFuncLists *wflists); -static List *add_volatile_sort_exprs(List *window_tlist, List *tlist, - List *activeWindows); +static List *make_windowInputTargetList(PlannerInfo *root, + List *tlist, List *activeWindows); static List *make_pathkeys_for_window(PlannerInfo *root, WindowClause *wc, - List *tlist, bool canonicalize); + List *tlist); static void get_column_info_for_window(PlannerInfo *root, WindowClause *wc, List *tlist, int numSortCols, AttrNumber *sortColIdx, @@ -189,7 +203,6 @@ standard_planner(Query *parse, int cursorOptions, ParamListInfo boundParams) glob = makeNode(PlannerGlobal); glob->boundParams = boundParams; - glob->paramlist = NIL; glob->subplans = NIL; glob->subroots = NIL; glob->rewindPlanIDs = NULL; @@ -198,6 +211,7 @@ standard_planner(Query *parse, int cursorOptions, ParamListInfo boundParams) glob->resultRelations = NIL; glob->relationOids = NIL; glob->invalItems = NIL; + glob->nParamExec = 0; glob->lastPHId = 0; glob->lastRowMarkId = 0; glob->transientPlan = false; @@ -216,7 +230,7 @@ standard_planner(Query *parse, int cursorOptions, ParamListInfo boundParams) /* * We document cursor_tuple_fraction as simply being a fraction, which - * means the edge cases 0 and 1 have to be treated specially here. We + * means the edge cases 0 and 1 have to be treated specially here. We * convert 1 to 0 ("all the tuples") and 0 to a very small fraction. */ if (tuple_fraction >= 1.0) @@ -314,12 +328,12 @@ standard_planner(Query *parse, int cursorOptions, ParamListInfo boundParams) result->rowMarks = glob->finalrowmarks; result->relationOids = glob->relationOids; result->invalItems = glob->invalItems; - result->nParamExec = list_length(glob->paramlist); #ifdef XCP result->distributionType = LOCATOR_TYPE_NONE; result->distributionKey = InvalidAttrNumber; result->distributionNodes = NULL; #endif + result->nParamExec = glob->nParamExec; return result; } @@ -361,6 +375,7 @@ subquery_planner(PlannerGlobal *glob, Query *parse, int num_old_subplans = list_length(glob->subplans); PlannerInfo *root; Plan *plan; + List *newWithCheckOptions; List *newHaving; bool hasOuterJoins; ListCell *l; @@ -372,6 +387,7 @@ subquery_planner(PlannerGlobal *glob, Query *parse, root->glob = glob; root->query_level = parent_root ? parent_root->query_level + 1 : 1; root->parent_root = parent_root; + root->plan_params = NIL; root->planner_cxt = CurrentMemoryContext; root->init_plans = NIL; root->cte_plan_ids = NIL; @@ -421,7 +437,7 @@ subquery_planner(PlannerGlobal *glob, Query *parse, * query. */ parse->jointree = (FromExpr *) - pull_up_subqueries(root, (Node *) parse->jointree, NULL, NULL); + pull_up_subqueries(root, (Node *) parse->jointree); /* * If this is a simple UNION ALL query, flatten it into an appendrel. We @@ -435,10 +451,12 @@ subquery_planner(PlannerGlobal *glob, Query *parse, /* * Detect whether any rangetable entries are RTE_JOIN kind; if not, we can * avoid the expense of doing flatten_join_alias_vars(). Also check for - * outer joins --- if none, we can skip reduce_outer_joins(). This must be - * done after we have done pull_up_subqueries, of course. + * outer joins --- if none, we can skip reduce_outer_joins(). And check + * for LATERAL RTEs, too. This must be done after we have done + * pull_up_subqueries(), of course. */ root->hasJoinRTEs = false; + root->hasLateralRTEs = false; hasOuterJoins = false; foreach(l, parse->rtable) { @@ -448,16 +466,14 @@ subquery_planner(PlannerGlobal *glob, Query *parse, { root->hasJoinRTEs = true; if (IS_OUTER_JOIN(rte->jointype)) - { hasOuterJoins = true; - /* Can quit scanning once we find an outer join */ - break; - } } + if (rte->lateral) + root->hasLateralRTEs = true; } /* - * Preprocess RowMark information. We need to do this after subquery + * Preprocess RowMark information. We need to do this after subquery * pullup (so that all non-inherited RTEs are present) and before * inheritance expansion (so that the info is available for * expand_inherited_tables to examine and modify). @@ -515,6 +531,18 @@ subquery_planner(PlannerGlobal *glob, Query *parse, preprocess_expression(root, (Node *) parse->targetList, EXPRKIND_TARGET); + newWithCheckOptions = NIL; + foreach(l, parse->withCheckOptions) + { + WithCheckOption *wco = (WithCheckOption *) lfirst(l); + + wco->qual = preprocess_expression(root, wco->qual, + EXPRKIND_QUAL); + if (wco->qual != NULL) + newWithCheckOptions = lappend(newWithCheckOptions, wco); + } + parse->withCheckOptions = newWithCheckOptions; + parse->returningList = (List *) preprocess_expression(root, (Node *) parse->returningList, EXPRKIND_TARGET); @@ -544,18 +572,38 @@ subquery_planner(PlannerGlobal *glob, Query *parse, preprocess_expression(root, (Node *) root->append_rel_list, EXPRKIND_APPINFO); - /* Also need to preprocess expressions for function and values RTEs */ + /* Also need to preprocess expressions within RTEs */ foreach(l, parse->rtable) { RangeTblEntry *rte = (RangeTblEntry *) lfirst(l); + int kind; - if (rte->rtekind == RTE_FUNCTION) - rte->funcexpr = preprocess_expression(root, rte->funcexpr, - EXPRKIND_RTFUNC); + if (rte->rtekind == RTE_SUBQUERY) + { + /* + * We don't want to do all preprocessing yet on the subquery's + * expressions, since that will happen when we plan it. But if it + * contains any join aliases of our level, those have to get + * expanded now, because planning of the subquery won't do it. + * That's only possible if the subquery is LATERAL. + */ + if (rte->lateral && root->hasJoinRTEs) + rte->subquery = (Query *) + flatten_join_alias_vars(root, (Node *) rte->subquery); + } + else if (rte->rtekind == RTE_FUNCTION) + { + /* Preprocess the function expression(s) fully */ + kind = rte->lateral ? EXPRKIND_RTFUNC_LATERAL : EXPRKIND_RTFUNC; + rte->functions = (List *) preprocess_expression(root, (Node *) rte->functions, kind); + } else if (rte->rtekind == RTE_VALUES) + { + /* Preprocess the values lists fully */ + kind = rte->lateral ? EXPRKIND_VALUES_LATERAL : EXPRKIND_VALUES; rte->values_lists = (List *) - preprocess_expression(root, (Node *) rte->values_lists, - EXPRKIND_VALUES); + preprocess_expression(root, (Node *) rte->values_lists, kind); + } } /* @@ -566,7 +614,7 @@ subquery_planner(PlannerGlobal *glob, Query *parse, * to execute that we're better off doing it only once per group, despite * the loss of selectivity. This is hard to estimate short of doing the * entire planning process twice, so we use a heuristic: clauses - * containing subplans are left in HAVING. Otherwise, we move or copy the + * containing subplans are left in HAVING. Otherwise, we move or copy the * HAVING clause into WHERE, in hopes of eliminating tuples before * aggregation instead of after. * @@ -633,21 +681,28 @@ subquery_planner(PlannerGlobal *glob, Query *parse, /* If it's not SELECT, we need a ModifyTable node */ if (parse->commandType != CMD_SELECT) { + List *withCheckOptionLists; List *returningLists; List *rowMarks; /* - * Set up the RETURNING list-of-lists, if needed. + * Set up the WITH CHECK OPTION and RETURNING lists-of-lists, if + * needed. */ + if (parse->withCheckOptions) + withCheckOptionLists = list_make1(parse->withCheckOptions); + else + withCheckOptionLists = NIL; + if (parse->returningList) returningLists = list_make1(parse->returningList); else returningLists = NIL; /* - * If there was a FOR UPDATE/SHARE clause, the LockRows node will - * have dealt with fetching non-locked marked rows, else we need - * to have ModifyTable do that. + * If there was a FOR [KEY] UPDATE/SHARE clause, the LockRows node + * will have dealt with fetching non-locked marked rows, else we + * need to have ModifyTable do that. */ if (parse->rowMarks) rowMarks = NIL; @@ -661,10 +716,12 @@ subquery_planner(PlannerGlobal *glob, Query *parse, errmsg("INSERT/UPDATE/DELETE is not supported in subquery"))); #endif - plan = (Plan *) make_modifytable(parse->commandType, + plan = (Plan *) make_modifytable(root, + parse->commandType, parse->canSetTag, list_make1_int(parse->resultRelation), list_make1(plan), + withCheckOptionLists, returningLists, rowMarks, SS_assign_special_param(root)); @@ -677,7 +734,7 @@ subquery_planner(PlannerGlobal *glob, Query *parse, * and attach the initPlans to the top plan node. */ if (list_length(glob->subplans) != num_old_subplans || - root->glob->paramlist != NIL) + root->glob->nParamExec > 0) SS_finalize_plan(root, plan, true); /* Return internal info if caller wants it */ @@ -753,7 +810,7 @@ subquery_planner(PlannerGlobal *glob, Query *parse, * preprocess_expression * Do subquery_planner's preprocessing work for an expression, * which can be a targetlist, a WHERE clause (including JOIN/ON - * conditions), or a HAVING clause. + * conditions), a HAVING clause, or a few other things. */ static Node * preprocess_expression(PlannerInfo *root, Node *expr, int kind) @@ -768,12 +825,13 @@ preprocess_expression(PlannerInfo *root, Node *expr, int kind) /* * If the query has any join RTEs, replace join alias variables with - * base-relation variables. We must do this before sublink processing, - * else sublinks expanded out from join aliases wouldn't get processed. We - * can skip it in VALUES lists, however, since they can't contain any Vars - * at all. + * base-relation variables. We must do this before sublink processing, + * else sublinks expanded out from join aliases would not get processed. + * We can skip it in non-lateral RTE functions and VALUES lists, however, + * since they can't contain any Vars of the current query level. */ - if (root->hasJoinRTEs && kind != EXPRKIND_VALUES) + if (root->hasJoinRTEs && + !(kind == EXPRKIND_RTFUNC || kind == EXPRKIND_VALUES)) expr = flatten_join_alias_vars(root, expr); /* @@ -870,6 +928,23 @@ preprocess_qual_conditions(PlannerInfo *root, Node *jtnode) } /* + * preprocess_phv_expression + * Do preprocessing on a PlaceHolderVar expression that's been pulled up. + * + * If a LATERAL subquery references an output of another subquery, and that + * output must be wrapped in a PlaceHolderVar because of an intermediate outer + * join, then we'll push the PlaceHolderVar expression down into the subquery + * and later pull it back up during find_lateral_references, which runs after + * subquery_planner has preprocessed all the expressions that were in the + * current query level to start with. So we need to preprocess it then. + */ +Expr * +preprocess_phv_expression(PlannerInfo *root, Expr *expr) +{ + return (Expr *) preprocess_expression(root, (Node *) expr, EXPRKIND_PHV); +} + +/* * inheritance_planner * Generate a plan in the case where the result relation is an * inheritance set. @@ -894,6 +969,7 @@ inheritance_planner(PlannerInfo *root) RelOptInfo **save_rel_array = NULL; List *subplans = NIL; List *resultRelations = NIL; + List *withCheckOptionLists = NIL; List *returningLists = NIL; List *rowMarks; ListCell *lc; @@ -950,6 +1026,13 @@ inheritance_planner(PlannerInfo *root) subroot.rowMarks = (List *) copyObject(root->rowMarks); /* + * The append_rel_list likewise might contain references to subquery + * RTEs (if any subqueries were flattenable UNION ALLs). So prepare + * to apply ChangeVarNodes to that, too. + */ + subroot.append_rel_list = (List *) copyObject(root->append_rel_list); + + /* * Add placeholders to the child Query's rangetable list to fill the * RT indexes already reserved for subqueries in previous children. * These won't be referenced, so there's no need to make them very @@ -989,6 +1072,7 @@ inheritance_planner(PlannerInfo *root) newrti = list_length(subroot.parse->rtable) + 1; ChangeVarNodes((Node *) subroot.parse, rti, newrti, 0); ChangeVarNodes((Node *) subroot.rowMarks, rti, newrti, 0); + ChangeVarNodes((Node *) subroot.append_rel_list, rti, newrti, 0); rte = copyObject(rte); subroot.parse->rtable = lappend(subroot.parse->rtable, rte); @@ -997,9 +1081,9 @@ inheritance_planner(PlannerInfo *root) } } - /* We needn't modify the child's append_rel_list */ - /* There shouldn't be any OJ info to translate, as yet */ + /* There shouldn't be any OJ or LATERAL info to translate, as yet */ Assert(subroot.join_info_list == NIL); + Assert(subroot.lateral_info_list == NIL); /* and we haven't created PlaceHolderInfos, either */ Assert(subroot.placeholder_list == NIL); /* hack to mark target relation as an inheritance partition */ @@ -1009,6 +1093,12 @@ inheritance_planner(PlannerInfo *root) subplan = grouping_planner(&subroot, 0.0 /* retrieve all tuples */ ); /* + * Planning may have modified the query result relation (if there were + * security barrier quals on the result RTE). + */ + appinfo->child_relid = subroot.parse->resultRelation; + + /* * If this child rel was excluded by constraint exclusion, exclude it * from the result plan. */ @@ -1058,9 +1148,41 @@ inheritance_planner(PlannerInfo *root) if (final_rtable == NIL) final_rtable = subroot.parse->rtable; else - final_rtable = list_concat(final_rtable, + { + List *tmp_rtable = NIL; + ListCell *cell1, + *cell2; + + /* + * Check to see if any of the original RTEs were turned into + * subqueries during planning. Currently, this should only ever + * happen due to securityQuals being involved which push a + * relation down under a subquery, to ensure that the security + * barrier quals are evaluated first. + * + * When this happens, we want to use the new subqueries in the + * final rtable. + */ + forboth(cell1, final_rtable, cell2, subroot.parse->rtable) + { + RangeTblEntry *rte1 = (RangeTblEntry *) lfirst(cell1); + RangeTblEntry *rte2 = (RangeTblEntry *) lfirst(cell2); + + if (rte1->rtekind == RTE_RELATION && + rte2->rtekind == RTE_SUBQUERY) + { + /* Should only be when there are securityQuals today */ + Assert(rte1->securityQuals != NIL); + tmp_rtable = lappend(tmp_rtable, rte2); + } + else + tmp_rtable = lappend(tmp_rtable, rte1); + } + + final_rtable = list_concat(tmp_rtable, list_copy_tail(subroot.parse->rtable, list_length(final_rtable))); + } /* * We need to collect all the RelOptInfos from all child plans into @@ -1086,7 +1208,10 @@ inheritance_planner(PlannerInfo *root) /* Build list of target-relation RT indexes */ resultRelations = lappend_int(resultRelations, appinfo->child_relid); - /* Build list of per-relation RETURNING targetlists */ + /* Build lists of per-relation WCO and RETURNING targetlists */ + if (parse->withCheckOptions) + withCheckOptionLists = lappend(withCheckOptionLists, + subroot.parse->withCheckOptions); if (parse->returningList) returningLists = lappend(returningLists, subroot.parse->returningList); @@ -1120,8 +1245,8 @@ inheritance_planner(PlannerInfo *root) root->simple_rel_array = save_rel_array; /* - * If there was a FOR UPDATE/SHARE clause, the LockRows node will have - * dealt with fetching non-locked marked rows, else we need to have + * If there was a FOR [KEY] UPDATE/SHARE clause, the LockRows node will + * have dealt with fetching non-locked marked rows, else we need to have * ModifyTable do that. */ if (parse->rowMarks) @@ -1130,10 +1255,12 @@ inheritance_planner(PlannerInfo *root) rowMarks = root->rowMarks; /* And last, tack on a ModifyTable node to do the UPDATE/DELETE work */ - return (Plan *) make_modifytable(parse->commandType, + return (Plan *) make_modifytable(root, + parse->commandType, parse->canSetTag, resultRelations, subplans, + withCheckOptionLists, returningLists, rowMarks, SS_assign_special_param(root)); @@ -1195,7 +1322,7 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) /* * If there's a top-level ORDER BY, assume we have to fetch all the - * tuples. This might be too simplistic given all the hackery below + * tuples. This might be too simplistic given all the hackery below * to possibly avoid the sort; but the odds of accurate estimates here * are pretty low anyway. */ @@ -1218,12 +1345,11 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) */ current_pathkeys = make_pathkeys_for_sortclauses(root, set_sortclauses, - result_plan->targetlist, - true); + result_plan->targetlist); /* * We should not need to call preprocess_targetlist, since we must be - * in a SELECT query node. Instead, use the targetlist returned by + * in a SELECT query node. Instead, use the targetlist returned by * plan_set_operations (since this tells whether it returned any * resjunk columns!), and transfer any sort key information from the * original tlist. @@ -1234,13 +1360,17 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) tlist); /* - * Can't handle FOR UPDATE/SHARE here (parser should have checked - * already, but let's make sure). + * Can't handle FOR [KEY] UPDATE/SHARE here (parser should have + * checked already, but let's make sure). */ if (parse->rowMarks) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("SELECT FOR UPDATE/SHARE is not allowed with UNION/INTERSECT/EXCEPT"))); + /*------ + translator: %s is a SQL row locking clause such as FOR UPDATE */ + errmsg("%s is not allowed with UNION/INTERSECT/EXCEPT", + LCS_asString(((RowMarkClause *) + linitial(parse->rowMarks))->strength)))); /* * Calculate pathkeys that represent result ordering requirements @@ -1248,17 +1378,16 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) Assert(parse->distinctClause == NIL); root->sort_pathkeys = make_pathkeys_for_sortclauses(root, parse->sortClause, - tlist, - true); + tlist); } else { /* No set operations, do regular planning */ List *sub_tlist; - double sub_limit_tuples; AttrNumber *groupColIdx = NULL; bool need_tlist_eval = true; - QualCost tlist_cost; + standard_qp_extra qp_extra; + RelOptInfo *final_rel; Path *cheapest_path; Path *sorted_path; Path *best_path; @@ -1285,6 +1414,12 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) tlist = preprocess_targetlist(root, tlist); /* + * Expand any rangetable entries that have security barrier quals. + * This may add new security barrier subquery RTEs to the rangetable. + */ + expand_security_quals(root, tlist); + + /* * Locate any window functions in the tlist. (We don't need to look * anywhere else, since expressions used in ORDER BY will be in there * too.) Note that they could all have been eliminated by constant @@ -1334,81 +1469,8 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) preprocess_minmax_aggregates(root, tlist); } - /* - * Calculate pathkeys that represent grouping/ordering requirements. - * Stash them in PlannerInfo so that query_planner can canonicalize - * them after EquivalenceClasses have been formed. The sortClause is - * certainly sort-able, but GROUP BY and DISTINCT might not be, in - * which case we just leave their pathkeys empty. - */ - if (parse->groupClause && - grouping_is_sortable(parse->groupClause)) - root->group_pathkeys = - make_pathkeys_for_sortclauses(root, - parse->groupClause, - tlist, - false); - else - root->group_pathkeys = NIL; - - /* We consider only the first (bottom) window in pathkeys logic */ - if (activeWindows != NIL) - { - WindowClause *wc = (WindowClause *) linitial(activeWindows); - - root->window_pathkeys = make_pathkeys_for_window(root, - wc, - tlist, - false); - } - else - root->window_pathkeys = NIL; - - if (parse->distinctClause && - grouping_is_sortable(parse->distinctClause)) - root->distinct_pathkeys = - make_pathkeys_for_sortclauses(root, - parse->distinctClause, - tlist, - false); - else - root->distinct_pathkeys = NIL; - - root->sort_pathkeys = - make_pathkeys_for_sortclauses(root, - parse->sortClause, - tlist, - false); - - /* - * Figure out whether we want a sorted result from query_planner. - * - * If we have a sortable GROUP BY clause, then we want a result sorted - * properly for grouping. Otherwise, if we have window functions to - * evaluate, we try to sort for the first window. Otherwise, if - * there's a sortable DISTINCT clause that's more rigorous than the - * ORDER BY clause, we try to produce output that's sufficiently well - * sorted for the DISTINCT. Otherwise, if there is an ORDER BY - * clause, we want to sort by the ORDER BY clause. - * - * Note: if we have both ORDER BY and GROUP BY, and ORDER BY is a - * superset of GROUP BY, it would be tempting to request sort by ORDER - * BY --- but that might just leave us failing to exploit an available - * sort order at all. Needs more thought. The choice for DISTINCT - * versus ORDER BY is much easier, since we know that the parser - * ensured that one is a superset of the other. - */ - if (root->group_pathkeys) - root->query_pathkeys = root->group_pathkeys; - else if (root->window_pathkeys) - root->query_pathkeys = root->window_pathkeys; - else if (list_length(root->distinct_pathkeys) > - list_length(root->sort_pathkeys)) - root->query_pathkeys = root->distinct_pathkeys; - else if (root->sort_pathkeys) - root->query_pathkeys = root->sort_pathkeys; - else - root->query_pathkeys = NIL; + /* Make tuple_fraction accessible to lower-level routines */ + root->tuple_fraction = tuple_fraction; /* * Figure out whether there's a hard limit on the number of rows that @@ -1421,35 +1483,174 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) parse->hasAggs || parse->hasWindowFuncs || root->hasHavingQual) - sub_limit_tuples = -1.0; + root->limit_tuples = -1.0; else - sub_limit_tuples = limit_tuples; + root->limit_tuples = limit_tuples; + + /* Set up data needed by standard_qp_callback */ + qp_extra.tlist = tlist; + qp_extra.activeWindows = activeWindows; /* * Generate the best unsorted and presorted paths for this Query (but - * note there may not be any presorted path). query_planner will also - * estimate the number of groups in the query, and canonicalize all - * the pathkeys. + * note there may not be any presorted paths). We also generate (in + * standard_qp_callback) pathkey representations of the query's sort + * clause, distinct clause, etc. */ - query_planner(root, sub_tlist, tuple_fraction, sub_limit_tuples, - &cheapest_path, &sorted_path, &dNumGroups); + final_rel = query_planner(root, sub_tlist, + standard_qp_callback, &qp_extra); /* - * Extract rowcount and width estimates for possible use in grouping - * decisions. Beware here of the possibility that - * cheapest_path->parent is NULL (ie, there is no FROM clause). + * Extract rowcount and width estimates for use below. */ - if (cheapest_path->parent) + path_rows = final_rel->rows; + path_width = final_rel->width; + + /* + * If there's grouping going on, estimate the number of result groups. + * We couldn't do this any earlier because it depends on relation size + * estimates that are created within query_planner(). + * + * Then convert tuple_fraction to fractional form if it is absolute, + * and if grouping or aggregation is involved, adjust tuple_fraction + * to describe the fraction of the underlying un-aggregated tuples + * that will be fetched. + */ + dNumGroups = 1; /* in case not grouping */ + + if (parse->groupClause) + { + List *groupExprs; + + groupExprs = get_sortgrouplist_exprs(parse->groupClause, + parse->targetList); + dNumGroups = estimate_num_groups(root, groupExprs, path_rows); + + /* + * In GROUP BY mode, an absolute LIMIT is relative to the number + * of groups not the number of tuples. If the caller gave us a + * fraction, keep it as-is. (In both cases, we are effectively + * assuming that all the groups are about the same size.) + */ + if (tuple_fraction >= 1.0) + tuple_fraction /= dNumGroups; + + /* + * If both GROUP BY and ORDER BY are specified, we will need two + * levels of sort --- and, therefore, certainly need to read all + * the tuples --- unless ORDER BY is a subset of GROUP BY. + * Likewise if we have both DISTINCT and GROUP BY, or if we have a + * window specification not compatible with the GROUP BY. + */ + if (!pathkeys_contained_in(root->sort_pathkeys, + root->group_pathkeys) || + !pathkeys_contained_in(root->distinct_pathkeys, + root->group_pathkeys) || + !pathkeys_contained_in(root->window_pathkeys, + root->group_pathkeys)) + tuple_fraction = 0.0; + } + else if (parse->hasAggs || root->hasHavingQual) { - path_rows = cheapest_path->parent->rows; - path_width = cheapest_path->parent->width; + /* + * Ungrouped aggregate will certainly want to read all the tuples, + * and it will deliver a single result row (so leave dNumGroups + * set to 1). + */ + tuple_fraction = 0.0; + } + else if (parse->distinctClause) + { + /* + * Since there was no grouping or aggregation, it's reasonable to + * assume the UNIQUE filter has effects comparable to GROUP BY. + * (If DISTINCT is used with grouping, we ignore its effects for + * rowcount estimation purposes; this amounts to assuming the + * grouped rows are distinct already.) + */ + List *distinctExprs; + + distinctExprs = get_sortgrouplist_exprs(parse->distinctClause, + parse->targetList); + dNumGroups = estimate_num_groups(root, distinctExprs, path_rows); + + /* + * Adjust tuple_fraction the same way as for GROUP BY, too. + */ + if (tuple_fraction >= 1.0) + tuple_fraction /= dNumGroups; } else { - path_rows = 1; /* assume non-set result */ - path_width = 100; /* arbitrary */ + /* + * Plain non-grouped, non-aggregated query: an absolute tuple + * fraction can be divided by the number of tuples. + */ + if (tuple_fraction >= 1.0) + tuple_fraction /= path_rows; } + /* + * Pick out the cheapest-total path as well as the cheapest presorted + * path for the requested pathkeys (if there is one). We should take + * the tuple fraction into account when selecting the cheapest + * presorted path, but not when selecting the cheapest-total path, + * since if we have to sort then we'll have to fetch all the tuples. + * (But there's a special case: if query_pathkeys is NIL, meaning + * order doesn't matter, then the "cheapest presorted" path will be + * the cheapest overall for the tuple fraction.) + */ + cheapest_path = final_rel->cheapest_total_path; + + sorted_path = + get_cheapest_fractional_path_for_pathkeys(final_rel->pathlist, + root->query_pathkeys, + NULL, + tuple_fraction); + + /* Don't consider same path in both guises; just wastes effort */ + if (sorted_path == cheapest_path) + sorted_path = NULL; + + /* + * Forget about the presorted path if it would be cheaper to sort the + * cheapest-total path. Here we need consider only the behavior at + * the tuple_fraction point. Also, limit_tuples is only relevant if + * not grouping/aggregating, so use root->limit_tuples in the + * cost_sort call. + */ + if (sorted_path) + { + Path sort_path; /* dummy for result of cost_sort */ + + if (root->query_pathkeys == NIL || + pathkeys_contained_in(root->query_pathkeys, + cheapest_path->pathkeys)) + { + /* No sort needed for cheapest path */ + sort_path.startup_cost = cheapest_path->startup_cost; + sort_path.total_cost = cheapest_path->total_cost; + } + else + { + /* Figure cost for sorting */ + cost_sort(&sort_path, root, root->query_pathkeys, + cheapest_path->total_cost, + path_rows, path_width, + 0.0, work_mem, root->limit_tuples); + } + + if (compare_fractional_path_costs(sorted_path, &sort_path, + tuple_fraction) > 0) + { + /* Presorted path is a loser */ + sorted_path = NULL; + } + } + + /* + * Consider whether we want to use hashing instead of sorting. + */ if (parse->groupClause) { /* @@ -1488,7 +1689,7 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) /* * Select the best path. If we are doing hashed grouping, we will * always read all the input tuples, so use the cheapest-total path. - * Otherwise, trust query_planner's decision about which to use. + * Otherwise, the comparison above is correct. */ if (use_hashed_grouping || use_hashed_distinct || !sorted_path) best_path = cheapest_path; @@ -1550,10 +1751,12 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) { /* * If the top-level plan node is one that cannot do expression - * evaluation, we must insert a Result node to project the + * evaluation and its existing target list isn't already what + * we need, we must insert a Result node to project the * desired tlist. */ - if (!is_projection_capable_plan(result_plan)) + if (!is_projection_capable_plan(result_plan) && + !tlist_same_exprs(sub_tlist, result_plan->targetlist)) { result_plan = (Plan *) make_result(root, sub_tlist, @@ -1579,27 +1782,9 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) /* * Also, account for the cost of evaluation of the sub_tlist. - * - * Up to now, we have only been dealing with "flat" tlists, - * containing just Vars. So their evaluation cost is zero - * according to the model used by cost_qual_eval() (or if you - * prefer, the cost is factored into cpu_tuple_cost). Thus we - * can avoid accounting for tlist cost throughout - * query_planner() and subroutines. But now we've inserted a - * tlist that might contain actual operators, sub-selects, etc - * --- so we'd better account for its cost. - * - * Below this point, any tlist eval cost for added-on nodes - * should be accounted for as we create those nodes. - * Presently, of the node types we can add on, only Agg, - * WindowAgg, and Group project new tlists (the rest just copy - * their input tuples) --- so make_agg(), make_windowagg() and - * make_group() are responsible for computing the added cost. + * See comments for add_tlist_costs_to_plan() for more info. */ - cost_qual_eval(&tlist_cost, sub_tlist, root); - result_plan->startup_cost += tlist_cost.startup; - result_plan->total_cost += tlist_cost.startup + - tlist_cost.per_tuple * result_plan->plan_rows; + add_tlist_costs_to_plan(root, result_plan, sub_tlist); } else { @@ -1742,7 +1927,7 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) * Furthermore, there cannot be any variables in either HAVING * or the targetlist, so we actually do not need the FROM * table at all! We can just throw away the plan-so-far and - * generate a Result node. This is a sufficiently unusual + * generate a Result node. This is a sufficiently unusual * corner case that it's not worth contorting the structure of * this routine to avoid having to generate the plan in the * first place. @@ -1786,10 +1971,13 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) * If the top-level plan node is one that cannot do expression * evaluation, we must insert a Result node to project the desired * tlist. (In some cases this might not really be required, but - * it's not worth trying to avoid it.) Note that on second and - * subsequent passes through the following loop, the top-level - * node will be a WindowAgg which we know can project; so we only - * need to check once. + * it's not worth trying to avoid it. In particular, think not to + * skip adding the Result if the initial window_tlist matches the + * top-level plan node's output, because we might change the tlist + * inside the following loop.) Note that on second and subsequent + * passes through the following loop, the top-level node will be a + * WindowAgg which we know can project; so we only need to check + * once. */ if (!is_projection_capable_plan(result_plan)) { @@ -1801,31 +1989,25 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) /* * The "base" targetlist for all steps of the windowing process is - * a flat tlist of all Vars and Aggs needed in the result. (In + * a flat tlist of all Vars and Aggs needed in the result. (In * some cases we wouldn't need to propagate all of these all the * way to the top, since they might only be needed as inputs to * WindowFuncs. It's probably not worth trying to optimize that - * though.) We also need any volatile sort expressions, because - * make_sort_from_pathkeys won't add those on its own, and anyway - * we want them evaluated only once at the bottom of the stack. As - * we climb up the stack, we add outputs for the WindowFuncs - * computed at each level. Also, each input tlist has to present - * all the columns needed to sort the data for the next WindowAgg - * step. That's handled internally by make_sort_from_pathkeys, - * but we need the copyObject steps here to ensure that each plan - * node has a separately modifiable tlist. - * - * Note: it's essential here to use PVC_INCLUDE_AGGREGATES so that - * Vars mentioned only in aggregate expressions aren't pulled out - * as separate targetlist entries. Otherwise we could be putting - * ungrouped Vars directly into an Agg node's tlist, resulting in - * undefined behavior. + * though.) We also add window partitioning and sorting + * expressions to the base tlist, to ensure they're computed only + * once at the bottom of the stack (that's critical for volatile + * functions). As we climb up the stack, we'll add outputs for + * the WindowFuncs computed at each level. + */ + window_tlist = make_windowInputTargetList(root, + tlist, + activeWindows); + + /* + * The copyObject steps here are needed to ensure that each plan + * node has a separately modifiable tlist. (XXX wouldn't a + * shallow list copy do for that?) */ - window_tlist = flatten_tlist(tlist, - PVC_INCLUDE_AGGREGATES, - PVC_INCLUDE_PLACEHOLDERS); - window_tlist = add_volatile_sort_exprs(window_tlist, tlist, - activeWindows); result_plan->targetlist = (List *) copyObject(window_tlist); #ifdef XCP /* @@ -1856,17 +2038,18 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) window_pathkeys = make_pathkeys_for_window(root, wc, - tlist, - true); + tlist); /* * This is a bit tricky: we build a sort node even if we don't * really have to sort. Even when no explicit sort is needed, * we need to have suitable resjunk items added to the input * plan's tlist for any partitioning or ordering columns that - * aren't plain Vars. Furthermore, this way we can use - * existing infrastructure to identify which input columns are - * the interesting ones. + * aren't plain Vars. (In theory, make_windowInputTargetList + * should have provided all such columns, but let's not assume + * that here.) Furthermore, this way we can use existing + * infrastructure to identify which input columns are the + * interesting ones. */ if (window_pathkeys) { @@ -1973,7 +2156,7 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) * If there was grouping or aggregation, use the current number of * rows as the estimated number of DISTINCT rows (ie, assume the * result was already mostly unique). If not, use the number of - * distinct-groups calculated by query_planner. + * distinct-groups calculated previously. */ if (parse->groupClause || root->hasHavingQual || parse->hasAggs) dNumDistinctRows = result_plan->plan_rows; @@ -2102,9 +2285,9 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) } /* - * If there is a FOR UPDATE/SHARE clause, add the LockRows node. (Note: we - * intentionally test parse->rowMarks not root->rowMarks here. If there - * are only non-locking rowmarks, they should be handled by the + * If there is a FOR [KEY] UPDATE/SHARE clause, add the LockRows node. + * (Note: we intentionally test parse->rowMarks not root->rowMarks here. + * If there are only non-locking rowmarks, they should be handled by the * ModifyTable node instead.) */ if (parse->rowMarks) @@ -2123,7 +2306,7 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) /* * Finally, if there is a LIMIT/OFFSET clause, add the LIMIT node. */ - if (parse->limitCount || parse->limitOffset) + if (limit_needed(parse)) { #ifdef XCP /* We should put Limit on top of distributed results */ @@ -2301,6 +2484,61 @@ grouping_planner(PlannerInfo *root, double tuple_fraction) } /* + * add_tlist_costs_to_plan + * + * Estimate the execution costs associated with evaluating the targetlist + * expressions, and add them to the cost estimates for the Plan node. + * + * If the tlist contains set-returning functions, also inflate the Plan's cost + * and plan_rows estimates accordingly. (Hence, this must be called *after* + * any logic that uses plan_rows to, eg, estimate qual evaluation costs.) + * + * Note: during initial stages of planning, we mostly consider plan nodes with + * "flat" tlists, containing just Vars. So their evaluation cost is zero + * according to the model used by cost_qual_eval() (or if you prefer, the cost + * is factored into cpu_tuple_cost). Thus we can avoid accounting for tlist + * cost throughout query_planner() and subroutines. But once we apply a + * tlist that might contain actual operators, sub-selects, etc, we'd better + * account for its cost. Any set-returning functions in the tlist must also + * affect the estimated rowcount. + * + * Once grouping_planner() has applied a general tlist to the topmost + * scan/join plan node, any tlist eval cost for added-on nodes should be + * accounted for as we create those nodes. Presently, of the node types we + * can add on later, only Agg, WindowAgg, and Group project new tlists (the + * rest just copy their input tuples) --- so make_agg(), make_windowagg() and + * make_group() are responsible for calling this function to account for their + * tlist costs. + */ +void +add_tlist_costs_to_plan(PlannerInfo *root, Plan *plan, List *tlist) +{ + QualCost tlist_cost; + double tlist_rows; + + cost_qual_eval(&tlist_cost, tlist, root); + plan->startup_cost += tlist_cost.startup; + plan->total_cost += tlist_cost.startup + + tlist_cost.per_tuple * plan->plan_rows; + + tlist_rows = tlist_returns_set_rows(tlist); + if (tlist_rows > 1) + { + /* + * We assume that execution costs of the tlist proper were all + * accounted for by cost_qual_eval. However, it still seems + * appropriate to charge something more for the executor's general + * costs of processing the added tuples. The cost is probably less + * than cpu_tuple_cost, though, so we arbitrarily use half of that. + */ + plan->total_cost += plan->plan_rows * (tlist_rows - 1) * + cpu_tuple_cost / 2; + + plan->plan_rows *= tlist_rows; + } +} + +/* * Detect whether a plan node is a "dummy" plan created when a relation * is deemed not to need scanning due to constraint exclusion. * @@ -2392,17 +2630,19 @@ preprocess_rowmarks(PlannerInfo *root) if (parse->rowMarks) { /* - * We've got trouble if FOR UPDATE/SHARE appears inside grouping, - * since grouping renders a reference to individual tuple CTIDs - * invalid. This is also checked at parse time, but that's + * We've got trouble if FOR [KEY] UPDATE/SHARE appears inside + * grouping, since grouping renders a reference to individual tuple + * CTIDs invalid. This is also checked at parse time, but that's * insufficient because of rule substitution, query pullup, etc. */ - CheckSelectLocking(parse); + CheckSelectLocking(parse, ((RowMarkClause *) + linitial(parse->rowMarks))->strength); } else { /* - * We only need rowmarks for UPDATE, DELETE, or FOR UPDATE/SHARE. + * We only need rowmarks for UPDATE, DELETE, or FOR [KEY] + * UPDATE/SHARE. */ if (parse->commandType != CMD_UPDATE && parse->commandType != CMD_DELETE) @@ -2412,7 +2652,7 @@ preprocess_rowmarks(PlannerInfo *root) /* * We need to have rowmarks for all base relations except the target. We * make a bitmapset of all base rels and then remove the items we don't - * need or have FOR UPDATE/SHARE marks for. + * need or have FOR [KEY] UPDATE/SHARE marks for. */ rels = get_base_rel_indexes((Node *) parse->jointree); if (parse->resultRelation) @@ -2429,8 +2669,8 @@ preprocess_rowmarks(PlannerInfo *root) PlanRowMark *newrc; /* - * Currently, it is syntactically impossible to have FOR UPDATE - * applied to an update/delete target rel. If that ever becomes + * Currently, it is syntactically impossible to have FOR UPDATE et al + * applied to an update/delete target rel. If that ever becomes * possible, we should drop the target from the PlanRowMark list. */ Assert(rc->rti != parse->resultRelation); @@ -2444,15 +2684,35 @@ preprocess_rowmarks(PlannerInfo *root) if (rte->rtekind != RTE_RELATION) continue; + /* + * Similarly, ignore RowMarkClauses for foreign tables; foreign tables + * will instead get ROW_MARK_COPY items in the next loop. (FDWs might + * choose to do something special while fetching their rows, but that + * is of no concern here.) + */ + if (rte->relkind == RELKIND_FOREIGN_TABLE) + continue; + rels = bms_del_member(rels, rc->rti); newrc = makeNode(PlanRowMark); newrc->rti = newrc->prti = rc->rti; newrc->rowmarkId = ++(root->glob->lastRowMarkId); - if (rc->forUpdate) - newrc->markType = ROW_MARK_EXCLUSIVE; - else - newrc->markType = ROW_MARK_SHARE; + switch (rc->strength) + { + case LCS_FORUPDATE: + newrc->markType = ROW_MARK_EXCLUSIVE; + break; + case LCS_FORNOKEYUPDATE: + newrc->markType = ROW_MARK_NOKEYEXCLUSIVE; + break; + case LCS_FORSHARE: + newrc->markType = ROW_MARK_SHARE; + break; + case LCS_FORKEYSHARE: + newrc->markType = ROW_MARK_KEYSHARE; + break; + } newrc->noWait = rc->noWait; newrc->isParent = false; @@ -2533,7 +2793,7 @@ separate_rowmarks(PlannerInfo *root) * preprocess_limit - do pre-estimation for LIMIT and/or OFFSET clauses * * We try to estimate the values of the LIMIT/OFFSET clauses, and pass the - * results back in *count_est and *offset_est. These variables are set to + * results back in *count_est and *offset_est. These variables are set to * 0 if the corresponding clause is not present, and -1 if it's present * but we couldn't estimate the value for it. (The "0" convention is OK * for OFFSET but a little bit bogus for LIMIT: effectively we estimate @@ -2542,7 +2802,7 @@ separate_rowmarks(PlannerInfo *root) * be passed to make_limit, which see if you change this code. * * The return value is the suitably adjusted tuple_fraction to use for - * planning the query. This adjustment is not overridable, since it reflects + * planning the query. This adjustment is not overridable, since it reflects * plan actions that grouping_planner() will certainly take, not assumptions * about context. */ @@ -2666,7 +2926,7 @@ preprocess_limit(PlannerInfo *root, double tuple_fraction, else if (*offset_est != 0 && tuple_fraction > 0.0) { /* - * We have an OFFSET but no LIMIT. This acts entirely differently + * We have an OFFSET but no LIMIT. This acts entirely differently * from the LIMIT case: here, we need to increase rather than decrease * the caller's tuple_fraction, because the OFFSET acts to cause more * tuples to be fetched instead of fewer. This only matters if we got @@ -2681,7 +2941,7 @@ preprocess_limit(PlannerInfo *root, double tuple_fraction, /* * If we have absolute counts from both caller and OFFSET, add them - * together; likewise if they are both fractional. If one is + * together; likewise if they are both fractional. If one is * fractional and the other absolute, we want to take the larger, and * we heuristically assume that's the fractional one. */ @@ -2717,6 +2977,60 @@ preprocess_limit(PlannerInfo *root, double tuple_fraction, return tuple_fraction; } +/* + * limit_needed - do we actually need a Limit plan node? + * + * If we have constant-zero OFFSET and constant-null LIMIT, we can skip adding + * a Limit node. This is worth checking for because "OFFSET 0" is a common + * locution for an optimization fence. (Because other places in the planner + * merely check whether parse->limitOffset isn't NULL, it will still work as + * an optimization fence --- we're just suppressing unnecessary run-time + * overhead.) + * + * This might look like it could be merged into preprocess_limit, but there's + * a key distinction: here we need hard constants in OFFSET/LIMIT, whereas + * in preprocess_limit it's good enough to consider estimated values. + */ +static bool +limit_needed(Query *parse) +{ + Node *node; + + node = parse->limitCount; + if (node) + { + if (IsA(node, Const)) + { + /* NULL indicates LIMIT ALL, ie, no limit */ + if (!((Const *) node)->constisnull) + return true; /* LIMIT with a constant value */ + } + else + return true; /* non-constant LIMIT */ + } + + node = parse->limitOffset; + if (node) + { + if (IsA(node, Const)) + { + /* Treat NULL as no offset; the executor would too */ + if (!((Const *) node)->constisnull) + { + int64 offset = DatumGetInt64(((Const *) node)->constvalue); + + /* Executor would treat less-than-zero same as zero */ + if (offset > 0) + return true; /* OFFSET with a positive value */ + } + } + else + return true; /* non-constant OFFSET */ + } + + return false; /* don't need a Limit plan node */ +} + /* * preprocess_groupclause - do preparatory work on GROUP BY clause @@ -2806,6 +3120,88 @@ preprocess_groupclause(PlannerInfo *root) } /* + * Compute query_pathkeys and other pathkeys during plan generation + */ +static void +standard_qp_callback(PlannerInfo *root, void *extra) +{ + Query *parse = root->parse; + standard_qp_extra *qp_extra = (standard_qp_extra *) extra; + List *tlist = qp_extra->tlist; + List *activeWindows = qp_extra->activeWindows; + + /* + * Calculate pathkeys that represent grouping/ordering requirements. The + * sortClause is certainly sort-able, but GROUP BY and DISTINCT might not + * be, in which case we just leave their pathkeys empty. + */ + if (parse->groupClause && + grouping_is_sortable(parse->groupClause)) + root->group_pathkeys = + make_pathkeys_for_sortclauses(root, + parse->groupClause, + tlist); + else + root->group_pathkeys = NIL; + + /* We consider only the first (bottom) window in pathkeys logic */ + if (activeWindows != NIL) + { + WindowClause *wc = (WindowClause *) linitial(activeWindows); + + root->window_pathkeys = make_pathkeys_for_window(root, + wc, + tlist); + } + else + root->window_pathkeys = NIL; + + if (parse->distinctClause && + grouping_is_sortable(parse->distinctClause)) + root->distinct_pathkeys = + make_pathkeys_for_sortclauses(root, + parse->distinctClause, + tlist); + else + root->distinct_pathkeys = NIL; + + root->sort_pathkeys = + make_pathkeys_for_sortclauses(root, + parse->sortClause, + tlist); + + /* + * Figure out whether we want a sorted result from query_planner. + * + * If we have a sortable GROUP BY clause, then we want a result sorted + * properly for grouping. Otherwise, if we have window functions to + * evaluate, we try to sort for the first window. Otherwise, if there's a + * sortable DISTINCT clause that's more rigorous than the ORDER BY clause, + * we try to produce output that's sufficiently well sorted for the + * DISTINCT. Otherwise, if there is an ORDER BY clause, we want to sort + * by the ORDER BY clause. + * + * Note: if we have both ORDER BY and GROUP BY, and ORDER BY is a superset + * of GROUP BY, it would be tempting to request sort by ORDER BY --- but + * that might just leave us failing to exploit an available sort order at + * all. Needs more thought. The choice for DISTINCT versus ORDER BY is + * much easier, since we know that the parser ensured that one is a + * superset of the other. + */ + if (root->group_pathkeys) + root->query_pathkeys = root->group_pathkeys; + else if (root->window_pathkeys) + root->query_pathkeys = root->window_pathkeys; + else if (list_length(root->distinct_pathkeys) > + list_length(root->sort_pathkeys)) + root->query_pathkeys = root->distinct_pathkeys; + else if (root->sort_pathkeys) + root->query_pathkeys = root->sort_pathkeys; + else + root->query_pathkeys = NIL; +} + +/* * choose_hashed_grouping - should we use hashed grouping? * * Returns TRUE to select hashing, FALSE to select sorting. @@ -2829,9 +3225,11 @@ choose_hashed_grouping(PlannerInfo *root, /* * Executor doesn't support hashed aggregation with DISTINCT or ORDER BY - * aggregates. (Doing so would imply storing *all* the input values in + * aggregates. (Doing so would imply storing *all* the input values in * the hash table, and/or running many sorts in parallel, either of which - * seems like a certain loser.) + * seems like a certain loser.) We similarly don't support ordered-set + * aggregates in hashed aggregation, but that case is included in the + * numOrderedAggs count. */ can_hash = (agg_costs->numOrderedAggs == 0 && grouping_is_hashable(parse->groupClause)); @@ -2892,8 +3290,8 @@ choose_hashed_grouping(PlannerInfo *root, * We need to consider cheapest_path + hashagg [+ final sort] versus * either cheapest_path [+ sort] + group or agg [+ final sort] or * presorted_path + group or agg [+ final sort] where brackets indicate a - * step that may not be needed. We assume query_planner() will have - * returned a presorted path only if it's a winner compared to + * step that may not be needed. We assume grouping_planner() will have + * passed us a presorted path only if it's a winner compared to * cheapest_path for this purpose. * * These path variables are dummies that just hold cost fields; we don't @@ -2946,12 +3344,8 @@ choose_hashed_grouping(PlannerInfo *root, 0.0, work_mem, limit_tuples); /* - * Now make the decision using the top-level tuple fraction. First we - * have to convert an absolute count (LIMIT) into fractional form. + * Now make the decision using the top-level tuple fraction. */ - if (tuple_fraction >= 1.0) - tuple_fraction /= dNumGroups; - if (compare_fractional_path_costs(&hashed_p, &sorted_p, tuple_fraction) < 0) { @@ -2971,7 +3365,7 @@ choose_hashed_grouping(PlannerInfo *root, * pass in the costs as individual variables.) * * But note that making the two choices independently is a bit bogus in - * itself. If the two could be combined into a single choice operation + * itself. If the two could be combined into a single choice operation * it'd probably be better, but that seems far too unwieldy to be practical, * especially considering that the combination of GROUP BY and DISTINCT * isn't very common in real queries. By separating them, we are giving @@ -3031,7 +3425,11 @@ choose_hashed_distinct(PlannerInfo *root, * Don't do it if it doesn't look like the hashtable will fit into * work_mem. */ + + /* Estimate per-hash-entry space at tuple width... */ hashentrysize = MAXALIGN(path_width) + MAXALIGN(sizeof(MinimalTupleData)); + /* plus the per-hash-entry overhead */ + hashentrysize += hash_agg_entry_size(0); if (hashentrysize * dNumDistinctRows > work_mem * 1024L) return false; @@ -3064,7 +3462,7 @@ choose_hashed_distinct(PlannerInfo *root, 0.0, work_mem, limit_tuples); /* - * Now for the GROUP case. See comments in grouping_planner about the + * Now for the GROUP case. See comments in grouping_planner about the * sorting choices here --- this code should match that code. */ sorted_p.startup_cost = sorted_startup_cost; @@ -3097,12 +3495,8 @@ choose_hashed_distinct(PlannerInfo *root, 0.0, work_mem, limit_tuples); /* - * Now make the decision using the top-level tuple fraction. First we - * have to convert an absolute count (LIMIT) into fractional form. + * Now make the decision using the top-level tuple fraction. */ - if (tuple_fraction >= 1.0) - tuple_fraction /= dNumDistinctRows; - if (compare_fractional_path_costs(&hashed_p, &sorted_p, tuple_fraction) < 0) { @@ -3147,7 +3541,8 @@ choose_hashed_distinct(PlannerInfo *root, * 'groupColIdx' receives an array of column numbers for the GROUP BY * expressions (if there are any) in the returned target list. * 'need_tlist_eval' is set true if we really need to evaluate the - * returned tlist as-is. + * returned tlist as-is. (Note: locate_grouping_columns assumes + * that if this is FALSE, all grouping columns are simple Vars.) * * The result is the targetlist to be passed to query_planner. */ @@ -3257,7 +3652,7 @@ make_subplanTargetList(PlannerInfo *root, * add them to the result tlist if not already present. (A Var used * directly as a GROUP BY item will be present already.) Note this * includes Vars used in resjunk items, so we are covering the needs of - * ORDER BY and window specifications. Vars used within Aggrefs will be + * ORDER BY and window specifications. Vars used within Aggrefs will be * pulled out here, too. */ non_group_vars = pull_var_clause((Node *) non_group_cols, @@ -3308,8 +3703,9 @@ get_grouping_column_index(Query *parse, TargetEntry *tle) * Locate grouping columns in the tlist chosen by create_plan. * * This is only needed if we don't use the sub_tlist chosen by - * make_subplanTargetList. We have to forget the column indexes found + * make_subplanTargetList. We have to forget the column indexes found * by that routine and re-locate the grouping exprs in the real sub_tlist. + * We assume the grouping exprs are just Vars (see make_subplanTargetList). */ static void locate_grouping_columns(PlannerInfo *root, @@ -3333,11 +3729,24 @@ locate_grouping_columns(PlannerInfo *root, foreach(gl, root->parse->groupClause) { SortGroupClause *grpcl = (SortGroupClause *) lfirst(gl); - Node *groupexpr = get_sortgroupclause_expr(grpcl, tlist); - TargetEntry *te = tlist_member(groupexpr, sub_tlist); + Var *groupexpr = (Var *) get_sortgroupclause_expr(grpcl, tlist); + TargetEntry *te; + /* + * The grouping column returned by create_plan might not have the same + * typmod as the original Var. (This can happen in cases where a + * set-returning function has been inlined, so that we now have more + * knowledge about what it returns than we did when the original Var + * was created.) So we can't use tlist_member() to search the tlist; + * instead use tlist_member_match_var. For safety, still check that + * the vartype matches. + */ + if (!(groupexpr && IsA(groupexpr, Var))) + elog(ERROR, "grouping column is not a Var as expected"); + te = tlist_member_match_var(groupexpr, sub_tlist); if (!te) elog(ERROR, "failed to locate grouping columns"); + Assert(((Var *) te->expr)->vartype == groupexpr->vartype); groupColIdx[keyno++] = te->resno; } } @@ -3450,18 +3859,57 @@ select_active_windows(PlannerInfo *root, WindowFuncLists *wflists) } /* - * add_volatile_sort_exprs - * Identify any volatile sort/group expressions used by the active - * windows, and add them to window_tlist if not already present. - * Return the modified window_tlist. + * make_windowInputTargetList + * Generate appropriate target list for initial input to WindowAgg nodes. + * + * When grouping_planner inserts one or more WindowAgg nodes into the plan, + * this function computes the initial target list to be computed by the node + * just below the first WindowAgg. This list must contain all values needed + * to evaluate the window functions, compute the final target list, and + * perform any required final sort step. If multiple WindowAggs are needed, + * each intermediate one adds its window function results onto this tlist; + * only the topmost WindowAgg computes the actual desired target list. + * + * This function is much like make_subplanTargetList, though not quite enough + * like it to share code. As in that function, we flatten most expressions + * into their component variables. But we do not want to flatten window + * PARTITION BY/ORDER BY clauses, since that might result in multiple + * evaluations of them, which would be bad (possibly even resulting in + * inconsistent answers, if they contain volatile functions). Also, we must + * not flatten GROUP BY clauses that were left unflattened by + * make_subplanTargetList, because we may no longer have access to the + * individual Vars in them. + * + * Another key difference from make_subplanTargetList is that we don't flatten + * Aggref expressions, since those are to be computed below the window + * functions and just referenced like Vars above that. + * + * 'tlist' is the query's final target list. + * 'activeWindows' is the list of active windows previously identified by + * select_active_windows. + * + * The result is the targetlist to be computed by the plan node immediately + * below the first WindowAgg node. */ static List * -add_volatile_sort_exprs(List *window_tlist, List *tlist, List *activeWindows) +make_windowInputTargetList(PlannerInfo *root, + List *tlist, + List *activeWindows) { - Bitmapset *sgrefs = NULL; + Query *parse = root->parse; + Bitmapset *sgrefs; + List *new_tlist; + List *flattenable_cols; + List *flattenable_vars; ListCell *lc; - /* First, collect the sortgrouprefs of the windows into a bitmapset */ + Assert(parse->hasWindowFuncs); + + /* + * Collect the sortgroupref numbers of window PARTITION/ORDER BY clauses + * into a bitmapset for convenient reference below. + */ + sgrefs = NULL; foreach(lc, activeWindows) { WindowClause *wc = (WindowClause *) lfirst(lc); @@ -3481,34 +3929,74 @@ add_volatile_sort_exprs(List *window_tlist, List *tlist, List *activeWindows) } } + /* Add in sortgroupref numbers of GROUP BY clauses, too */ + foreach(lc, parse->groupClause) + { + SortGroupClause *grpcl = (SortGroupClause *) lfirst(lc); + + sgrefs = bms_add_member(sgrefs, grpcl->tleSortGroupRef); + } + /* - * Now scan the original tlist to find the referenced expressions. Any - * that are volatile must be added to window_tlist. - * - * Note: we know that the input window_tlist contains no items marked with - * ressortgrouprefs, so we don't have to worry about collisions of the - * reference numbers. + * Construct a tlist containing all the non-flattenable tlist items, and + * save aside the others for a moment. */ + new_tlist = NIL; + flattenable_cols = NIL; + foreach(lc, tlist) { TargetEntry *tle = (TargetEntry *) lfirst(lc); + /* + * Don't want to deconstruct window clauses or GROUP BY items. (Note + * that such items can't contain window functions, so it's okay to + * compute them below the WindowAgg nodes.) + */ if (tle->ressortgroupref != 0 && - bms_is_member(tle->ressortgroupref, sgrefs) && - contain_volatile_functions((Node *) tle->expr)) + bms_is_member(tle->ressortgroupref, sgrefs)) { + /* Don't want to deconstruct this value, so add to new_tlist */ TargetEntry *newtle; newtle = makeTargetEntry(tle->expr, - list_length(window_tlist) + 1, + list_length(new_tlist) + 1, NULL, false); + /* Preserve its sortgroupref marking, in case it's volatile */ newtle->ressortgroupref = tle->ressortgroupref; - window_tlist = lappend(window_tlist, newtle); + new_tlist = lappend(new_tlist, newtle); + } + else + { + /* + * Column is to be flattened, so just remember the expression for + * later call to pull_var_clause. There's no need for + * pull_var_clause to examine the TargetEntry node itself. + */ + flattenable_cols = lappend(flattenable_cols, tle->expr); } } - return window_tlist; + /* + * Pull out all the Vars and Aggrefs mentioned in flattenable columns, and + * add them to the result tlist if not already present. (Some might be + * there already because they're used directly as window/group clauses.) + * + * Note: it's essential to use PVC_INCLUDE_AGGREGATES here, so that the + * Aggrefs are placed in the Agg node's tlist and not left to be computed + * at higher levels. + */ + flattenable_vars = pull_var_clause((Node *) flattenable_cols, + PVC_INCLUDE_AGGREGATES, + PVC_INCLUDE_PLACEHOLDERS); + new_tlist = add_to_flat_tlist(new_tlist, flattenable_vars); + + /* clean up cruft */ + list_free(flattenable_vars); + list_free(flattenable_cols); + + return new_tlist; } /* @@ -3522,7 +4010,7 @@ add_volatile_sort_exprs(List *window_tlist, List *tlist, List *activeWindows) */ static List * make_pathkeys_for_window(PlannerInfo *root, WindowClause *wc, - List *tlist, bool canonicalize) + List *tlist) { List *window_pathkeys; List *window_sortclauses; @@ -3544,8 +4032,7 @@ make_pathkeys_for_window(PlannerInfo *root, WindowClause *wc, list_copy(wc->orderClause)); window_pathkeys = make_pathkeys_for_sortclauses(root, window_sortclauses, - tlist, - canonicalize); + tlist); list_free(window_sortclauses); return window_pathkeys; } @@ -3558,7 +4045,7 @@ make_pathkeys_for_window(PlannerInfo *root, WindowClause *wc, * This depends on the behavior of make_pathkeys_for_window()! * * We are given the target WindowClause and an array of the input column - * numbers associated with the resulting pathkeys. In the easy case, there + * numbers associated with the resulting pathkeys. In the easy case, there * are the same number of pathkey columns as partitioning + ordering columns * and we just have to copy some data around. However, it's possible that * some of the original partitioning + ordering columns were eliminated as @@ -3570,7 +4057,7 @@ make_pathkeys_for_window(PlannerInfo *root, WindowClause *wc, * determine which keys are significant. * * The method used here is a bit brute-force: add the sort columns to a list - * one at a time and note when the resulting pathkey list gets longer. But + * one at a time and note when the resulting pathkey list gets longer. But * it's a sufficiently uncommon case that a faster way doesn't seem worth * the amount of code refactoring that'd be needed. *---------- @@ -3623,8 +4110,7 @@ get_column_info_for_window(PlannerInfo *root, WindowClause *wc, List *tlist, sortclauses = lappend(sortclauses, sgc); new_pathkeys = make_pathkeys_for_sortclauses(root, sortclauses, - tlist, - true); + tlist); if (list_length(new_pathkeys) > list_length(pathkeys)) { /* this sort clause is actually significant */ @@ -3642,8 +4128,7 @@ get_column_info_for_window(PlannerInfo *root, WindowClause *wc, List *tlist, sortclauses = lappend(sortclauses, sgc); new_pathkeys = make_pathkeys_for_sortclauses(root, sortclauses, - tlist, - true); + tlist); if (list_length(new_pathkeys) > list_length(pathkeys)) { /* this sort clause is actually significant */ @@ -3745,7 +4230,8 @@ plan_cluster_use_sort(Oid tableOid, Oid indexOid) rte = makeNode(RangeTblEntry); rte->rtekind = RTE_RELATION; rte->relid = tableOid; - rte->relkind = RELKIND_RELATION; + rte->relkind = RELKIND_RELATION; /* Don't be too picky. */ + rte->lateral = false; rte->inh = false; rte->inFromCl = true; query->rtable = list_make1(rte); diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index 691b6d0909..177927935b 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -9,7 +9,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * Portions Copyright (c) 2012-2014, TransLattice, Inc. - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -26,6 +26,7 @@ #include "nodes/nodeFuncs.h" #include "optimizer/pathnode.h" #include "optimizer/planmain.h" +#include "optimizer/planner.h" #include "optimizer/tlist.h" #include "tcop/utility.h" #include "utils/lsyscache.h" @@ -100,6 +101,10 @@ typedef struct #define fix_scan_list(root, lst, rtoffset) \ ((List *) fix_scan_expr(root, (Node *) (lst), rtoffset)) +static void add_rtes_to_flat_rtable(PlannerInfo *root, bool recursing); +static void flatten_unplanned_rtes(PlannerGlobal *glob, RangeTblEntry *rte); +static bool flatten_rtes_walker(Node *node, PlannerGlobal *glob); +static void add_rte_to_flat_rtable(PlannerGlobal *glob, RangeTblEntry *rte); static Plan *set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset); static Plan *set_indexonlyscan_references(PlannerInfo *root, IndexOnlyScan *plan, @@ -185,7 +190,7 @@ static void set_remotesubplan_references(PlannerInfo *root, Plan *plan, int rtof /* * set_plan_references * - * This is the final processing pass of the planner/optimizer. The plan + * This is the final processing pass of the planner/optimizer. The plan * tree is complete; we just have to adjust some representational details * for the convenience of the executor: * @@ -229,7 +234,7 @@ static void set_remotesubplan_references(PlannerInfo *root, Plan *plan, int rtof * and root->glob->invalItems (for everything else). * * Notice that we modify Plan nodes in-place, but use expression_tree_mutator - * to process targetlist and qual expressions. We can assume that the Plan + * to process targetlist and qual expressions. We can assume that the Plan * nodes were just built by the planner and are not multiply referenced, but * it's not so safe to assume that for expression tree nodes. */ @@ -241,63 +246,11 @@ set_plan_references(PlannerInfo *root, Plan *plan) ListCell *lc; /* - * In the flat rangetable, we zero out substructure pointers that are not - * needed by the executor; this reduces the storage space and copying cost - * for cached plans. We keep only the alias and eref Alias fields, which - * are needed by EXPLAIN, and the selectedCols and modifiedCols bitmaps, - * which are needed for executor-startup permissions checking and for - * trigger event checking. + * Add all the query's RTEs to the flattened rangetable. The live ones + * will have their rangetable indexes increased by rtoffset. (Additional + * RTEs, not referenced by the Plan tree, might get added after those.) */ - foreach(lc, root->parse->rtable) - { - RangeTblEntry *rte = (RangeTblEntry *) lfirst(lc); - RangeTblEntry *newrte; - - /* flat copy to duplicate all the scalar fields */ - newrte = (RangeTblEntry *) palloc(sizeof(RangeTblEntry)); - memcpy(newrte, rte, sizeof(RangeTblEntry)); - - /* zap unneeded sub-structure */ - newrte->subquery = NULL; - newrte->joinaliasvars = NIL; - newrte->funcexpr = NULL; - newrte->funccoltypes = NIL; - newrte->funccoltypmods = NIL; - newrte->funccolcollations = NIL; - newrte->values_lists = NIL; - newrte->values_collations = NIL; - newrte->ctecoltypes = NIL; - newrte->ctecoltypmods = NIL; - newrte->ctecolcollations = NIL; - - glob->finalrtable = lappend(glob->finalrtable, newrte); - - /* - * If it's a plain relation RTE, add the table to relationOids. - * - * We do this even though the RTE might be unreferenced in the plan - * tree; this would correspond to cases such as views that were - * expanded, child tables that were eliminated by constraint - * exclusion, etc. Schema invalidation on such a rel must still force - * rebuilding of the plan. - * - * Note we don't bother to avoid duplicate list entries. We could, - * but it would probably cost more cycles than it would save. - */ - if (newrte->rtekind == RTE_RELATION) - glob->relationOids = lappend_oid(glob->relationOids, - newrte->relid); - } - - /* - * Check for RT index overflow; it's very unlikely, but if it did happen, - * the executor would get confused by varnos that match the special varno - * values. - */ - if (IS_SPECIAL_VARNO(list_length(glob->finalrtable))) - ereport(ERROR, - (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), - errmsg("too many range table entries"))); + add_rtes_to_flat_rtable(root, false); /* * Adjust RT indexes of PlanRowMarks and add to final rowmarks list @@ -325,6 +278,190 @@ set_plan_references(PlannerInfo *root, Plan *plan) } /* + * Extract RangeTblEntries from the plan's rangetable, and add to flat rtable + * + * This can recurse into subquery plans; "recursing" is true if so. + */ +static void +add_rtes_to_flat_rtable(PlannerInfo *root, bool recursing) +{ + PlannerGlobal *glob = root->glob; + Index rti; + ListCell *lc; + + /* + * Add the query's own RTEs to the flattened rangetable. + * + * At top level, we must add all RTEs so that their indexes in the + * flattened rangetable match up with their original indexes. When + * recursing, we only care about extracting relation RTEs. + */ + foreach(lc, root->parse->rtable) + { + RangeTblEntry *rte = (RangeTblEntry *) lfirst(lc); + + if (!recursing || rte->rtekind == RTE_RELATION) + add_rte_to_flat_rtable(glob, rte); + } + + /* + * If there are any dead subqueries, they are not referenced in the Plan + * tree, so we must add RTEs contained in them to the flattened rtable + * separately. (If we failed to do this, the executor would not perform + * expected permission checks for tables mentioned in such subqueries.) + * + * Note: this pass over the rangetable can't be combined with the previous + * one, because that would mess up the numbering of the live RTEs in the + * flattened rangetable. + */ + rti = 1; + foreach(lc, root->parse->rtable) + { + RangeTblEntry *rte = (RangeTblEntry *) lfirst(lc); + + /* + * We should ignore inheritance-parent RTEs: their contents have been + * pulled up into our rangetable already. Also ignore any subquery + * RTEs without matching RelOptInfos, as they likewise have been + * pulled up. + */ + if (rte->rtekind == RTE_SUBQUERY && !rte->inh && + rti < root->simple_rel_array_size) + { + RelOptInfo *rel = root->simple_rel_array[rti]; + + if (rel != NULL) + { + Assert(rel->relid == rti); /* sanity check on array */ + + /* + * The subquery might never have been planned at all, if it + * was excluded on the basis of self-contradictory constraints + * in our query level. In this case apply + * flatten_unplanned_rtes. + * + * If it was planned but the plan is dummy, we assume that it + * has been omitted from our plan tree (see + * set_subquery_pathlist), and recurse to pull up its RTEs. + * + * Otherwise, it should be represented by a SubqueryScan node + * somewhere in our plan tree, and we'll pull up its RTEs when + * we process that plan node. + * + * However, if we're recursing, then we should pull up RTEs + * whether the subplan is dummy or not, because we've found + * that some upper query level is treating this one as dummy, + * and so we won't scan this level's plan tree at all. + */ + if (rel->subplan == NULL) + flatten_unplanned_rtes(glob, rte); + else if (recursing || is_dummy_plan(rel->subplan)) + { + Assert(rel->subroot != NULL); + add_rtes_to_flat_rtable(rel->subroot, true); + } + } + } + rti++; + } +} + +/* + * Extract RangeTblEntries from a subquery that was never planned at all + */ +static void +flatten_unplanned_rtes(PlannerGlobal *glob, RangeTblEntry *rte) +{ + /* Use query_tree_walker to find all RTEs in the parse tree */ + (void) query_tree_walker(rte->subquery, + flatten_rtes_walker, + (void *) glob, + QTW_EXAMINE_RTES); +} + +static bool +flatten_rtes_walker(Node *node, PlannerGlobal *glob) +{ + if (node == NULL) + return false; + if (IsA(node, RangeTblEntry)) + { + RangeTblEntry *rte = (RangeTblEntry *) node; + + /* As above, we need only save relation RTEs */ + if (rte->rtekind == RTE_RELATION) + add_rte_to_flat_rtable(glob, rte); + return false; + } + if (IsA(node, Query)) + { + /* Recurse into subselects */ + return query_tree_walker((Query *) node, + flatten_rtes_walker, + (void *) glob, + QTW_EXAMINE_RTES); + } + return expression_tree_walker(node, flatten_rtes_walker, + (void *) glob); +} + +/* + * Add (a copy of) the given RTE to the final rangetable + * + * In the flat rangetable, we zero out substructure pointers that are not + * needed by the executor; this reduces the storage space and copying cost + * for cached plans. We keep only the alias and eref Alias fields, which + * are needed by EXPLAIN, and the selectedCols and modifiedCols bitmaps, + * which are needed for executor-startup permissions checking and for + * trigger event checking. + */ +static void +add_rte_to_flat_rtable(PlannerGlobal *glob, RangeTblEntry *rte) +{ + RangeTblEntry *newrte; + + /* flat copy to duplicate all the scalar fields */ + newrte = (RangeTblEntry *) palloc(sizeof(RangeTblEntry)); + memcpy(newrte, rte, sizeof(RangeTblEntry)); + + /* zap unneeded sub-structure */ + newrte->subquery = NULL; + newrte->joinaliasvars = NIL; + newrte->functions = NIL; + newrte->values_lists = NIL; + newrte->values_collations = NIL; + newrte->ctecoltypes = NIL; + newrte->ctecoltypmods = NIL; + newrte->ctecolcollations = NIL; + + glob->finalrtable = lappend(glob->finalrtable, newrte); + + /* + * Check for RT index overflow; it's very unlikely, but if it did happen, + * the executor would get confused by varnos that match the special varno + * values. + */ + if (IS_SPECIAL_VARNO(list_length(glob->finalrtable))) + ereport(ERROR, + (errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED), + errmsg("too many range table entries"))); + + /* + * If it's a plain relation RTE, add the table to relationOids. + * + * We do this even though the RTE might be unreferenced in the plan tree; + * this would correspond to cases such as views that were expanded, child + * tables that were eliminated by constraint exclusion, etc. Schema + * invalidation on such a rel must still force rebuilding of the plan. + * + * Note we don't bother to avoid making duplicate list entries. We could, + * but it would probably cost more cycles than it would save. + */ + if (newrte->rtekind == RTE_RELATION) + glob->relationOids = lappend_oid(glob->relationOids, newrte->relid); +} + +/* * set_plan_refs: recurse through the Plan nodes of a single subquery level */ static Plan * @@ -431,8 +568,8 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) fix_scan_list(root, splan->scan.plan.targetlist, rtoffset); splan->scan.plan.qual = fix_scan_list(root, splan->scan.plan.qual, rtoffset); - splan->funcexpr = - fix_scan_expr(root, splan->funcexpr, rtoffset); + splan->functions = + fix_scan_list(root, splan->functions, rtoffset); } break; case T_ValuesScan: @@ -526,7 +663,7 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) /* * These plan types don't actually bother to evaluate their * targetlists, because they just return their unmodified input - * tuples. Even though the targetlist won't be used by the + * tuples. Even though the targetlist won't be used by the * executor, we fix it up for possible use by EXPLAIN (not to * mention ease of debugging --- wrong varnos are very confusing). */ @@ -544,7 +681,7 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) /* * Like the plan types above, LockRows doesn't evaluate its - * tlist or quals. But we have to fix up the RT indexes in + * tlist or quals. But we have to fix up the RT indexes in * its rowmarks. */ set_dummy_tlist_references(plan, rtoffset); @@ -662,7 +799,7 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) * Set up the visible plan targetlist as being the same as * the first RETURNING list. This is for the use of * EXPLAIN; the executor won't pay any attention to the - * targetlist. We postpone this step until here so that + * targetlist. We postpone this step until here so that * we don't have to do set_returning_clause_references() * twice on identical targetlists. */ @@ -905,7 +1042,7 @@ set_subqueryscan_references(PlannerInfo *root, else { /* - * Keep the SubqueryScan node. We have to do the processing that + * Keep the SubqueryScan node. We have to do the processing that * set_plan_references would otherwise have done on it. Notice we do * not do set_upper_references() here, because a SubqueryScan will * always have been created with correct references to its subplan's @@ -1418,7 +1555,7 @@ set_dummy_tlist_references(Plan *plan, int rtoffset) * * In most cases, subplan tlists will be "flat" tlists with only Vars, * so we try to optimize that case by extracting information about Vars - * in advance. Matching a parent tlist to a child is still an O(N^2) + * in advance. Matching a parent tlist to a child is still an O(N^2) * operation, but at least with a much smaller constant factor than plain * tlist_member() searches. * @@ -1914,7 +2051,7 @@ fix_upper_expr_mutator(Node *node, fix_upper_expr_context *context) * adjust any Vars that refer to other tables to reference junk tlist * entries in the top subplan's targetlist. Vars referencing the result * table should be left alone, however (the executor will evaluate them - * using the actual heap tuple, after firing triggers if any). In the + * using the actual heap tuple, after firing triggers if any). In the * adjusted RETURNING list, result-table Vars will have their original * varno (plus rtoffset), but Vars for other rels will have varno OUTER_VAR. * @@ -2118,7 +2255,7 @@ extract_query_dependencies_walker(Node *node, PlannerInfo *context) Query *query = (Query *) node; ListCell *lc; - while (query->commandType == CMD_UTILITY) + if (query->commandType == CMD_UTILITY) { /* * Ignore utility statements, except those (such as EXPLAIN) that diff --git a/src/backend/optimizer/plan/subselect.c b/src/backend/optimizer/plan/subselect.c index ea5363ab07..de70719e72 100644 --- a/src/backend/optimizer/plan/subselect.c +++ b/src/backend/optimizer/plan/subselect.c @@ -8,7 +8,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * Portions Copyright (c) 2012-2014, TransLattice, Inc. - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * IDENTIFICATION @@ -18,6 +18,7 @@ */ #include "postgres.h" +#include "access/htup_details.h" #include "catalog/pg_operator.h" #include "catalog/pg_type.h" #include "executor/executor.h" @@ -60,6 +61,7 @@ typedef struct finalize_primnode_context static Node *build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot, + List *plan_params, SubLinkType subLinkType, Node *testexpr, bool adjust_testexpr, bool unknownEqFalse); static List *generate_subquery_params(PlannerInfo *root, List *tlist, @@ -88,35 +90,42 @@ static bool finalize_primnode(Node *node, finalize_primnode_context *context); /* - * Select a PARAM_EXEC number to identify the given Var. - * If the Var already has a param slot, return that one. + * Select a PARAM_EXEC number to identify the given Var as a parameter for + * the current subquery, or for a nestloop's inner scan. + * If the Var already has a param in the current context, return that one. */ static int assign_param_for_var(PlannerInfo *root, Var *var) { ListCell *ppl; PlannerParamItem *pitem; - Index abslevel; - int i; + Index levelsup; - abslevel = root->query_level - var->varlevelsup; + /* Find the query level the Var belongs to */ + for (levelsup = var->varlevelsup; levelsup > 0; levelsup--) + root = root->parent_root; - /* If there's already a paramlist entry for this same Var, just use it */ - i = 0; - foreach(ppl, root->glob->paramlist) + /* If there's already a matching PlannerParamItem there, just use it */ + foreach(ppl, root->plan_params) { pitem = (PlannerParamItem *) lfirst(ppl); - if (pitem->abslevel == abslevel && IsA(pitem->item, Var)) + if (IsA(pitem->item, Var)) { Var *pvar = (Var *) pitem->item; + /* + * This comparison must match _equalVar(), except for ignoring + * varlevelsup. Note that _equalVar() ignores the location. + */ if (pvar->varno == var->varno && pvar->varattno == var->varattno && pvar->vartype == var->vartype && - pvar->vartypmod == var->vartypmod) - return i; + pvar->vartypmod == var->vartypmod && + pvar->varcollid == var->varcollid && + pvar->varnoold == var->varnoold && + pvar->varoattno == var->varoattno) + return pitem->paramId; } - i++; } /* Nope, so make a new one */ @@ -125,12 +134,11 @@ assign_param_for_var(PlannerInfo *root, Var *var) pitem = makeNode(PlannerParamItem); pitem->item = (Node *) var; - pitem->abslevel = abslevel; + pitem->paramId = root->glob->nParamExec++; - root->glob->paramlist = lappend(root->glob->paramlist, pitem); + root->plan_params = lappend(root->plan_params, pitem); - /* i is already the correct list index for the new item */ - return i; + return pitem->paramId; } /* @@ -145,16 +153,7 @@ replace_outer_var(PlannerInfo *root, Var *var) Assert(var->varlevelsup > 0 && var->varlevelsup < root->query_level); - /* - * Find the Var in root->glob->paramlist, or add it if not present. - * - * NOTE: in sufficiently complex querytrees, it is possible for the same - * varno/abslevel to refer to different RTEs in different parts of the - * parsetree, so that different fields might end up sharing the same Param - * number. As long as we check the vartype/typmod as well, I believe that - * this sort of aliasing will cause no trouble. The correct field should - * get stored into the Param slot at execution in each part of the tree. - */ + /* Find the Var in the appropriate plan_params, or add it if not present */ i = assign_param_for_var(root, var); retval = makeNode(Param); @@ -163,7 +162,7 @@ replace_outer_var(PlannerInfo *root, Var *var) retval->paramtype = var->vartype; retval->paramtypmod = var->vartypmod; retval->paramcollid = var->varcollid; - retval->location = -1; + retval->location = var->location; return retval; } @@ -172,8 +171,7 @@ replace_outer_var(PlannerInfo *root, Var *var) * Generate a Param node to replace the given Var, which will be supplied * from an upper NestLoop join node. * - * Because we allow nestloop and subquery Params to alias each other, - * this is effectively the same as replace_outer_var, except that we expect + * This is effectively the same as replace_outer_var, except that we expect * the Var to be local to the current query level. */ Param * @@ -192,14 +190,15 @@ assign_nestloop_param_var(PlannerInfo *root, Var *var) retval->paramtype = var->vartype; retval->paramtypmod = var->vartypmod; retval->paramcollid = var->varcollid; - retval->location = -1; + retval->location = var->location; return retval; } /* - * Select a PARAM_EXEC number to identify the given PlaceHolderVar. - * If the PlaceHolderVar already has a param slot, return that one. + * Select a PARAM_EXEC number to identify the given PlaceHolderVar as a + * parameter for the current subquery, or for a nestloop's inner scan. + * If the PHV already has a param in the current context, return that one. * * This is just like assign_param_for_var, except for PlaceHolderVars. */ @@ -208,25 +207,24 @@ assign_param_for_placeholdervar(PlannerInfo *root, PlaceHolderVar *phv) { ListCell *ppl; PlannerParamItem *pitem; - Index abslevel; - int i; + Index levelsup; - abslevel = root->query_level - phv->phlevelsup; + /* Find the query level the PHV belongs to */ + for (levelsup = phv->phlevelsup; levelsup > 0; levelsup--) + root = root->parent_root; - /* If there's already a paramlist entry for this same PHV, just use it */ - i = 0; - foreach(ppl, root->glob->paramlist) + /* If there's already a matching PlannerParamItem there, just use it */ + foreach(ppl, root->plan_params) { pitem = (PlannerParamItem *) lfirst(ppl); - if (pitem->abslevel == abslevel && IsA(pitem->item, PlaceHolderVar)) + if (IsA(pitem->item, PlaceHolderVar)) { PlaceHolderVar *pphv = (PlaceHolderVar *) pitem->item; /* We assume comparing the PHIDs is sufficient */ if (pphv->phid == phv->phid) - return i; + return pitem->paramId; } - i++; } /* Nope, so make a new one */ @@ -239,12 +237,11 @@ assign_param_for_placeholdervar(PlannerInfo *root, PlaceHolderVar *phv) pitem = makeNode(PlannerParamItem); pitem->item = (Node *) phv; - pitem->abslevel = abslevel; + pitem->paramId = root->glob->nParamExec++; - root->glob->paramlist = lappend(root->glob->paramlist, pitem); + root->plan_params = lappend(root->plan_params, pitem); - /* i is already the correct list index for the new item */ - return i; + return pitem->paramId; } /* @@ -261,10 +258,7 @@ replace_outer_placeholdervar(PlannerInfo *root, PlaceHolderVar *phv) Assert(phv->phlevelsup > 0 && phv->phlevelsup < root->query_level); - /* - * Find the PlaceHolderVar in root->glob->paramlist, or add it if not - * present. - */ + /* Find the PHV in the appropriate plan_params, or add it if not present */ i = assign_param_for_placeholdervar(root, phv); retval = makeNode(Param); @@ -314,11 +308,13 @@ replace_outer_agg(PlannerInfo *root, Aggref *agg) { Param *retval; PlannerParamItem *pitem; - Index abslevel; - int i; + Index levelsup; Assert(agg->agglevelsup > 0 && agg->agglevelsup < root->query_level); - abslevel = root->query_level - agg->agglevelsup; + + /* Find the query level the Aggref belongs to */ + for (levelsup = agg->agglevelsup; levelsup > 0; levelsup--) + root = root->parent_root; /* * It does not seem worthwhile to try to match duplicate outer aggs. Just @@ -330,18 +326,17 @@ replace_outer_agg(PlannerInfo *root, Aggref *agg) pitem = makeNode(PlannerParamItem); pitem->item = (Node *) agg; - pitem->abslevel = abslevel; + pitem->paramId = root->glob->nParamExec++; - root->glob->paramlist = lappend(root->glob->paramlist, pitem); - i = list_length(root->glob->paramlist) - 1; + root->plan_params = lappend(root->plan_params, pitem); retval = makeNode(Param); retval->paramkind = PARAM_EXEC; - retval->paramid = i; + retval->paramid = pitem->paramId; retval->paramtype = agg->aggtype; retval->paramtypmod = -1; retval->paramcollid = agg->aggcollid; - retval->location = -1; + retval->location = agg->location; return retval; } @@ -349,29 +344,24 @@ replace_outer_agg(PlannerInfo *root, Aggref *agg) /* * Generate a new Param node that will not conflict with any other. * - * This is used to allocate PARAM_EXEC slots for subplan outputs. + * This is used to create Params representing subplan outputs. + * We don't need to build a PlannerParamItem for such a Param, but we do + * need to record the PARAM_EXEC slot number as being allocated. */ static Param * generate_new_param(PlannerInfo *root, Oid paramtype, int32 paramtypmod, Oid paramcollation) { Param *retval; - PlannerParamItem *pitem; retval = makeNode(Param); retval->paramkind = PARAM_EXEC; - retval->paramid = list_length(root->glob->paramlist); + retval->paramid = root->glob->nParamExec++; retval->paramtype = paramtype; retval->paramtypmod = paramtypmod; retval->paramcollid = paramcollation; retval->location = -1; - pitem = makeNode(PlannerParamItem); - pitem->item = (Node *) retval; - pitem->abslevel = root->query_level; - - root->glob->paramlist = lappend(root->glob->paramlist, pitem); - return retval; } @@ -380,17 +370,13 @@ generate_new_param(PlannerInfo *root, Oid paramtype, int32 paramtypmod, * is not actually used to carry a value at runtime). Such parameters are * used for special runtime signaling purposes, such as connecting a * recursive union node to its worktable scan node or forcing plan - * re-evaluation within the EvalPlanQual mechanism. + * re-evaluation within the EvalPlanQual mechanism. No actual Param node + * exists with this ID, however. */ int SS_assign_special_param(PlannerInfo *root) { - Param *param; - - /* We generate a Param of datatype INTERNAL */ - param = generate_new_param(root, INTERNALOID, -1, InvalidOid); - /* ... but the caller only cares about its ID */ - return param->paramid; + return root->glob->nParamExec++; } /* @@ -451,10 +437,11 @@ make_subplan(PlannerInfo *root, Query *orig_subquery, SubLinkType subLinkType, double tuple_fraction; Plan *plan; PlannerInfo *subroot; + List *plan_params; Node *result; /* - * Copy the source Query node. This is a quick and dirty kluge to resolve + * Copy the source Query node. This is a quick and dirty kluge to resolve * the fact that the parser can generate trees with multiple links to the * same sub-Query node, but the planner wants to scribble on the Query. * Try to clean this up when we do querytree redesign... @@ -479,7 +466,7 @@ make_subplan(PlannerInfo *root, Query *orig_subquery, SubLinkType subLinkType, * path/costsize.c. * * XXX If an ANY subplan is uncorrelated, build_subplan may decide to hash - * its output. In that case it would've been better to specify full + * its output. In that case it would've been better to specify full * retrieval. At present, however, we can only check hashability after * we've made the subplan :-(. (Determining whether it'll fit in work_mem * is the really hard part.) Therefore, we don't want to be too @@ -494,6 +481,9 @@ make_subplan(PlannerInfo *root, Query *orig_subquery, SubLinkType subLinkType, else tuple_fraction = 0.0; /* default behavior */ + /* plan_params should not be in use in current query level */ + Assert(root->plan_params == NIL); + /* * Generate the plan for the subquery. */ @@ -518,8 +508,12 @@ make_subplan(PlannerInfo *root, Query *orig_subquery, SubLinkType subLinkType, } #endif + /* Isolate the params needed by this specific subplan */ + plan_params = root->plan_params; + root->plan_params = NIL; + /* And convert to SubPlan or InitPlan format. */ - result = build_subplan(root, plan, subroot, + result = build_subplan(root, plan, subroot, plan_params, subLinkType, testexpr, true, isTopQual); #ifdef PGXC /* This is not necessary for a PGXC Coordinator, we just need one plan */ @@ -530,7 +524,7 @@ make_subplan(PlannerInfo *root, Query *orig_subquery, SubLinkType subLinkType, /* * If it's a correlated EXISTS with an unimportant targetlist, we might be * able to transform it to the equivalent of an IN and then implement it - * by hashing. We don't have enough information yet to tell which way is + * by hashing. We don't have enough information yet to tell which way is * likely to be better (it depends on the expected number of executions of * the EXISTS qual, and we are much too early in planning the outer query * to be able to guess that). So we generate both plans, if possible, and @@ -557,6 +551,10 @@ make_subplan(PlannerInfo *root, Query *orig_subquery, SubLinkType subLinkType, false, 0.0, &subroot); + /* Isolate the params needed by this specific subplan */ + plan_params = root->plan_params; + root->plan_params = NIL; + /* Now we can check if it'll fit in work_mem */ if (subplan_is_hashable(plan)) { @@ -565,6 +563,7 @@ make_subplan(PlannerInfo *root, Query *orig_subquery, SubLinkType subLinkType, /* OK, convert to SubPlan format. */ hashplan = (SubPlan *) build_subplan(root, plan, subroot, + plan_params, ANY_SUBLINK, newtestexpr, false, true); /* Check we got what we expected */ @@ -593,14 +592,14 @@ make_subplan(PlannerInfo *root, Query *orig_subquery, SubLinkType subLinkType, */ static Node * build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot, + List *plan_params, SubLinkType subLinkType, Node *testexpr, bool adjust_testexpr, bool unknownEqFalse) { Node *result; SubPlan *splan; bool isInitPlan; - Bitmapset *tmpset; - int paramid; + ListCell *lc; /* * Initialize the SubPlan node. Note plan_id, plan_name, and cost fields @@ -622,36 +621,26 @@ build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot, * Make parParam and args lists of param IDs and expressions that current * query level will pass to this child plan. */ - tmpset = bms_copy(plan->extParam); - while ((paramid = bms_first_member(tmpset)) >= 0) + foreach(lc, plan_params) { - PlannerParamItem *pitem = list_nth(root->glob->paramlist, paramid); - - if (pitem->abslevel == root->query_level) - { - Node *arg; + PlannerParamItem *pitem = (PlannerParamItem *) lfirst(lc); + Node *arg = pitem->item; - /* - * The Var, PlaceHolderVar, or Aggref has already been adjusted to - * have the correct varlevelsup, phlevelsup, or agglevelsup. We - * probably don't even need to copy it again, but be safe. - */ - arg = copyObject(pitem->item); - - /* - * If it's a PlaceHolderVar or Aggref, its arguments might contain - * SubLinks, which have not yet been processed (see the comments - * for SS_replace_correlation_vars). Do that now. - */ - if (IsA(arg, PlaceHolderVar) || - IsA(arg, Aggref)) - arg = SS_process_sublinks(root, arg, false); + /* + * The Var, PlaceHolderVar, or Aggref has already been adjusted to + * have the correct varlevelsup, phlevelsup, or agglevelsup. + * + * If it's a PlaceHolderVar or Aggref, its arguments might contain + * SubLinks, which have not yet been processed (see the comments for + * SS_replace_correlation_vars). Do that now. + */ + if (IsA(arg, PlaceHolderVar) || + IsA(arg, Aggref)) + arg = SS_process_sublinks(root, arg, false); - splan->parParam = lappend_int(splan->parParam, paramid); - splan->args = lappend(splan->args, arg); - } + splan->parParam = lappend_int(splan->parParam, pitem->paramId); + splan->args = lappend(splan->args, arg); } - bms_free(tmpset); /* * Un-correlated or undirect correlated plans of EXISTS, EXPR, ARRAY, or @@ -763,7 +752,7 @@ build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot, * Otherwise, we have the option to tack a Material node onto the top * of the subplan, to reduce the cost of reading it repeatedly. This * is pointless for a direct-correlated subplan, since we'd have to - * recompute its results each time anyway. For uncorrelated/undirect + * recompute its results each time anyway. For uncorrelated/undirect * correlated subplans, we add Material unless the subplan's top plan * node would materialize its output anyway. Also, if enable_material * is false, then the user does not want us to materialize anything @@ -789,10 +778,10 @@ build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot, /* * A parameterless subplan (not initplan) should be prepared to handle - * REWIND efficiently. If it has direct parameters then there's no point + * REWIND efficiently. If it has direct parameters then there's no point * since it'll be reset on each scan anyway; and if it's an initplan then * there's no point since it won't get re-run without parameter changes - * anyway. The input of a hashed subplan doesn't need REWIND either. + * anyway. The input of a hashed subplan doesn't need REWIND either. */ if (splan->parParam == NIL && !isInitPlan && !splan->useHashTable) root->glob->rewindPlanIDs = bms_add_member(root->glob->rewindPlanIDs, @@ -817,10 +806,7 @@ build_subplan(PlannerInfo *root, Plan *plan, PlannerInfo *subroot, sprintf(splan->plan_name + offset, ")"); } else - { - splan->plan_name = palloc(32); - sprintf(splan->plan_name, "SubPlan %d", splan->plan_id); - } + splan->plan_name = psprintf("SubPlan %d", splan->plan_id); /* Lastly, fill in the cost estimates for use later */ cost_subplan(root, splan, plan); @@ -892,14 +878,9 @@ generate_subquery_vars(PlannerInfo *root, List *tlist, Index varno) /* * convert_testexpr: convert the testexpr given by the parser into * actually executable form. This entails replacing PARAM_SUBLINK Params - * with Params or Vars representing the results of the sub-select. The + * with Params or Vars representing the results of the sub-select. The * nodes to be substituted are passed in as the List result from * generate_subquery_params or generate_subquery_vars. - * - * The given testexpr has already been recursively processed by - * process_sublinks_mutator. Hence it can no longer contain any - * PARAM_SUBLINK Params for lower SubLink nodes; we can safely assume that - * any we find are for our own level of SubLink. */ static Node * convert_testexpr(PlannerInfo *root, @@ -938,6 +919,28 @@ convert_testexpr_mutator(Node *node, param->paramid - 1)); } } + if (IsA(node, SubLink)) + { + /* + * If we come across a nested SubLink, it is neither necessary nor + * correct to recurse into it: any PARAM_SUBLINKs we might find inside + * belong to the inner SubLink not the outer. So just return it as-is. + * + * This reasoning depends on the assumption that nothing will pull + * subexpressions into or out of the testexpr field of a SubLink, at + * least not without replacing PARAM_SUBLINKs first. If we did want + * to do that we'd need to rethink the parser-output representation + * altogether, since currently PARAM_SUBLINKs are only unique per + * SubLink not globally across the query. The whole point of + * replacing them with Vars or PARAM_EXEC nodes is to make them + * globally unique before they escape from the SubLink's testexpr. + * + * Note: this can't happen when called during SS_process_sublinks, + * because that recursively processes inner SubLinks first. It can + * happen when called from convert_ANY_sublink_to_join, though. + */ + return node; + } return expression_tree_mutator(node, convert_testexpr_mutator, (void *) context); @@ -977,7 +980,7 @@ testexpr_is_hashable(Node *testexpr) * * The combining operators must be hashable and strict. The need for * hashability is obvious, since we want to use hashing. Without - * strictness, behavior in the presence of nulls is too unpredictable. We + * strictness, behavior in the presence of nulls is too unpredictable. We * actually must assume even more than plain strictness: they can't yield * NULL for non-null inputs, either (see nodeSubplan.c). However, hash * indexes and hash joins assume that too. @@ -1072,9 +1075,7 @@ SS_process_ctes(PlannerInfo *root) Plan *plan; PlannerInfo *subroot; SubPlan *splan; - Bitmapset *tmpset; int paramid; - Param *prm; /* * Ignore SELECT CTEs that are not actually referenced anywhere. @@ -1087,11 +1088,14 @@ SS_process_ctes(PlannerInfo *root) } /* - * Copy the source Query node. Probably not necessary, but let's keep + * Copy the source Query node. Probably not necessary, but let's keep * this similar to make_subplan. */ subquery = (Query *) copyObject(cte->ctequery); + /* plan_params should not be in use in current query level */ + Assert(root->plan_params == NIL); + /* * Generate the plan for the CTE query. Always plan for full * retrieval --- we don't have enough info to predict otherwise. @@ -1118,7 +1122,15 @@ SS_process_ctes(PlannerInfo *root) #endif /* - * Make a SubPlan node for it. This is just enough unlike + * Since the current query level doesn't yet contain any RTEs, it + * should not be possible for the CTE to have requested parameters of + * this level. + */ + if (root->plan_params) + elog(ERROR, "unexpected outer reference in CTE query"); + + /* + * Make a SubPlan node for it. This is just enough unlike * build_subplan that we can't share code. * * Note plan_id, plan_name, and cost fields are set further down. @@ -1136,35 +1148,22 @@ SS_process_ctes(PlannerInfo *root) splan->args = NIL; /* - * Make parParam and args lists of param IDs and expressions that - * current query level will pass to this child plan. Even though this - * is an initplan, there could be side-references to earlier - * initplan's outputs, specifically their CTE output parameters. + * The node can't have any inputs (since it's an initplan), so the + * parParam and args lists remain empty. (It could contain references + * to earlier CTEs' output param IDs, but CTE outputs are not + * propagated via the args list.) */ - tmpset = bms_copy(plan->extParam); - while ((paramid = bms_first_member(tmpset)) >= 0) - { - PlannerParamItem *pitem = list_nth(root->glob->paramlist, paramid); - - if (pitem->abslevel == root->query_level) - { - prm = (Param *) pitem->item; - if (!IsA(prm, Param) || - prm->paramtype != INTERNALOID) - elog(ERROR, "bogus local parameter passed to WITH query"); - - splan->parParam = lappend_int(splan->parParam, paramid); - splan->args = lappend(splan->args, copyObject(prm)); - } - } - bms_free(tmpset); /* - * Assign a param to represent the query output. We only really care - * about reserving a parameter ID number. + * Assign a param ID to represent the CTE's output. No ordinary + * "evaluation" of this param slot ever happens, but we use the param + * ID for setParam/chgParam signaling just as if the CTE plan were + * returning a simple scalar output. (Also, the executor abuses the + * ParamExecData slot for this param ID for communication among + * multiple CteScan nodes that might be scanning this CTE.) */ - prm = generate_new_param(root, INTERNALOID, -1, InvalidOid); - splan->setParam = list_make1_int(prm->paramid); + paramid = SS_assign_special_param(root); + splan->setParam = list_make1_int(paramid); /* * Add the subplan and its PlannerInfo to the global lists. @@ -1178,8 +1177,7 @@ SS_process_ctes(PlannerInfo *root) root->cte_plan_ids = lappend_int(root->cte_plan_ids, splan->plan_id); /* Label the subplan for EXPLAIN purposes */ - splan->plan_name = palloc(4 + strlen(cte->ctename) + 1); - sprintf(splan->plan_name, "CTE %s", cte->ctename); + splan->plan_name = psprintf("CTE %s", cte->ctename); /* Lastly, fill in the cost estimates for use later */ cost_subplan(root, splan, plan); @@ -1275,6 +1273,7 @@ convert_ANY_sublink_to_join(PlannerInfo *root, SubLink *sublink, rte = addRangeTableEntryForSubquery(NULL, subselect, makeAlias("ANY_subquery", NIL), + false, false); parse->rtable = lappend(parse->rtable, rte); rtindex = list_length(parse->rtable); @@ -1354,7 +1353,7 @@ convert_EXISTS_sublink_to_join(PlannerInfo *root, SubLink *sublink, /* * See if the subquery can be simplified based on the knowledge that it's - * being used in EXISTS(). If we aren't able to get rid of its + * being used in EXISTS(). If we aren't able to get rid of its * targetlist, we have to fail, because the pullup operation leaves us * with noplace to evaluate the targetlist. */ @@ -1403,9 +1402,9 @@ convert_EXISTS_sublink_to_join(PlannerInfo *root, SubLink *sublink, * what pull_up_subqueries has to go through. * * In fact, it's even easier than what convert_ANY_sublink_to_join has to - * do. The machinations of simplify_EXISTS_query ensured that there is + * do. The machinations of simplify_EXISTS_query ensured that there is * nothing interesting in the subquery except an rtable and jointree, and - * even the jointree FromExpr no longer has quals. So we can just append + * even the jointree FromExpr no longer has quals. So we can just append * the rtable to our own and use the FromExpr in our jointree. But first, * adjust all level-zero varnos in the subquery to account for the rtable * merger. @@ -1536,7 +1535,7 @@ simplify_EXISTS_query(Query *query) * * On success, the modified subselect is returned, and we store a suitable * upper-level test expression at *testexpr, plus a list of the subselect's - * output Params at *paramIds. (The test expression is already Param-ified + * output Params at *paramIds. (The test expression is already Param-ified * and hence need not go through convert_testexpr, which is why we have to * deal with the Param IDs specially.) * @@ -1699,7 +1698,7 @@ convert_EXISTS_to_ANY(PlannerInfo *root, Query *subselect, return NULL; /* - * Also reject sublinks in the stuff we intend to pull up. (It might be + * Also reject sublinks in the stuff we intend to pull up. (It might be * possible to support this, but doesn't seem worth the complication.) */ if (contain_subplans((Node *) leftargs)) @@ -1901,7 +1900,7 @@ process_sublinks_mutator(Node *node, process_sublinks_context *context) * is needed for a bare List.) * * Anywhere within the top-level AND/OR clause structure, we can tell - * make_subplan() that NULL and FALSE are interchangeable. So isTopQual + * make_subplan() that NULL and FALSE are interchangeable. So isTopQual * propagates down in both cases. (Note that this is unlike the meaning * of "top level qual" used in most other places in Postgres.) */ @@ -1974,7 +1973,7 @@ SS_finalize_plan(PlannerInfo *root, Plan *plan, bool attach_initplans) *initExtParam, *initSetParam; Cost initplan_cost; - int paramid; + PlannerInfo *proot; ListCell *l; /* @@ -2006,31 +2005,36 @@ SS_finalize_plan(PlannerInfo *root, Plan *plan, bool attach_initplans) /* * Now determine the set of params that are validly referenceable in this * query level; to wit, those available from outer query levels plus the - * output parameters of any initPlans. (We do not include output - * parameters of regular subplans. Those should only appear within the + * output parameters of any local initPlans. (We do not include output + * parameters of regular subplans. Those should only appear within the * testexpr of SubPlan nodes, and are taken care of locally within * finalize_primnode. Likewise, special parameters that are generated by * nodes such as ModifyTable are handled within finalize_plan.) - * - * Note: this is a bit overly generous since some parameters of upper - * query levels might belong to query subtrees that don't include this - * query, or might be nestloop params that won't be passed down at all. - * However, valid_params is only a debugging crosscheck, so it doesn't - * seem worth expending lots of cycles to try to be exact. */ valid_params = bms_copy(initSetParam); - paramid = 0; - foreach(l, root->glob->paramlist) + for (proot = root->parent_root; proot != NULL; proot = proot->parent_root) { - PlannerParamItem *pitem = (PlannerParamItem *) lfirst(l); - - if (pitem->abslevel < root->query_level) + /* Include ordinary Var/PHV/Aggref params */ + foreach(l, proot->plan_params) { - /* valid outer-level parameter */ - valid_params = bms_add_member(valid_params, paramid); + PlannerParamItem *pitem = (PlannerParamItem *) lfirst(l); + + valid_params = bms_add_member(valid_params, pitem->paramId); } + /* Include any outputs of outer-level initPlans */ + foreach(l, proot->init_plans) + { + SubPlan *initsubplan = (SubPlan *) lfirst(l); + ListCell *l2; - paramid++; + foreach(l2, initsubplan->setParam) + { + valid_params = bms_add_member(valid_params, lfirst_int(l2)); + } + } + /* Include worktable ID, if a recursive query is being planned */ + if (proot->wt_param_id >= 0) + valid_params = bms_add_member(valid_params, proot->wt_param_id); } /* @@ -2178,7 +2182,7 @@ finalize_plan(PlannerInfo *root, Plan *plan, Bitmapset *valid_params, /* * In a SubqueryScan, SS_finalize_plan has already been run on the * subplan by the inner invocation of subquery_planner, so there's - * no need to do it again. Instead, just pull out the subplan's + * no need to do it again. Instead, just pull out the subplan's * extParams list, which represents the params it needs from my * level and higher levels. */ @@ -2189,9 +2193,37 @@ finalize_plan(PlannerInfo *root, Plan *plan, Bitmapset *valid_params, break; case T_FunctionScan: - finalize_primnode(((FunctionScan *) plan)->funcexpr, - &context); - context.paramids = bms_add_members(context.paramids, scan_params); + { + FunctionScan *fscan = (FunctionScan *) plan; + ListCell *lc; + + /* + * Call finalize_primnode independently on each function + * expression, so that we can record which params are + * referenced in each, in order to decide which need + * re-evaluating during rescan. + */ + foreach(lc, fscan->functions) + { + RangeTblFunction *rtfunc = (RangeTblFunction *) lfirst(lc); + finalize_primnode_context funccontext; + + funccontext = context; + funccontext.paramids = NULL; + + finalize_primnode(rtfunc->funcexpr, &funccontext); + + /* remember results for execution */ + rtfunc->funcparams = funccontext.paramids; + + /* add the function's params to the overall set */ + context.paramids = bms_add_members(context.paramids, + funccontext.paramids); + } + + context.paramids = bms_add_members(context.paramids, + scan_params); + } break; case T_ValuesScan: @@ -2523,7 +2555,7 @@ finalize_primnode(Node *node, finalize_primnode_context *context) /* * Remove any param IDs of output parameters of the subplan that were - * referenced in the testexpr. These are not interesting for + * referenced in the testexpr. These are not interesting for * parameter change signaling since we always re-evaluate the subplan. * Note that this wouldn't work too well if there might be uses of the * same param IDs elsewhere in the plan, but that can't happen because @@ -2620,9 +2652,8 @@ SS_make_initplan_from_plan(PlannerInfo *root, Plan *plan, node->setParam = list_make1_int(prm->paramid); /* Label the subplan for EXPLAIN purposes */ - node->plan_name = palloc(64); - sprintf(node->plan_name, "InitPlan %d (returns $%d)", - node->plan_id, prm->paramid); + node->plan_name = psprintf("InitPlan %d (returns $%d)", + node->plan_id, prm->paramid); return prm; } diff --git a/src/backend/optimizer/prep/Makefile b/src/backend/optimizer/prep/Makefile index 86301bfbd3..5195d9b0ba 100644 --- a/src/backend/optimizer/prep/Makefile +++ b/src/backend/optimizer/prep/Makefile @@ -12,6 +12,6 @@ subdir = src/backend/optimizer/prep top_builddir = ../../../.. include $(top_builddir)/src/Makefile.global -OBJS = prepjointree.o prepqual.o preptlist.o prepunion.o +OBJS = prepjointree.o prepqual.o prepsecurity.o preptlist.o prepunion.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/prep/prepjointree.c b/src/backend/optimizer/prep/prepjointree.c index c5262fba26..88c6d23c27 100644 --- a/src/backend/optimizer/prep/prepjointree.c +++ b/src/backend/optimizer/prep/prepjointree.c @@ -12,7 +12,7 @@ * reduce_outer_joins * * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -44,6 +44,8 @@ typedef struct pullup_replace_vars_context PlannerInfo *root; List *targetlist; /* tlist of subquery being pulled up */ RangeTblEntry *target_rte; /* RTE of subquery */ + Relids relids; /* relids within subquery, as numbered after + * pullup (set only if target_rte->lateral) */ bool *outer_hasSubLinks; /* -> outer query's hasSubLinks */ int varno; /* varno of subquery */ bool need_phvs; /* do we need PlaceHolderVars? */ @@ -63,9 +65,14 @@ static Node *pull_up_sublinks_jointree_recurse(PlannerInfo *root, Node *jtnode, static Node *pull_up_sublinks_qual_recurse(PlannerInfo *root, Node *node, Node **jtlink1, Relids available_rels1, Node **jtlink2, Relids available_rels2); +static Node *pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode, + JoinExpr *lowest_outer_join, + JoinExpr *lowest_nulling_outer_join, + AppendRelInfo *containing_appendrel); static Node *pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, JoinExpr *lowest_outer_join, + JoinExpr *lowest_nulling_outer_join, AppendRelInfo *containing_appendrel); static Node *pull_up_simple_union_all(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte); @@ -74,18 +81,23 @@ static void pull_up_union_leaf_queries(Node *setOp, PlannerInfo *root, int childRToffset); static void make_setop_translation_list(Query *query, Index newvarno, List **translated_vars); -static bool is_simple_subquery(Query *subquery); +static bool is_simple_subquery(Query *subquery, RangeTblEntry *rte, + JoinExpr *lowest_outer_join); static bool is_simple_union_all(Query *subquery); static bool is_simple_union_all_recurse(Node *setOp, Query *setOpQuery, List *colTypes); static bool is_safe_append_member(Query *subquery); +static bool jointree_contains_lateral_outer_refs(Node *jtnode, bool restricted, + Relids safe_upper_varnos); static void replace_vars_in_jointree(Node *jtnode, pullup_replace_vars_context *context, - JoinExpr *lowest_outer_join); + JoinExpr *lowest_nulling_outer_join); static Node *pullup_replace_vars(Node *expr, pullup_replace_vars_context *context); static Node *pullup_replace_vars_callback(Var *var, replace_rte_variables_context *context); +static Query *pullup_replace_vars_subquery(Query *query, + pullup_replace_vars_context *context); static reduce_outer_joins_state *reduce_outer_joins_pass1(Node *jtnode); static void reduce_outer_joins_pass2(Node *jtnode, reduce_outer_joins_state *state, @@ -107,7 +119,7 @@ static Node *find_jointree_node_for_rel(Node *jtnode, int relid); * * A clause "foo op ANY (sub-SELECT)" can be processed by pulling the * sub-SELECT up to become a rangetable entry and treating the implied - * comparisons as quals of a semijoin. However, this optimization *only* + * comparisons as quals of a semijoin. However, this optimization *only* * works at the top level of WHERE or a JOIN/ON clause, because we cannot * distinguish whether the ANY ought to return FALSE or NULL in cases * involving NULL inputs. Also, in an outer join's ON clause we can only @@ -124,7 +136,7 @@ static Node *find_jointree_node_for_rel(Node *jtnode, int relid); * transformations if any are found. * * This routine has to run before preprocess_expression(), so the quals - * clauses are not yet reduced to implicit-AND format. That means we need + * clauses are not yet reduced to implicit-AND format. That means we need * to recursively search through explicit AND clauses, which are * probably only binary ANDs. We stop as soon as we hit a non-AND item. */ @@ -278,7 +290,7 @@ pull_up_sublinks_jointree_recurse(PlannerInfo *root, Node *jtnode, /* * Although we could include the pulled-up subqueries in the returned * relids, there's no need since upper quals couldn't refer to their - * outputs anyway. But we *do* need to include the join's own rtindex + * outputs anyway. But we *do* need to include the join's own rtindex * because we haven't yet collapsed join alias variables, so upper * levels would mistakenly think they couldn't use references to this * join. @@ -571,10 +583,7 @@ inline_set_returning_functions(PlannerInfo *root) /* Successful expansion, replace the rtable entry */ rte->rtekind = RTE_SUBQUERY; rte->subquery = funcquery; - rte->funcexpr = NULL; - rte->funccoltypes = NIL; - rte->funccoltypmods = NIL; - rte->funccolcollations = NIL; + rte->functions = NIL; } } } @@ -588,10 +597,28 @@ inline_set_returning_functions(PlannerInfo *root) * Also, subqueries that are simple UNION ALL structures can be * converted into "append relations". * + * This recursively processes the jointree and returns a modified jointree. + */ +Node * +pull_up_subqueries(PlannerInfo *root, Node *jtnode) +{ + /* Start off with no containing join nor appendrel */ + return pull_up_subqueries_recurse(root, jtnode, NULL, NULL, NULL); +} + +/* + * pull_up_subqueries_recurse + * Recursive guts of pull_up_subqueries. + * + * If this jointree node is within either side of an outer join, then + * lowest_outer_join references the lowest such JoinExpr node; otherwise + * it is NULL. We use this to constrain the effects of LATERAL subqueries. + * * If this jointree node is within the nullable side of an outer join, then - * lowest_outer_join references the lowest such JoinExpr node; otherwise it - * is NULL. This forces use of the PlaceHolderVar mechanism for references - * to non-nullable targetlist items, but only for references above that join. + * lowest_nulling_outer_join references the lowest such JoinExpr node; + * otherwise it is NULL. This forces use of the PlaceHolderVar mechanism for + * references to non-nullable targetlist items, but only for references above + * that join. * * If we are looking at a member subquery of an append relation, * containing_appendrel describes that relation; else it is NULL. @@ -606,15 +633,17 @@ inline_set_returning_functions(PlannerInfo *root) * subquery RangeTblRef entries will be replaced. Also, we can't turn * pullup_replace_vars loose on the whole jointree, because it'll return a * mutated copy of the tree; we have to invoke it just on the quals, instead. - * This behavior is what makes it reasonable to pass lowest_outer_join as a - * pointer rather than some more-indirect way of identifying the lowest OJ. - * Likewise, we don't replace append_rel_list members but only their - * substructure, so the containing_appendrel reference is safe to use. + * This behavior is what makes it reasonable to pass lowest_outer_join and + * lowest_nulling_outer_join as pointers rather than some more-indirect way + * of identifying the lowest OJs. Likewise, we don't replace append_rel_list + * members but only their substructure, so the containing_appendrel reference + * is safe to use. */ -Node * -pull_up_subqueries(PlannerInfo *root, Node *jtnode, - JoinExpr *lowest_outer_join, - AppendRelInfo *containing_appendrel) +static Node * +pull_up_subqueries_recurse(PlannerInfo *root, Node *jtnode, + JoinExpr *lowest_outer_join, + JoinExpr *lowest_nulling_outer_join, + AppendRelInfo *containing_appendrel) { if (jtnode == NULL) return NULL; @@ -631,12 +660,12 @@ pull_up_subqueries(PlannerInfo *root, Node *jtnode, * unless is_safe_append_member says so. */ if (rte->rtekind == RTE_SUBQUERY && - is_simple_subquery(rte->subquery) && - !rte->security_barrier && + is_simple_subquery(rte->subquery, rte, lowest_outer_join) && (containing_appendrel == NULL || is_safe_append_member(rte->subquery))) return pull_up_simple_subquery(root, jtnode, rte, lowest_outer_join, + lowest_nulling_outer_join, containing_appendrel); /* @@ -661,8 +690,10 @@ pull_up_subqueries(PlannerInfo *root, Node *jtnode, Assert(containing_appendrel == NULL); foreach(l, f->fromlist) - lfirst(l) = pull_up_subqueries(root, lfirst(l), - lowest_outer_join, NULL); + lfirst(l) = pull_up_subqueries_recurse(root, lfirst(l), + lowest_outer_join, + lowest_nulling_outer_join, + NULL); } else if (IsA(jtnode, JoinExpr)) { @@ -673,30 +704,46 @@ pull_up_subqueries(PlannerInfo *root, Node *jtnode, switch (j->jointype) { case JOIN_INNER: - j->larg = pull_up_subqueries(root, j->larg, - lowest_outer_join, NULL); - j->rarg = pull_up_subqueries(root, j->rarg, - lowest_outer_join, NULL); + j->larg = pull_up_subqueries_recurse(root, j->larg, + lowest_outer_join, + lowest_nulling_outer_join, + NULL); + j->rarg = pull_up_subqueries_recurse(root, j->rarg, + lowest_outer_join, + lowest_nulling_outer_join, + NULL); break; case JOIN_LEFT: case JOIN_SEMI: case JOIN_ANTI: - j->larg = pull_up_subqueries(root, j->larg, - lowest_outer_join, NULL); - j->rarg = pull_up_subqueries(root, j->rarg, - j, NULL); + j->larg = pull_up_subqueries_recurse(root, j->larg, + j, + lowest_nulling_outer_join, + NULL); + j->rarg = pull_up_subqueries_recurse(root, j->rarg, + j, + j, + NULL); break; case JOIN_FULL: - j->larg = pull_up_subqueries(root, j->larg, - j, NULL); - j->rarg = pull_up_subqueries(root, j->rarg, - j, NULL); + j->larg = pull_up_subqueries_recurse(root, j->larg, + j, + j, + NULL); + j->rarg = pull_up_subqueries_recurse(root, j->rarg, + j, + j, + NULL); break; case JOIN_RIGHT: - j->larg = pull_up_subqueries(root, j->larg, - j, NULL); - j->rarg = pull_up_subqueries(root, j->rarg, - lowest_outer_join, NULL); + j->larg = pull_up_subqueries_recurse(root, j->larg, + j, + j, + NULL); + j->rarg = pull_up_subqueries_recurse(root, j->rarg, + j, + lowest_nulling_outer_join, + NULL); break; default: elog(ERROR, "unrecognized join type: %d", @@ -715,16 +762,17 @@ pull_up_subqueries(PlannerInfo *root, Node *jtnode, * Attempt to pull up a single simple subquery. * * jtnode is a RangeTblRef that has been tentatively identified as a simple - * subquery by pull_up_subqueries. We return the replacement jointree node, + * subquery by pull_up_subqueries. We return the replacement jointree node, * or jtnode itself if we determine that the subquery can't be pulled up after * all. * * rte is the RangeTblEntry referenced by jtnode. Remaining parameters are - * as for pull_up_subqueries. + * as for pull_up_subqueries_recurse. */ static Node * pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, JoinExpr *lowest_outer_join, + JoinExpr *lowest_nulling_outer_join, AppendRelInfo *containing_appendrel) { Query *parse = root->parse; @@ -747,7 +795,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, * Create a PlannerInfo data structure for this subquery. * * NOTE: the next few steps should match the first processing in - * subquery_planner(). Can we refactor to avoid code duplication, or + * subquery_planner(). Can we refactor to avoid code duplication, or * would that just make things uglier? */ subroot = makeNode(PlannerInfo); @@ -755,6 +803,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, subroot->glob = root->glob; subroot->query_level = root->query_level; subroot->parent_root = root->parent_root; + subroot->plan_params = NIL; subroot->planner_cxt = CurrentMemoryContext; subroot->init_plans = NIL; subroot->cte_plan_ids = NIL; @@ -792,18 +841,18 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, * handling an appendrel member. */ subquery->jointree = (FromExpr *) - pull_up_subqueries(subroot, (Node *) subquery->jointree, NULL, NULL); + pull_up_subqueries_recurse(subroot, (Node *) subquery->jointree, + NULL, NULL, NULL); /* * Now we must recheck whether the subquery is still simple enough to pull - * up. If not, abandon processing it. + * up. If not, abandon processing it. * * We don't really need to recheck all the conditions involved, but it's * easier just to keep this "if" looking the same as the one in - * pull_up_subqueries. + * pull_up_subqueries_recurse. */ - if (is_simple_subquery(subquery) && - !rte->security_barrier && + if (is_simple_subquery(subquery, rte, lowest_outer_join) && (containing_appendrel == NULL || is_safe_append_member(subquery))) { /* good to go */ @@ -814,7 +863,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, * Give up, return unmodified RangeTblRef. * * Note: The work we just did will be redone when the subquery gets - * planned on its own. Perhaps we could avoid that by storing the + * planned on its own. Perhaps we could avoid that by storing the * modified subquery back into the rangetable, but I'm not gonna risk * it now. */ @@ -822,6 +871,18 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, } /* + * We must flatten any join alias Vars in the subquery's targetlist, + * because pulling up the subquery's subqueries might have changed their + * expansions into arbitrary expressions, which could affect + * pullup_replace_vars' decisions about whether PlaceHolderVar wrappers + * are needed for tlist entries. (Likely it'd be better to do + * flatten_join_alias_vars on the whole query tree at some earlier stage, + * maybe even in the rewriter; but for now let's just fix this case here.) + */ + subquery->targetList = (List *) + flatten_join_alias_vars(subroot, (Node *) subquery->targetList); + + /* * Adjust level-0 varnos in subquery so that we can append its rangetable * to upper query's. We have to fix the subquery's append_rel_list as * well. @@ -840,17 +901,22 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, /* * The subquery's targetlist items are now in the appropriate form to * insert into the top query, but if we are under an outer join then - * non-nullable items may have to be turned into PlaceHolderVars. If we - * are dealing with an appendrel member then anything that's not a simple - * Var has to be turned into a PlaceHolderVar. Set up appropriate context - * data for pullup_replace_vars. + * non-nullable items and lateral references may have to be turned into + * PlaceHolderVars. If we are dealing with an appendrel member then + * anything that's not a simple Var has to be turned into a + * PlaceHolderVar. Set up required context data for pullup_replace_vars. */ rvcontext.root = root; rvcontext.targetlist = subquery->targetList; rvcontext.target_rte = rte; + if (rte->lateral) + rvcontext.relids = get_relids_in_jointree((Node *) subquery->jointree, + true); + else /* won't need relids */ + rvcontext.relids = NULL; rvcontext.outer_hasSubLinks = &parse->hasSubLinks; rvcontext.varno = varno; - rvcontext.need_phvs = (lowest_outer_join != NULL || + rvcontext.need_phvs = (lowest_nulling_outer_join != NULL || containing_appendrel != NULL); rvcontext.wrap_non_vars = (containing_appendrel != NULL); /* initialize cache array with indexes 0 .. length(tlist) */ @@ -863,7 +929,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, * replace any of the jointree structure. (This'd be a lot cleaner if we * could use query_tree_mutator.) We have to use PHVs in the targetList, * returningList, and havingQual, since those are certainly above any - * outer join. replace_vars_in_jointree tracks its location in the + * outer join. replace_vars_in_jointree tracks its location in the * jointree and uses PHVs or not appropriately. */ parse->targetList = (List *) @@ -871,7 +937,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, parse->returningList = (List *) pullup_replace_vars((Node *) parse->returningList, &rvcontext); replace_vars_in_jointree((Node *) parse->jointree, &rvcontext, - lowest_outer_join); + lowest_nulling_outer_join); Assert(parse->setOperations == NULL); parse->havingQual = pullup_replace_vars(parse->havingQual, &rvcontext); @@ -898,10 +964,10 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, * Replace references in the joinaliasvars lists of join RTEs. * * You might think that we could avoid using PHVs for alias vars of joins - * below lowest_outer_join, but that doesn't work because the alias vars - * could be referenced above that join; we need the PHVs to be present in - * such references after the alias vars get flattened. (It might be worth - * trying to be smarter here, someday.) + * below lowest_nulling_outer_join, but that doesn't work because the + * alias vars could be referenced above that join; we need the PHVs to be + * present in such references after the alias vars get flattened. (It + * might be worth trying to be smarter here, someday.) */ foreach(lc, parse->rtable) { @@ -914,6 +980,38 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, } /* + * If the subquery had a LATERAL marker, propagate that to any of its + * child RTEs that could possibly now contain lateral cross-references. + * The children might or might not contain any actual lateral + * cross-references, but we have to mark the pulled-up child RTEs so that + * later planner stages will check for such. + */ + if (rte->lateral) + { + foreach(lc, subquery->rtable) + { + RangeTblEntry *child_rte = (RangeTblEntry *) lfirst(lc); + + switch (child_rte->rtekind) + { + case RTE_SUBQUERY: + case RTE_FUNCTION: + case RTE_VALUES: + child_rte->lateral = true; + break; + case RTE_RELATION: + case RTE_JOIN: + case RTE_CTE: +#ifdef XCP + case RTE_REMOTE_DUMMY: +#endif + /* these can't contain any lateral references */ + break; + } + } + } + + /* * Now append the adjusted rtable entries to upper query. (We hold off * until after fixing the upper rtable entries; no point in running that * code on the subquery ones too.) @@ -954,11 +1052,14 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, subroot->append_rel_list); /* - * We don't have to do the equivalent bookkeeping for outer-join info, - * because that hasn't been set up yet. placeholder_list likewise. + * We don't have to do the equivalent bookkeeping for outer-join or + * LATERAL info, because that hasn't been set up yet. placeholder_list + * likewise. */ Assert(root->join_info_list == NIL); Assert(subroot->join_info_list == NIL); + Assert(root->lateral_info_list == NIL); + Assert(subroot->lateral_info_list == NIL); Assert(root->placeholder_list == NIL); Assert(subroot->placeholder_list == NIL); @@ -990,7 +1091,7 @@ pull_up_simple_subquery(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte, * Pull up a single simple UNION ALL subquery. * * jtnode is a RangeTblRef that has been identified as a simple UNION ALL - * subquery by pull_up_subqueries. We pull up the leaf subqueries and + * subquery by pull_up_subqueries. We pull up the leaf subqueries and * build an "append relation" for the union set. The result value is just * jtnode, since we don't actually need to change the query jointree. */ @@ -999,20 +1100,45 @@ pull_up_simple_union_all(PlannerInfo *root, Node *jtnode, RangeTblEntry *rte) { int varno = ((RangeTblRef *) jtnode)->rtindex; Query *subquery = rte->subquery; - int rtoffset; + int rtoffset = list_length(root->parse->rtable); List *rtable; /* - * Append child RTEs to parent rtable. - * + * Make a modifiable copy of the subquery's rtable, so we can adjust + * upper-level Vars in it. There are no such Vars in the setOperations + * tree proper, so fixing the rtable should be sufficient. + */ + rtable = copyObject(subquery->rtable); + + /* * Upper-level vars in subquery are now one level closer to their parent * than before. We don't have to worry about offsetting varnos, though, - * because any such vars must refer to stuff above the level of the query - * we are pulling into. + * because the UNION leaf queries can't cross-reference each other. */ - rtoffset = list_length(root->parse->rtable); - rtable = copyObject(subquery->rtable); IncrementVarSublevelsUp_rtable(rtable, -1, 1); + + /* + * If the UNION ALL subquery had a LATERAL marker, propagate that to all + * its children. The individual children might or might not contain any + * actual lateral cross-references, but we have to mark the pulled-up + * child RTEs so that later planner stages will check for such. + */ + if (rte->lateral) + { + ListCell *rt; + + foreach(rt, rtable) + { + RangeTblEntry *child_rte = (RangeTblEntry *) lfirst(rt); + + Assert(child_rte->rtekind == RTE_SUBQUERY); + child_rte->lateral = true; + } + } + + /* + * Append child RTEs to parent rtable. + */ root->parse->rtable = list_concat(root->parse->rtable, rtable); /* @@ -1083,11 +1209,12 @@ pull_up_union_leaf_queries(Node *setOp, PlannerInfo *root, int parentRTindex, * must build the AppendRelInfo first, because this will modify it.) * Note that we can pass NULL for containing-join info even if we're * actually under an outer join, because the child's expressions - * aren't going to propagate up above the join. + * aren't going to propagate up to the join. */ rtr = makeNode(RangeTblRef); rtr->rtindex = childRTindex; - (void) pull_up_subqueries(root, (Node *) rtr, NULL, appinfo); + (void) pull_up_subqueries_recurse(root, (Node *) rtr, + NULL, NULL, appinfo); } else if (IsA(setOp, SetOperationStmt)) { @@ -1137,9 +1264,15 @@ make_setop_translation_list(Query *query, Index newvarno, * is_simple_subquery * Check a subquery in the range table to see if it's simple enough * to pull up into the parent query. + * + * rte is the RTE_SUBQUERY RangeTblEntry that contained the subquery. + * (Note subquery is not necessarily equal to rte->subquery; it could be a + * processed copy of that.) + * lowest_outer_join is the lowest outer join above the subquery, or NULL. */ static bool -is_simple_subquery(Query *subquery) +is_simple_subquery(Query *subquery, RangeTblEntry *rte, + JoinExpr *lowest_outer_join) { /* * Let's just make sure it's a valid subselect ... @@ -1180,8 +1313,71 @@ is_simple_subquery(Query *subquery) return false; /* + * Don't pull up if the RTE represents a security-barrier view; we + * couldn't prevent information leakage once the RTE's Vars are scattered + * about in the upper query. + */ + if (rte->security_barrier) + return false; + + /* + * If the subquery is LATERAL, check for pullup restrictions from that. + */ + if (rte->lateral) + { + bool restricted; + Relids safe_upper_varnos; + + /* + * The subquery's WHERE and JOIN/ON quals mustn't contain any lateral + * references to rels outside a higher outer join (including the case + * where the outer join is within the subquery itself). In such a + * case, pulling up would result in a situation where we need to + * postpone quals from below an outer join to above it, which is + * probably completely wrong and in any case is a complication that + * doesn't seem worth addressing at the moment. + */ + if (lowest_outer_join != NULL) + { + restricted = true; + safe_upper_varnos = get_relids_in_jointree((Node *) lowest_outer_join, + true); + } + else + { + restricted = false; + safe_upper_varnos = NULL; /* doesn't matter */ + } + + if (jointree_contains_lateral_outer_refs((Node *) subquery->jointree, + restricted, safe_upper_varnos)) + return false; + + /* + * If there's an outer join above the LATERAL subquery, also disallow + * pullup if the subquery's targetlist has any references to rels + * outside the outer join, since these might get pulled into quals + * above the subquery (but in or below the outer join) and then lead + * to qual-postponement issues similar to the case checked for above. + * (We wouldn't need to prevent pullup if no such references appear in + * outer-query quals, but we don't have enough info here to check + * that. Also, maybe this restriction could be removed if we forced + * such refs to be wrapped in PlaceHolderVars, even when they're below + * the nearest outer join? But it's a pretty hokey usage, so not + * clear this is worth sweating over.) + */ + if (lowest_outer_join != NULL) + { + Relids lvarnos = pull_varnos_of_level((Node *) subquery->targetList, 1); + + if (!bms_is_subset(lvarnos, safe_upper_varnos)) + return false; + } + } + + /* * Don't pull up a subquery that has any set-returning functions in its - * targetlist. Otherwise we might well wind up inserting set-returning + * targetlist. Otherwise we might well wind up inserting set-returning * functions into places where they mustn't go, such as quals of higher * queries. */ @@ -1190,7 +1386,7 @@ is_simple_subquery(Query *subquery) /* * Don't pull up a subquery that has any volatile functions in its - * targetlist. Otherwise we might introduce multiple evaluations of these + * targetlist. Otherwise we might introduce multiple evaluations of these * functions, if they get copied to multiple places in the upper query, * leading to surprising results. (Note: the PlaceHolderVar mechanism * doesn't quite guarantee single evaluation; else we could pull up anyway @@ -1200,15 +1396,16 @@ is_simple_subquery(Query *subquery) return false; /* - * Hack: don't try to pull up a subquery with an empty jointree. - * query_planner() will correctly generate a Result plan for a jointree - * that's totally empty, but I don't think the right things happen if an - * empty FromExpr appears lower down in a jointree. It would pose a - * problem for the PlaceHolderVar mechanism too, since we'd have no way to - * identify where to evaluate a PHV coming out of the subquery. Not worth - * working hard on this, just to collapse SubqueryScan/Result into Result; - * especially since the SubqueryScan can often be optimized away by - * setrefs.c anyway. + * Don't pull up a subquery with an empty jointree. query_planner() will + * correctly generate a Result plan for a jointree that's totally empty, + * but we can't cope with an empty FromExpr appearing lower down in a + * jointree: we identify join rels via baserelid sets, so we couldn't + * distinguish a join containing such a FromExpr from one without it. This + * would for example break the PlaceHolderVar mechanism, since we'd have + * no way to identify where to evaluate a PHV coming out of the subquery. + * Not worth working hard on this, just to collapse SubqueryScan/Result + * into Result; especially since the SubqueryScan can often be optimized + * away by setrefs.c anyway. */ if (subquery->jointree->fromlist == NIL) return false; @@ -1326,23 +1523,143 @@ is_safe_append_member(Query *subquery) } /* + * jointree_contains_lateral_outer_refs + * Check for disallowed lateral references in a jointree's quals + * + * If restricted is false, all level-1 Vars are allowed (but we still must + * search the jointree, since it might contain outer joins below which there + * will be restrictions). If restricted is true, return TRUE when any qual + * in the jointree contains level-1 Vars coming from outside the rels listed + * in safe_upper_varnos. + */ +static bool +jointree_contains_lateral_outer_refs(Node *jtnode, bool restricted, + Relids safe_upper_varnos) +{ + if (jtnode == NULL) + return false; + if (IsA(jtnode, RangeTblRef)) + return false; + else if (IsA(jtnode, FromExpr)) + { + FromExpr *f = (FromExpr *) jtnode; + ListCell *l; + + /* First, recurse to check child joins */ + foreach(l, f->fromlist) + { + if (jointree_contains_lateral_outer_refs(lfirst(l), + restricted, + safe_upper_varnos)) + return true; + } + + /* Then check the top-level quals */ + if (restricted && + !bms_is_subset(pull_varnos_of_level(f->quals, 1), + safe_upper_varnos)) + return true; + } + else if (IsA(jtnode, JoinExpr)) + { + JoinExpr *j = (JoinExpr *) jtnode; + + /* + * If this is an outer join, we mustn't allow any upper lateral + * references in or below it. + */ + if (j->jointype != JOIN_INNER) + { + restricted = true; + safe_upper_varnos = NULL; + } + + /* Check the child joins */ + if (jointree_contains_lateral_outer_refs(j->larg, + restricted, + safe_upper_varnos)) + return true; + if (jointree_contains_lateral_outer_refs(j->rarg, + restricted, + safe_upper_varnos)) + return true; + + /* Check the JOIN's qual clauses */ + if (restricted && + !bms_is_subset(pull_varnos_of_level(j->quals, 1), + safe_upper_varnos)) + return true; + } + else + elog(ERROR, "unrecognized node type: %d", + (int) nodeTag(jtnode)); + return false; +} + +/* * Helper routine for pull_up_subqueries: do pullup_replace_vars on every * expression in the jointree, without changing the jointree structure itself. * Ugly, but there's no other way... * - * If we are at or below lowest_outer_join, we can suppress use of + * If we are at or below lowest_nulling_outer_join, we can suppress use of * PlaceHolderVars wrapped around the replacement expressions. */ static void replace_vars_in_jointree(Node *jtnode, pullup_replace_vars_context *context, - JoinExpr *lowest_outer_join) + JoinExpr *lowest_nulling_outer_join) { if (jtnode == NULL) return; if (IsA(jtnode, RangeTblRef)) { - /* nothing to do here */ + /* + * If the RangeTblRef refers to a LATERAL subquery (that isn't the + * same subquery we're pulling up), it might contain references to the + * target subquery, which we must replace. We drive this from the + * jointree scan, rather than a scan of the rtable, for a couple of + * reasons: we can avoid processing no-longer-referenced RTEs, and we + * can use the appropriate setting of need_phvs depending on whether + * the RTE is above possibly-nulling outer joins or not. + */ + int varno = ((RangeTblRef *) jtnode)->rtindex; + + if (varno != context->varno) /* ignore target subquery itself */ + { + RangeTblEntry *rte = rt_fetch(varno, context->root->parse->rtable); + + Assert(rte != context->target_rte); + if (rte->lateral) + { + switch (rte->rtekind) + { + case RTE_SUBQUERY: + rte->subquery = + pullup_replace_vars_subquery(rte->subquery, + context); + break; + case RTE_FUNCTION: + rte->functions = (List *) + pullup_replace_vars((Node *) rte->functions, + context); + break; + case RTE_VALUES: + rte->values_lists = (List *) + pullup_replace_vars((Node *) rte->values_lists, + context); + break; + case RTE_RELATION: + case RTE_JOIN: + case RTE_CTE: +#ifdef XCP + case RTE_REMOTE_DUMMY: +#endif + /* these shouldn't be marked LATERAL */ + Assert(false); + break; + } + } + } } else if (IsA(jtnode, FromExpr)) { @@ -1350,7 +1667,8 @@ replace_vars_in_jointree(Node *jtnode, ListCell *l; foreach(l, f->fromlist) - replace_vars_in_jointree(lfirst(l), context, lowest_outer_join); + replace_vars_in_jointree(lfirst(l), context, + lowest_nulling_outer_join); f->quals = pullup_replace_vars(f->quals, context); } else if (IsA(jtnode, JoinExpr)) @@ -1358,14 +1676,14 @@ replace_vars_in_jointree(Node *jtnode, JoinExpr *j = (JoinExpr *) jtnode; bool save_need_phvs = context->need_phvs; - if (j == lowest_outer_join) + if (j == lowest_nulling_outer_join) { /* no more PHVs in or below this join */ context->need_phvs = false; - lowest_outer_join = NULL; + lowest_nulling_outer_join = NULL; } - replace_vars_in_jointree(j->larg, context, lowest_outer_join); - replace_vars_in_jointree(j->rarg, context, lowest_outer_join); + replace_vars_in_jointree(j->larg, context, lowest_nulling_outer_join); + replace_vars_in_jointree(j->rarg, context, lowest_nulling_outer_join); j->quals = pullup_replace_vars(j->quals, context); /* @@ -1462,7 +1780,7 @@ pullup_replace_vars_callback(Var *var, /* * Insert PlaceHolderVar if needed. Notice that we are wrapping one * PlaceHolderVar around the whole RowExpr, rather than putting one - * around each element of the row. This is because we need the + * around each element of the row. This is because we need the * expression to yield NULL, not ROW(NULL,NULL,...) when it is forced * to null by an outer join. */ @@ -1504,8 +1822,18 @@ pullup_replace_vars_callback(Var *var, if (newnode && IsA(newnode, Var) && ((Var *) newnode)->varlevelsup == 0) { - /* Simple Vars always escape being wrapped */ - wrap = false; + /* + * Simple Vars always escape being wrapped, unless they are + * lateral references to something outside the subquery being + * pulled up. (Even then, we could omit the PlaceHolderVar if + * the referenced rel is under the same lowest outer join, but + * it doesn't seem worth the trouble to check that.) + */ + if (rcon->target_rte->lateral && + !bms_is_member(((Var *) newnode)->varno, rcon->relids)) + wrap = true; + else + wrap = false; } else if (newnode && IsA(newnode, PlaceHolderVar) && ((PlaceHolderVar *) newnode)->phlevelsup == 0) @@ -1521,9 +1849,10 @@ pullup_replace_vars_callback(Var *var, else { /* - * If it contains a Var of current level, and does not contain - * any non-strict constructs, then it's certainly nullable so - * we don't need to insert a PlaceHolderVar. + * If it contains a Var of the subquery being pulled up, and + * does not contain any non-strict constructs, then it's + * certainly nullable so we don't need to insert a + * PlaceHolderVar. * * This analysis could be tighter: in particular, a non-strict * construct hidden within a lower-level PlaceHolderVar is not @@ -1532,8 +1861,14 @@ pullup_replace_vars_callback(Var *var, * * Note: in future maybe we should insert a PlaceHolderVar * anyway, if the tlist item is expensive to evaluate? + * + * For a LATERAL subquery, we have to check the actual var + * membership of the node, but if it's non-lateral then any + * level-zero var must belong to the subquery. */ - if (contain_vars_of_level((Node *) newnode, 0) && + if ((rcon->target_rte->lateral ? + bms_overlap(pull_varnos((Node *) newnode), rcon->relids) : + contain_vars_of_level((Node *) newnode, 0)) && !contain_nonstrict_functions((Node *) newnode)) { /* No wrap needed */ @@ -1554,7 +1889,7 @@ pullup_replace_vars_callback(Var *var, /* * Cache it if possible (ie, if the attno is in range, which it - * probably always should be). We can cache the value even if we + * probably always should be). We can cache the value even if we * decided we didn't need a PHV, since this result will be * suitable for any request that has need_phvs. */ @@ -1571,6 +1906,25 @@ pullup_replace_vars_callback(Var *var, return newnode; } +/* + * Apply pullup variable replacement to a subquery + * + * This needs to be different from pullup_replace_vars() because + * replace_rte_variables will think that it shouldn't increment sublevels_up + * before entering the Query; so we need to call it with sublevels_up == 1. + */ +static Query * +pullup_replace_vars_subquery(Query *query, + pullup_replace_vars_context *context) +{ + Assert(IsA(query, Query)); + return (Query *) replace_rte_variables((Node *) query, + context->varno, 1, + pullup_replace_vars_callback, + (void *) context, + NULL); +} + /* * flatten_simple_union_all @@ -1578,7 +1932,7 @@ pullup_replace_vars_callback(Var *var, * * If a query's setOperations tree consists entirely of simple UNION ALL * operations, flatten it into an append relation, which we can process more - * intelligently than the general setops case. Otherwise, do nothing. + * intelligently than the general setops case. Otherwise, do nothing. * * In most cases, this can succeed only for a top-level query, because for a * subquery in FROM, the parent query's invocation of pull_up_subqueries would @@ -1690,7 +2044,7 @@ flatten_simple_union_all(PlannerInfo *root) * SELECT ... FROM a LEFT JOIN b ON (a.x = b.y) WHERE b.y IS NULL; * If the join clause is strict for b.y, then only null-extended rows could * pass the upper WHERE, and we can conclude that what the query is really - * specifying is an anti-semijoin. We change the join type from JOIN_LEFT + * specifying is an anti-semijoin. We change the join type from JOIN_LEFT * to JOIN_ANTI. The IS NULL clause then becomes redundant, and must be * removed to prevent bogus selectivity calculations, but we leave it to * distribute_qual_to_rels to get rid of such clauses. @@ -1930,7 +2284,7 @@ reduce_outer_joins_pass2(Node *jtnode, /* * See if we can reduce JOIN_LEFT to JOIN_ANTI. This is the case if * the join's own quals are strict for any var that was forced null by - * higher qual levels. NOTE: there are other ways that we could + * higher qual levels. NOTE: there are other ways that we could * detect an anti-join, in particular if we were to check whether Vars * coming from the RHS must be non-null because of table constraints. * That seems complicated and expensive though (in particular, one @@ -2088,7 +2442,7 @@ reduce_outer_joins_pass2(Node *jtnode, * pulled-up relid, and change them to reference the replacement relid(s). * * NOTE: although this has the form of a walker, we cheat and modify the - * nodes in-place. This should be OK since the tree was copied by + * nodes in-place. This should be OK since the tree was copied by * pullup_replace_vars earlier. Avoid scribbling on the original values of * the bitmapsets, though, because expression_tree_mutator doesn't copy those. */ @@ -2134,6 +2488,7 @@ substitute_multiple_relids_walker(Node *node, } /* Shouldn't need to handle planner auxiliary nodes here */ Assert(!IsA(node, SpecialJoinInfo)); + Assert(!IsA(node, LateralJoinInfo)); Assert(!IsA(node, AppendRelInfo)); Assert(!IsA(node, PlaceHolderInfo)); Assert(!IsA(node, MinMaxAggInfo)); diff --git a/src/backend/optimizer/prep/prepqual.c b/src/backend/optimizer/prep/prepqual.c index 3760fdc645..2a24938d84 100644 --- a/src/backend/optimizer/prep/prepqual.c +++ b/src/backend/optimizer/prep/prepqual.c @@ -20,7 +20,7 @@ * tree after local transformations that might introduce nested AND/ORs. * * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -54,12 +54,12 @@ static Expr *process_duplicate_ors(List *orlist); * Although this can be invoked on its own, it's mainly intended as a helper * for eval_const_expressions(), and that context drives several design * decisions. In particular, if the input is already AND/OR flat, we must - * preserve that property. We also don't bother to recurse in situations + * preserve that property. We also don't bother to recurse in situations * where we can assume that lower-level executions of eval_const_expressions * would already have simplified sub-clauses of the input. * * The difference between this and a simple make_notclause() is that this - * tries to get rid of the NOT node by logical simplification. It's clearly + * tries to get rid of the NOT node by logical simplification. It's clearly * always a win if the NOT node can be eliminated altogether. However, our * use of DeMorgan's laws could result in having more NOT nodes rather than * fewer. We do that unconditionally anyway, because in WHERE clauses it's @@ -152,7 +152,7 @@ negate_clause(Node *node) * those properties. For example, if no direct child of * the given AND clause is an AND or a NOT-above-OR, then * the recursive calls of negate_clause() can't return any - * OR clauses. So we needn't call pull_ors() before + * OR clauses. So we needn't call pull_ors() before * building a new OR clause. Similarly for the OR case. *-------------------- */ @@ -293,7 +293,7 @@ canonicalize_qual(Expr *qual) /* * Pull up redundant subclauses in OR-of-AND trees. We do this only * within the top-level AND/OR structure; there's no point in looking - * deeper. + * deeper. Also remove any NULL constants in the top-level structure. */ newqual = find_duplicate_ors(qual); @@ -374,7 +374,7 @@ pull_ors(List *orlist) * * This may seem like a fairly useless activity, but it turns out to be * applicable to many machine-generated queries, and there are also queries - * in some of the TPC benchmarks that need it. This was in fact almost the + * in some of the TPC benchmarks that need it. This was in fact almost the * sole useful side-effect of the old prepqual code that tried to force * the query into canonical AND-of-ORs form: the canonical equivalent of * ((A AND B) OR (A AND C)) @@ -393,7 +393,14 @@ pull_ors(List *orlist) * OR clauses to which the inverse OR distributive law might apply. * Only the top-level AND/OR structure is searched. * - * Returns the modified qualification. AND/OR flatness is preserved. + * While at it, we remove any NULL constants within the top-level AND/OR + * structure, eg "x OR NULL::boolean" is reduced to "x". In general that + * would change the result, so eval_const_expressions can't do it; but at + * top level of WHERE, we don't need to distinguish between FALSE and NULL + * results, so it's valid to treat NULL::boolean the same as FALSE and then + * simplify AND/OR accordingly. + * + * Returns the modified qualification. AND/OR flatness is preserved. */ static Expr * find_duplicate_ors(Expr *qual) @@ -405,12 +412,30 @@ find_duplicate_ors(Expr *qual) /* Recurse */ foreach(temp, ((BoolExpr *) qual)->args) - orlist = lappend(orlist, find_duplicate_ors(lfirst(temp))); + { + Expr *arg = (Expr *) lfirst(temp); - /* - * Don't need pull_ors() since this routine will never introduce an OR - * where there wasn't one before. - */ + arg = find_duplicate_ors(arg); + + /* Get rid of any constant inputs */ + if (arg && IsA(arg, Const)) + { + Const *carg = (Const *) arg; + + /* Drop constant FALSE or NULL */ + if (carg->constisnull || !DatumGetBool(carg->constvalue)) + continue; + /* constant TRUE, so OR reduces to TRUE */ + return arg; + } + + orlist = lappend(orlist, arg); + } + + /* Flatten any ORs pulled up to just below here */ + orlist = pull_ors(orlist); + + /* Now we can look for duplicate ORs */ return process_duplicate_ors(orlist); } else if (and_clause((Node *) qual)) @@ -420,10 +445,38 @@ find_duplicate_ors(Expr *qual) /* Recurse */ foreach(temp, ((BoolExpr *) qual)->args) - andlist = lappend(andlist, find_duplicate_ors(lfirst(temp))); + { + Expr *arg = (Expr *) lfirst(temp); + + arg = find_duplicate_ors(arg); + + /* Get rid of any constant inputs */ + if (arg && IsA(arg, Const)) + { + Const *carg = (Const *) arg; + + /* Drop constant TRUE */ + if (!carg->constisnull && DatumGetBool(carg->constvalue)) + continue; + /* constant FALSE or NULL, so AND reduces to FALSE */ + return (Expr *) makeBoolConst(false, false); + } + + andlist = lappend(andlist, arg); + } + /* Flatten any ANDs introduced just below here */ andlist = pull_ands(andlist); - /* The AND list can't get shorter, so result is always an AND */ + + /* AND of no inputs reduces to TRUE */ + if (andlist == NIL) + return (Expr *) makeBoolConst(true, false); + + /* Single-expression AND just reduces to that expression */ + if (list_length(andlist) == 1) + return (Expr *) linitial(andlist); + + /* Else we still need an AND node */ return make_andclause(andlist); } else @@ -447,11 +500,13 @@ process_duplicate_ors(List *orlist) List *neworlist; ListCell *temp; + /* OR of no inputs reduces to FALSE */ if (orlist == NIL) - return NULL; /* probably can't happen */ - if (list_length(orlist) == 1) /* single-expression OR (can this - * happen?) */ - return linitial(orlist); + return (Expr *) makeBoolConst(false, false); + + /* Single-expression OR just reduces to that expression */ + if (list_length(orlist) == 1) + return (Expr *) linitial(orlist); /* * Choose the shortest AND clause as the reference list --- obviously, any diff --git a/src/backend/optimizer/prep/prepsecurity.c b/src/backend/optimizer/prep/prepsecurity.c new file mode 100644 index 0000000000..dd7f9003a2 --- /dev/null +++ b/src/backend/optimizer/prep/prepsecurity.c @@ -0,0 +1,470 @@ +/*------------------------------------------------------------------------- + * + * prepsecurity.c + * Routines for preprocessing security barrier quals. + * + * Portions Copyright (c) 1996-2013, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/optimizer/prep/prepsecurity.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/heapam.h" +#include "access/sysattr.h" +#include "catalog/heap.h" +#include "nodes/makefuncs.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/prep.h" +#include "parser/analyze.h" +#include "parser/parsetree.h" +#include "rewrite/rewriteManip.h" +#include "utils/rel.h" + + +typedef struct +{ + int rt_index; /* Index of security barrier RTE */ + int sublevels_up; /* Current nesting depth */ + Relation rel; /* RTE relation at rt_index */ + List *targetlist; /* Targetlist for new subquery RTE */ + List *colnames; /* Column names in subquery RTE */ + List *vars_processed; /* List of Vars already processed */ +} security_barrier_replace_vars_context; + +static void expand_security_qual(PlannerInfo *root, List *tlist, int rt_index, + RangeTblEntry *rte, Node *qual); + +static void security_barrier_replace_vars(Node *node, + security_barrier_replace_vars_context *context); + +static bool security_barrier_replace_vars_walker(Node *node, + security_barrier_replace_vars_context *context); + + +/* + * expand_security_quals - + * expands any security barrier quals on RTEs in the query rtable, turning + * them into security barrier subqueries. + * + * Any given RTE may have multiple security barrier quals in a list, from which + * we create a set of nested subqueries to isolate each security barrier from + * the others, providing protection against malicious user-defined security + * barriers. The first security barrier qual in the list will be used in the + * innermost subquery. + */ +void +expand_security_quals(PlannerInfo *root, List *tlist) +{ + Query *parse = root->parse; + int rt_index; + ListCell *cell; + + /* + * Process each RTE in the rtable list. + * + * We only ever modify entries in place and append to the rtable, so it is + * safe to use a foreach loop here. + */ + rt_index = 0; + foreach(cell, parse->rtable) + { + RangeTblEntry *rte = (RangeTblEntry *) lfirst(cell); + + rt_index++; + + if (rte->securityQuals == NIL) + continue; + + /* + * Ignore any RTEs that aren't used in the query (such RTEs may be + * present for permissions checks). + */ + if (rt_index != parse->resultRelation && + !rangeTableEntry_used((Node *) parse, rt_index, 0)) + continue; + + /* + * If this RTE is the target then we need to make a copy of it before + * expanding it. The unexpanded copy will become the new target, and + * the original RTE will be expanded to become the source of rows to + * update/delete. + */ + if (rt_index == parse->resultRelation) + { + RangeTblEntry *newrte = copyObject(rte); + + parse->rtable = lappend(parse->rtable, newrte); + parse->resultRelation = list_length(parse->rtable); + + /* + * Wipe out any copied security barrier quals on the new target to + * prevent infinite recursion. + */ + newrte->securityQuals = NIL; + + /* + * There's no need to do permissions checks twice, so wipe out the + * permissions info for the original RTE (we prefer to keep the + * bits set on the result RTE). + */ + rte->requiredPerms = 0; + rte->checkAsUser = InvalidOid; + rte->selectedCols = NULL; + rte->modifiedCols = NULL; + + /* + * For the most part, Vars referencing the original relation + * should remain as they are, meaning that they pull OLD values + * from the expanded RTE. But in the RETURNING list and in any + * WITH CHECK OPTION quals, we want such Vars to represent NEW + * values, so change them to reference the new RTE. + */ + ChangeVarNodes((Node *) parse->returningList, rt_index, + parse->resultRelation, 0); + + ChangeVarNodes((Node *) parse->withCheckOptions, rt_index, + parse->resultRelation, 0); + } + + /* + * Process each security barrier qual in turn, starting with the + * innermost one (the first in the list) and working outwards. + * + * We remove each qual from the list before processing it, so that its + * variables aren't modified by expand_security_qual. Also we don't + * necessarily want the attributes referred to by the qual to be + * exposed by the newly built subquery. + */ + while (rte->securityQuals != NIL) + { + Node *qual = (Node *) linitial(rte->securityQuals); + + rte->securityQuals = list_delete_first(rte->securityQuals); + + ChangeVarNodes(qual, rt_index, 1, 0); + expand_security_qual(root, tlist, rt_index, rte, qual); + } + } +} + + +/* + * expand_security_qual - + * expand the specified security barrier qual on a query RTE, turning the + * RTE into a security barrier subquery. + */ +static void +expand_security_qual(PlannerInfo *root, List *tlist, int rt_index, + RangeTblEntry *rte, Node *qual) +{ + Query *parse = root->parse; + Oid relid = rte->relid; + Query *subquery; + RangeTblEntry *subrte; + RangeTblRef *subrtr; + PlanRowMark *rc; + security_barrier_replace_vars_context context; + ListCell *cell; + + /* + * There should only be 2 possible cases: + * + * 1. A relation RTE, which we turn into a subquery RTE containing all + * referenced columns. + * + * 2. A subquery RTE (either from a prior call to this function or from an + * expanded view). In this case we build a new subquery on top of it to + * isolate this security barrier qual from any other quals. + */ + switch (rte->rtekind) + { + case RTE_RELATION: + + /* + * Turn the relation RTE into a security barrier subquery RTE, + * moving all permissions checks down into the subquery. + */ + subquery = makeNode(Query); + subquery->commandType = CMD_SELECT; + subquery->querySource = QSRC_INSTEAD_RULE; + + subrte = copyObject(rte); + subrte->inFromCl = true; + subrte->securityQuals = NIL; + subquery->rtable = list_make1(subrte); + + subrtr = makeNode(RangeTblRef); + subrtr->rtindex = 1; + subquery->jointree = makeFromExpr(list_make1(subrtr), qual); + subquery->hasSubLinks = checkExprHasSubLink(qual); + + rte->rtekind = RTE_SUBQUERY; + rte->relid = InvalidOid; + rte->subquery = subquery; + rte->security_barrier = true; + rte->inh = false; /* must not be set for a subquery */ + + /* the permissions checks have now been moved down */ + rte->requiredPerms = 0; + rte->checkAsUser = InvalidOid; + rte->selectedCols = NULL; + rte->modifiedCols = NULL; + + /* + * Now deal with any PlanRowMark on this RTE by requesting a lock + * of the same strength on the RTE copied down to the subquery. + * + * Note that we can't push the user-defined quals down since they + * may included untrusted functions and that means that we will + * end up locking all rows which pass the securityQuals, even if + * those rows don't pass the user-defined quals. This is + * currently documented behavior, but it'd be nice to come up with + * a better solution some day. + */ + rc = get_plan_rowmark(root->rowMarks, rt_index); + if (rc != NULL) + { + switch (rc->markType) + { + case ROW_MARK_EXCLUSIVE: + applyLockingClause(subquery, 1, LCS_FORUPDATE, + rc->noWait, false); + break; + case ROW_MARK_NOKEYEXCLUSIVE: + applyLockingClause(subquery, 1, LCS_FORNOKEYUPDATE, + rc->noWait, false); + break; + case ROW_MARK_SHARE: + applyLockingClause(subquery, 1, LCS_FORSHARE, + rc->noWait, false); + break; + case ROW_MARK_KEYSHARE: + applyLockingClause(subquery, 1, LCS_FORKEYSHARE, + rc->noWait, false); + break; + case ROW_MARK_REFERENCE: + case ROW_MARK_COPY: + /* No locking needed */ + break; + } + root->rowMarks = list_delete(root->rowMarks, rc); + } + + /* + * Replace any variables in the outer query that refer to the + * original relation RTE with references to columns that we will + * expose in the new subquery, building the subquery's targetlist + * as we go. + */ + context.rt_index = rt_index; + context.sublevels_up = 0; + context.rel = heap_open(relid, NoLock); + context.targetlist = NIL; + context.colnames = NIL; + context.vars_processed = NIL; + + security_barrier_replace_vars((Node *) parse, &context); + security_barrier_replace_vars((Node *) tlist, &context); + + heap_close(context.rel, NoLock); + + /* Now we know what columns the subquery needs to expose */ + rte->subquery->targetList = context.targetlist; + rte->eref = makeAlias(rte->eref->aliasname, context.colnames); + + break; + + case RTE_SUBQUERY: + + /* + * Build a new subquery that includes all the same columns as the + * original subquery. + */ + subquery = makeNode(Query); + subquery->commandType = CMD_SELECT; + subquery->querySource = QSRC_INSTEAD_RULE; + subquery->targetList = NIL; + + foreach(cell, rte->subquery->targetList) + { + TargetEntry *tle; + Var *var; + + tle = (TargetEntry *) lfirst(cell); + var = makeVarFromTargetEntry(1, tle); + + tle = makeTargetEntry((Expr *) var, + list_length(subquery->targetList) + 1, + pstrdup(tle->resname), + tle->resjunk); + subquery->targetList = lappend(subquery->targetList, tle); + } + + subrte = makeNode(RangeTblEntry); + subrte->rtekind = RTE_SUBQUERY; + subrte->subquery = rte->subquery; + subrte->security_barrier = rte->security_barrier; + subrte->eref = copyObject(rte->eref); + subrte->inFromCl = true; + subquery->rtable = list_make1(subrte); + + subrtr = makeNode(RangeTblRef); + subrtr->rtindex = 1; + subquery->jointree = makeFromExpr(list_make1(subrtr), qual); + subquery->hasSubLinks = checkExprHasSubLink(qual); + + rte->subquery = subquery; + rte->security_barrier = true; + + break; + + default: + elog(ERROR, "invalid range table entry for security barrier qual"); + } +} + + +/* + * security_barrier_replace_vars - + * Apply security barrier variable replacement to an expression tree. + * + * This also builds/updates a targetlist with entries for each replacement + * variable that needs to be exposed by the security barrier subquery RTE. + * + * NOTE: although this has the form of a walker, we cheat and modify the + * nodes in-place. The given expression tree should have been copied + * earlier to ensure that no unwanted side-effects occur! + */ +static void +security_barrier_replace_vars(Node *node, + security_barrier_replace_vars_context *context) +{ + /* + * Must be prepared to start with a Query or a bare expression tree; if + * it's a Query, go straight to query_tree_walker to make sure that + * sublevels_up doesn't get incremented prematurely. + */ + if (node && IsA(node, Query)) + query_tree_walker((Query *) node, + security_barrier_replace_vars_walker, + (void *) context, 0); + else + security_barrier_replace_vars_walker(node, context); +} + +static bool +security_barrier_replace_vars_walker(Node *node, + security_barrier_replace_vars_context *context) +{ + if (node == NULL) + return false; + + if (IsA(node, Var)) + { + Var *var = (Var *) node; + + /* + * Note that the same Var may be present in different lists, so we + * need to take care not to process it multiple times. + */ + if (var->varno == context->rt_index && + var->varlevelsup == context->sublevels_up && + !list_member_ptr(context->vars_processed, var)) + { + /* + * Found a matching variable. Make sure that it is in the subquery + * targetlist and map its attno accordingly. + */ + AttrNumber attno; + ListCell *l; + TargetEntry *tle; + char *attname; + Var *newvar; + + /* Search for the base attribute in the subquery targetlist */ + attno = InvalidAttrNumber; + foreach(l, context->targetlist) + { + tle = (TargetEntry *) lfirst(l); + attno++; + + Assert(IsA(tle->expr, Var)); + if (((Var *) tle->expr)->varattno == var->varattno && + ((Var *) tle->expr)->varcollid == var->varcollid) + { + /* Map the variable onto this subquery targetlist entry */ + var->varattno = attno; + return false; + } + } + + /* Not in the subquery targetlist, so add it. Get its name. */ + if (var->varattno < 0) + { + Form_pg_attribute att_tup; + + att_tup = SystemAttributeDefinition(var->varattno, + context->rel->rd_rel->relhasoids); + attname = NameStr(att_tup->attname); + } + else if (var->varattno == InvalidAttrNumber) + { + attname = "wholerow"; + } + else if (var->varattno <= context->rel->rd_att->natts) + { + Form_pg_attribute att_tup; + + att_tup = context->rel->rd_att->attrs[var->varattno - 1]; + attname = NameStr(att_tup->attname); + } + else + { + elog(ERROR, "invalid attribute number %d in security_barrier_replace_vars", var->varattno); + } + + /* New variable for subquery targetlist */ + newvar = copyObject(var); + newvar->varno = 1; + + attno = list_length(context->targetlist) + 1; + tle = makeTargetEntry((Expr *) newvar, + attno, + pstrdup(attname), + false); + + context->targetlist = lappend(context->targetlist, tle); + + context->colnames = lappend(context->colnames, + makeString(pstrdup(attname))); + + /* Update the outer query's variable */ + var->varattno = attno; + + /* Remember this Var so that we don't process it again */ + context->vars_processed = lappend(context->vars_processed, var); + } + return false; + } + + if (IsA(node, Query)) + { + /* Recurse into subselects */ + bool result; + + context->sublevels_up++; + result = query_tree_walker((Query *) node, + security_barrier_replace_vars_walker, + (void *) context, 0); + context->sublevels_up--; + return result; + } + + return expression_tree_walker(node, security_barrier_replace_vars_walker, + (void *) context); +} diff --git a/src/backend/optimizer/prep/preptlist.c b/src/backend/optimizer/prep/preptlist.c index e355f349f2..ec482045b9 100644 --- a/src/backend/optimizer/prep/preptlist.c +++ b/src/backend/optimizer/prep/preptlist.c @@ -4,9 +4,10 @@ * Routines to preprocess the parse tree target list * * For INSERT and UPDATE queries, the targetlist must contain an entry for - * each attribute of the target relation in the correct order. For all query + * each attribute of the target relation in the correct order. For all query * types, we may need to add junk tlist entries for Vars used in the RETURNING - * list and row ID information needed for EvalPlanQual checking. + * list and row ID information needed for SELECT FOR UPDATE locking and/or + * EvalPlanQual checking. * * NOTE: the rewriter's rewriteTargetListIU and rewriteTargetListUD * routines also do preprocessing of the targetlist. The division of labor @@ -18,7 +19,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * Portions Copyright (c) 2012-2014, TransLattice, Inc. - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * IDENTIFICATION @@ -204,7 +205,7 @@ preprocess_targetlist(PlannerInfo *root, List *tlist) /* * Add necessary junk columns for rowmarked rels. These values are needed * for locking of rels selected FOR UPDATE/SHARE, and to do EvalPlanQual - * rechecking. See comments for PlanRowMark in plannodes.h. + * rechecking. See comments for PlanRowMark in plannodes.h. */ foreach(lc, root->rowMarks) { @@ -269,7 +270,7 @@ preprocess_targetlist(PlannerInfo *root, List *tlist) /* * If the query has a RETURNING list, add resjunk entries for any Vars * used in RETURNING that belong to other relations. We need to do this - * to make these Vars available for the RETURNING calculation. Vars that + * to make these Vars available for the RETURNING calculation. Vars that * belong to the result rel don't need to be added, because they will be * made to refer to the actual heap tuple. */ @@ -377,9 +378,9 @@ expand_targetlist(List *tlist, int command_type, * When generating a NULL constant for a dropped column, we label * it INT4 (any other guaranteed-to-exist datatype would do as * well). We can't label it with the dropped column's datatype - * since that might not exist anymore. It does not really matter + * since that might not exist anymore. It does not really matter * what we claim the type is, since NULL is NULL --- its - * representation is datatype-independent. This could perhaps + * representation is datatype-independent. This could perhaps * confuse code comparing the finished plan to the target * relation, however. */ @@ -461,7 +462,7 @@ expand_targetlist(List *tlist, int command_type, /* * The remaining tlist entries should be resjunk; append them all to the * end of the new tlist, making sure they have resnos higher than the last - * real attribute. (Note: although the rewriter already did such + * real attribute. (Note: although the rewriter already did such * renumbering, we have to do it again here in case we are doing an UPDATE * in a table with dropped columns, or an inheritance child table with * extra columns.) diff --git a/src/backend/optimizer/prep/prepunion.c b/src/backend/optimizer/prep/prepunion.c index 47e331cb59..458e89b0e2 100644 --- a/src/backend/optimizer/prep/prepunion.c +++ b/src/backend/optimizer/prep/prepunion.c @@ -6,14 +6,14 @@ * * There are two code paths in the planner for set-operation queries. * If a subquery consists entirely of simple UNION ALL operations, it - * is converted into an "append relation". Otherwise, it is handled + * is converted into an "append relation". Otherwise, it is handled * by the general code in this module (plan_set_operations and its * subroutines). There is some support code here for the append-relation * case, but most of the heavy lifting for that is done elsewhere, * notably in prepjointree.c and allpaths.c. * * There is also some code here to support planning of queries that use - * inheritance (SELECT FROM foo*). Inheritance trees are converted into + * inheritance (SELECT FROM foo*). Inheritance trees are converted into * append relations, and thenceforth share code with the UNION ALL case. * * @@ -22,7 +22,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * Portions Copyright (c) 2012-2014, TransLattice, Inc. - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -33,8 +33,10 @@ */ #include "postgres.h" +#include <limits.h> #include "access/heapam.h" +#include "access/htup_details.h" #include "access/sysattr.h" #include "catalog/pg_inherits_fn.h" #include "catalog/pg_type.h" @@ -58,6 +60,7 @@ typedef struct { PlannerInfo *root; AppendRelInfo *appinfo; + int sublevels_up; } adjust_appendrel_attrs_context; static Plan *recurse_set_operations(Node *setOp, PlannerInfo *root, @@ -241,6 +244,9 @@ recurse_set_operations(Node *setOp, PlannerInfo *root, */ rel = build_simple_rel(root, rtr->rtindex, RELOPT_BASEREL); + /* plan_params should not be in use in current query level */ + Assert(root->plan_params == NIL); + /* * Generate plan for primitive subquery */ @@ -267,6 +273,13 @@ recurse_set_operations(Node *setOp, PlannerInfo *root, root->recursiveOk = subroot->recursiveOk; /* + * It should not be possible for the primitive query to contain any + * cross-references to other primitive queries in the setop tree. + */ + if (root->plan_params) + elog(ERROR, "unexpected outer reference in set operation subquery"); + + /* * Estimate number of groups if caller wants it. If the subquery used * grouping or aggregation, its output is probably mostly unique * anyway; otherwise do statistical estimation. @@ -582,7 +595,7 @@ generate_nonunion_plan(SetOperationStmt *op, PlannerInfo *root, * * The tlist for an Append plan isn't important as far as the Append is * concerned, but we must make it look real anyway for the benefit of the - * next plan level up. In fact, it has to be real enough that the flag + * next plan level up. In fact, it has to be real enough that the flag * column is shown as a variable not a constant, else setrefs.c will get * confused. */ @@ -975,7 +988,7 @@ generate_setop_tlist(List *colTypes, List *colCollations, * Ensure the tlist entry's exposed collation matches the set-op. This * is necessary because plan_set_operations() reports the result * ordering as a list of SortGroupClauses, which don't carry collation - * themselves but just refer to tlist entries. If we don't show the + * themselves but just refer to tlist entries. If we don't show the * right collation then planner.c might do the wrong thing in * higher-level queries. * @@ -988,7 +1001,7 @@ generate_setop_tlist(List *colTypes, List *colCollations, exprType(expr), exprTypmod(expr), colColl, - COERCE_DONTCARE); + COERCE_IMPLICIT_CAST); } tle = makeTargetEntry((Expr *) expr, @@ -1005,7 +1018,7 @@ generate_setop_tlist(List *colTypes, List *colCollations, expr = (Node *) makeConst(INT4OID, -1, InvalidOid, - sizeof(int4), + sizeof(int32), Int32GetDatum(flag), false, true); @@ -1189,7 +1202,7 @@ generate_setop_grouplist(SetOperationStmt *op, List *targetlist) /* * expand_inherited_tables * Expand each rangetable entry that represents an inheritance set - * into an "append relation". At the conclusion of this process, + * into an "append relation". At the conclusion of this process, * the "inh" flag is set in all and only those RTEs that are append * relation parents. */ @@ -1221,7 +1234,7 @@ expand_inherited_tables(PlannerInfo *root) * Check whether a rangetable entry represents an inheritance set. * If so, add entries for all the child tables to the query's * rangetable, and build AppendRelInfo nodes for all the child tables - * and add them to root->append_rel_list. If not, clear the entry's + * and add them to root->append_rel_list. If not, clear the entry's * "inh" flag to prevent later code from looking for AppendRelInfos. * * Note that the original RTE is considered to represent the whole @@ -1532,7 +1545,7 @@ make_inh_translation_list(Relation oldrelation, Relation newrelation, * parent rel's attribute numbering to the child's. * * The only surprise here is that we don't translate a parent whole-row - * reference into a child whole-row reference. That would mean requiring + * reference into a child whole-row reference. That would mean requiring * permissions on all child columns, which is overly strict, since the * query is really only going to reference the inherited columns. Instead * we set the per-column bits for all inherited columns. @@ -1586,8 +1599,9 @@ translate_col_privs(const Bitmapset *parent_privs, * child rel instead. We also update rtindexes appearing outside Vars, * such as resultRelation and jointree relids. * - * Note: this is only applied after conversion of sublinks to subplans, - * so we don't need to cope with recursion into sub-queries. + * Note: this is applied after conversion of sublinks to subplans in the + * query jointree, but there may still be sublinks in the security barrier + * quals of RTEs, so we do need to cope with recursion into sub-queries. * * Note: this is not hugely different from what pullup_replace_vars() does; * maybe we should try to fold the two routines together. @@ -1600,9 +1614,12 @@ adjust_appendrel_attrs(PlannerInfo *root, Node *node, AppendRelInfo *appinfo) context.root = root; context.appinfo = appinfo; + context.sublevels_up = 0; /* - * Must be prepared to start with a Query or a bare expression tree. + * Must be prepared to start with a Query or a bare expression tree; if + * it's a Query, go straight to query_tree_walker to make sure that + * sublevels_up doesn't get incremented prematurely. */ if (node && IsA(node, Query)) { @@ -1641,7 +1658,7 @@ adjust_appendrel_attrs_mutator(Node *node, { Var *var = (Var *) copyObject(node); - if (var->varlevelsup == 0 && + if (var->varlevelsup == context->sublevels_up && var->varno == appinfo->parent_relid) { var->varno = appinfo->child_relid; @@ -1658,6 +1675,7 @@ adjust_appendrel_attrs_mutator(Node *node, if (newnode == NULL) elog(ERROR, "attribute %d of relation \"%s\" does not exist", var->varattno, get_rel_name(appinfo->parent_reloid)); + ((Var *) newnode)->varlevelsup += context->sublevels_up; return newnode; } else if (var->varattno == 0) @@ -1700,10 +1718,17 @@ adjust_appendrel_attrs_mutator(Node *node, RowExpr *rowexpr; List *fields; RangeTblEntry *rte; + ListCell *lc; rte = rt_fetch(appinfo->parent_relid, context->root->parse->rtable); fields = (List *) copyObject(appinfo->translated_vars); + foreach(lc, fields) + { + Var *field = (Var *) lfirst(lc); + + field->varlevelsup += context->sublevels_up; + } rowexpr = makeNode(RowExpr); rowexpr->args = fields; rowexpr->row_typeid = var->vartype; @@ -1722,7 +1747,8 @@ adjust_appendrel_attrs_mutator(Node *node, { CurrentOfExpr *cexpr = (CurrentOfExpr *) copyObject(node); - if (cexpr->cvarno == appinfo->parent_relid) + if (context->sublevels_up == 0 && + cexpr->cvarno == appinfo->parent_relid) cexpr->cvarno = appinfo->child_relid; return (Node *) cexpr; } @@ -1730,7 +1756,8 @@ adjust_appendrel_attrs_mutator(Node *node, { RangeTblRef *rtr = (RangeTblRef *) copyObject(node); - if (rtr->rtindex == appinfo->parent_relid) + if (context->sublevels_up == 0 && + rtr->rtindex == appinfo->parent_relid) rtr->rtindex = appinfo->child_relid; return (Node *) rtr; } @@ -1743,7 +1770,8 @@ adjust_appendrel_attrs_mutator(Node *node, adjust_appendrel_attrs_mutator, (void *) context); /* now fix JoinExpr's rtindex (probably never happens) */ - if (j->rtindex == appinfo->parent_relid) + if (context->sublevels_up == 0 && + j->rtindex == appinfo->parent_relid) j->rtindex = appinfo->child_relid; return (Node *) j; } @@ -1756,7 +1784,7 @@ adjust_appendrel_attrs_mutator(Node *node, adjust_appendrel_attrs_mutator, (void *) context); /* now fix PlaceHolderVar's relid sets */ - if (phv->phlevelsup == 0) + if (phv->phlevelsup == context->sublevels_up) phv->phrels = adjust_relid_set(phv->phrels, appinfo->parent_relid, appinfo->child_relid); @@ -1764,6 +1792,7 @@ adjust_appendrel_attrs_mutator(Node *node, } /* Shouldn't need to handle planner auxiliary nodes here */ Assert(!IsA(node, SpecialJoinInfo)); + Assert(!IsA(node, LateralJoinInfo)); Assert(!IsA(node, AppendRelInfo)); Assert(!IsA(node, PlaceHolderInfo)); Assert(!IsA(node, MinMaxAggInfo)); @@ -1827,12 +1856,29 @@ adjust_appendrel_attrs_mutator(Node *node, return (Node *) newinfo; } - /* - * NOTE: we do not need to recurse into sublinks, because they should - * already have been converted to subplans before we see them. - */ - Assert(!IsA(node, SubLink)); - Assert(!IsA(node, Query)); + if (IsA(node, Query)) + { + /* + * Recurse into sublink subqueries. This should only be possible in + * security barrier quals of top-level RTEs. All other sublinks should + * have already been converted to subplans during expression + * preprocessing, but this doesn't happen for security barrier quals, + * since they are destined to become quals of a subquery RTE, which + * will be recursively planned, and so should not be preprocessed at + * this stage. + * + * We don't explicitly Assert() for securityQuals here simply because + * it's not trivial to do so. + */ + Query *newnode; + + context->sublevels_up++; + newnode = query_tree_mutator((Query *) node, + adjust_appendrel_attrs_mutator, + (void *) context, 0); + context->sublevels_up--; + return (Node *) newnode; + } return expression_tree_mutator(node, adjust_appendrel_attrs_mutator, (void *) context); @@ -1860,7 +1906,7 @@ adjust_relid_set(Relids relids, Index oldrelid, Index newrelid) * * The expressions have already been fixed, but we have to make sure that * the target resnos match the child table (they may not, in the case of - * a column that was added after-the-fact by ALTER TABLE). In some cases + * a column that was added after-the-fact by ALTER TABLE). In some cases * this can force us to re-order the tlist to preserve resno ordering. * (We do all this work in special cases so that preptlist.c is fast for * the typical case.) diff --git a/src/backend/optimizer/util/Makefile b/src/backend/optimizer/util/Makefile index 3b2d16b635..c54d0a690d 100644 --- a/src/backend/optimizer/util/Makefile +++ b/src/backend/optimizer/util/Makefile @@ -12,7 +12,7 @@ subdir = src/backend/optimizer/util top_builddir = ../../../.. include $(top_builddir)/src/Makefile.global -OBJS = clauses.o joininfo.o pathnode.o placeholder.o plancat.o predtest.o \ - relnode.o restrictinfo.o tlist.o var.o +OBJS = clauses.o joininfo.o orclauses.o pathnode.o placeholder.o \ + plancat.o predtest.o relnode.o restrictinfo.o tlist.o var.o include $(top_srcdir)/src/backend/common.mk diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c index 73f5e11abe..97dacaaac1 100644 --- a/src/backend/optimizer/util/clauses.c +++ b/src/backend/optimizer/util/clauses.c @@ -3,7 +3,7 @@ * clauses.c * routines to manipulate qualification clauses * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -19,6 +19,7 @@ #include "postgres.h" +#include "access/htup_details.h" #include "catalog/pg_aggregate.h" #include "catalog/pg_language.h" #include "catalog/pg_operator.h" @@ -36,6 +37,7 @@ #include "optimizer/prep.h" #include "optimizer/var.h" #include "parser/analyze.h" +#include "parser/parse_agg.h" #include "parser/parse_coerce.h" #include "parser/parse_func.h" #include "rewrite/rewriteManip.h" @@ -43,6 +45,7 @@ #include "utils/acl.h" #include "utils/builtins.h" #include "utils/datum.h" +#include "utils/fmgroids.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/syscache.h" @@ -92,12 +95,12 @@ static bool expression_returns_set_rows_walker(Node *node, double *count); static bool contain_subplans_walker(Node *node, void *context); static bool contain_mutable_functions_walker(Node *node, void *context); static bool contain_volatile_functions_walker(Node *node, void *context); +static bool contain_volatile_functions_not_nextval_walker(Node *node, void *context); static bool contain_nonstrict_functions_walker(Node *node, void *context); static bool contain_leaky_functions_walker(Node *node, void *context); static Relids find_nonnullable_rels_walker(Node *node, bool top_level); static List *find_nonnullable_vars_walker(Node *node, bool top_level); static bool is_strict_saop(ScalarArrayOpExpr *expr, bool falseOK); -static bool set_coercionform_dontcare_walker(Node *node, void *context); static Node *eval_const_expressions_mutator(Node *node, eval_const_expressions_context *context); static List *simplify_or_arguments(List *args, @@ -110,7 +113,7 @@ static Node *simplify_boolean_equality(Oid opno, List *args); static Expr *simplify_function(Oid funcid, Oid result_type, int32 result_typmod, Oid result_collid, Oid input_collid, List **args_p, - bool process_args, bool allow_non_const, + bool funcvariadic, bool process_args, bool allow_non_const, eval_const_expressions_context *context); static List *expand_function_arguments(List *args, Oid result_type, HeapTuple func_tuple); @@ -121,10 +124,12 @@ static void recheck_cast_function_args(List *args, Oid result_type, HeapTuple func_tuple); static Expr *evaluate_function(Oid funcid, Oid result_type, int32 result_typmod, Oid result_collid, Oid input_collid, List *args, + bool funcvariadic, HeapTuple func_tuple, eval_const_expressions_context *context); static Expr *inline_function(Oid funcid, Oid result_type, Oid result_collid, Oid input_collid, List *args, + bool funcvariadic, HeapTuple func_tuple, eval_const_expressions_context *context); static Node *substitute_actual_parameters(Node *expr, int nargs, List *args, @@ -459,14 +464,18 @@ count_agg_clauses_walker(Node *node, count_agg_clauses_context *context) Oid aggtransfn; Oid aggfinalfn; Oid aggtranstype; + int32 aggtransspace; QualCost argcosts; - Oid *inputTypes; + Oid inputTypes[FUNC_MAX_ARGS]; int numArguments; - ListCell *l; Assert(aggref->agglevelsup == 0); - /* fetch info about aggregate from pg_aggregate */ + /* + * Fetch info about aggregate from pg_aggregate. Note it's correct to + * ignore the moving-aggregate variant, since what we're concerned + * with here is aggregates not window functions. + */ aggTuple = SearchSysCache1(AGGFNOID, ObjectIdGetDatum(aggref->aggfnoid)); if (!HeapTupleIsValid(aggTuple)) @@ -476,9 +485,10 @@ count_agg_clauses_walker(Node *node, count_agg_clauses_context *context) aggtransfn = aggform->aggtransfn; aggfinalfn = aggform->aggfinalfn; aggtranstype = aggform->aggtranstype; + aggtransspace = aggform->aggtransspace; ReleaseSysCache(aggTuple); - /* count it */ + /* count it; note ordered-set aggs always have nonempty aggorder */ costs->numAggs++; if (aggref->aggorder != NIL || aggref->aggdistinct != NIL) costs->numOrderedAggs++; @@ -493,59 +503,75 @@ count_agg_clauses_walker(Node *node, count_agg_clauses_context *context) costs->transCost.startup += argcosts.startup; costs->transCost.per_tuple += argcosts.per_tuple; - /* extract argument types (ignoring any ORDER BY expressions) */ - inputTypes = (Oid *) palloc(sizeof(Oid) * list_length(aggref->args)); - numArguments = 0; - foreach(l, aggref->args) + /* + * Add any filter's cost to per-input-row costs. + * + * XXX Ideally we should reduce input expression costs according to + * filter selectivity, but it's not clear it's worth the trouble. + */ + if (aggref->aggfilter) { - TargetEntry *tle = (TargetEntry *) lfirst(l); - - if (!tle->resjunk) - inputTypes[numArguments++] = exprType((Node *) tle->expr); + cost_qual_eval_node(&argcosts, (Node *) aggref->aggfilter, + context->root); + costs->transCost.startup += argcosts.startup; + costs->transCost.per_tuple += argcosts.per_tuple; } - /* resolve actual type of transition state, if polymorphic */ - if (IsPolymorphicType(aggtranstype)) + /* + * If there are direct arguments, treat their evaluation cost like the + * cost of the finalfn. + */ + if (aggref->aggdirectargs) { - /* have to fetch the agg's declared input types... */ - Oid *declaredArgTypes; - int agg_nargs; - - (void) get_func_signature(aggref->aggfnoid, - &declaredArgTypes, &agg_nargs); - Assert(agg_nargs == numArguments); - aggtranstype = enforce_generic_type_consistency(inputTypes, - declaredArgTypes, - agg_nargs, - aggtranstype, - false); - pfree(declaredArgTypes); + cost_qual_eval_node(&argcosts, (Node *) aggref->aggdirectargs, + context->root); + costs->transCost.startup += argcosts.startup; + costs->finalCost += argcosts.per_tuple; } + /* extract argument types (ignoring any ORDER BY expressions) */ + numArguments = get_aggregate_argtypes(aggref, inputTypes); + + /* resolve actual type of transition state, if polymorphic */ + aggtranstype = resolve_aggregate_transtype(aggref->aggfnoid, + aggtranstype, + inputTypes, + numArguments); + /* * If the transition type is pass-by-value then it doesn't add - * anything to the required size of the hashtable. If it is + * anything to the required size of the hashtable. If it is * pass-by-reference then we have to add the estimated size of the * value itself, plus palloc overhead. */ if (!get_typbyval(aggtranstype)) { - int32 aggtranstypmod; int32 avgwidth; - /* - * If transition state is of same type as first input, assume it's - * the same typmod (same width) as well. This works for cases - * like MAX/MIN and is probably somewhat reasonable otherwise. - */ - if (numArguments > 0 && aggtranstype == inputTypes[0]) - aggtranstypmod = exprTypmod((Node *) linitial(aggref->args)); + /* Use average width if aggregate definition gave one */ + if (aggtransspace > 0) + avgwidth = aggtransspace; else - aggtranstypmod = -1; + { + /* + * If transition state is of same type as first aggregated + * input, assume it's the same typmod (same width) as well. + * This works for cases like MAX/MIN and is probably somewhat + * reasonable otherwise. + */ + int numdirectargs = list_length(aggref->aggdirectargs); + int32 aggtranstypmod; - avgwidth = get_typavgwidth(aggtranstype, aggtranstypmod); - avgwidth = MAXALIGN(avgwidth); + if (numArguments > numdirectargs && + aggtranstype == inputTypes[numdirectargs]) + aggtranstypmod = exprTypmod((Node *) linitial(aggref->args)); + else + aggtranstypmod = -1; + + avgwidth = get_typavgwidth(aggtranstype, aggtranstypmod); + } + avgwidth = MAXALIGN(avgwidth); costs->transitionSpace += avgwidth + 2 * sizeof(void *); } else if (aggtranstype == INTERNALOID) @@ -553,25 +579,24 @@ count_agg_clauses_walker(Node *node, count_agg_clauses_context *context) /* * INTERNAL transition type is a special case: although INTERNAL * is pass-by-value, it's almost certainly being used as a pointer - * to some large data structure. We assume usage of + * to some large data structure. The aggregate definition can + * provide an estimate of the size. If it doesn't, then we assume * ALLOCSET_DEFAULT_INITSIZE, which is a good guess if the data is * being kept in a private memory context, as is done by * array_agg() for instance. */ - costs->transitionSpace += ALLOCSET_DEFAULT_INITSIZE; + if (aggtransspace > 0) + costs->transitionSpace += aggtransspace; + else + costs->transitionSpace += ALLOCSET_DEFAULT_INITSIZE; } /* - * Complain if the aggregate's arguments contain any aggregates; - * nested agg functions are semantically nonsensical. - */ - if (contain_agg_clause((Node *) aggref->args)) - ereport(ERROR, - (errcode(ERRCODE_GROUPING_ERROR), - errmsg("aggregate function calls cannot be nested"))); - - /* - * Having checked that, we need not recurse into the argument. + * We assume that the parser checked that there are no aggregates (of + * this level anyway) in the aggregated arguments, direct arguments, + * or filter clause. Hence, we need not recurse into any of them. (If + * either the parser or the planner screws up on this point, the + * executor will still catch it; see ExecInitExpr.) */ return false; } @@ -596,7 +621,7 @@ count_agg_clauses_walker(Node *node, count_agg_clauses_context *context) bool contain_window_function(Node *clause) { - return checkExprHasWindowFuncs(clause); + return contain_windowfuncs(clause); } /* @@ -636,16 +661,10 @@ find_window_functions_walker(Node *node, WindowFuncLists *lists) lists->numWindowFuncs++; /* - * Complain if the window function's arguments contain window - * functions - */ - if (contain_window_function((Node *) wfunc->args)) - ereport(ERROR, - (errcode(ERRCODE_WINDOWING_ERROR), - errmsg("window function calls cannot be nested"))); - - /* - * Having checked that, we need not recurse into the argument. + * We assume that the parser checked that there are no window + * functions in the arguments or filter clause. Hence, we need not + * recurse into them. (If either the parser or the planner screws up + * on this point, the executor will still catch it; see ExecInitExpr.) */ return false; } @@ -661,10 +680,12 @@ find_window_functions_walker(Node *node, WindowFuncLists *lists) /* * expression_returns_set_rows - * Estimate the number of rows in a set result. + * Estimate the number of rows returned by a set-returning expression. + * The result is 1 if there are no set-returning functions. * * We use the product of the rowcount estimates of all the functions in - * the given tree. The result is 1 if there are no set-returning functions. + * the given tree (this corresponds to the behavior of ExecMakeFunctionResult + * for nested set-returning functions). * * Note: keep this in sync with expression_returns_set() in nodes/nodeFuncs.c. */ @@ -674,7 +695,7 @@ expression_returns_set_rows(Node *clause) double result = 1; (void) expression_returns_set_rows_walker(clause, &result); - return result; + return clamp_row_est(result); } static bool @@ -736,6 +757,40 @@ expression_returns_set_rows_walker(Node *node, double *count) (void *) count); } +/* + * tlist_returns_set_rows + * Estimate the number of rows returned by a set-returning targetlist. + * The result is 1 if there are no set-returning functions. + * + * Here, the result is the largest rowcount estimate of any of the tlist's + * expressions, not the product as you would get from naively applying + * expression_returns_set_rows() to the whole tlist. The behavior actually + * implemented by ExecTargetList produces a number of rows equal to the least + * common multiple of the expression rowcounts, so that the product would be + * a worst-case estimate that is typically not realistic. Taking the max as + * we do here is a best-case estimate that might not be realistic either, + * but it's probably closer for typical usages. We don't try to compute the + * actual LCM because we're working with very approximate estimates, so their + * LCM would be unduly noisy. + */ +double +tlist_returns_set_rows(List *tlist) +{ + double result = 1; + ListCell *lc; + + foreach(lc, tlist) + { + TargetEntry *tle = (TargetEntry *) lfirst(lc); + double colresult; + + colresult = expression_returns_set_rows((Node *) tle->expr); + if (result < colresult) + result = colresult; + } + return result; +} + /***************************************************************************** * Subplan clause manipulation @@ -780,12 +835,12 @@ contain_subplans_walker(Node *node, void *context) * Recursively search for mutable functions within a clause. * * Returns true if any mutable function (or operator implemented by a - * mutable function) is found. This test is needed so that we don't + * mutable function) is found. This test is needed so that we don't * mistakenly think that something like "WHERE random() < 0.5" can be treated * as a constant qualification. * - * XXX we do not examine sub-selects to see if they contain uses of - * mutable functions. It's not real clear if that is correct or not... + * We will recursively look into Query nodes (i.e., SubLink sub-selects) + * but not into SubPlans. See comments for contain_volatile_functions(). */ bool contain_mutable_functions(Node *clause) @@ -882,6 +937,13 @@ contain_mutable_functions_walker(Node *node, void *context) } /* else fall through to check args */ } + else if (IsA(node, Query)) + { + /* Recurse into subselects */ + return query_tree_walker((Query *) node, + contain_mutable_functions_walker, + context, 0); + } return expression_tree_walker(node, contain_mutable_functions_walker, context); } @@ -896,11 +958,18 @@ contain_mutable_functions_walker(Node *node, void *context) * Recursively search for volatile functions within a clause. * * Returns true if any volatile function (or operator implemented by a - * volatile function) is found. This test prevents invalid conversions - * of volatile expressions into indexscan quals. + * volatile function) is found. This test prevents, for example, + * invalid conversions of volatile expressions into indexscan quals. * - * XXX we do not examine sub-selects to see if they contain uses of - * volatile functions. It's not real clear if that is correct or not... + * We will recursively look into Query nodes (i.e., SubLink sub-selects) + * but not into SubPlans. This is a bit odd, but intentional. If we are + * looking at a SubLink, we are probably deciding whether a query tree + * transformation is safe, and a contained sub-select should affect that; + * for example, duplicating a sub-select containing a volatile function + * would be bad. However, once we've got to the stage of having SubPlans, + * subsequent planning need not consider volatility within those, since + * the executor won't change its evaluation rules for a SubPlan based on + * volatility. */ bool contain_volatile_functions(Node *clause) @@ -908,6 +977,19 @@ contain_volatile_functions(Node *clause) return contain_volatile_functions_walker(clause, NULL); } +bool +contain_volatile_functions_not_nextval(Node *clause) +{ + return contain_volatile_functions_not_nextval_walker(clause, NULL); +} + +/* + * General purpose code for checking expression volatility. + * + * Special purpose code for use in COPY is almost identical to this, + * so any changes here may also be needed in other contain_volatile... + * functions. + */ static bool contain_volatile_functions_walker(Node *node, void *context) { @@ -998,10 +1080,118 @@ contain_volatile_functions_walker(Node *node, void *context) } /* else fall through to check args */ } + else if (IsA(node, Query)) + { + /* Recurse into subselects */ + return query_tree_walker((Query *) node, + contain_volatile_functions_walker, + context, 0); + } return expression_tree_walker(node, contain_volatile_functions_walker, context); } +/* + * Special purpose version of contain_volatile_functions for use in COPY + */ +static bool +contain_volatile_functions_not_nextval_walker(Node *node, void *context) +{ + if (node == NULL) + return false; + if (IsA(node, FuncExpr)) + { + FuncExpr *expr = (FuncExpr *) node; + + /* + * For this case only, we want to ignore the volatility of the + * nextval() function, since some callers want this. + */ + if (expr->funcid != F_NEXTVAL_OID && + func_volatile(expr->funcid) == PROVOLATILE_VOLATILE) + return true; + /* else fall through to check args */ + } + else if (IsA(node, OpExpr)) + { + OpExpr *expr = (OpExpr *) node; + + set_opfuncid(expr); + if (func_volatile(expr->opfuncid) == PROVOLATILE_VOLATILE) + return true; + /* else fall through to check args */ + } + else if (IsA(node, DistinctExpr)) + { + DistinctExpr *expr = (DistinctExpr *) node; + + set_opfuncid((OpExpr *) expr); /* rely on struct equivalence */ + if (func_volatile(expr->opfuncid) == PROVOLATILE_VOLATILE) + return true; + /* else fall through to check args */ + } + else if (IsA(node, NullIfExpr)) + { + NullIfExpr *expr = (NullIfExpr *) node; + + set_opfuncid((OpExpr *) expr); /* rely on struct equivalence */ + if (func_volatile(expr->opfuncid) == PROVOLATILE_VOLATILE) + return true; + /* else fall through to check args */ + } + else if (IsA(node, ScalarArrayOpExpr)) + { + ScalarArrayOpExpr *expr = (ScalarArrayOpExpr *) node; + + set_sa_opfuncid(expr); + if (func_volatile(expr->opfuncid) == PROVOLATILE_VOLATILE) + return true; + /* else fall through to check args */ + } + else if (IsA(node, CoerceViaIO)) + { + CoerceViaIO *expr = (CoerceViaIO *) node; + Oid iofunc; + Oid typioparam; + bool typisvarlena; + + /* check the result type's input function */ + getTypeInputInfo(expr->resulttype, + &iofunc, &typioparam); + if (func_volatile(iofunc) == PROVOLATILE_VOLATILE) + return true; + /* check the input type's output function */ + getTypeOutputInfo(exprType((Node *) expr->arg), + &iofunc, &typisvarlena); + if (func_volatile(iofunc) == PROVOLATILE_VOLATILE) + return true; + /* else fall through to check args */ + } + else if (IsA(node, ArrayCoerceExpr)) + { + ArrayCoerceExpr *expr = (ArrayCoerceExpr *) node; + + if (OidIsValid(expr->elemfuncid) && + func_volatile(expr->elemfuncid) == PROVOLATILE_VOLATILE) + return true; + /* else fall through to check args */ + } + else if (IsA(node, RowCompareExpr)) + { + /* RowCompare probably can't have volatile ops, but check anyway */ + RowCompareExpr *rcexpr = (RowCompareExpr *) node; + ListCell *opid; + + foreach(opid, rcexpr->opnos) + { + if (op_volatile(lfirst_oid(opid)) == PROVOLATILE_VOLATILE) + return true; + } + /* else fall through to check args */ + } + return expression_tree_walker(node, contain_volatile_functions_not_nextval_walker, + context); +} /***************************************************************************** * Check clauses for nonstrict functions @@ -1017,7 +1207,7 @@ contain_volatile_functions_walker(Node *node, void *context) * The idea here is that the caller has verified that the expression contains * one or more Var or Param nodes (as appropriate for the caller's need), and * now wishes to prove that the expression result will be NULL if any of these - * inputs is NULL. If we return false, then the proof succeeded. + * inputs is NULL. If we return false, then the proof succeeded. */ bool contain_nonstrict_functions(Node *clause) @@ -1136,7 +1326,7 @@ contain_nonstrict_functions_walker(Node *node, void *context) * Recursively search for leaky functions within a clause. * * Returns true if any function call with side-effect may be present in the - * clause. Qualifiers from outside the a security_barrier view should not + * clause. Qualifiers from outside the a security_barrier view should not * be pushed down into the view, lest the contents of tuples intended to be * filtered out be revealed via side effects. */ @@ -1275,7 +1465,7 @@ contain_leaky_functions_walker(Node *node, void *context) * * Returns the set of all Relids that are referenced in the clause in such * a way that the clause cannot possibly return TRUE if any of these Relids - * is an all-NULL row. (It is OK to err on the side of conservatism; hence + * is an all-NULL row. (It is OK to err on the side of conservatism; hence * the analysis here is simplistic.) * * The semantics here are subtly different from contain_nonstrict_functions: @@ -1381,7 +1571,7 @@ find_nonnullable_rels_walker(Node *node, bool top_level) * could be FALSE (hence not NULL). However, if *all* the * arms produce NULL then the result is NULL, so we can take * the intersection of the sets of nonnullable rels, just as - * for OR. Fall through to share code. + * for OR. Fall through to share code. */ /* FALL THRU */ case OR_EXPR: @@ -1589,7 +1779,7 @@ find_nonnullable_vars_walker(Node *node, bool top_level) * could be FALSE (hence not NULL). However, if *all* the * arms produce NULL then the result is NULL, so we can take * the intersection of the sets of nonnullable vars, just as - * for OR. Fall through to share code. + * for OR. Fall through to share code. */ /* FALL THRU */ case OR_EXPR: @@ -1859,7 +2049,7 @@ is_strict_saop(ScalarArrayOpExpr *expr, bool falseOK) * variables of the current query level and no uses of volatile functions. * Such an expr is not necessarily a true constant: it can still contain * Params and outer-level Vars, not to mention functions whose results - * may vary from one statement to the next. However, the expr's value + * may vary from one statement to the next. However, the expr's value * will be constant over any one scan of the current query, so it can be * used as, eg, an indexscan key. * @@ -2022,103 +2212,6 @@ CommuteRowCompareExpr(RowCompareExpr *clause) } /* - * strip_implicit_coercions: remove implicit coercions at top level of tree - * - * Note: there isn't any useful thing we can do with a RowExpr here, so - * just return it unchanged, even if it's marked as an implicit coercion. - */ -Node * -strip_implicit_coercions(Node *node) -{ - if (node == NULL) - return NULL; - if (IsA(node, FuncExpr)) - { - FuncExpr *f = (FuncExpr *) node; - - if (f->funcformat == COERCE_IMPLICIT_CAST) - return strip_implicit_coercions(linitial(f->args)); - } - else if (IsA(node, RelabelType)) - { - RelabelType *r = (RelabelType *) node; - - if (r->relabelformat == COERCE_IMPLICIT_CAST) - return strip_implicit_coercions((Node *) r->arg); - } - else if (IsA(node, CoerceViaIO)) - { - CoerceViaIO *c = (CoerceViaIO *) node; - - if (c->coerceformat == COERCE_IMPLICIT_CAST) - return strip_implicit_coercions((Node *) c->arg); - } - else if (IsA(node, ArrayCoerceExpr)) - { - ArrayCoerceExpr *c = (ArrayCoerceExpr *) node; - - if (c->coerceformat == COERCE_IMPLICIT_CAST) - return strip_implicit_coercions((Node *) c->arg); - } - else if (IsA(node, ConvertRowtypeExpr)) - { - ConvertRowtypeExpr *c = (ConvertRowtypeExpr *) node; - - if (c->convertformat == COERCE_IMPLICIT_CAST) - return strip_implicit_coercions((Node *) c->arg); - } - else if (IsA(node, CoerceToDomain)) - { - CoerceToDomain *c = (CoerceToDomain *) node; - - if (c->coercionformat == COERCE_IMPLICIT_CAST) - return strip_implicit_coercions((Node *) c->arg); - } - return node; -} - -/* - * set_coercionform_dontcare: set all CoercionForm fields to COERCE_DONTCARE - * - * This is used to make index expressions and index predicates more easily - * comparable to clauses of queries. CoercionForm is not semantically - * significant (for cases where it does matter, the significant info is - * coded into the coercion function arguments) so we can ignore it during - * comparisons. Thus, for example, an index on "foo::int4" can match an - * implicit coercion to int4. - * - * Caution: the passed expression tree is modified in-place. - */ -void -set_coercionform_dontcare(Node *node) -{ - (void) set_coercionform_dontcare_walker(node, NULL); -} - -static bool -set_coercionform_dontcare_walker(Node *node, void *context) -{ - if (node == NULL) - return false; - if (IsA(node, FuncExpr)) - ((FuncExpr *) node)->funcformat = COERCE_DONTCARE; - else if (IsA(node, RelabelType)) - ((RelabelType *) node)->relabelformat = COERCE_DONTCARE; - else if (IsA(node, CoerceViaIO)) - ((CoerceViaIO *) node)->coerceformat = COERCE_DONTCARE; - else if (IsA(node, ArrayCoerceExpr)) - ((ArrayCoerceExpr *) node)->coerceformat = COERCE_DONTCARE; - else if (IsA(node, ConvertRowtypeExpr)) - ((ConvertRowtypeExpr *) node)->convertformat = COERCE_DONTCARE; - else if (IsA(node, RowExpr)) - ((RowExpr *) node)->row_format = COERCE_DONTCARE; - else if (IsA(node, CoerceToDomain)) - ((CoerceToDomain *) node)->coercionformat = COERCE_DONTCARE; - return expression_tree_walker(node, set_coercionform_dontcare_walker, - context); -} - -/* * Helper for eval_const_expressions: check that datatype of an attribute * is still what it was when the expression was parsed. This is needed to * guard against improper simplification after ALTER COLUMN TYPE. (XXX we @@ -2162,7 +2255,7 @@ rowtype_field_matches(Oid rowtypeid, int fieldnum, * expression tree, for example "2 + 2" => "4". More interestingly, * we can reduce certain boolean expressions even when they contain * non-constant subexpressions: "x OR true" => "true" no matter what - * the subexpression x is. (XXX We assume that no such subexpression + * the subexpression x is. (XXX We assume that no such subexpression * will have important side-effects, which is not necessarily a good * assumption in the presence of user-defined functions; do we need a * pg_proc flag that prevents discarding the execution of a function?) @@ -2175,7 +2268,7 @@ rowtype_field_matches(Oid rowtypeid, int fieldnum, * * Whenever a function is eliminated from the expression by means of * constant-expression evaluation or inlining, we add the function to - * root->glob->invalItems. This ensures the plan is known to depend on + * root->glob->invalItems. This ensures the plan is known to depend on * such functions, even though they aren't referenced anymore. * * We assume that the tree has already been type-checked and contains @@ -2299,6 +2392,56 @@ eval_const_expressions_mutator(Node *node, */ return (Node *) copyObject(param); } + case T_WindowFunc: + { + WindowFunc *expr = (WindowFunc *) node; + Oid funcid = expr->winfnoid; + List *args; + Expr *aggfilter; + HeapTuple func_tuple; + WindowFunc *newexpr; + + /* + * We can't really simplify a WindowFunc node, but we mustn't + * just fall through to the default processing, because we + * have to apply expand_function_arguments to its argument + * list. That takes care of inserting default arguments and + * expanding named-argument notation. + */ + func_tuple = SearchSysCache1(PROCOID, ObjectIdGetDatum(funcid)); + if (!HeapTupleIsValid(func_tuple)) + elog(ERROR, "cache lookup failed for function %u", funcid); + + args = expand_function_arguments(expr->args, expr->wintype, + func_tuple); + + ReleaseSysCache(func_tuple); + + /* Now, recursively simplify the args (which are a List) */ + args = (List *) + expression_tree_mutator((Node *) args, + eval_const_expressions_mutator, + (void *) context); + /* ... and the filter expression, which isn't */ + aggfilter = (Expr *) + eval_const_expressions_mutator((Node *) expr->aggfilter, + context); + + /* And build the replacement WindowFunc node */ + newexpr = makeNode(WindowFunc); + newexpr->winfnoid = expr->winfnoid; + newexpr->wintype = expr->wintype; + newexpr->wincollid = expr->wincollid; + newexpr->inputcollid = expr->inputcollid; + newexpr->args = args; + newexpr->aggfilter = aggfilter; + newexpr->winref = expr->winref; + newexpr->winstar = expr->winstar; + newexpr->winagg = expr->winagg; + newexpr->location = expr->location; + + return (Node *) newexpr; + } case T_FuncExpr: { FuncExpr *expr = (FuncExpr *) node; @@ -2308,7 +2451,7 @@ eval_const_expressions_mutator(Node *node, /* * Code for op/func reduction is pretty bulky, so split it out - * as a separate function. Note: exprTypmod normally returns + * as a separate function. Note: exprTypmod normally returns * -1 for a FuncExpr, but not when the node is recognizably a * length coercion; we want to preserve the typmod in the * eventual Const if so. @@ -2319,6 +2462,7 @@ eval_const_expressions_mutator(Node *node, expr->funccollid, expr->inputcollid, &args, + expr->funcvariadic, true, true, context); @@ -2335,6 +2479,7 @@ eval_const_expressions_mutator(Node *node, newexpr->funcid = expr->funcid; newexpr->funcresulttype = expr->funcresulttype; newexpr->funcretset = expr->funcretset; + newexpr->funcvariadic = expr->funcvariadic; newexpr->funcformat = expr->funcformat; newexpr->funccollid = expr->funccollid; newexpr->inputcollid = expr->inputcollid; @@ -2350,7 +2495,7 @@ eval_const_expressions_mutator(Node *node, OpExpr *newexpr; /* - * Need to get OID of underlying function. Okay to scribble + * Need to get OID of underlying function. Okay to scribble * on input to this extent. */ set_opfuncid(expr); @@ -2364,6 +2509,7 @@ eval_const_expressions_mutator(Node *node, expr->opcollid, expr->inputcollid, &args, + false, true, true, context); @@ -2452,7 +2598,7 @@ eval_const_expressions_mutator(Node *node, /* (NOT okay to try to inline it, though!) */ /* - * Need to get OID of underlying function. Okay to + * Need to get OID of underlying function. Okay to * scribble on input to this extent. */ set_opfuncid((OpExpr *) expr); /* rely on struct @@ -2469,6 +2615,7 @@ eval_const_expressions_mutator(Node *node, &args, false, false, + false, context); if (simple) /* successfully simplified it */ { @@ -2670,6 +2817,7 @@ eval_const_expressions_mutator(Node *node, InvalidOid, InvalidOid, &args, + false, true, true, context); @@ -2702,6 +2850,7 @@ eval_const_expressions_mutator(Node *node, InvalidOid, &args, false, + false, true, context); if (simple) /* successfully simplified input fn */ @@ -2794,7 +2943,7 @@ eval_const_expressions_mutator(Node *node, relabel->resulttype = exprType(arg); relabel->resulttypmod = exprTypmod(arg); relabel->resultcollid = collate->collOid; - relabel->relabelformat = COERCE_DONTCARE; + relabel->relabelformat = COERCE_IMPLICIT_CAST; relabel->location = collate->location; /* Don't create stacked RelabelTypes */ @@ -2814,13 +2963,13 @@ eval_const_expressions_mutator(Node *node, * TRUE: drop all remaining alternatives * If the first non-FALSE alternative is a constant TRUE, * we can simplify the entire CASE to that alternative's - * expression. If there are no non-FALSE alternatives, + * expression. If there are no non-FALSE alternatives, * we simplify the entire CASE to the default result (ELSE). * * If we have a simple-form CASE with constant test * expression, we substitute the constant value for contained * CaseTestExpr placeholder nodes, so that we have the - * opportunity to reduce constant test conditions. For + * opportunity to reduce constant test conditions. For * example this allows * CASE 0 WHEN 0 THEN 1 ELSE 1/0 END * to reduce to 1 rather than drawing a divide-by-0 error. @@ -3042,7 +3191,7 @@ eval_const_expressions_mutator(Node *node, { /* * We can optimize field selection from a whole-row Var into a - * simple Var. (This case won't be generated directly by the + * simple Var. (This case won't be generated directly by the * parser, because ParseComplexProjection short-circuits it. * But it can arise while simplifying functions.) Also, we * can optimize field selection from a RowExpr construct. @@ -3300,7 +3449,7 @@ simplify_or_arguments(List *args, /* * Since the parser considers OR to be a binary operator, long OR lists * become deeply nested expressions. We must flatten these into long - * argument lists of a single OR operator. To avoid blowing out the stack + * argument lists of a single OR operator. To avoid blowing out the stack * with recursion of eval_const_expressions, we resort to some tenseness * here: we keep a list of not-yet-processed inputs, and handle flattening * of nested ORs by prepending to the to-do list instead of recursing. @@ -3348,7 +3497,7 @@ simplify_or_arguments(List *args, } /* - * OK, we have a const-simplified non-OR argument. Process it per + * OK, we have a const-simplified non-OR argument. Process it per * comments above. */ if (IsA(arg, Const)) @@ -3570,7 +3719,7 @@ simplify_boolean_equality(Oid opno, List *args) static Expr * simplify_function(Oid funcid, Oid result_type, int32 result_typmod, Oid result_collid, Oid input_collid, List **args_p, - bool process_args, bool allow_non_const, + bool funcvariadic, bool process_args, bool allow_non_const, eval_const_expressions_context *context) { List *args = *args_p; @@ -3583,7 +3732,7 @@ simplify_function(Oid funcid, Oid result_type, int32 result_typmod, * deliver a constant result, use a transform function to generate a * substitute node tree, or expand in-line the body of the function * definition (which only works for simple SQL-language functions, but - * that is a common case). Each case needs access to the function's + * that is a common case). Each case needs access to the function's * pg_proc tuple, so fetch it just once. * * Note: the allow_non_const flag suppresses both the second and third @@ -3614,13 +3763,14 @@ simplify_function(Oid funcid, Oid result_type, int32 result_typmod, /* Now attempt simplification of the function call proper. */ newexpr = evaluate_function(funcid, result_type, result_typmod, - result_collid, input_collid, args, + result_collid, input_collid, + args, funcvariadic, func_tuple, context); if (!newexpr && allow_non_const && OidIsValid(func_form->protransform)) { /* - * Build a dummy FuncExpr node containing the simplified arg list. We + * Build a dummy FuncExpr node containing the simplified arg list. We * use this approach to present a uniform interface to the transform * function regardless of how the function is actually being invoked. */ @@ -3630,7 +3780,8 @@ simplify_function(Oid funcid, Oid result_type, int32 result_typmod, fexpr.funcid = funcid; fexpr.funcresulttype = result_type; fexpr.funcretset = func_form->proretset; - fexpr.funcformat = COERCE_DONTCARE; + fexpr.funcvariadic = funcvariadic; + fexpr.funcformat = COERCE_EXPLICIT_CALL; fexpr.funccollid = result_collid; fexpr.inputcollid = input_collid; fexpr.args = args; @@ -3643,7 +3794,7 @@ simplify_function(Oid funcid, Oid result_type, int32 result_typmod, if (!newexpr && allow_non_const) newexpr = inline_function(funcid, result_type, result_collid, - input_collid, args, + input_collid, args, funcvariadic, func_tuple, context); ReleaseSysCache(func_tuple); @@ -3827,7 +3978,7 @@ fetch_function_defaults(HeapTuple func_tuple) * * It is possible for some of the defaulted arguments to be polymorphic; * therefore we can't assume that the default expressions have the correct - * data types already. We have to re-resolve polymorphics and do coercion + * data types already. We have to re-resolve polymorphics and do coercion * just like the parser did. * * This should be a no-op if there are no polymorphic arguments, @@ -3883,6 +4034,7 @@ recheck_cast_function_args(List *args, Oid result_type, HeapTuple func_tuple) static Expr * evaluate_function(Oid funcid, Oid result_type, int32 result_typmod, Oid result_collid, Oid input_collid, List *args, + bool funcvariadic, HeapTuple func_tuple, eval_const_expressions_context *context) { @@ -3964,7 +4116,8 @@ evaluate_function(Oid funcid, Oid result_type, int32 result_typmod, newexpr->funcid = funcid; newexpr->funcresulttype = result_type; newexpr->funcretset = false; - newexpr->funcformat = COERCE_DONTCARE; /* doesn't matter */ + newexpr->funcvariadic = funcvariadic; + newexpr->funcformat = COERCE_EXPLICIT_CALL; /* doesn't matter */ newexpr->funccollid = result_collid; /* doesn't matter */ newexpr->inputcollid = input_collid; newexpr->args = args; @@ -3988,7 +4141,7 @@ evaluate_function(Oid funcid, Oid result_type, int32 result_typmod, * do not re-expand them. Also, if a parameter is used more than once * in the SQL-function body, we require it not to contain any volatile * functions (volatiles might deliver inconsistent answers) nor to be - * unreasonably expensive to evaluate. The expensiveness check not only + * unreasonably expensive to evaluate. The expensiveness check not only * prevents us from doing multiple evaluations of an expensive parameter * at runtime, but is a safety value to limit growth of an expression due * to repeated inlining. @@ -4006,6 +4159,7 @@ evaluate_function(Oid funcid, Oid result_type, int32 result_typmod, static Expr * inline_function(Oid funcid, Oid result_type, Oid result_collid, Oid input_collid, List *args, + bool funcvariadic, HeapTuple func_tuple, eval_const_expressions_context *context) { @@ -4030,7 +4184,7 @@ inline_function(Oid funcid, Oid result_type, Oid result_collid, /* * Forget it if the function is not SQL-language or has other showstopper - * properties. (The nargs check is just paranoia.) + * properties. (The nargs check is just paranoia.) */ if (funcform->prolang != SQLlanguageId || funcform->prosecdef || @@ -4094,7 +4248,8 @@ inline_function(Oid funcid, Oid result_type, Oid result_collid, fexpr->funcid = funcid; fexpr->funcresulttype = result_type; fexpr->funcretset = false; - fexpr->funcformat = COERCE_DONTCARE; /* doesn't matter */ + fexpr->funcvariadic = funcvariadic; + fexpr->funcformat = COERCE_EXPLICIT_CALL; /* doesn't matter */ fexpr->funccollid = result_collid; /* doesn't matter */ fexpr->inputcollid = input_collid; fexpr->args = args; @@ -4107,7 +4262,7 @@ inline_function(Oid funcid, Oid result_type, Oid result_collid, /* * We just do parsing and parse analysis, not rewriting, because rewriting * will not affect table-free-SELECT-only queries, which is all that we - * care about. Also, we can punt as soon as we detect more than one + * care about. Also, we can punt as soon as we detect more than one * command in the function body. */ raw_parsetree_list = pg_parse_query(src); @@ -4149,7 +4304,7 @@ inline_function(Oid funcid, Oid result_type, Oid result_collid, /* * Make sure the function (still) returns what it's declared to. This * will raise an error if wrong, but that's okay since the function would - * fail at runtime anyway. Note that check_sql_fn_retval will also insert + * fail at runtime anyway. Note that check_sql_fn_retval will also insert * a RelabelType if needed to make the tlist expression match the declared * type of the function. * @@ -4194,7 +4349,7 @@ inline_function(Oid funcid, Oid result_type, Oid result_collid, /* * We may be able to do it; there are still checks on parameter usage to * make, but those are most easily done in combination with the actual - * substitution of the inputs. So start building expression with inputs + * substitution of the inputs. So start building expression with inputs * substituted. */ usecounts = (int *) palloc0(funcform->pronargs * sizeof(int)); @@ -4394,7 +4549,7 @@ evaluate_expr(Expr *expr, Oid result_type, int32 result_typmod, fix_opfuncids((Node *) expr); /* - * Prepare expr for execution. (Note: we can't use ExecPrepareExpr + * Prepare expr for execution. (Note: we can't use ExecPrepareExpr * because it'd result in recursively invoking eval_const_expressions.) */ exprstate = ExecInitExpr(expr, NULL); @@ -4460,6 +4615,7 @@ evaluate_expr(Expr *expr, Oid result_type, int32 result_typmod, Query * inline_set_returning_function(PlannerInfo *root, RangeTblEntry *rte) { + RangeTblFunction *rtfunc; FuncExpr *fexpr; Oid func_oid; HeapTuple func_tuple; @@ -4488,10 +4644,19 @@ inline_set_returning_function(PlannerInfo *root, RangeTblEntry *rte) */ check_stack_depth(); - /* Fail if FROM item isn't a simple FuncExpr */ - fexpr = (FuncExpr *) rte->funcexpr; - if (fexpr == NULL || !IsA(fexpr, FuncExpr)) + /* Fail if the RTE has ORDINALITY - we don't implement that here. */ + if (rte->funcordinality) + return NULL; + + /* Fail if RTE isn't a single, simple FuncExpr */ + if (list_length(rte->functions) != 1) return NULL; + rtfunc = (RangeTblFunction *) linitial(rte->functions); + + if (!IsA(rtfunc->funcexpr, FuncExpr)) + return NULL; + fexpr = (FuncExpr *) rtfunc->funcexpr; + func_oid = fexpr->funcid; /* @@ -4506,7 +4671,7 @@ inline_set_returning_function(PlannerInfo *root, RangeTblEntry *rte) * Refuse to inline if the arguments contain any volatile functions or * sub-selects. Volatile functions are rejected because inlining may * result in the arguments being evaluated multiple times, risking a - * change in behavior. Sub-selects are rejected partly for implementation + * change in behavior. Sub-selects are rejected partly for implementation * reasons (pushing them down another level might change their behavior) * and partly because they're likely to be expensive and so multiple * evaluation would be bad. @@ -4533,7 +4698,7 @@ inline_set_returning_function(PlannerInfo *root, RangeTblEntry *rte) /* * Forget it if the function is not SQL-language or has other showstopper - * properties. In particular it mustn't be declared STRICT, since we + * properties. In particular it mustn't be declared STRICT, since we * couldn't enforce that. It also mustn't be VOLATILE, because that is * supposed to cause it to be executed with its own snapshot, rather than * sharing the snapshot of the calling query. (Rechecking proretset is @@ -4563,9 +4728,9 @@ inline_set_returning_function(PlannerInfo *root, RangeTblEntry *rte) /* * When we call eval_const_expressions below, it might try to add items to - * root->glob->invalItems. Since it is running in the temp context, those + * root->glob->invalItems. Since it is running in the temp context, those * items will be in that context, and will need to be copied out if we're - * successful. Temporarily reset the list so that we can keep those items + * successful. Temporarily reset the list so that we can keep those items * separate from the pre-existing list contents. */ saveInvalItems = root->glob->invalItems; @@ -4595,7 +4760,7 @@ inline_set_returning_function(PlannerInfo *root, RangeTblEntry *rte) /* * Run eval_const_expressions on the function call. This is necessary to * ensure that named-argument notation is converted to positional notation - * and any default arguments are inserted. It's a bit of overkill for the + * and any default arguments are inserted. It's a bit of overkill for the * arguments, since they'll get processed again later, but no harm will be * done. */ @@ -4647,7 +4812,7 @@ inline_set_returning_function(PlannerInfo *root, RangeTblEntry *rte) /* * Make sure the function (still) returns what it's declared to. This * will raise an error if wrong, but that's okay since the function would - * fail at runtime anyway. Note that check_sql_fn_retval will also insert + * fail at runtime anyway. Note that check_sql_fn_retval will also insert * RelabelType(s) and/or NULL columns if needed to make the tlist * expression(s) match the declared type of the function. * @@ -4680,7 +4845,8 @@ inline_set_returning_function(PlannerInfo *root, RangeTblEntry *rte) */ if (fexpr->funcresulttype == RECORDOID && get_func_result_type(func_oid, NULL, NULL) == TYPEFUNC_RECORD && - !tlist_matches_coltypelist(querytree->targetList, rte->funccoltypes)) + !tlist_matches_coltypelist(querytree->targetList, + rtfunc->funccoltypes)) goto fail; /* diff --git a/src/backend/optimizer/util/joininfo.c b/src/backend/optimizer/util/joininfo.c index 20d57c5140..0418946d71 100644 --- a/src/backend/optimizer/util/joininfo.c +++ b/src/backend/optimizer/util/joininfo.c @@ -3,7 +3,7 @@ * joininfo.c * joininfo list manipulation routines * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -83,7 +83,7 @@ have_relevant_joinclause(PlannerInfo *root, * Add 'restrictinfo' to the joininfo list of each relation it requires. * * Note that the same copy of the restrictinfo node is linked to by all the - * lists it is in. This allows us to exploit caching of information about + * lists it is in. This allows us to exploit caching of information about * the restriction clause (but we must be careful that the information does * not depend on context). * diff --git a/src/backend/optimizer/util/orclauses.c b/src/backend/optimizer/util/orclauses.c new file mode 100644 index 0000000000..9e954d0d35 --- /dev/null +++ b/src/backend/optimizer/util/orclauses.c @@ -0,0 +1,343 @@ +/*------------------------------------------------------------------------- + * + * orclauses.c + * Routines to extract restriction OR clauses from join OR clauses + * + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * + * IDENTIFICATION + * src/backend/optimizer/util/orclauses.c + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "optimizer/clauses.h" +#include "optimizer/cost.h" +#include "optimizer/orclauses.h" +#include "optimizer/restrictinfo.h" + + +static bool is_safe_restriction_clause_for(RestrictInfo *rinfo, RelOptInfo *rel); +static Expr *extract_or_clause(RestrictInfo *or_rinfo, RelOptInfo *rel); +static void consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel, + Expr *orclause, RestrictInfo *join_or_rinfo); + + +/* + * extract_restriction_or_clauses + * Examine join OR-of-AND clauses to see if any useful restriction OR + * clauses can be extracted. If so, add them to the query. + * + * Although a join clause must reference multiple relations overall, + * an OR of ANDs clause might contain sub-clauses that reference just one + * relation and can be used to build a restriction clause for that rel. + * For example consider + * WHERE ((a.x = 42 AND b.y = 43) OR (a.x = 44 AND b.z = 45)); + * We can transform this into + * WHERE ((a.x = 42 AND b.y = 43) OR (a.x = 44 AND b.z = 45)) + * AND (a.x = 42 OR a.x = 44) + * AND (b.y = 43 OR b.z = 45); + * which allows the latter clauses to be applied during the scans of a and b, + * perhaps as index qualifications, and in any case reducing the number of + * rows arriving at the join. In essence this is a partial transformation to + * CNF (AND of ORs format). It is not complete, however, because we do not + * unravel the original OR --- doing so would usually bloat the qualification + * expression to little gain. + * + * The added quals are partially redundant with the original OR, and therefore + * would cause the size of the joinrel to be underestimated when it is finally + * formed. (This would be true of a full transformation to CNF as well; the + * fault is not really in the transformation, but in clauselist_selectivity's + * inability to recognize redundant conditions.) We can compensate for this + * redundancy by changing the cached selectivity of the original OR clause, + * cancelling out the (valid) reduction in the estimated sizes of the base + * relations so that the estimated joinrel size remains the same. This is + * a MAJOR HACK: it depends on the fact that clause selectivities are cached + * and on the fact that the same RestrictInfo node will appear in every + * joininfo list that might be used when the joinrel is formed. + * And it doesn't work in cases where the size estimation is nonlinear + * (i.e., outer and IN joins). But it beats not doing anything. + * + * We examine each base relation to see if join clauses associated with it + * contain extractable restriction conditions. If so, add those conditions + * to the rel's baserestrictinfo and update the cached selectivities of the + * join clauses. Note that the same join clause will be examined afresh + * from the point of view of each baserel that participates in it, so its + * cached selectivity may get updated multiple times. + */ +void +extract_restriction_or_clauses(PlannerInfo *root) +{ + Index rti; + + /* Examine each baserel for potential join OR clauses */ + for (rti = 1; rti < root->simple_rel_array_size; rti++) + { + RelOptInfo *rel = root->simple_rel_array[rti]; + ListCell *lc; + + /* there may be empty slots corresponding to non-baserel RTEs */ + if (rel == NULL) + continue; + + Assert(rel->relid == rti); /* sanity check on array */ + + /* ignore RTEs that are "other rels" */ + if (rel->reloptkind != RELOPT_BASEREL) + continue; + + /* + * Find potentially interesting OR joinclauses. We can use any + * joinclause that is considered safe to move to this rel by the + * parameterized-path machinery, even though what we are going to do + * with it is not exactly a parameterized path. + * + * However, it seems best to ignore clauses that have been marked + * redundant (by setting norm_selec > 1). That likely can't happen + * for OR clauses, but let's be safe. + */ + foreach(lc, rel->joininfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + if (restriction_is_or_clause(rinfo) && + join_clause_is_movable_to(rinfo, rel) && + rinfo->norm_selec <= 1) + { + /* Try to extract a qual for this rel only */ + Expr *orclause = extract_or_clause(rinfo, rel); + + /* + * If successful, decide whether we want to use the clause, + * and insert it into the rel's restrictinfo list if so. + */ + if (orclause) + consider_new_or_clause(root, rel, orclause, rinfo); + } + } + } +} + +/* + * Is the given primitive (non-OR) RestrictInfo safe to move to the rel? + */ +static bool +is_safe_restriction_clause_for(RestrictInfo *rinfo, RelOptInfo *rel) +{ + /* + * We want clauses that mention the rel, and only the rel. So in + * particular pseudoconstant clauses can be rejected quickly. Then check + * the clause's Var membership. + */ + if (rinfo->pseudoconstant) + return false; + if (!bms_equal(rinfo->clause_relids, rel->relids)) + return false; + + /* We don't want extra evaluations of any volatile functions */ + if (contain_volatile_functions((Node *) rinfo->clause)) + return false; + + return true; +} + +/* + * Try to extract a restriction clause mentioning only "rel" from the given + * join OR-clause. + * + * We must be able to extract at least one qual for this rel from each of + * the arms of the OR, else we can't use it. + * + * Returns an OR clause (not a RestrictInfo!) pertaining to rel, or NULL + * if no OR clause could be extracted. + */ +static Expr * +extract_or_clause(RestrictInfo *or_rinfo, RelOptInfo *rel) +{ + List *clauselist = NIL; + ListCell *lc; + + /* + * Scan each arm of the input OR clause. Notice we descend into + * or_rinfo->orclause, which has RestrictInfo nodes embedded below the + * toplevel OR/AND structure. This is useful because we can use the info + * in those nodes to make is_safe_restriction_clause_for()'s checks + * cheaper. We'll strip those nodes from the returned tree, though, + * meaning that fresh ones will be built if the clause is accepted as a + * restriction clause. This might seem wasteful --- couldn't we re-use + * the existing RestrictInfos? But that'd require assuming that + * selectivity and other cached data is computed exactly the same way for + * a restriction clause as for a join clause, which seems undesirable. + */ + Assert(or_clause((Node *) or_rinfo->orclause)); + foreach(lc, ((BoolExpr *) or_rinfo->orclause)->args) + { + Node *orarg = (Node *) lfirst(lc); + List *subclauses = NIL; + + /* OR arguments should be ANDs or sub-RestrictInfos */ + if (and_clause(orarg)) + { + List *andargs = ((BoolExpr *) orarg)->args; + ListCell *lc2; + + foreach(lc2, andargs) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc2); + + Assert(IsA(rinfo, RestrictInfo)); + if (restriction_is_or_clause(rinfo)) + { + /* + * Recurse to deal with nested OR. Note we *must* recurse + * here, this isn't just overly-tense optimization: we + * have to descend far enough to find and strip all + * RestrictInfos in the expression. + */ + Expr *suborclause; + + suborclause = extract_or_clause(rinfo, rel); + if (suborclause) + subclauses = lappend(subclauses, suborclause); + } + else if (is_safe_restriction_clause_for(rinfo, rel)) + subclauses = lappend(subclauses, rinfo->clause); + } + } + else + { + Assert(IsA(orarg, RestrictInfo)); + Assert(!restriction_is_or_clause((RestrictInfo *) orarg)); + if (is_safe_restriction_clause_for((RestrictInfo *) orarg, rel)) + subclauses = lappend(subclauses, + ((RestrictInfo *) orarg)->clause); + } + + /* + * If nothing could be extracted from this arm, we can't do anything + * with this OR clause. + */ + if (subclauses == NIL) + return NULL; + + /* + * OK, add subclause(s) to the result OR. If we found more than one, + * we need an AND node. + */ + clauselist = lappend(clauselist, make_ands_explicit(subclauses)); + } + + /* + * If we got a restriction clause from every arm, wrap them up in an OR + * node. (In theory the OR node might be unnecessary, if there was only + * one arm --- but then the input OR node was also redundant.) + */ + if (clauselist != NIL) + return make_orclause(clauselist); + return NULL; +} + +/* + * Consider whether a successfully-extracted restriction OR clause is + * actually worth using. If so, add it to the planner's data structures, + * and adjust the original join clause (join_or_rinfo) to compensate. + */ +static void +consider_new_or_clause(PlannerInfo *root, RelOptInfo *rel, + Expr *orclause, RestrictInfo *join_or_rinfo) +{ + RestrictInfo *or_rinfo; + Selectivity or_selec, + orig_selec; + + /* + * Build a RestrictInfo from the new OR clause. We can assume it's valid + * as a base restriction clause. + */ + or_rinfo = make_restrictinfo(orclause, + true, + false, + false, + NULL, + NULL, + NULL); + + /* + * Estimate its selectivity. (We could have done this earlier, but doing + * it on the RestrictInfo representation allows the result to get cached, + * saving work later.) + */ + or_selec = clause_selectivity(root, (Node *) or_rinfo, + 0, JOIN_INNER, NULL); + + /* + * The clause is only worth adding to the query if it rejects a useful + * fraction of the base relation's rows; otherwise, it's just going to + * cause duplicate computation (since we will still have to check the + * original OR clause when the join is formed). Somewhat arbitrarily, we + * set the selectivity threshold at 0.9. + */ + if (or_selec > 0.9) + return; /* forget it */ + + /* + * OK, add it to the rel's restriction-clause list. + */ + rel->baserestrictinfo = lappend(rel->baserestrictinfo, or_rinfo); + + /* + * Adjust the original join OR clause's cached selectivity to compensate + * for the selectivity of the added (but redundant) lower-level qual. This + * should result in the join rel getting approximately the same rows + * estimate as it would have gotten without all these shenanigans. + * + * XXX major hack alert: this depends on the assumption that the + * selectivity will stay cached. + * + * XXX another major hack: we adjust only norm_selec, the cached + * selectivity for JOIN_INNER semantics, even though the join clause + * might've been an outer-join clause. This is partly because we can't + * easily identify the relevant SpecialJoinInfo here, and partly because + * the linearity assumption we're making would fail anyway. (If it is an + * outer-join clause, "rel" must be on the nullable side, else we'd not + * have gotten here. So the computation of the join size is going to be + * quite nonlinear with respect to the size of "rel", so it's not clear + * how we ought to adjust outer_selec even if we could compute its + * original value correctly.) + */ + if (or_selec > 0) + { + SpecialJoinInfo sjinfo; + + /* + * Make up a SpecialJoinInfo for JOIN_INNER semantics. (Compare + * approx_tuple_count() in costsize.c.) + */ + sjinfo.type = T_SpecialJoinInfo; + sjinfo.min_lefthand = bms_difference(join_or_rinfo->clause_relids, + rel->relids); + sjinfo.min_righthand = rel->relids; + sjinfo.syn_lefthand = sjinfo.min_lefthand; + sjinfo.syn_righthand = sjinfo.min_righthand; + sjinfo.jointype = JOIN_INNER; + /* we don't bother trying to make the remaining fields valid */ + sjinfo.lhs_strict = false; + sjinfo.delay_upper_joins = false; + sjinfo.join_quals = NIL; + + /* Compute inner-join size */ + orig_selec = clause_selectivity(root, (Node *) join_or_rinfo, + 0, JOIN_INNER, &sjinfo); + + /* And hack cached selectivity so join size remains the same */ + join_or_rinfo->norm_selec = orig_selec / or_selec; + /* ensure result stays in sane range, in particular not "redundant" */ + if (join_or_rinfo->norm_selec > 1) + join_or_rinfo->norm_selec = 1; + /* as explained above, we don't touch outer_selec */ + } +} diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index 13c77c8544..ec4bfdb49a 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -8,7 +8,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * Portions Copyright (c) 2012-2014, TransLattice, Inc. - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -50,7 +50,6 @@ typedef enum COSTS_DIFFERENT /* neither path dominates the other on cost */ } PathCostComparison; -static void add_parameterized_path(RelOptInfo *parent_rel, Path *new_path); static List *translate_sub_tlist(List *tlist, int relid); static bool query_is_distinct_for(Query *query, List *colnos, List *opids); static Oid distinct_col_search(int colno, List *colnos, List *opids); @@ -149,20 +148,30 @@ compare_fractional_path_costs(Path *path1, Path *path2, * * The fuzz_factor argument must be 1.0 plus delta, where delta is the * fraction of the smaller cost that is considered to be a significant - * difference. For example, fuzz_factor = 1.01 makes the fuzziness limit + * difference. For example, fuzz_factor = 1.01 makes the fuzziness limit * be 1% of the smaller cost. * * The two paths are said to have "equal" costs if both startup and total - * costs are fuzzily the same. Path1 is said to be better than path2 if + * costs are fuzzily the same. Path1 is said to be better than path2 if * it has fuzzily better startup cost and fuzzily no worse total cost, * or if it has fuzzily better total cost and fuzzily no worse startup cost. * Path2 is better than path1 if the reverse holds. Finally, if one path * is fuzzily better than the other on startup cost and fuzzily worse on * total cost, we just say that their costs are "different", since neither * dominates the other across the whole performance spectrum. + * + * If consider_startup is false, then we don't care about keeping paths with + * good startup cost, so we'll never return COSTS_DIFFERENT. + * + * This function also includes special hacks to support a policy enforced + * by its sole caller, add_path(): paths that have any parameterization + * cannot win comparisons on the grounds of having cheaper startup cost, + * since we deem only total cost to be of interest for a parameterized path. + * (Unparameterized paths are more common, so we check for this case last.) */ static PathCostComparison -compare_path_costs_fuzzily(Path *path1, Path *path2, double fuzz_factor) +compare_path_costs_fuzzily(Path *path1, Path *path2, double fuzz_factor, + bool consider_startup) { /* * Check total cost first since it's more likely to be different; many @@ -171,7 +180,9 @@ compare_path_costs_fuzzily(Path *path1, Path *path2, double fuzz_factor) if (path1->total_cost > path2->total_cost * fuzz_factor) { /* path1 fuzzily worse on total cost */ - if (path2->startup_cost > path1->startup_cost * fuzz_factor) + if (consider_startup && + path2->startup_cost > path1->startup_cost * fuzz_factor && + path1->param_info == NULL) { /* ... but path2 fuzzily worse on startup, so DIFFERENT */ return COSTS_DIFFERENT; @@ -182,7 +193,9 @@ compare_path_costs_fuzzily(Path *path1, Path *path2, double fuzz_factor) if (path2->total_cost > path1->total_cost * fuzz_factor) { /* path2 fuzzily worse on total cost */ - if (path1->startup_cost > path2->startup_cost * fuzz_factor) + if (consider_startup && + path1->startup_cost > path2->startup_cost * fuzz_factor && + path2->param_info == NULL) { /* ... but path1 fuzzily worse on startup, so DIFFERENT */ return COSTS_DIFFERENT; @@ -191,12 +204,15 @@ compare_path_costs_fuzzily(Path *path1, Path *path2, double fuzz_factor) return COSTS_BETTER1; } /* fuzzily the same on total cost */ - if (path1->startup_cost > path2->startup_cost * fuzz_factor) + /* (so we may as well compare startup cost, even if !consider_startup) */ + if (path1->startup_cost > path2->startup_cost * fuzz_factor && + path2->param_info == NULL) { /* ... but path1 fuzzily worse on startup, so path2 wins */ return COSTS_BETTER2; } - if (path2->startup_cost > path1->startup_cost * fuzz_factor) + if (path2->startup_cost > path1->startup_cost * fuzz_factor && + path1->param_info == NULL) { /* ... but path2 fuzzily worse on startup, so path1 wins */ return COSTS_BETTER1; @@ -210,11 +226,19 @@ compare_path_costs_fuzzily(Path *path1, Path *path2, double fuzz_factor) * Find the minimum-cost paths from among a relation's paths, * and save them in the rel's cheapest-path fields. * - * Only unparameterized paths are considered candidates for cheapest_startup - * and cheapest_total. The cheapest_parameterized_paths list collects paths - * that are cheapest-total for their parameterization (i.e., there is no - * cheaper path with the same or weaker parameterization). This list always - * includes the unparameterized cheapest-total path, too. + * cheapest_total_path is normally the cheapest-total-cost unparameterized + * path; but if there are no unparameterized paths, we assign it to be the + * best (cheapest least-parameterized) parameterized path. However, only + * unparameterized paths are considered candidates for cheapest_startup_path, + * so that will be NULL if there are no unparameterized paths. + * + * The cheapest_parameterized_paths list collects all parameterized paths + * that have survived the add_path() tournament for this relation. (Since + * add_path ignores pathkeys and startup cost for a parameterized path, + * these will be paths that have best total cost or best row count for their + * parameterization.) cheapest_parameterized_paths always includes the + * cheapest-total unparameterized path, too, if there is one; the users of + * that list find it more convenient if that's included. * * This is normally called only after we've finished constructing the path * list for the rel node. @@ -224,74 +248,119 @@ set_cheapest(RelOptInfo *parent_rel) { Path *cheapest_startup_path; Path *cheapest_total_path; - bool have_parameterized_paths; + Path *best_param_path; + List *parameterized_paths; ListCell *p; Assert(IsA(parent_rel, RelOptInfo)); - cheapest_startup_path = cheapest_total_path = NULL; - have_parameterized_paths = false; + if (parent_rel->pathlist == NIL) + elog(ERROR, "could not devise a query plan for the given query"); + + cheapest_startup_path = cheapest_total_path = best_param_path = NULL; + parameterized_paths = NIL; foreach(p, parent_rel->pathlist) { Path *path = (Path *) lfirst(p); int cmp; - /* We only consider unparameterized paths in this step */ if (path->param_info) { - have_parameterized_paths = true; - continue; - } + /* Parameterized path, so add it to parameterized_paths */ + parameterized_paths = lappend(parameterized_paths, path); - if (cheapest_total_path == NULL) - { - cheapest_startup_path = cheapest_total_path = path; - continue; + /* + * If we have an unparameterized cheapest-total, we no longer care + * about finding the best parameterized path, so move on. + */ + if (cheapest_total_path) + continue; + + /* + * Otherwise, track the best parameterized path, which is the one + * with least total cost among those of the minimum + * parameterization. + */ + if (best_param_path == NULL) + best_param_path = path; + else + { + switch (bms_subset_compare(PATH_REQ_OUTER(path), + PATH_REQ_OUTER(best_param_path))) + { + case BMS_EQUAL: + /* keep the cheaper one */ + if (compare_path_costs(path, best_param_path, + TOTAL_COST) < 0) + best_param_path = path; + break; + case BMS_SUBSET1: + /* new path is less-parameterized */ + best_param_path = path; + break; + case BMS_SUBSET2: + /* old path is less-parameterized, keep it */ + break; + case BMS_DIFFERENT: + + /* + * This means that neither path has the least possible + * parameterization for the rel. We'll sit on the old + * path until something better comes along. + */ + break; + } + } } + else + { + /* Unparameterized path, so consider it for cheapest slots */ + if (cheapest_total_path == NULL) + { + cheapest_startup_path = cheapest_total_path = path; + continue; + } - /* - * If we find two paths of identical costs, try to keep the - * better-sorted one. The paths might have unrelated sort orderings, - * in which case we can only guess which might be better to keep, but - * if one is superior then we definitely should keep that one. - */ - cmp = compare_path_costs(cheapest_startup_path, path, STARTUP_COST); - if (cmp > 0 || - (cmp == 0 && - compare_pathkeys(cheapest_startup_path->pathkeys, - path->pathkeys) == PATHKEYS_BETTER2)) - cheapest_startup_path = path; - - cmp = compare_path_costs(cheapest_total_path, path, TOTAL_COST); - if (cmp > 0 || - (cmp == 0 && - compare_pathkeys(cheapest_total_path->pathkeys, - path->pathkeys) == PATHKEYS_BETTER2)) - cheapest_total_path = path; + /* + * If we find two paths of identical costs, try to keep the + * better-sorted one. The paths might have unrelated sort + * orderings, in which case we can only guess which might be + * better to keep, but if one is superior then we definitely + * should keep that one. + */ + cmp = compare_path_costs(cheapest_startup_path, path, STARTUP_COST); + if (cmp > 0 || + (cmp == 0 && + compare_pathkeys(cheapest_startup_path->pathkeys, + path->pathkeys) == PATHKEYS_BETTER2)) + cheapest_startup_path = path; + + cmp = compare_path_costs(cheapest_total_path, path, TOTAL_COST); + if (cmp > 0 || + (cmp == 0 && + compare_pathkeys(cheapest_total_path->pathkeys, + path->pathkeys) == PATHKEYS_BETTER2)) + cheapest_total_path = path; + } } + /* Add cheapest unparameterized path, if any, to parameterized_paths */ + if (cheapest_total_path) + parameterized_paths = lcons(cheapest_total_path, parameterized_paths); + + /* + * If there is no unparameterized path, use the best parameterized path as + * cheapest_total_path (but not as cheapest_startup_path). + */ if (cheapest_total_path == NULL) - elog(ERROR, "could not devise a query plan for the given query"); + cheapest_total_path = best_param_path; + Assert(cheapest_total_path != NULL); parent_rel->cheapest_startup_path = cheapest_startup_path; parent_rel->cheapest_total_path = cheapest_total_path; parent_rel->cheapest_unique_path = NULL; /* computed only if needed */ - - /* Seed the parameterized-paths list with the cheapest total */ - parent_rel->cheapest_parameterized_paths = list_make1(cheapest_total_path); - - /* And, if there are any parameterized paths, add them in one at a time */ - if (have_parameterized_paths) - { - foreach(p, parent_rel->pathlist) - { - Path *path = (Path *) lfirst(p); - - if (path->param_info) - add_parameterized_path(parent_rel, path); - } - } + parent_rel->cheapest_parameterized_paths = parameterized_paths; } /* @@ -313,11 +382,15 @@ set_cheapest(RelOptInfo *parent_rel) * one parameterization can seldom dominate a path of another. But such * cases do arise, so we make the full set of checks anyway. * - * There is one policy decision embedded in this function, along with its - * sibling add_path_precheck: we treat all parameterized paths as having - * NIL pathkeys, so that they compete only on cost. This is to reduce - * the number of parameterized paths that are kept. See discussion in - * src/backend/optimizer/README. + * There are two policy decisions embedded in this function, along with + * its sibling add_path_precheck: we treat all parameterized paths as + * having NIL pathkeys, and we ignore their startup costs, so that they + * compete only on parameterization, total cost and rowcount. This is to + * reduce the number of parameterized paths that are kept. See discussion + * in src/backend/optimizer/README. + * + * Another policy that is enforced here is that we only consider cheap + * startup cost to be interesting if parent_rel->consider_startup is true. * * The pathlist is kept sorted by total_cost, with cheaper paths * at the front. Within this routine, that's simply a speed hack: @@ -379,10 +452,11 @@ add_path(RelOptInfo *parent_rel, Path *new_path) p1_next = lnext(p1); /* - * Do a fuzzy cost comparison with 1% fuzziness limit. (XXX does this + * Do a fuzzy cost comparison with 1% fuzziness limit. (XXX does this * percentage need to be user-configurable?) */ - costcmp = compare_path_costs_fuzzily(new_path, old_path, 1.01); + costcmp = compare_path_costs_fuzzily(new_path, old_path, 1.01, + parent_rel->consider_startup); /* * If the two paths compare differently for startup and total cost, @@ -447,8 +521,10 @@ add_path(RelOptInfo *parent_rel, Path *new_path) remove_old = true; /* new dominates old */ else if (new_path->rows > old_path->rows) accept_new = false; /* old dominates new */ - else if (compare_path_costs_fuzzily(new_path, old_path, - 1.0000000001) == COSTS_BETTER1) + else if (compare_path_costs_fuzzily(new_path, + old_path, + 1.0000000001, + parent_rel->consider_startup) == COSTS_BETTER1) remove_old = true; /* new dominates old */ else accept_new = false; /* old equals or @@ -552,7 +628,7 @@ add_path(RelOptInfo *parent_rel, Path *new_path) * and have lower bounds for its costs. * * Note that we do not know the path's rowcount, since getting an estimate for - * that is too expensive to do before prechecking. We assume here that paths + * that is too expensive to do before prechecking. We assume here that paths * of a superset parameterization will generate fewer rows; if that holds, * then paths with different parameterizations cannot dominate each other * and so we can simply ignore existing paths of another parameterization. @@ -570,7 +646,7 @@ add_path_precheck(RelOptInfo *parent_rel, List *new_path_pathkeys; ListCell *p1; - /* Pretend parameterized paths have no pathkeys, per add_path comment */ + /* Pretend parameterized paths have no pathkeys, per add_path policy */ new_path_pathkeys = required_outer ? NIL : pathkeys; foreach(p1, parent_rel->pathlist) @@ -590,8 +666,10 @@ add_path_precheck(RelOptInfo *parent_rel, */ if (total_cost >= old_path->total_cost) { - if (startup_cost >= old_path->startup_cost) + /* can win on startup cost only if unparameterized */ + if (startup_cost >= old_path->startup_cost || required_outer) { + /* new path does not win on cost, so check pathkeys... */ List *old_path_pathkeys; old_path_pathkeys = old_path->param_info ? NIL : old_path->pathkeys; @@ -600,6 +678,7 @@ add_path_precheck(RelOptInfo *parent_rel, if (keyscmp == PATHKEYS_EQUAL || keyscmp == PATHKEYS_BETTER2) { + /* new path does not win on pathkeys... */ if (bms_equal(required_outer, PATH_REQ_OUTER(old_path))) { /* Found an old path that dominates the new one */ @@ -622,123 +701,6 @@ add_path_precheck(RelOptInfo *parent_rel, return true; } -/* - * add_parameterized_path - * Consider a parameterized implementation path for the specified rel, - * and add it to the rel's cheapest_parameterized_paths list if it - * belongs there, removing any old entries that it dominates. - * - * This is essentially a cut-down form of add_path(): we do not care - * about startup cost or sort ordering, only total cost, rowcount, and - * parameterization. Also, we must not recycle rejected paths, since - * they will still be present in the rel's pathlist. - * - * 'parent_rel' is the relation entry to which the path corresponds. - * 'new_path' is a parameterized path for parent_rel. - * - * Returns nothing, but modifies parent_rel->cheapest_parameterized_paths. - */ -static void -add_parameterized_path(RelOptInfo *parent_rel, Path *new_path) -{ - bool accept_new = true; /* unless we find a superior old path */ - ListCell *insert_after = NULL; /* where to insert new item */ - ListCell *p1; - ListCell *p1_prev; - ListCell *p1_next; - - /* - * Loop to check proposed new path against old paths. Note it is possible - * for more than one old path to be tossed out because new_path dominates - * it. - * - * We can't use foreach here because the loop body may delete the current - * list cell. - */ - p1_prev = NULL; - for (p1 = list_head(parent_rel->cheapest_parameterized_paths); - p1 != NULL; p1 = p1_next) - { - Path *old_path = (Path *) lfirst(p1); - bool remove_old = false; /* unless new proves superior */ - int costcmp; - BMS_Comparison outercmp; - - p1_next = lnext(p1); - - costcmp = compare_path_costs(new_path, old_path, TOTAL_COST); - outercmp = bms_subset_compare(PATH_REQ_OUTER(new_path), - PATH_REQ_OUTER(old_path)); - if (outercmp != BMS_DIFFERENT) - { - if (costcmp < 0) - { - if (outercmp != BMS_SUBSET2 && - new_path->rows <= old_path->rows) - remove_old = true; /* new dominates old */ - } - else if (costcmp > 0) - { - if (outercmp != BMS_SUBSET1 && - new_path->rows >= old_path->rows) - accept_new = false; /* old dominates new */ - } - else if (outercmp == BMS_SUBSET1 && - new_path->rows <= old_path->rows) - remove_old = true; /* new dominates old */ - else if (outercmp == BMS_SUBSET2 && - new_path->rows >= old_path->rows) - accept_new = false; /* old dominates new */ - else if (new_path->rows < old_path->rows) - remove_old = true; /* new dominates old */ - else - { - /* Same cost, rows, and param rels; arbitrarily keep old */ - accept_new = false; /* old equals or dominates new */ - } - } - - /* - * Remove current element from cheapest_parameterized_paths if - * dominated by new. - */ - if (remove_old) - { - parent_rel->cheapest_parameterized_paths = - list_delete_cell(parent_rel->cheapest_parameterized_paths, - p1, p1_prev); - /* p1_prev does not advance */ - } - else - { - /* new belongs after this old path if it has cost >= old's */ - if (costcmp >= 0) - insert_after = p1; - /* p1_prev advances */ - p1_prev = p1; - } - - /* - * If we found an old path that dominates new_path, we can quit - * scanning the list; we will not add new_path, and we assume new_path - * cannot dominate any other elements of the list. - */ - if (!accept_new) - break; - } - - if (accept_new) - { - /* Accept the new path: insert it at proper place in list */ - if (insert_after) - lappend_cell(parent_rel->cheapest_parameterized_paths, - insert_after, new_path); - else - parent_rel->cheapest_parameterized_paths = - lcons(new_path, parent_rel->cheapest_parameterized_paths); - } -} - /***************************************************************************** * PATH NODE CREATION ROUTINES @@ -1890,13 +1852,15 @@ create_bitmap_or_path(PlannerInfo *root, * Creates a path corresponding to a scan by TID, returning the pathnode. */ TidPath * -create_tidscan_path(PlannerInfo *root, RelOptInfo *rel, List *tidquals) +create_tidscan_path(PlannerInfo *root, RelOptInfo *rel, List *tidquals, + Relids required_outer) { TidPath *pathnode = makeNode(TidPath); pathnode->path.pathtype = T_TidScan; pathnode->path.parent = rel; - pathnode->path.param_info = NULL; /* never parameterized at present */ + pathnode->path.param_info = get_baserel_parampathinfo(root, rel, + required_outer); pathnode->path.pathkeys = NIL; /* always unordered */ pathnode->tidquals = tidquals; @@ -1908,7 +1872,8 @@ create_tidscan_path(PlannerInfo *root, RelOptInfo *rel, List *tidquals) elog(ERROR, "could not perform TID scan on remote relation"); #endif - cost_tidscan(&pathnode->path, root, rel, tidquals); + cost_tidscan(&pathnode->path, root, rel, tidquals, + pathnode->path.param_info); return pathnode; } @@ -2003,7 +1968,7 @@ create_append_path(RelOptInfo *rel, List *subpaths, Relids required_outer) * Compute rows and costs as sums of subplan rows and costs. We charge * nothing extra for the Append itself, which perhaps is too optimistic, * but since it doesn't do any selection or projection, it is a pretty - * cheap node. If you change this, see also make_append(). + * cheap node. If you change this, see also make_append(). */ pathnode->path.rows = 0; pathnode->path.startup_cost = 0; @@ -2180,7 +2145,7 @@ create_result_path(List *quals) pathnode->path.pathtype = T_Result; pathnode->path.parent = NULL; - pathnode->path.param_info = NULL; + pathnode->path.param_info = NULL; /* there are no other rels... */ pathnode->path.pathkeys = NIL; pathnode->quals = quals; @@ -2648,7 +2613,7 @@ translate_sub_tlist(List *tlist, int relid) * * colnos is an integer list of output column numbers (resno's). We are * interested in whether rows consisting of just these columns are certain - * to be distinct. "Distinctness" is defined according to whether the + * to be distinct. "Distinctness" is defined according to whether the * corresponding upper-level equality operators listed in opids would think * the values are distinct. (Note: the opids entries could be cross-type * operators, and thus not exactly the equality operators that the subquery @@ -2769,7 +2734,7 @@ query_is_distinct_for(Query *query, List *colnos, List *opids) * distinct_col_search - subroutine for query_is_distinct_for * * If colno is in colnos, return the corresponding element of opids, - * else return InvalidOid. (We expect colnos does not contain duplicates, + * else return InvalidOid. (We expect colnos does not contain duplicates, * so the result is well-defined.) */ static Oid @@ -2823,16 +2788,18 @@ create_subqueryscan_path(PlannerInfo *root, RelOptInfo *rel, * returning the pathnode. */ Path * -create_functionscan_path(PlannerInfo *root, RelOptInfo *rel) +create_functionscan_path(PlannerInfo *root, RelOptInfo *rel, + List *pathkeys, Relids required_outer) { Path *pathnode = makeNode(Path); pathnode->pathtype = T_FunctionScan; pathnode->parent = rel; - pathnode->param_info = NULL; /* never parameterized at present */ - pathnode->pathkeys = NIL; /* for now, assume unordered result */ + pathnode->param_info = get_baserel_parampathinfo(root, rel, + required_outer); + pathnode->pathkeys = pathkeys; - cost_functionscan(pathnode, root, rel); + cost_functionscan(pathnode, root, rel, pathnode->param_info); return pathnode; } @@ -2843,16 +2810,18 @@ create_functionscan_path(PlannerInfo *root, RelOptInfo *rel) * returning the pathnode. */ Path * -create_valuesscan_path(PlannerInfo *root, RelOptInfo *rel) +create_valuesscan_path(PlannerInfo *root, RelOptInfo *rel, + Relids required_outer) { Path *pathnode = makeNode(Path); pathnode->pathtype = T_ValuesScan; pathnode->parent = rel; - pathnode->param_info = NULL; /* never parameterized at present */ + pathnode->param_info = get_baserel_parampathinfo(root, rel, + required_outer); pathnode->pathkeys = NIL; /* result is always unordered */ - cost_valuesscan(pathnode, root, rel); + cost_valuesscan(pathnode, root, rel, pathnode->param_info); return pathnode; } @@ -2863,16 +2832,17 @@ create_valuesscan_path(PlannerInfo *root, RelOptInfo *rel) * returning the pathnode. */ Path * -create_ctescan_path(PlannerInfo *root, RelOptInfo *rel) +create_ctescan_path(PlannerInfo *root, RelOptInfo *rel, Relids required_outer) { Path *pathnode = makeNode(Path); pathnode->pathtype = T_CteScan; pathnode->parent = rel; - pathnode->param_info = NULL; /* never parameterized at present */ + pathnode->param_info = get_baserel_parampathinfo(root, rel, + required_outer); pathnode->pathkeys = NIL; /* XXX for now, result is always unordered */ - cost_ctescan(pathnode, root, rel); + cost_ctescan(pathnode, root, rel, pathnode->param_info); return pathnode; } @@ -2883,17 +2853,19 @@ create_ctescan_path(PlannerInfo *root, RelOptInfo *rel) * returning the pathnode. */ Path * -create_worktablescan_path(PlannerInfo *root, RelOptInfo *rel) +create_worktablescan_path(PlannerInfo *root, RelOptInfo *rel, + Relids required_outer) { Path *pathnode = makeNode(Path); pathnode->pathtype = T_WorkTableScan; pathnode->parent = rel; - pathnode->param_info = NULL; /* never parameterized at present */ + pathnode->param_info = get_baserel_parampathinfo(root, rel, + required_outer); pathnode->pathkeys = NIL; /* result is always unordered */ /* Cost is the same as for a regular CTE scan */ - cost_ctescan(pathnode, root, rel); + cost_ctescan(pathnode, root, rel, pathnode->param_info); return pathnode; } @@ -3241,10 +3213,10 @@ create_hashjoin_path(PlannerInfo *root, /* * A hashjoin never has pathkeys, since its output ordering is - * unpredictable due to possible batching. XXX If the inner relation is + * unpredictable due to possible batching. XXX If the inner relation is * small enough, we could instruct the executor that it must not batch, * and then we could assume that the output inherits the outer relation's - * ordering, which might save a sort step. However there is considerable + * ordering, which might save a sort step. However there is considerable * downside if our estimate of the inner relation size is badly off. For * the moment we don't risk it. (Note also that if we wanted to take this * seriously, joinpath.c would have to consider many more paths for the @@ -3286,7 +3258,7 @@ create_hashjoin_path(PlannerInfo *root, * same parameterization level, ensuring that they all enforce the same set * of join quals (and thus that that parameterization can be attributed to * an append path built from such paths). Currently, only a few path types - * are supported here, though more could be added at need. We return NULL + * are supported here, though more could be added at need. We return NULL * if we can't reparameterize the given path. * * Note: we intentionally do not pass created paths to add_path(); it would @@ -3318,7 +3290,7 @@ reparameterize_path(PlannerInfo *root, Path *path, /* * We can't use create_index_path directly, and would not want * to because it would re-compute the indexqual conditions - * which is wasted effort. Instead we hack things a bit: + * which is wasted effort. Instead we hack things a bit: * flat-copy the path node, revise its param_info, and redo * the cost estimate. */ diff --git a/src/backend/optimizer/util/placeholder.c b/src/backend/optimizer/util/placeholder.c index e05c8ddef1..8d7c4feca4 100644 --- a/src/backend/optimizer/util/placeholder.c +++ b/src/backend/optimizer/util/placeholder.c @@ -4,7 +4,7 @@ * PlaceHolderVar and PlaceHolderInfo manipulation routines * * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -23,9 +23,8 @@ #include "utils/lsyscache.h" /* Local functions */ -static Relids find_placeholders_recurse(PlannerInfo *root, Node *jtnode); -static void mark_placeholders_in_expr(PlannerInfo *root, Node *expr, - Relids relids); +static void find_placeholders_recurse(PlannerInfo *root, Node *jtnode); +static void find_placeholders_in_expr(PlannerInfo *root, Node *expr); /* @@ -61,7 +60,7 @@ make_placeholder_expr(PlannerInfo *root, Expr *expr, Relids phrels) * We build PlaceHolderInfos only for PHVs that are still present in the * simplified query passed to query_planner(). * - * Note: this should only be called after query_planner() has started. Also, + * Note: this should only be called after query_planner() has started. Also, * create_new_ph must not be TRUE after deconstruct_jointree begins, because * make_outerjoininfo assumes that we already know about all placeholders. */ @@ -70,6 +69,7 @@ find_placeholder_info(PlannerInfo *root, PlaceHolderVar *phv, bool create_new_ph) { PlaceHolderInfo *phinfo; + Relids rels_used; ListCell *lc; /* if this ever isn't true, we'd need to be able to look in parent lists */ @@ -90,16 +90,39 @@ find_placeholder_info(PlannerInfo *root, PlaceHolderVar *phv, phinfo->phid = phv->phid; phinfo->ph_var = copyObject(phv); - phinfo->ph_eval_at = pull_varnos((Node *) phv); + + /* + * Any referenced rels that are outside the PHV's syntactic scope are + * LATERAL references, which should be included in ph_lateral but not in + * ph_eval_at. If no referenced rels are within the syntactic scope, + * force evaluation at the syntactic location. + */ + rels_used = pull_varnos((Node *) phv->phexpr); + phinfo->ph_lateral = bms_difference(rels_used, phv->phrels); + if (bms_is_empty(phinfo->ph_lateral)) + phinfo->ph_lateral = NULL; /* make it exactly NULL if empty */ + phinfo->ph_eval_at = bms_int_members(rels_used, phv->phrels); + /* If no contained vars, force evaluation at syntactic location */ + if (bms_is_empty(phinfo->ph_eval_at)) + { + phinfo->ph_eval_at = bms_copy(phv->phrels); + Assert(!bms_is_empty(phinfo->ph_eval_at)); + } /* ph_eval_at may change later, see update_placeholder_eval_levels */ phinfo->ph_needed = NULL; /* initially it's unused */ - phinfo->ph_may_need = NULL; /* for the moment, estimate width using just the datatype info */ phinfo->ph_width = get_typavgwidth(exprType((Node *) phv->phexpr), exprTypmod((Node *) phv->phexpr)); root->placeholder_list = lappend(root->placeholder_list, phinfo); + /* + * The PHV's contained expression may contain other, lower-level PHVs. We + * now know we need to get those into the PlaceHolderInfo list, too, so we + * may as well do that immediately. + */ + find_placeholders_in_expr(root, (Node *) phinfo->ph_var->phexpr); + return phinfo; } @@ -109,6 +132,12 @@ find_placeholder_info(PlannerInfo *root, PlaceHolderVar *phv, * * We don't need to look at the targetlist because build_base_rel_tlists() * will already have made entries for any PHVs in the tlist. + * + * This is called before we begin deconstruct_jointree. Once we begin + * deconstruct_jointree, all active placeholders must be present in + * root->placeholder_list, because make_outerjoininfo and + * update_placeholder_eval_levels require this info to be available + * while we crawl up the join tree. */ void find_placeholders_in_jointree(PlannerInfo *root) @@ -119,7 +148,7 @@ find_placeholders_in_jointree(PlannerInfo *root) /* Start recursion at top of jointree */ Assert(root->parse->jointree != NULL && IsA(root->parse->jointree, FromExpr)); - (void) find_placeholders_recurse(root, (Node *) root->parse->jointree); + find_placeholders_recurse(root, (Node *) root->parse->jointree); } } @@ -128,23 +157,15 @@ find_placeholders_in_jointree(PlannerInfo *root) * One recursion level of find_placeholders_in_jointree. * * jtnode is the current jointree node to examine. - * - * The result is the set of base Relids contained in or below jtnode. - * This is just an internal convenience, it's not used at the top level. */ -static Relids +static void find_placeholders_recurse(PlannerInfo *root, Node *jtnode) { - Relids jtrelids; - if (jtnode == NULL) - return NULL; + return; if (IsA(jtnode, RangeTblRef)) { - int varno = ((RangeTblRef *) jtnode)->rtindex; - - /* No quals to deal with, just return correct result */ - jtrelids = bms_make_singleton(varno); + /* No quals to deal with here */ } else if (IsA(jtnode, FromExpr)) { @@ -152,57 +173,43 @@ find_placeholders_recurse(PlannerInfo *root, Node *jtnode) ListCell *l; /* - * First, recurse to handle child joins, and form their relid set. + * First, recurse to handle child joins. */ - jtrelids = NULL; foreach(l, f->fromlist) { - Relids sub_relids; - - sub_relids = find_placeholders_recurse(root, lfirst(l)); - jtrelids = bms_join(jtrelids, sub_relids); + find_placeholders_recurse(root, lfirst(l)); } /* * Now process the top-level quals. */ - mark_placeholders_in_expr(root, f->quals, jtrelids); + find_placeholders_in_expr(root, f->quals); } else if (IsA(jtnode, JoinExpr)) { JoinExpr *j = (JoinExpr *) jtnode; - Relids leftids, - rightids; /* - * First, recurse to handle child joins, and form their relid set. + * First, recurse to handle child joins. */ - leftids = find_placeholders_recurse(root, j->larg); - rightids = find_placeholders_recurse(root, j->rarg); - jtrelids = bms_join(leftids, rightids); + find_placeholders_recurse(root, j->larg); + find_placeholders_recurse(root, j->rarg); /* Process the qual clauses */ - mark_placeholders_in_expr(root, j->quals, jtrelids); + find_placeholders_in_expr(root, j->quals); } else - { elog(ERROR, "unrecognized node type: %d", (int) nodeTag(jtnode)); - jtrelids = NULL; /* keep compiler quiet */ - } - return jtrelids; } /* - * mark_placeholders_in_expr - * Find all PlaceHolderVars in the given expression, and mark them - * as possibly needed at the specified join level. - * - * relids is the syntactic join level to mark as the "maybe needed" level - * for each PlaceHolderVar found in the expression. + * find_placeholders_in_expr + * Find all PlaceHolderVars in the given expression, and create + * PlaceHolderInfo entries for them. */ static void -mark_placeholders_in_expr(PlannerInfo *root, Node *expr, Relids relids) +find_placeholders_in_expr(PlannerInfo *root, Node *expr) { List *vars; ListCell *vl; @@ -217,62 +224,25 @@ mark_placeholders_in_expr(PlannerInfo *root, Node *expr, Relids relids) foreach(vl, vars) { PlaceHolderVar *phv = (PlaceHolderVar *) lfirst(vl); - PlaceHolderInfo *phinfo; /* Ignore any plain Vars */ if (!IsA(phv, PlaceHolderVar)) continue; /* Create a PlaceHolderInfo entry if there's not one already */ - phinfo = find_placeholder_info(root, phv, true); - - /* Mark it, and recursively process any contained placeholders */ - mark_placeholder_maybe_needed(root, phinfo, relids); + (void) find_placeholder_info(root, phv, true); } list_free(vars); } /* - * mark_placeholder_maybe_needed - * Mark a placeholder as possibly needed at the specified join level. - * - * relids is the syntactic join level to mark as the "maybe needed" level - * for the placeholder. - * - * This is called during an initial scan of the query's targetlist and quals - * before we begin deconstruct_jointree. Once we begin deconstruct_jointree, - * all active placeholders must be present in root->placeholder_list with - * their correct ph_may_need values, because make_outerjoininfo and - * update_placeholder_eval_levels require this info to be available while - * we crawl up the join tree. - */ -void -mark_placeholder_maybe_needed(PlannerInfo *root, PlaceHolderInfo *phinfo, - Relids relids) -{ - /* Mark the PHV as possibly needed at the given syntactic level */ - phinfo->ph_may_need = bms_add_members(phinfo->ph_may_need, relids); - - /* - * This is a bit tricky: the PHV's contained expression may contain other, - * lower-level PHVs. We need to get those into the PlaceHolderInfo list, - * but they aren't going to be needed where the outer PHV is referenced. - * Rather, they'll be needed where the outer PHV is evaluated. We can - * estimate that (conservatively) as the syntactic location of the PHV's - * expression. Recurse to take care of any such PHVs. - */ - mark_placeholders_in_expr(root, (Node *) phinfo->ph_var->phexpr, - phinfo->ph_var->phrels); -} - -/* * update_placeholder_eval_levels * Adjust the target evaluation levels for placeholders * * The initial eval_at level set by find_placeholder_info was the set of * rels used in the placeholder's expression (or the whole subselect below * the placeholder's syntactic location, if the expr is variable-free). - * If the subselect contains any outer joins that can null any of those rels, + * If the query contains any outer joins that can null any of those rels, * we must delay evaluation to above those joins. * * We repeat this operation each time we add another outer join to @@ -352,6 +322,9 @@ update_placeholder_eval_levels(PlannerInfo *root, SpecialJoinInfo *new_sjinfo) } } while (found_some); + /* Can't move the PHV's eval_at level to above its syntactic level */ + Assert(bms_is_subset(eval_at, syn_level)); + phinfo->ph_eval_at = eval_at; } } @@ -362,11 +335,14 @@ update_placeholder_eval_levels(PlannerInfo *root, SpecialJoinInfo *new_sjinfo) * * This is called after we've finished determining the eval_at levels for * all placeholders. We need to make sure that all vars and placeholders - * needed to evaluate each placeholder will be available at the join level - * where the evaluation will be done. Note that this loop can have - * side-effects on the ph_needed sets of other PlaceHolderInfos; that's okay - * because we don't examine ph_needed here, so there are no ordering issues - * to worry about. + * needed to evaluate each placeholder will be available at the scan or join + * level where the evaluation will be done. (It might seem that scan-level + * evaluations aren't interesting, but that's not so: a LATERAL reference + * within a placeholder's expression needs to cause the referenced var or + * placeholder to be marked as needed in the scan where it's evaluated.) + * Note that this loop can have side-effects on the ph_needed sets of other + * PlaceHolderInfos; that's okay because we don't examine ph_needed here, so + * there are no ordering issues to worry about. */ void fix_placeholder_input_needed_levels(PlannerInfo *root) @@ -376,27 +352,23 @@ fix_placeholder_input_needed_levels(PlannerInfo *root) foreach(lc, root->placeholder_list) { PlaceHolderInfo *phinfo = (PlaceHolderInfo *) lfirst(lc); - Relids eval_at = phinfo->ph_eval_at; - - /* No work unless it'll be evaluated above baserel level */ - if (bms_membership(eval_at) == BMS_MULTIPLE) - { - List *vars = pull_var_clause((Node *) phinfo->ph_var->phexpr, - PVC_RECURSE_AGGREGATES, - PVC_INCLUDE_PLACEHOLDERS); + List *vars = pull_var_clause((Node *) phinfo->ph_var->phexpr, + PVC_RECURSE_AGGREGATES, + PVC_INCLUDE_PLACEHOLDERS); - add_vars_to_targetlist(root, vars, eval_at, false); - list_free(vars); - } + add_vars_to_targetlist(root, vars, phinfo->ph_eval_at, false); + list_free(vars); } } /* * add_placeholders_to_base_rels - * Add any required PlaceHolderVars to base rels' targetlists. + * Add any required PlaceHolderVars to base rels' targetlists, and + * update lateral_vars lists for lateral references contained in them. * * If any placeholder can be computed at a base rel and is needed above it, - * add it to that rel's targetlist. This might look like it could be merged + * add it to that rel's targetlist, and add any lateral references it requires + * to the rel's lateral_vars list. This might look like it could be merged * with fix_placeholder_input_needed_levels, but it must be separate because * join removal happens in between, and can change the ph_eval_at sets. There * is essentially the same logic in add_placeholders_to_joinrel, but we can't @@ -417,9 +389,47 @@ add_placeholders_to_base_rels(PlannerInfo *root) int varno = bms_singleton_member(eval_at); RelOptInfo *rel = find_base_rel(root, varno); - if (bms_nonempty_difference(phinfo->ph_needed, rel->relids)) + /* add it to reltargetlist if needed above the rel scan level */ + if (bms_nonempty_difference(phinfo->ph_needed, eval_at)) rel->reltargetlist = lappend(rel->reltargetlist, copyObject(phinfo->ph_var)); + /* if there are lateral refs in it, add them to lateral_vars */ + if (phinfo->ph_lateral != NULL) + { + List *vars = pull_var_clause((Node *) phinfo->ph_var->phexpr, + PVC_RECURSE_AGGREGATES, + PVC_INCLUDE_PLACEHOLDERS); + ListCell *lc2; + + foreach(lc2, vars) + { + Node *node = (Node *) lfirst(lc2); + + if (IsA(node, Var)) + { + Var *var = (Var *) node; + + if (var->varno != varno) + rel->lateral_vars = lappend(rel->lateral_vars, + var); + } + else if (IsA(node, PlaceHolderVar)) + { + PlaceHolderVar *other_phv = (PlaceHolderVar *) node; + PlaceHolderInfo *other_phi; + + other_phi = find_placeholder_info(root, other_phv, + false); + if (!bms_is_subset(other_phi->ph_eval_at, eval_at)) + rel->lateral_vars = lappend(rel->lateral_vars, + other_phv); + } + else + Assert(false); + } + + list_free(vars); + } } } } diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index bc7e8a6096..1b7f4b64da 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -9,7 +9,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. * * Portions Copyright (c) 2012-2014, TransLattice, Inc. - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -24,10 +24,14 @@ #include "access/genam.h" #include "access/heapam.h" +#include "access/htup_details.h" +#include "access/nbtree.h" #include "access/sysattr.h" #include "access/transam.h" +#include "access/xlog.h" #include "catalog/catalog.h" #include "catalog/heap.h" +#include "foreign/fdwapi.h" #include "miscadmin.h" #include "nodes/makefuncs.h" #include "optimizer/clauses.h" @@ -71,6 +75,7 @@ static List *build_index_tlist(PlannerInfo *root, IndexOptInfo *index, * min_attr lowest valid AttrNumber * max_attr highest valid AttrNumber * indexlist list of IndexOptInfos for relation's indexes + * fdwroutine if it's a foreign table, the FDW function pointers * pages number of pages * tuples number of tuples * @@ -130,7 +135,7 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, * Don't bother with indexes for an inheritance parent, either. */ if (inhparent || - (IgnoreSystemIndexes && IsSystemClass(relation->rd_rel))) + (IgnoreSystemIndexes && IsSystemRelation(relation))) hasindex = false; else hasindex = relation->rd_rel->relhasindex; @@ -175,9 +180,10 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, * Ignore invalid indexes, since they can't safely be used for * queries. Note that this is OK because the data structure we * are constructing is only used by the planner --- the executor - * still needs to insert into "invalid" indexes! + * still needs to insert into "invalid" indexes, if they're marked + * IndexIsReady. */ - if (!index->indisvalid) + if (!IndexIsValid(index)) { index_close(indexRelation, NoLock); continue; @@ -366,6 +372,17 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, info->tuples = rel->tuples; } + if (info->relam == BTREE_AM_OID) + { + /* For btrees, get tree height while we have the index open */ + info->tree_height = _bt_getrootheight(indexRelation); + } + else + { + /* For other index types, just set it to "unknown" for now */ + info->tree_height = -1; + } + index_close(indexRelation, NoLock); indexinfos = lcons(info, indexinfos); @@ -376,6 +393,12 @@ get_relation_info(PlannerInfo *root, Oid relationObjectId, bool inhparent, rel->indexlist = indexinfos; + /* Grab the fdwroutine info using the relcache, while we have it */ + if (relation->rd_rel->relkind == RELKIND_FOREIGN_TABLE) + rel->fdwroutine = GetFdwRoutineForRelation(relation, true); + else + rel->fdwroutine = NULL; + heap_close(relation, NoLock); /* @@ -437,6 +460,7 @@ estimate_rel_size(Relation rel, int32 *attr_widths, #endif #endif case RELKIND_INDEX: + case RELKIND_MATVIEW: case RELKIND_TOASTVALUE: #ifdef XCP if (IS_PGXC_COORDINATOR && rel->rd_locator_info != NULL) @@ -458,12 +482,12 @@ estimate_rel_size(Relation rel, int32 *attr_widths, * minimum size estimate of 10 pages. The idea here is to avoid * assuming a newly-created table is really small, even if it * currently is, because that may not be true once some data gets - * loaded into it. Once a vacuum or analyze cycle has been done + * loaded into it. Once a vacuum or analyze cycle has been done * on it, it's more reasonable to believe the size is somewhat * stable. * * (Note that this is only an issue if the plan gets cached and - * used again after the table has been filled. What we're trying + * used again after the table has been filled. What we're trying * to avoid is using a nestloop-type plan on a table that has * grown substantially since the plan was made. Normally, * autovacuum/autoanalyze will occur once enough inserts have @@ -472,7 +496,7 @@ estimate_rel_size(Relation rel, int32 *attr_widths, * such as temporary tables.) * * We approximate "never vacuumed" by "has relpages = 0", which - * means this will also fire on genuinely empty relations. Not + * means this will also fire on genuinely empty relations. Not * great, but fortunately that's a seldom-seen case in the real * world, and it shouldn't degrade the quality of the plan too * much anyway to err in this direction. @@ -718,12 +742,6 @@ get_relation_constraints(PlannerInfo *root, cexpr = (Node *) canonicalize_qual((Expr *) cexpr); - /* - * Also mark any coercion format fields as "don't care", so that - * we can match to both explicit and implicit coercions. - */ - set_coercionform_dontcare(cexpr); - /* Fix Vars to have the desired varno */ if (varno != 1) ChangeVarNodes(cexpr, 1, varno, 0); @@ -823,7 +841,7 @@ relation_excluded_by_constraints(PlannerInfo *root, return false; /* - * OK to fetch the constraint expressions. Include "col IS NOT NULL" + * OK to fetch the constraint expressions. Include "col IS NOT NULL" * expressions for attnotnull columns, in case we can refute those. */ constraint_pred = get_relation_constraints(root, rte->relid, rel, true); @@ -871,7 +889,7 @@ relation_excluded_by_constraints(PlannerInfo *root, * Exception: if there are any dropped columns, we punt and return NIL. * Ideally we would like to handle the dropped-column case too. However this * creates problems for ExecTypeFromTL, which may be asked to build a tupdesc - * for a tlist that includes vars of no-longer-existent types. In theory we + * for a tlist that includes vars of no-longer-existent types. In theory we * could dig out the required info from the pg_attribute entries of the * relation, but that data is not readily available to ExecTypeFromTL. * For now, we don't apply the physical-tlist optimization when there are @@ -1065,6 +1083,7 @@ Selectivity restriction_selectivity(PlannerInfo *root, Oid operatorid, List *args, + Oid inputcollid, int varRelid) { RegProcedure oprrest = get_oprrest(operatorid); @@ -1077,11 +1096,12 @@ restriction_selectivity(PlannerInfo *root, if (!oprrest) return (Selectivity) 0.5; - result = DatumGetFloat8(OidFunctionCall4(oprrest, - PointerGetDatum(root), - ObjectIdGetDatum(operatorid), - PointerGetDatum(args), - Int32GetDatum(varRelid))); + result = DatumGetFloat8(OidFunctionCall4Coll(oprrest, + inputcollid, + PointerGetDatum(root), + ObjectIdGetDatum(operatorid), + PointerGetDatum(args), + Int32GetDatum(varRelid))); if (result < 0.0 || result > 1.0) elog(ERROR, "invalid restriction selectivity: %f", result); @@ -1100,6 +1120,7 @@ Selectivity join_selectivity(PlannerInfo *root, Oid operatorid, List *args, + Oid inputcollid, JoinType jointype, SpecialJoinInfo *sjinfo) { @@ -1113,12 +1134,13 @@ join_selectivity(PlannerInfo *root, if (!oprjoin) return (Selectivity) 0.5; - result = DatumGetFloat8(OidFunctionCall5(oprjoin, - PointerGetDatum(root), - ObjectIdGetDatum(operatorid), - PointerGetDatum(args), - Int16GetDatum(jointype), - PointerGetDatum(sjinfo))); + result = DatumGetFloat8(OidFunctionCall5Coll(oprjoin, + inputcollid, + PointerGetDatum(root), + ObjectIdGetDatum(operatorid), + PointerGetDatum(args), + Int16GetDatum(jointype), + PointerGetDatum(sjinfo))); if (result < 0.0 || result > 1.0) elog(ERROR, "invalid join selectivity: %f", result); diff --git a/src/backend/optimizer/util/predtest.c b/src/backend/optimizer/util/predtest.c index 65d191e5d3..9d61a4d71c 100644 --- a/src/backend/optimizer/util/predtest.c +++ b/src/backend/optimizer/util/predtest.c @@ -4,7 +4,7 @@ * Routines to attempt to prove logical implications between predicate * expressions. * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -133,7 +133,7 @@ predicate_implied_by(List *predicate_list, List *restrictinfo_list) /* * If either input is a single-element list, replace it with its lone - * member; this avoids one useless level of AND-recursion. We only need + * member; this avoids one useless level of AND-recursion. We only need * to worry about this at top level, since eval_const_expressions should * have gotten rid of any trivial ANDs or ORs below that. */ @@ -191,7 +191,7 @@ predicate_refuted_by(List *predicate_list, List *restrictinfo_list) /* * If either input is a single-element list, replace it with its lone - * member; this avoids one useless level of AND-recursion. We only need + * member; this avoids one useless level of AND-recursion. We only need * to worry about this at top level, since eval_const_expressions should * have gotten rid of any trivial ANDs or ORs below that. */ @@ -225,7 +225,7 @@ predicate_refuted_by(List *predicate_list, List *restrictinfo_list) * OR-expr A => AND-expr B iff: A => each of B's components * OR-expr A => OR-expr B iff: each of A's components => any of B's * - * An "atom" is anything other than an AND or OR node. Notice that we don't + * An "atom" is anything other than an AND or OR node. Notice that we don't * have any special logic to handle NOT nodes; these should have been pushed * down or eliminated where feasible by prepqual.c. * @@ -658,7 +658,7 @@ predicate_refuted_by_recurse(Node *clause, Node *predicate) * We cannot make the stronger conclusion that B is refuted if B * implies A's arg; that would only prove that B is not-TRUE, not * that it's not NULL either. Hence use equal() rather than - * predicate_implied_by_recurse(). We could do the latter if we + * predicate_implied_by_recurse(). We could do the latter if we * ever had a need for the weak form of refutation. */ not_arg = extract_strong_not_arg(clause); @@ -820,7 +820,7 @@ predicate_classify(Node *clause, PredIterInfo info) } /* - * PredIterInfo routines for iterating over regular Lists. The iteration + * PredIterInfo routines for iterating over regular Lists. The iteration * state variable is the next ListCell to visit. */ static void @@ -1014,13 +1014,13 @@ arrayexpr_cleanup_fn(PredIterInfo info) * implies another: * * A simple and general way is to see if they are equal(); this works for any - * kind of expression. (Actually, there is an implied assumption that the + * kind of expression. (Actually, there is an implied assumption that the * functions in the expression are immutable, ie dependent only on their input * arguments --- but this was checked for the predicate by the caller.) * * When the predicate is of the form "foo IS NOT NULL", we can conclude that * the predicate is implied if the clause is a strict operator or function - * that has "foo" as an input. In this case the clause must yield NULL when + * that has "foo" as an input. In this case the clause must yield NULL when * "foo" is NULL, which we can take as equivalent to FALSE because we know * we are within an AND/OR subtree of a WHERE clause. (Again, "foo" is * already known immutable, so the clause will certainly always fail.) @@ -1244,7 +1244,7 @@ list_member_strip(List *list, Expr *datum) * * The strategy numbers defined by btree indexes (see access/skey.h) are: * (1) < (2) <= (3) = (4) >= (5) > - * and in addition we use (6) to represent <>. <> is not a btree-indexable + * and in addition we use (6) to represent <>. <> is not a btree-indexable * operator, but we assume here that if an equality operator of a btree * opfamily has a negator operator, the negator behaves as <> for the opfamily. * (This convention is also known to get_op_btree_interpretation().) @@ -1328,7 +1328,7 @@ static const StrategyNumber BT_refute_table[6][6] = { * if not able to prove it. * * What we look for here is binary boolean opclauses of the form - * "foo op constant", where "foo" is the same in both clauses. The operators + * "foo op constant", where "foo" is the same in both clauses. The operators * and constants can be different but the operators must be in the same btree * operator family. We use the above operator implication tables to * derive implications between nonidentical clauses. (Note: "foo" is known @@ -1418,7 +1418,7 @@ btree_predicate_proof(Expr *predicate, Node *clause, bool refute_it) /* * Check for matching subexpressions on the non-Const sides. We used to * only allow a simple Var, but it's about as easy to allow any - * expression. Remember we already know that the pred expression does not + * expression. Remember we already know that the pred expression does not * contain any non-immutable functions, so identical expressions should * yield identical results. */ @@ -1690,7 +1690,7 @@ get_btree_test_op(Oid pred_op, Oid clause_op, bool refute_it) * Last check: test_op must be immutable. * * Note that we require only the test_op to be immutable, not the - * original clause_op. (pred_op is assumed to have been checked + * original clause_op. (pred_op is assumed to have been checked * immutable by the caller.) Essentially we are assuming that the * opfamily is consistent even if it contains operators that are * merely stable. diff --git a/src/backend/optimizer/util/relnode.c b/src/backend/optimizer/util/relnode.c index 5e96460e81..9e78e144bc 100644 --- a/src/backend/optimizer/util/relnode.c +++ b/src/backend/optimizer/util/relnode.c @@ -3,7 +3,7 @@ * relnode.c * Relation-node lookup/construction routines * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -102,6 +102,8 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptKind reloptkind) rel->relids = bms_make_singleton(relid); rel->rows = 0; rel->width = 0; + /* cheap startup cost is interesting iff not all tuples to be retrieved */ + rel->consider_startup = (root->tuple_fraction > 0); rel->reltargetlist = NIL; rel->pathlist = NIL; rel->ppilist = NIL; @@ -112,12 +114,16 @@ build_simple_rel(PlannerInfo *root, int relid, RelOptKind reloptkind) rel->relid = relid; rel->rtekind = rte->rtekind; /* min_attr, max_attr, attr_needed, attr_widths are set below */ + rel->lateral_vars = NIL; + rel->lateral_relids = NULL; + rel->lateral_referencers = NULL; rel->indexlist = NIL; rel->pages = 0; rel->tuples = 0; rel->allvisfrac = 0; rel->subplan = NULL; rel->subroot = NULL; + rel->subplan_params = NIL; rel->fdwroutine = NULL; rel->fdw_private = NULL; rel->baserestrictinfo = NIL; @@ -259,7 +265,7 @@ RelOptInfo * find_join_rel(PlannerInfo *root, Relids relids) { /* - * Switch to using hash lookup when list grows "too long". The threshold + * Switch to using hash lookup when list grows "too long". The threshold * is arbitrary and is known only here. */ if (!root->join_rel_hash && list_length(root->join_rel_list) > 32) @@ -355,6 +361,8 @@ build_join_rel(PlannerInfo *root, joinrel->relids = bms_copy(joinrelids); joinrel->rows = 0; joinrel->width = 0; + /* cheap startup cost is interesting iff not all tuples to be retrieved */ + joinrel->consider_startup = (root->tuple_fraction > 0); joinrel->reltargetlist = NIL; joinrel->pathlist = NIL; joinrel->ppilist = NIL; @@ -368,12 +376,16 @@ build_join_rel(PlannerInfo *root, joinrel->max_attr = 0; joinrel->attr_needed = NULL; joinrel->attr_widths = NULL; + joinrel->lateral_vars = NIL; + joinrel->lateral_relids = NULL; + joinrel->lateral_referencers = NULL; joinrel->indexlist = NIL; joinrel->pages = 0; joinrel->tuples = 0; joinrel->allvisfrac = 0; joinrel->subplan = NULL; joinrel->subroot = NULL; + joinrel->subplan_params = NIL; joinrel->fdwroutine = NULL; joinrel->fdw_private = NULL; joinrel->baserestrictinfo = NIL; @@ -439,7 +451,7 @@ build_join_rel(PlannerInfo *root, /* * Also, if dynamic-programming join search is active, add the new joinrel - * to the appropriate sublist. Note: you might think the Assert on number + * to the appropriate sublist. Note: you might think the Assert on number * of members should be for equality, but some of the level 1 rels might * have been joinrels already, so we can only assert <=. */ @@ -475,8 +487,7 @@ build_joinrel_tlist(PlannerInfo *root, RelOptInfo *joinrel, foreach(vars, input_rel->reltargetlist) { - Node *origvar = (Node *) lfirst(vars); - Var *var; + Var *var = (Var *) lfirst(vars); RelOptInfo *baserel; int ndx; @@ -484,22 +495,17 @@ build_joinrel_tlist(PlannerInfo *root, RelOptInfo *joinrel, * Ignore PlaceHolderVars in the input tlists; we'll make our own * decisions about whether to copy them. */ - if (IsA(origvar, PlaceHolderVar)) + if (IsA(var, PlaceHolderVar)) continue; /* - * We can't run into any child RowExprs here, but we could find a - * whole-row Var with a ConvertRowtypeExpr atop it. + * Otherwise, anything in a baserel or joinrel targetlist ought to be + * a Var. (More general cases can only appear in appendrel child + * rels, which will never be seen here.) */ - var = (Var *) origvar; - while (!IsA(var, Var)) - { - if (IsA(var, ConvertRowtypeExpr)) - var = (Var *) ((ConvertRowtypeExpr *) var)->arg; - else - elog(ERROR, "unexpected node type in reltargetlist: %d", - (int) nodeTag(var)); - } + if (!IsA(var, Var)) + elog(ERROR, "unexpected node type in reltargetlist: %d", + (int) nodeTag(var)); /* Get the Var's original base rel */ baserel = find_base_rel(root, var->varno); @@ -509,7 +515,7 @@ build_joinrel_tlist(PlannerInfo *root, RelOptInfo *joinrel, if (bms_nonempty_difference(baserel->attr_needed[ndx], relids)) { /* Yup, add it to the output */ - joinrel->reltargetlist = lappend(joinrel->reltargetlist, origvar); + joinrel->reltargetlist = lappend(joinrel->reltargetlist, var); joinrel->width += baserel->attr_widths[ndx]; } } @@ -526,7 +532,7 @@ build_joinrel_tlist(PlannerInfo *root, RelOptInfo *joinrel, * the join list need only be computed once for any join RelOptInfo. * The join list is fully determined by the set of rels making up the * joinrel, so we should get the same results (up to ordering) from any - * candidate pair of sub-relations. But the restriction list is whatever + * candidate pair of sub-relations. But the restriction list is whatever * is not handled in the sub-relations, so it depends on which * sub-relations are considered. * @@ -535,7 +541,7 @@ build_joinrel_tlist(PlannerInfo *root, RelOptInfo *joinrel, * we put it into the joininfo list for the joinrel. Otherwise, * the clause is now a restrict clause for the joined relation, and we * return it to the caller of build_joinrel_restrictlist() to be stored in - * join paths made from this pair of sub-relations. (It will not need to + * join paths made from this pair of sub-relations. (It will not need to * be considered further up the join tree.) * * In many case we will find the same RestrictInfos in both input @@ -554,7 +560,7 @@ build_joinrel_tlist(PlannerInfo *root, RelOptInfo *joinrel, * * NB: Formerly, we made deep(!) copies of each input RestrictInfo to pass * up to the join relation. I believe this is no longer necessary, because - * RestrictInfo nodes are no longer context-dependent. Instead, just include + * RestrictInfo nodes are no longer context-dependent. Instead, just include * the original nodes in the lists made for the join relation. */ static List * @@ -574,7 +580,7 @@ build_joinrel_restrictlist(PlannerInfo *root, result = subbuild_joinrel_restrictlist(joinrel, inner_rel->joininfo, result); /* - * Add on any clauses derived from EquivalenceClasses. These cannot be + * Add on any clauses derived from EquivalenceClasses. These cannot be * redundant with the clauses in the joininfo lists, so don't bother * checking. */ @@ -676,6 +682,36 @@ subbuild_joinrel_joinlist(RelOptInfo *joinrel, /* + * build_empty_join_rel + * Build a dummy join relation describing an empty set of base rels. + * + * This is used for queries with empty FROM clauses, such as "SELECT 2+2" or + * "INSERT INTO foo VALUES(...)". We don't try very hard to make the empty + * joinrel completely valid, since no real planning will be done with it --- + * we just need it to carry a simple Result path out of query_planner(). + */ +RelOptInfo * +build_empty_join_rel(PlannerInfo *root) +{ + RelOptInfo *joinrel; + + /* The dummy join relation should be the only one ... */ + Assert(root->join_rel_list == NIL); + + joinrel = makeNode(RelOptInfo); + joinrel->reloptkind = RELOPT_JOINREL; + joinrel->relids = NULL; /* empty set */ + joinrel->rows = 1; /* we produce one row for such cases */ + joinrel->width = 0; /* it contains no Vars */ + joinrel->rtekind = RTE_JOIN; + + root->join_rel_list = lappend(root->join_rel_list, joinrel); + + return joinrel; +} + + +/* * find_childrel_appendrelinfo * Get the AppendRelInfo associated with an appendrel child rel. * @@ -912,7 +948,7 @@ get_joinrel_parampathinfo(PlannerInfo *root, RelOptInfo *joinrel, *restrict_clauses); /* - * And now we can build the ParamPathInfo. No point in saving the + * And now we can build the ParamPathInfo. No point in saving the * input-pair-dependent clause list, though. * * Note: in GEQO mode, we'll be called in a temporary memory context, but @@ -932,8 +968,8 @@ get_joinrel_parampathinfo(PlannerInfo *root, RelOptInfo *joinrel, * Get the ParamPathInfo for a parameterized path for an append relation. * * For an append relation, the rowcount estimate will just be the sum of - * the estimates for its children. However, we still need a ParamPathInfo - * to flag the fact that the path requires parameters. So this just creates + * the estimates for its children. However, we still need a ParamPathInfo + * to flag the fact that the path requires parameters. So this just creates * a suitable struct with zero ppi_rows (and no ppi_clauses either, since * the Append node isn't responsible for checking quals). */ diff --git a/src/backend/optimizer/util/restrictinfo.c b/src/backend/optimizer/util/restrictinfo.c index 48e96ab5d7..e861ce6657 100644 --- a/src/backend/optimizer/util/restrictinfo.c +++ b/src/backend/optimizer/util/restrictinfo.c @@ -3,7 +3,7 @@ * restrictinfo.c * RestrictInfo node manipulation routines. * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -15,7 +15,6 @@ #include "postgres.h" #include "optimizer/clauses.h" -#include "optimizer/predtest.h" #include "optimizer/restrictinfo.h" #include "optimizer/var.h" @@ -88,192 +87,6 @@ make_restrictinfo(Expr *clause, } /* - * make_restrictinfo_from_bitmapqual - * - * Given the bitmapqual Path structure for a bitmap indexscan, generate - * RestrictInfo node(s) equivalent to the condition represented by the - * indexclauses of the Path structure. - * - * The result is a List (effectively, implicit-AND representation) of - * RestrictInfos. - * - * The caller must pass is_pushed_down, but we assume outerjoin_delayed - * and pseudoconstant are false while outer_relids and nullable_relids - * are NULL (no other kind of qual should ever get into a bitmapqual). - * - * If include_predicates is true, we add any partial index predicates to - * the explicit index quals. When this is not true, we return a condition - * that might be weaker than the actual scan represents. - * - * To do this through the normal make_restrictinfo() API, callers would have - * to strip off the RestrictInfo nodes present in the indexclauses lists, and - * then make_restrictinfo() would have to build new ones. It's better to have - * a specialized routine to allow sharing of RestrictInfos. - * - * The qual manipulations here are much the same as in create_bitmap_subplan; - * keep the two routines in sync! - */ -List * -make_restrictinfo_from_bitmapqual(Path *bitmapqual, - bool is_pushed_down, - bool include_predicates) -{ - List *result; - ListCell *l; - - if (IsA(bitmapqual, BitmapAndPath)) - { - BitmapAndPath *apath = (BitmapAndPath *) bitmapqual; - - /* - * There may well be redundant quals among the subplans, since a - * top-level WHERE qual might have gotten used to form several - * different index quals. We don't try exceedingly hard to eliminate - * redundancies, but we do eliminate obvious duplicates by using - * list_concat_unique. - */ - result = NIL; - foreach(l, apath->bitmapquals) - { - List *sublist; - - sublist = make_restrictinfo_from_bitmapqual((Path *) lfirst(l), - is_pushed_down, - include_predicates); - result = list_concat_unique(result, sublist); - } - } - else if (IsA(bitmapqual, BitmapOrPath)) - { - BitmapOrPath *opath = (BitmapOrPath *) bitmapqual; - List *withris = NIL; - List *withoutris = NIL; - - /* - * Here, we only detect qual-free subplans. A qual-free subplan would - * cause us to generate "... OR true ..." which we may as well reduce - * to just "true". We do not try to eliminate redundant subclauses - * because (a) it's not as likely as in the AND case, and (b) we might - * well be working with hundreds or even thousands of OR conditions, - * perhaps from a long IN list. The performance of list_append_unique - * would be unacceptable. - */ - foreach(l, opath->bitmapquals) - { - List *sublist; - - sublist = make_restrictinfo_from_bitmapqual((Path *) lfirst(l), - is_pushed_down, - include_predicates); - if (sublist == NIL) - { - /* - * If we find a qual-less subscan, it represents a constant - * TRUE, and hence the OR result is also constant TRUE, so we - * can stop here. - */ - return NIL; - } - - /* - * If the sublist contains multiple RestrictInfos, we create an - * AND subclause. If there's just one, we have to check if it's - * an OR clause, and if so flatten it to preserve AND/OR flatness - * of our output. - * - * We construct lists with and without sub-RestrictInfos, so as - * not to have to regenerate duplicate RestrictInfos below. - */ - if (list_length(sublist) > 1) - { - withris = lappend(withris, make_andclause(sublist)); - sublist = get_actual_clauses(sublist); - withoutris = lappend(withoutris, make_andclause(sublist)); - } - else - { - RestrictInfo *subri = (RestrictInfo *) linitial(sublist); - - Assert(IsA(subri, RestrictInfo)); - if (restriction_is_or_clause(subri)) - { - BoolExpr *subor = (BoolExpr *) subri->orclause; - - Assert(or_clause((Node *) subor)); - withris = list_concat(withris, - list_copy(subor->args)); - subor = (BoolExpr *) subri->clause; - Assert(or_clause((Node *) subor)); - withoutris = list_concat(withoutris, - list_copy(subor->args)); - } - else - { - withris = lappend(withris, subri); - withoutris = lappend(withoutris, subri->clause); - } - } - } - - /* - * Avoid generating one-element ORs, which could happen due to - * redundancy elimination or ScalarArrayOpExpr quals. - */ - if (list_length(withris) <= 1) - result = withris; - else - { - /* Here's the magic part not available to outside callers */ - result = - list_make1(make_restrictinfo_internal(make_orclause(withoutris), - make_orclause(withris), - is_pushed_down, - false, - false, - NULL, - NULL, - NULL)); - } - } - else if (IsA(bitmapqual, IndexPath)) - { - IndexPath *ipath = (IndexPath *) bitmapqual; - - result = list_copy(ipath->indexclauses); - if (include_predicates && ipath->indexinfo->indpred != NIL) - { - foreach(l, ipath->indexinfo->indpred) - { - Expr *pred = (Expr *) lfirst(l); - - /* - * We know that the index predicate must have been implied by - * the query condition as a whole, but it may or may not be - * implied by the conditions that got pushed into the - * bitmapqual. Avoid generating redundant conditions. - */ - if (!predicate_implied_by(list_make1(pred), result)) - result = lappend(result, - make_restrictinfo(pred, - is_pushed_down, - false, - false, - NULL, - NULL, - NULL)); - } - } - } - else - { - elog(ERROR, "unrecognized node type: %d", nodeTag(bitmapqual)); - result = NIL; /* keep compiler quiet */ - } - - return result; -} - -/* * make_restrictinfos_from_actual_clauses * * Given a list of implicitly-ANDed restriction clauses, produce a list @@ -397,7 +210,7 @@ make_restrictinfo_internal(Expr *clause, /* * Fill in all the cacheable fields with "not yet set" markers. None of - * these will be computed until/unless needed. Note in particular that we + * these will be computed until/unless needed. Note in particular that we * don't mark a binary opclause as mergejoinable or hashjoinable here; * that happens only if it appears in the right context (top level of a * joinclause list). @@ -651,24 +464,32 @@ extract_actual_join_clauses(List *restrictinfo_list, * outer join, as that would change the results (rows would be suppressed * rather than being null-extended). * - * And the target relation must not be in the clause's nullable_relids, i.e., + * Also the target relation must not be in the clause's nullable_relids, i.e., * there must not be an outer join below the clause that would null the Vars * coming from the target relation. Otherwise the clause might give results * different from what it would give at its normal semantic level. + * + * Also, the join clause must not use any relations that have LATERAL + * references to the target relation, since we could not put such rels on + * the outer side of a nestloop with the target relation. */ bool -join_clause_is_movable_to(RestrictInfo *rinfo, Index baserelid) +join_clause_is_movable_to(RestrictInfo *rinfo, RelOptInfo *baserel) { /* Clause must physically reference target rel */ - if (!bms_is_member(baserelid, rinfo->clause_relids)) + if (!bms_is_member(baserel->relid, rinfo->clause_relids)) return false; /* Cannot move an outer-join clause into the join's outer side */ - if (bms_is_member(baserelid, rinfo->outer_relids)) + if (bms_is_member(baserel->relid, rinfo->outer_relids)) return false; /* Target rel must not be nullable below the clause */ - if (bms_is_member(baserelid, rinfo->nullable_relids)) + if (bms_is_member(baserel->relid, rinfo->nullable_relids)) + return false; + + /* Clause must not use any rels with LATERAL references to this rel */ + if (bms_overlap(baserel->lateral_referencers, rinfo->clause_relids)) return false; return true; @@ -695,6 +516,11 @@ join_clause_is_movable_to(RestrictInfo *rinfo, Index baserelid) * not pushing the clause into its outer-join outer side, nor down into * a lower outer join's inner side. * + * There's no check here equivalent to join_clause_is_movable_to's test on + * lateral_referencers. We assume the caller wouldn't be inquiring unless + * it'd verified that the proposed outer rels don't have lateral references + * to the current rel(s). + * * Note: get_joinrel_parampathinfo depends on the fact that if * current_and_outer is NULL, this function will always return false * (since one or the other of the first two tests must fail). diff --git a/src/backend/optimizer/util/tlist.c b/src/backend/optimizer/util/tlist.c index 3031c404f5..f1f1be1b7f 100644 --- a/src/backend/optimizer/util/tlist.c +++ b/src/backend/optimizer/util/tlist.c @@ -3,7 +3,7 @@ * tlist.c * Target list manipulation routines * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -26,7 +26,7 @@ /* * tlist_member * Finds the (first) member of the given tlist whose expression is - * equal() to the given expression. Result is NULL if no such member. + * equal() to the given expression. Result is NULL if no such member. */ TargetEntry * tlist_member(Node *node, List *targetlist) @@ -72,6 +72,35 @@ tlist_member_ignore_relabel(Node *node, List *targetlist) } /* + * tlist_member_match_var + * Same as above, except that we match the provided Var on the basis + * of varno/varattno/varlevelsup only, rather than using full equal(). + * + * This is needed in some cases where we can't be sure of an exact typmod + * match. It's probably a good idea to check the vartype anyway, but + * we leave it to the caller to apply any suitable sanity checks. + */ +TargetEntry * +tlist_member_match_var(Var *var, List *targetlist) +{ + ListCell *temp; + + foreach(temp, targetlist) + { + TargetEntry *tlentry = (TargetEntry *) lfirst(temp); + Var *tlvar = (Var *) tlentry->expr; + + if (!tlvar || !IsA(tlvar, Var)) + continue; + if (var->varno == tlvar->varno && + var->varattno == tlvar->varattno && + var->varlevelsup == tlvar->varlevelsup) + return tlentry; + } + return NULL; +} + +/* * flatten_tlist * Create a target list that only contains unique variables. * @@ -159,6 +188,43 @@ get_tlist_exprs(List *tlist, bool includeJunk) /* + * tlist_same_exprs + * Check whether two target lists contain the same expressions + * + * Note: this function is used to decide whether it's safe to jam a new tlist + * into a non-projection-capable plan node. Obviously we can't do that unless + * the node's tlist shows it already returns the column values we want. + * However, we can ignore the TargetEntry attributes resname, ressortgroupref, + * resorigtbl, resorigcol, and resjunk, because those are only labelings that + * don't affect the row values computed by the node. (Moreover, if we didn't + * ignore them, we'd frequently fail to make the desired optimization, since + * the planner tends to not bother to make resname etc. valid in intermediate + * plan nodes.) Note that on success, the caller must still jam the desired + * tlist into the plan node, else it won't have the desired labeling fields. + */ +bool +tlist_same_exprs(List *tlist1, List *tlist2) +{ + ListCell *lc1, + *lc2; + + if (list_length(tlist1) != list_length(tlist2)) + return false; /* not same length, so can't match */ + + forboth(lc1, tlist1, lc2, tlist2) + { + TargetEntry *tle1 = (TargetEntry *) lfirst(lc1); + TargetEntry *tle2 = (TargetEntry *) lfirst(lc2); + + if (!equal(tle1->expr, tle2->expr)) + return false; + } + + return true; +} + + +/* * Does tlist have same output datatypes as listed in colTypes? * * Resjunk columns are ignored if junkOK is true; otherwise presence of diff --git a/src/backend/optimizer/util/var.c b/src/backend/optimizer/util/var.c index 9bc90c2531..d4f46b8d46 100644 --- a/src/backend/optimizer/util/var.c +++ b/src/backend/optimizer/util/var.c @@ -9,7 +9,7 @@ * contains variables. * * - * Portions Copyright (c) 1996-2012, PostgreSQL Global Development Group + * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group * Portions Copyright (c) 1994, Regents of the University of California * * @@ -42,22 +42,15 @@ typedef struct typedef struct { - int var_location; + List *vars; int sublevels_up; -} locate_var_of_level_context; +} pull_vars_context; typedef struct { int var_location; - int relid; int sublevels_up; -} locate_var_of_relation_context; - -typedef struct -{ - int min_varlevel; - int sublevels_up; -} find_minimum_var_level_context; +} locate_var_of_level_context; typedef struct { @@ -77,14 +70,11 @@ typedef struct static bool pull_varnos_walker(Node *node, pull_varnos_context *context); static bool pull_varattnos_walker(Node *node, pull_varattnos_context *context); +static bool pull_vars_walker(Node *node, pull_vars_context *context); static bool contain_var_clause_walker(Node *node, void *context); static bool contain_vars_of_level_walker(Node *node, int *sublevels_up); static bool locate_var_of_level_walker(Node *node, locate_var_of_level_context *context); -static bool locate_var_of_relation_walker(Node *node, - locate_var_of_relation_context *context); -static bool find_minimum_var_level_walker(Node *node, - find_minimum_var_level_context *context); static bool pull_var_clause_walker(Node *node, pull_var_clause_context *context); static Node *flatten_join_alias_vars_mutator(Node *node, @@ -122,6 +112,31 @@ pull_varnos(Node *node) return context.varnos; } +/* + * pull_varnos_of_level + * Create a set of all the distinct varnos present in a parsetree. + * Only Vars of the specified level are considered. + */ +Relids +pull_varnos_of_level(Node *node, int levelsup) +{ + pull_varnos_context context; + + context.varnos = NULL; + context.sublevels_up = levelsup; + + /* + * Must be prepared to start with a Query or a bare expression tree; if + * it's a Query, we don't want to increment sublevels_up. + */ + query_or_expression_tree_walker(node, + pull_varnos_walker, + (void *) &context, + 0); + + return context.varnos; +} + static bool pull_varnos_walker(Node *node, pull_varnos_context *context) { @@ -146,8 +161,13 @@ pull_varnos_walker(Node *node, pull_varnos_context *context) if (IsA(node, PlaceHolderVar)) { /* - * Normally, we can just take the varnos in the contained expression. - * But if it is variable-free, use the PHV's syntactic relids. + * A PlaceHolderVar acts as a variable of its syntactic scope, or + * lower than that if it references only a subset of the rels in its + * syntactic scope. It might also contain lateral references, but we + * should ignore such references when computing the set of varnos in + * an expression tree. Also, if the PHV contains no variables within + * its syntactic scope, it will be forced to be evaluated exactly at + * the syntactic scope, so take that as the relid set. */ PlaceHolderVar *phv = (PlaceHolderVar *) node; pull_varnos_context subcontext; @@ -155,12 +175,15 @@ pull_varnos_walker(Node *node, pull_varnos_context *context) subcontext.varnos = NULL; subcontext.sublevels_up = context->sublevels_up; (void) pull_varnos_walker((Node *) phv->phexpr, &subcontext); - - if (bms_is_empty(subcontext.varnos) && - phv->phlevelsup == context->sublevels_up) - context->varnos = bms_add_members(context->varnos, phv->phrels); - else - context->varnos = bms_join(context->varnos, subcontext.varnos); + if (phv->phlevelsup == context->sublevels_up) + { + subcontext.varnos = bms_int_members(subcontext.varnos, + phv->phrels); + if (bms_is_empty(subcontext.varnos)) + context->varnos = bms_add_members(context->varnos, + phv->phrels); + } + context->varnos = bms_join(context->varnos, subcontext.varnos); return false; } if (IsA(node, Query)) @@ -231,6 +254,71 @@ pull_varattnos_walker(Node *node, pull_varattnos_context *context) /* + * pull_vars_of_level + * Create a list of all Vars (and PlaceHolderVars) referencing the + * specified query level in the given parsetree. + * + * Caution: the Vars are not copied, only linked into the list. + */ +List * +pull_vars_of_level(Node *node, int levelsup) +{ + pull_vars_context context; + + context.vars = NIL; + context.sublevels_up = levelsup; + + /* + * Must be prepared to start with a Query or a bare expression tree; if + * it's a Query, we don't want to increment sublevels_up. + */ + query_or_expression_tree_walker(node, + pull_vars_walker, + (void *) &context, + 0); + + return context.vars; +} + +static bool +pull_vars_walker(Node *node, pull_vars_context *context) +{ + if (node == NULL) + return false; + if (IsA(node, Var)) + { + Var *var = (Var *) node; + + if (var->varlevelsup == context->sublevels_up) + context->vars = lappend(context->vars, var); + return false; + } + if (IsA(node, PlaceHolderVar)) + { + PlaceHolderVar *phv = (PlaceHolderVar *) node; + + if (phv->phlevelsup == context->sublevels_up) + context->vars = lappend(context->vars, phv); + /* we don't want to look into the contained expression */ + return false; + } + if (IsA(node, Query)) + { + /* Recurse into RTE subquery or not-yet-planned sublink subquery */ + bool result; + + context->sublevels_up++; + result = query_tree_walker((Query *) node, pull_vars_walker, + (void *) context, 0); + context->sublevels_up--; + return result; + } + return expression_tree_walker(node, pull_vars_walker, + (void *) context); +} + + +/* * contain_var_clause * Recursively scan a clause to discover whether it contains any Var nodes * (of the current query level). @@ -276,7 +364,7 @@ contain_var_clause_walker(Node *node, void *context) * * Returns true if any such Var found. * - * Will recurse into sublinks. Also, may be invoked directly on a Query. + * Will recurse into sublinks. Also, may be invoked directly on a Query. */ bool contain_vars_of_level(Node *node, int levelsup) @@ -336,10 +424,10 @@ contain_vars_of_level_walker(Node *node, int *sublevels_up) * Find the parse location of any Var of the specified query level. * * Returns -1 if no such Var is in the querytree, or if they all have - * unknown parse location. (The former case is probably caller error, + * unknown parse location. (The former case is probably caller error, * but we don't bother to distinguish it from the latter case.) * - * Will recurse into sublinks. Also, may be invoked directly on a Query. + * Will recurse into sublinks. Also, may be invoked directly on a Query. * * Note: it might seem appropriate to merge this functionality into * contain_vars_of_level, but that would complicate that function's API. @@ -406,229 +494,6 @@ locate_var_of_level_walker(Node *node, /* - * locate_var_of_relation - * Find the parse location of any Var of the specified relation. - * - * Returns -1 if no such Var is in the querytree, or if they all have - * unknown parse location. - * - * Will recurse into sublinks. Also, may be invoked directly on a Query. - */ -int -locate_var_of_relation(Node *node, int relid, int levelsup) -{ - locate_var_of_relation_context context; - - context.var_location = -1; /* in case we find nothing */ - context.relid = relid; - context.sublevels_up = levelsup; - - (void) query_or_expression_tree_walker(node, - locate_var_of_relation_walker, - (void *) &context, - 0); - - return context.var_location; -} - -static bool -locate_var_of_relation_walker(Node *node, - locate_var_of_relation_context *context) -{ - if (node == NULL) - return false; - if (IsA(node, Var)) - { - Var *var = (Var *) node; - - if (var->varno == context->relid && - var->varlevelsup == context->sublevels_up && - var->location >= 0) - { - context->var_location = var->location; - return true; /* abort tree traversal and return true */ - } - return false; - } - if (IsA(node, CurrentOfExpr)) - { - /* since CurrentOfExpr doesn't carry location, nothing we can do */ - return false; - } - /* No extra code needed for PlaceHolderVar; just look in contained expr */ - if (IsA(node, Query)) - { - /* Recurse into subselects */ - bool result; - - context->sublevels_up++; - result = query_tree_walker((Query *) node, - locate_var_of_relation_walker, - (void *) context, - 0); - context->sublevels_up--; - return result; - } - return expression_tree_walker(node, - locate_var_of_relation_walker, - (void *) context); -} - - -/* - * find_minimum_var_level - * Recursively scan a clause to find the lowest variable level it - * contains --- for example, zero is returned if there are any local - * variables, one if there are no local variables but there are - * one-level-up outer references, etc. Subqueries are scanned to see - * if they possess relevant outer references. (But any local variables - * within subqueries are not relevant.) - * - * -1 is returned if the clause has no variables at all. - * - * Will recurse into sublinks. Also, may be invoked directly on a Query. - */ -int -find_minimum_var_level(Node *node) -{ - find_minimum_var_level_context context; - - context.min_varlevel = -1; /* signifies nothing found yet */ - context.sublevels_up = 0; - - (void) query_or_expression_tree_walker(node, - find_minimum_var_level_walker, - (void *) &context, - 0); - - return context.min_varlevel; -} - -static bool -find_minimum_var_level_walker(Node *node, - find_minimum_var_level_context *context) -{ - if (node == NULL) - return false; - if (IsA(node, Var)) - { - int varlevelsup = ((Var *) node)->varlevelsup; - - /* convert levelsup to frame of reference of original query */ - varlevelsup -= context->sublevels_up; - /* ignore local vars of subqueries */ - if (varlevelsup >= 0) - { - if (context->min_varlevel < 0 || - context->min_varlevel > varlevelsup) - { - context->min_varlevel = varlevelsup; - - /* - * As soon as we find a local variable, we can abort the tree - * traversal, since min_varlevel is then certainly 0. - */ - if (varlevelsup == 0) - return true; - } - } - } - if (IsA(node, CurrentOfExpr)) - { - int varlevelsup = 0; - - /* convert levelsup to frame of reference of original query */ - varlevelsup -= context->sublevels_up; - /* ignore local vars of subqueries */ - if (varlevelsup >= 0) - { - if (context->min_varlevel < 0 || - context->min_varlevel > varlevelsup) - { - context->min_varlevel = varlevelsup; - - /* - * As soon as we find a local variable, we can abort the tree - * traversal, since min_varlevel is then certainly 0. - */ - if (varlevelsup == 0) - return true; - } - } - } - - /* - * An Aggref must be treated like a Var of its level. Normally we'd get - * the same result from looking at the Vars in the aggregate's argument, - * but this fails in the case of a Var-less aggregate call (COUNT(*)). - */ - if (IsA(node, Aggref)) - { - int agglevelsup = ((Aggref *) node)->agglevelsup; - - /* convert levelsup to frame of reference of original query */ - agglevelsup -= context->sublevels_up; - /* ignore local aggs of subqueries */ - if (agglevelsup >= 0) - { - if (context->min_varlevel < 0 || - context->min_varlevel > agglevelsup) - { - context->min_varlevel = agglevelsup; - - /* - * As soon as we find a local aggregate, we can abort the tree - * traversal, since min_varlevel is then certainly 0. - */ - if (agglevelsup == 0) - return true; - } - } - } - /* Likewise, make sure PlaceHolderVar is treated correctly */ - if (IsA(node, PlaceHolderVar)) - { - int phlevelsup = ((PlaceHolderVar *) node)->phlevelsup; - - /* convert levelsup to frame of reference of original query */ - phlevelsup -= context->sublevels_up; - /* ignore local vars of subqueries */ - if (phlevelsup >= 0) - { - if (context->min_varlevel < 0 || - context->min_varlevel > phlevelsup) - { - context->min_varlevel = phlevelsup; - - /* - * As soon as we find a local variable, we can abort the tree - * traversal, since min_varlevel is then certainly 0. - */ - if (phlevelsup == 0) - return true; - } - } - } - if (IsA(node, Query)) - { - /* Recurse into subselects */ - bool result; - - context->sublevels_up++; - result = query_tree_walker((Query *) node, - find_minimum_var_level_walker, - (void *) context, - 0); - context->sublevels_up--; - return result; - } - return expression_tree_walker(node, - find_minimum_var_level_walker, - (void *) context); -} - - -/* * pull_var_clause * Recursively pulls all Var nodes from an expression clause. * @@ -649,7 +514,7 @@ find_minimum_var_level_walker(Node *node, * Upper-level vars (with varlevelsup > 0) should not be seen here, * likewise for upper-level Aggrefs and PlaceHolderVars. * - * Returns list of nodes found. Note the nodes themselves are not + * Returns list of nodes found. Note the nodes themselves are not * copied, only referenced. * * Does not examine subqueries, therefore must only be used after reduction @@ -726,7 +591,7 @@ pull_var_clause_walker(Node *node, pull_var_clause_context *context) * flatten_join_alias_vars * Replace Vars that reference JOIN outputs with references to the original * relation variables instead. This allows quals involving such vars to be - * pushed down. Whole-row Vars that reference JOIN relations are expanded + * pushed down. Whole-row Vars that reference JOIN relations are expanded * into RowExpr constructs that name the individual output Vars. This * is necessary since we will not scan the JOIN as a base relation, which * is the only way that the executor can directly handle whole-row Vars. @@ -738,12 +603,14 @@ pull_var_clause_walker(Node *node, pull_var_clause_context *context) * entries might now be arbitrary expressions, not just Vars. This affects * this function in one important way: we might find ourselves inserting * SubLink expressions into subqueries, and we must make sure that their - * Query.hasSubLinks fields get set to TRUE if so. If there are any + * Query.hasSubLinks fields get set to TRUE if so. If there are any * SubLinks in the join alias lists, the outer Query should already have * hasSubLinks = TRUE, so this is only relevant to un-flattened subqueries. * * NOTE: this is used on not-yet-planned expressions. We do not expect it - * to be applied directly to a Query node. + * to be applied directly to the whole Query, so if we see a Query to start + * with, we do want to increment sublevels_up (this occurs for LATERAL + * subqueries). */ Node * flatten_join_alias_vars(PlannerInfo *root, Node *node) @@ -795,7 +662,7 @@ flatten_join_alias_vars_mutator(Node *node, newvar = (Node *) lfirst(lv); attnum++; /* Ignore dropped columns */ - if (IsA(newvar, Const)) + if (newvar == NULL) continue; newvar = copyObject(newvar); @@ -828,6 +695,7 @@ flatten_join_alias_vars_mutator(Node *node, /* Expand join alias reference */ Assert(var->varattno > 0); newvar = (Node *) list_nth(rte->joinaliasvars, var->varattno - 1); + Assert(newvar != NULL); newvar = copyObject(newvar); /* @@ -889,6 +757,7 @@ flatten_join_alias_vars_mutator(Node *node, Assert(!IsA(node, SubPlan)); /* Shouldn't need to handle these planner auxiliary nodes here */ Assert(!IsA(node, SpecialJoinInfo)); + Assert(!IsA(node, LateralJoinInfo)); Assert(!IsA(node, PlaceHolderInfo)); Assert(!IsA(node, MinMaxAggInfo)); |