Fix assorted oversights in range selectivity estimation.

calc_rangesel() failed outright when comparing range variables to empty
constant ranges with < or >=, as a result of missing cases in a switch.
It also produced a bogus estimate for > comparison to an empty range.

On top of that, the >= and > cases were mislabeled throughout.  For
nonempty constant ranges, they managed to produce the right answers
anyway as a result of counterbalancing typos.

Also, default_range_selectivity() omitted cases for elem <@ range,
range &< range, and range &> range, so that rather dubious defaults
were applied for these operators.

In passing, rearrange the code in rangesel() so that the elem <@ range
case is handled in a less opaque fashion.

Report and patch by Emre Hasegeli, some additional work by me

2 files changed, 36 insertions, 0 deletions

diff --git a/src/test/regress/expected/rangetypes.out b/src/test/regress/expected/rangetypes.out
index 39db9927291..35d0dd3f677 100644
--- a/src/test/regress/expected/rangetypes.out
+++ b/src/test/regress/expected/rangetypes.out
@@ -260,6 +260,11 @@ select * from numrange_test where nr = '[1.1, 2.2)';
  [1.1,2.2)
 (1 row)
 
+select * from numrange_test where nr < 'empty';
+ nr 
+----
+(0 rows)
+
 select * from numrange_test where nr < numrange(-1000.0, -1000.0,'[]');
   nr   
 -------
@@ -287,6 +292,33 @@ select * from numrange_test where nr < numrange(1000.0, 1001.0,'[]');
  [1.7,1.7]
 (6 rows)
 
+select * from numrange_test where nr <= 'empty';
+  nr   
+-------
+ empty
+(1 row)
+
+select * from numrange_test where nr >= 'empty';
+    nr     
+-----------
+ (,)
+ [3,)
+ (,5)
+ [1.1,2.2)
+ empty
+ [1.7,1.7]
+(6 rows)
+
+select * from numrange_test where nr > 'empty';
+    nr     
+-----------
+ (,)
+ [3,)
+ (,5)
+ [1.1,2.2)
+ [1.7,1.7]
+(5 rows)
+
 select * from numrange_test where nr > numrange(-1001.0, -1000.0,'[]');
     nr     
 -----------
diff --git a/src/test/regress/sql/rangetypes.sql b/src/test/regress/sql/rangetypes.sql
index fad843a405f..aa026cad0ab 100644
--- a/src/test/regress/sql/rangetypes.sql
+++ b/src/test/regress/sql/rangetypes.sql
@@ -67,9 +67,13 @@ SELECT * FROM numrange_test WHERE 1.9 <@ nr;
 select * from numrange_test where nr = 'empty';
 select * from numrange_test where nr = '(1.1, 2.2)';
 select * from numrange_test where nr = '[1.1, 2.2)';
+select * from numrange_test where nr < 'empty';
 select * from numrange_test where nr < numrange(-1000.0, -1000.0,'[]');
 select * from numrange_test where nr < numrange(0.0, 1.0,'[]');
 select * from numrange_test where nr < numrange(1000.0, 1001.0,'[]');
+select * from numrange_test where nr <= 'empty';
+select * from numrange_test where nr >= 'empty';
+select * from numrange_test where nr > 'empty';
 select * from numrange_test where nr > numrange(-1001.0, -1000.0,'[]');
 select * from numrange_test where nr > numrange(0.0, 1.0,'[]');
 select * from numrange_test where nr > numrange(1000.0, 1000.0,'[]');