comparison chaining has wrong source positions in most contexts (python 3.11.0rc2)

[15r10nk]

problem

comparison chaining has wrong source positions in most contexts

>>> cnd = True                                            
>>> a = 1 if 1 in 2 == 3 else 0                                                                
Traceback (most recent call last):                                                              
  File "example.py", line 43, in <module>                                                             
    a = 1 if 1 in 2 == 3 else 0                                              
        ^^^^^^^^^^^^^^^^^^^^^^^                                                                  
TypeError: argument of type 'int' is not iterable  

This bug requires two things to happen:

1. The comparison chain has to contain more than on comparison the type does not matter (==,in,<=,is,...)
2. The context has to be something other than an expression

no problem

The following examples work:
only one comparison

Traceback (most recent call last):
  File "example.py", line 4, in <module>
    if 5<"5":
       ^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

inside expression

>>> a=False or 4<"5"<6
Traceback (most recent call last):
  File "example.py", line 15, in <module>
    a=False or 4<"5"<6
               ^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

reason

The problem seems to be the positions in the compiled bytecode:

script:

import dis

def foo():
    if 1<2<"no int":
        return 1
    return 0

for p in dis.get_instructions(foo):
    print(p.positions, p.opname,p.argval)

output (Python 3.11.0rc2+):

Positions(lineno=4, end_lineno=4, col_offset=0, end_col_offset=0) RESUME 0
Positions(lineno=5, end_lineno=5, col_offset=7, end_col_offset=8) LOAD_CONST 1
Positions(lineno=5, end_lineno=5, col_offset=9, end_col_offset=10) LOAD_CONST 2
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) SWAP 2
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) COPY 2
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) COMPARE_OP < # range of the whole if
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) POP_JUMP_FORWARD_IF_FALSE 30
Positions(lineno=5, end_lineno=5, col_offset=11, end_col_offset=19) LOAD_CONST no int
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) COMPARE_OP < # range of the whole if
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) POP_JUMP_FORWARD_IF_FALSE 38
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) JUMP_FORWARD 34
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) POP_TOP None
Positions(lineno=5, end_lineno=6, col_offset=4, end_col_offset=16) JUMP_FORWARD 38
Positions(lineno=6, end_lineno=6, col_offset=15, end_col_offset=16) LOAD_CONST 1
Positions(lineno=6, end_lineno=6, col_offset=15, end_col_offset=16) RETURN_VALUE None
Positions(lineno=7, end_lineno=7, col_offset=11, end_col_offset=12) LOAD_CONST 0
Positions(lineno=7, end_lineno=7, col_offset=11, end_col_offset=12) RETURN_VALUE None

the generated ast looks fine (snippet):

            If(
               test=Compare(
                  left=Constant(
                     value=1,
                     lineno=5,
                     col_offset=7, # correct
                     end_lineno=5,
                     end_col_offset=8),
                  ops=[
                     Lt(),
                     Lt()],
                  comparators=[
                     Constant(
                        value=2,
                        lineno=5,
                        col_offset=9, # correct
                        end_lineno=5,
                        end_col_offset=10),
                     Constant(
                        value='no int',
                        lineno=5,
                        col_offset=11, # correct
                        end_lineno=5,
                        end_col_offset=19)],
                  lineno=5,
                  col_offset=7,
                  end_lineno=5,
                  end_col_offset=19),

other examples

here is a list of all the problems I found so far:

>>> if 4<"5"<6:
...     pass
...
Traceback (most recent call last):
  File "example.py", line 20, in <module>
    if 4<"5"<6:
TypeError: '<' not supported between instances of 'int' and 'str'
>>> while 4<"5"<6:
...     pass
...
Traceback (most recent call last):
  File "example.py", line 24, in <module>
    while 4<"5"<6:
TypeError: '<' not supported between instances of 'int' and 'str'

>>> assert 4<"5"<6
Traceback (most recent call last):
  File "example.py", line 27, in <module>
    assert 4<"5"<6
TypeError: '<' not supported between instances of 'int' and 'str'

>>> a=2+(5 if 4<"5"<6 else 3)
Traceback (most recent call last):
  File "example.py", line 29, in <module>
    a=2+(5 if 4<"5"<6 else 3)
         ^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

>>> a=[a for a in [1] if 4<"5"<a ]
Traceback (most recent call last):
  File "example.py", line 34, in <listcomp>
    a=[a for a in [1] if 4<"5"<a ]
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

>>> a={a for a in [1] if 4<"5"<a }
Traceback (most recent call last):
  File "example.py", line 35, in <setcomp>
    a={a for a in [1] if 4<"5"<a }
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

>>> a={a:a for a in [1] if 4<"5"<a }
Traceback (most recent call last):
  File "example.py", line 36, in <dictcomp>
    a={a:a for a in [1] if 4<"5"<a }
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

>>> a=list((a for a in [1] if 4<"5"<a ))
Traceback (most recent call last):
  File "example.py", line 37, in <genexpr>
    a=list((a for a in [1] if 4<"5"<a ))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

>>> cnd = True
>>> a = 1 if 1 in 2 == 3 else 0
Traceback (most recent call last):
  File "example.py", line 43, in <module>
    a = 1 if 1 in 2 == 3 else 0
        ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'int' is not iterable

>>> if True and 4<"5"<6:
...     pass
...
Traceback (most recent call last):
  File "example.py", line 45, in <module>
    if True and 4<"5"<6:
TypeError: '<' not supported between instances of 'int' and 'str'

expected result

The positions should always match the ast-node range:

>>> a=list((a for a in [1] if 4<"5"<a ))
Traceback (most recent call last):
  File "example.py", line 37, in <genexpr>
    a=list((a for a in [1] if 4<"5"<a ))
                              ^^^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

or, even better only the failing comparison:

>>> a=list((a for a in [1] if 4<"5"<a ))
Traceback (most recent call last):
  File "example.py", line 37, in <genexpr>
    a=list((a for a in [1] if 4<"5"<a ))
                              ^^^^^
TypeError: '<' not supported between instances of 'int' and 'str'

[15r10nk]

@brandtbucher I think this is related to #93691.
Do you know if there someone already working on this?

Does it makes sense for me to look into it? I have (almost) no cpython core developer experience and it would take me probably far longer than someone else.

[brandtbucher]

Thanks for the detailed report. This issue seems to stem from the fact that we have two implementations of comparisons in the bytecode compiler: one generic version (compiler_compare) and one optimized for branching conditions (in compiler_jump_if). The latter doesn’t update the location to the current node, which is probably an oversight.

I’ll have a fix up either today or tomorrow. It will probably need to wait for 3.11.1, though.

[brandtbucher]

Thanks again for the great bug report!