f-string string_parser.c regression: Assertion *str < end failed

[gpshead]

Build an interpreter with assertions enabled (--with-pydebug works) and have it try to parse the following: (execute it, paste it in a repl, call literal_eval, use -c, etc.)

./python -c 'f"{i="'

(gdb) run -c 'f"{i="'
... the usual ...
python: ../gpshead/Parser/string_parser.c:806: fstring_find_expr: Assertion `*str < end' failed.

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
49      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
#1  0x00007ffff7c86546 in __GI_abort () at abort.c:79
#2  0x00007ffff7c8642f in __assert_fail_base (fmt=0x7ffff7dfcdf8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x55555590ca64 "*str < end", file=0x55555590c3d0 "../gpshead/Parser/string_parser.c", line=806, function=<optimized out>)
    at assert.c:92
#3  0x00007ffff7c95222 in __GI___assert_fail (assertion=assertion@entry=0x55555590ca64 "*str < end", 
    file=file@entry=0x55555590c3d0 "../gpshead/Parser/string_parser.c", line=line@entry=806, 
    function=function@entry=0x55555590cc30 <__PRETTY_FUNCTION__.6> "fstring_find_expr") at assert.c:101
#4  0x00005555556af7f2 in fstring_find_expr (p=p@entry=0x7ffff7761840, str=str@entry=0x7fffffffcfe0, end=end@entry=0x7ffff773cd25 "\"", 
    raw=raw@entry=0, recurse_lvl=recurse_lvl@entry=0, expr_text=expr_text@entry=0x7fffffffcf50, expression=0x7fffffffcf58, 
    first_token=0x7ffff773ad40, t=0x7ffff773ad40, last_token=0x7ffff773ad40) at ../gpshead/Parser/string_parser.c:783
#5  0x00005555556af92e in fstring_find_literal_and_expr (p=p@entry=0x7ffff7761840, str=str@entry=0x7fffffffcfe0, 
    end=end@entry=0x7ffff773cd25 "\"", raw=raw@entry=0, recurse_lvl=recurse_lvl@entry=0, literal=literal@entry=0x7fffffffcf48, 
    expr_text=0x7fffffffcf50, expression=0x7fffffffcf58, first_token=0x7ffff773ad40, t=0x7ffff773ad40, last_token=0x7ffff773ad40)
    at ../gpshead/Parser/string_parser.c:917
#6  0x00005555556afb17 in _PyPegen_FstringParser_ConcatFstring (p=p@entry=0x7ffff7761840, state=state@entry=0x7fffffffcff0, 
    str=str@entry=0x7fffffffcfe0, end=0x7ffff773cd25 "\"", raw=0, recurse_lvl=recurse_lvl@entry=0, first_token=0x7ffff773ad40, t=0x7ffff773ad40, 
    last_token=0x7ffff773ad40) at ../gpshead/Parser/string_parser.c:1142
#7  0x00005555556480dd in _PyPegen_concatenate_strings (p=p@entry=0x7ffff7761840, strings=strings@entry=0x555555c43560)
    at ../gpshead/Parser/action_helpers.c:911
#8  0x0000555555652607 in strings_rule (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:15467
#9  0x0000555555661df1 in atom_rule (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:14273
#10 0x000055555567ac83 in t_primary_raw (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:18112
#11 0x000055555567afdf in t_primary_rule (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:17902
#12 0x000055555567f196 in single_subscript_attribute_target_rule (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:17791
#13 0x000055555567fbb0 in _tmp_12_rule (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:24111
#14 0x0000555555699f99 in assignment_rule (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:2241
#15 0x000055555569af35 in simple_stmt_rule (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:1648
#16 0x000055555569c67e in simple_stmts_rule (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:1543
#17 0x00005555556a2bfe in statement_rule (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:1366
#18 0x00005555556a2e61 in _loop1_3_rule (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:23542
#19 0x00005555556a3040 in statements_rule (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:1298
#20 0x00005555556aa782 in file_rule (p=0x7ffff7761840) at ../gpshead/Parser/parser.c:1057
#21 0x00005555556ad811 in _PyPegen_parse (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/parser.c:38559
#22 0x0000555555645162 in _PyPegen_run_parser (p=p@entry=0x7ffff7761840) at ../gpshead/Parser/pegen.c:817
#23 0x0000555555645445 in _PyPegen_run_parser_from_string (str=str@entry=0x7ffff788f8a0 "f\"{i=\"\n", start_rule=start_rule@entry=257, 
    filename_ob=filename_ob@entry='<string>', flags=flags@entry=0x7fffffffd628, arena=arena@entry=0x7ffff7787940)
    at ../gpshead/Parser/pegen.c:932
#24 0x00005555556afc7c in _PyParser_ASTFromString (str=str@entry=0x7ffff788f8a0 "f\"{i=\"\n", filename='<string>', mode=mode@entry=257, 
    flags=flags@entry=0x7fffffffd628, arena=arena@entry=0x7ffff7787940) at ../gpshead/Parser/peg_api.c:14
#25 0x000055555584fa0e in PyRun_StringFlags (str=str@entry=0x7ffff788f8a0 "f\"{i=\"\n", start=start@entry=257, 
    globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x555555bde730>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7907350>}, 
    locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <type at remote 0x555555bde730>, '__spec__': None, '__annotations__': {}, '__builtins__': <module at remote 0x7ffff7907350>}, flags=flags@entry=0x7fffffffd628) at ../gpshead/Python/pythonrun.c:1600
#26 0x000055555584fa8e in PyRun_SimpleStringFlags (command=0x7ffff788f8a0 "f\"{i=\"\n", flags=flags@entry=0x7fffffffd628)
    at ../gpshead/Python/pythonrun.c:486
#27 0x00005555558705a7 in pymain_run_command (command=<optimized out>) at ../gpshead/Modules/main.c:255

This comes from oss-fuzz https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=47764 aka https://oss-fuzz.com/testcase-detail/5010707545980928

I minimized the input to be trivial. Do we really not have a regression test for this? We will! 😄

A simplified version of the full oss-fuzz input is:

f'{i=}{i=}{i=}{i=}{i:m{i=}{i=}{i=}{i=}{h=}{i=}i{i=}{i=}{i=}{i=}{i=}{i=}{i=}{i=}{i=}{i:}{y5}{i=}{i=}{i=}{i='/{''x\ިr]\r\srr@\n\]\]\25}=}{i=}{i:}{i=}{i=}{i=}{E=XXXE,bqE=XX,E=bE,bqE}{i=}{i=}{i=}{}{i=}{i=}{i=}{i5}as{i=}Bi=}{i=}{i=}{i=}{i=}{i=}{i=}{i}}y}}}}}}}}}}}}}}6r\r\n\rrr\]srr@\n\r\]'\r\srr\]\r\r@\n\Y\r\.2n\]\r\r'\n\\r\n\r\]\r\srr@\n\]\]128r\r\n\rrr\]srr@\n\r\]\r\srr@\n\]\r\rx\ިr\srr@\n\r\]\r\srr@\n\]\r\n\]\r\r@\n\\r\n\r\]\r\srr@\n\]\]\0r\r\n\rrr\]srr@\n\r\]\r\srr@\n\]\r\rx\ިr\srr@\n\r\]\\r\]\r\srr@\n\]\r\\ddddddddddddddd\r\]\r\srr@\n\]\dddddddddddddddddd]\2r\]\r\r\~'

If you want extra torture, but it really appears to just be the simple unclosed } ?

I did not test assertion enabled release builds to determine what backports are needed. probably 3.11, unknown others. needs verifying.

[ericvsmith]

I can duplicate this on Windows on main. I'm working on a fix.

[ericvsmith]

Thanks for the catch! I should note that the code would have worked find in non-debug mode without the assert, but it's of course best to fix it to avoid the assert.

[ericvsmith]

Closed via #93419.

[vincedani]

Hello @gpshead, it looks like you are (or were) fuzzing this repository, and you’ve found some interesting bugs. 🥇

I would like to create a Python based test case reduction test suite that contains fuzzer generated outputs, and benchmark automatic test case reducers how they perform on Python inputs. It looks like to me you have opened this issue with the already reduced input that caused malfunction. Is it possible that you still have the output of the fuzzer, which is free of any reduction?

I’m also interested in these issues of yours:

• PEG parser segfault from ast.literal_eval
• assertion failure in Parser/string_parser.c

with the same motivation.

Thanks in advance,
Daniel