PEG parser segfault from ast.literal_eval

[gpshead]

BPO	46339
Nosy	@gpshead, @lysnikolaou, @pablogsal, @miss-islington, @kumaraditya303
PRs	bpo-46339: Fix crash in the parser when computing error text for multi-line f-strings #30529 [3.10] bpo-46339: Fix crash in the parser when computing error text for multi-line f-strings (GH-30529) #30542 bpo-46339: Include clarification on assert in 'get_error_line_from_tokenizer_buffers' #30545

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pablogsal'
closed_at = <Date 2022-01-20.13:09:49.067>
created_at = <Date 2022-01-11.02:56:11.794>
labels = ['interpreter-core', 'type-crash', '3.11']
title = 'PEG parser segfault from ast.literal_eval'
updated_at = <Date 2022-01-20.13:09:49.067>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2022-01-20.13:09:49.067>
actor = 'pablogsal'
assignee = 'pablogsal'
closed = True
closed_date = <Date 2022-01-20.13:09:49.067>
closer = 'pablogsal'
components = ['Parser']
creation = <Date 2022-01-11.02:56:11.794>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 46339
keywords = ['patch']
message_count = 7.0
messages = ['410270', '410275', '410291', '410294', '410313', '410852', '411023']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'lys.nikolaou', 'pablogsal', 'miss-islington', 'kumaraditya']
pr_nums = ['30529', '30542', '30545']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue46339'
versions = ['Python 3.11']

[gpshead]

cpython/b$ ./python 
Python 3.11.0a3+ (heads/main-dirty:081a214008, Jan 11 2022, 02:48:22) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.literal_eval('''F"""
... 
... 
... {6
... 0="""''')
Segmentation fault

this was discovered by oss-fuzz.

This may be a recent regression? It didn't happen on my build from a few weeks ago. worth testing on older versions.

[kumaraditya303]

I tested it on 3.10.1 and it didn't segfault on Windows 11 so seems like 3.11 regression.

[pablogsal]

Seems that the regression is caused by this commit:

6fa8b2c is the first bad commit
commit 6fa8b2c
Author: Pablo Galindo Salgado <Pablogsal@gmail.com>
Date: Sat Jan 8 00:23:40 2022 +0000

bpo-46237: Fix the line number of tokenizer errors inside f-strings (GH-30463)

Lib/test/test_exceptions.py | 12 ++++++++++++
.../2022-01-07-19-33-05.bpo-46237.9A6Hpq.rst | 2 ++
Parser/pegen.c | 8 ++++----
Parser/string_parser.c | 5 ++++-
4 files changed, 22 insertions(+), 5 deletions(-)
create mode 100644 Misc/NEWS.d/next/Core and Builtins/2022-01-07-19-33-05.bpo-46237.9A6Hpq.rst

[pablogsal]

The reason is that the line numbers are now correctly identified, and that causes existing code to not work correctly :(

[miss-islington]

New changeset cedec19 by Pablo Galindo Salgado in branch 'main':
bpo-46339: Fix crash in the parser when computing error text for multi-line f-strings (GH-30529)
cedec19

[pablogsal]

New changeset 8c2fd09 by Pablo Galindo Salgado in branch 'main':
bpo-46339: Include clarification on assert in 'get_error_line_from_tokenizer_buffers' (bpo-30545)
8c2fd09

[pablogsal]

New changeset 1fb1f5d by Miss Islington (bot) in branch '3.10':
[3.10] bpo-46339: Fix crash in the parser when computing error text for multi-line f-strings (GH-30529) (GH-30542)
1fb1f5d