Crash when using multiple capturing groups in re.Scanner

Bug report

Bug description:

The re.Scanner class crashes with a segmentation fault (memory corruption) when any pattern in the lexicon contains named capturing groups.

import re

lexicon = [
    ('(?P<group1>a)(?P<group2>b)', None),  # Named capturing groups
]

scanner = re.Scanner(lexicon)
result, leftover = scanner.scan("ab")  # Segmentation fault occurs here

zsh: segmentation fault  python3 test.py

Scanner should process the input normally and return scan results without crashing, but Python interpreter crashes with segmentation fault, indicating memory corruption.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

FTR, the traceback with faulthandler is:

Debug memory block at address p=0x7f5167149ea0: API 'm'
    16 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x7f5167149eb0 are not all FORBIDDENBYTE (0xfd):
        at tail+0: 0xf9 *** OUCH
        at tail+1: 0x3d *** OUCH
        at tail+2: 0x7a *** OUCH
        at tail+3: 0x66 *** OUCH
        at tail+4: 0x51 *** OUCH
        at tail+5: 0x7f *** OUCH
        at tail+6: 0x00 *** OUCH
        at tail+7: 0x00 *** OUCH
    Data at p: f8 3d 7a 66 51 7f 00 00 fa 3d 7a 66 51 7f 00 00

Memory block allocated at (most recent call first):
  File "/python/cpython/Lib/re/__init__.py", line 410

Fatal Python error: _PyMem_DebugRawFree: bad trailing pad byte
Python runtime state: initialized

Current thread 0x00007f5167222740 [python] (most recent call first):

Looks like a deallocation issue in the scanner as the function completes correctly.

So the segfault occurs in the destructor when trying to free the memory of state->mark (PyMem_Free((void*) state->mark);). Looks like either it was already freed and we have a double-free somewhere or something else is happening.

@serhiy-storchaka

Ok, I was able to patch this but only by messing a bit:

state->mark is allocated as state->mark = (const void **)PyMem_New(const void *, pattern->groups * 2); So it depends on the number of patterns.
The actual value here is pattern->groups == 1 when we're in the state that crash but I'm not sure it's correct here.
When putting other kind of RE in the lexicon, it doesn't even work (I get invalid SRE code for '(a)(b)(c)' for instance).
I get other crashes when having more capturing groups.

I think it's a more general issue here. My gut feeling is telling me that the crash is happening because we don't reset explicitly the memory that was allocated for the catching groups but I may be wrong (that is, in state_reset).

@serhiy-storchaka As you're more familiar with the state machine, can you investigate this please?

This can work by accident, if the number of the captured groups is less than the sequential number of the phrase (counting from 1). No capturing group is allowed in the first phrase, one capturing group can work in the second phrase, two in the third phrase, etc. This is because when the scanner handles group N for phrase N, previous N-1 groups are unused, and setting them will not conflict with group N and will not affect lastgroup.

For example, the code reported in #142220 worked for that reason, because the second phrase contains only one capturing group.

    arg_parser = re.Scanner([
                (r',', lambda x,t: ''),
                (r'.*?((?<!\\),)', lambda x,t: t[:-1]),
                (r'.*?\)', lambda x,t: t[:-1]),
        ])

I am not sure that this always works if such condition is satisfied, and in any case such condition is too artificial.

But there is a code in the wild that "works", and we should not break it in the 3.13 bugfix.

We can simply revert the Scanner change in 3.13. We can also raise an exception conditionally, or emit a warning, or make re.Scanner supporting arbitrary capturing groups, but I am not sure it is worth a hassle.

It is perhaps fine for 3.14, taking into account that re.Scanner is undocumented.

Revert "[3.13] pythongh-140797: Forbid capturing groups in re.Scanner…

@StanFromIreland

@StanFromIreland Seems a bit weird to me to mix two completely different topics in one ticket and close a perfectly natural report as a stigmatic "duplicate"...

It is perhaps fine for 3.14, taking into account that re.Scanner is undocumented.

@serhiy-storchaka given the code worked fine in 3.14 and was tested in several release candidates and then released into the wild in 3.14.0 final, I must say that I don't see any rationale for backporting to 3.14 any more than 3.13 (or rather let us say, any rationale to refrain from reverting the backport).

It's the same issue in both. People have already started using 3.14, and packaging and deploying scripts that work on 3.14.0, and this backport broke that. Given that, again, my argument in the ticket closed as a duplicate (??) was that instead of backporting a raise Exception('this is not allowed') for code that could work, it would be better to only backport a deprecation warning or something, I think this well applies to 3.14 too.

The "solution" for multiple capturing groups causing crashes, was simply to ban doing things that are useless anyway. No code was actually made to work -- the only difference was a clearer and more understandable error message (an exception instead of a crash) so nobody's code would actually work without fixing the code anyway.

That being the case, it is not really an urgent priority to change the type of error in released versions of CPython if it also changes working scripts into erroring scripts... and that rule applies to 3.14.1 as well.

And again, adding the exception on main but adding a deprecation warning on released stable versions seems quite reasonable. A deprecation warning can still be discovered in CI and responded to, but doesn't break scripts in the wild.

[3.13] Revert "[3.13] gh-140797: Forbid capturing groups in re.Scanne…

@hugovk

@hugovk, what do you say?

Let's also revert for 3.14.2, then we have a couple of months to decide on whether to reintroduce the fix, or deprecate first, or something else.

Revert "[3.14] pythongh-140797: Forbid capturing groups in re.Scanner…

I've opened #142309 to revert for 3.14.

[3.14] Revert "gh-140797: Forbid capturing groups in re.Scanner lexic…

I am closing this issue. #140984 fixed the original crash. #140944 improved error detection and perhaps saved from other errors, but since it broke some user code, the backports were reverted.

pythongh-140797: Forbid capturing groups in re.Scanner lexicon patter…

Python 3.14 compat

Don't use a capturing group in the re.Scanner lexicon

Navigation Menu

Repository navigation

Crash when using multiple capturing groups in `re.Scanner` #140797

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

36 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Crash when using multiple capturing groups in re.Scanner #140797

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

picnixz commented on Oct 30, 2025

picnixz commented on Oct 30, 2025

picnixz commented on Oct 30, 2025

picnixz commented on Oct 30, 2025

36 remaining items

serhiy-storchaka commented on Dec 3, 2025

eli-schwartz commented on Dec 5, 2025

serhiy-storchaka commented on Dec 5, 2025

hugovk commented on Dec 5, 2025

hugovk commented on Dec 5, 2025

serhiy-storchaka commented on Dec 5, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Crash when using multiple capturing groups in `re.Scanner` #140797