can not unparse code with ' in format_spec

[15r10nk]

Bug report

Bug description:

The following example shows that it is not possible to unparse a f-string with a ' in the format_spec, but such code can be generated when the f-string is double-quoted.

expected behaviour:
unparse should use different quotes if quotes are part of the format_spec.

This is only a problem in 3.12 and worked in 3.11

import ast

code="""f"{something:'}" """
print("original code:",code)

tree=ast.parse(code)
print("original tree:",ast.dump(tree,indent=2))

new_code=ast.unparse(tree)
print("unparsed code:",new_code)

ast.parse(new_code)

output (Python 3.12.0):

original code: f"{something:'}" 
original tree: Module(
  body=[
    Expr(
      value=JoinedStr(
        values=[
          FormattedValue(
            value=Name(id='something', ctx=Load()),
            conversion=-1,
            format_spec=JoinedStr(
              values=[
                Constant(value="'")]))]))],
  type_ignores=[])
unparsed code: f'{something:'}'
Traceback (most recent call last):
  File "/home/frank/projects/pysource-playground/pysource-codegen/codi.py", line 13, in <module>
    ast.parse(new_code)
  File "/home/frank/.pyenv/versions/3.12.0/lib/python3.12/ast.py", line 52, in parse
    return compile(source, filename, mode, flags,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<unknown>", line 1
    f'{something:'}'
                   ^
SyntaxError: unterminated string literal (detected at line 1)

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-112364: correct unparsing of backslashes and quotes #115696
• [3.12] gh-112364: Correct unparsing of backslashes and quotes in ast.… #115782

[Eclips4]

cc @pablogsal @lysnikolaou

[pablogsal]

Will take a look this week

[pablogsal]

I took a look at this issue and I think switching to a different set of quotes is too complex so the best way foward is that unparse scapes the quote. @isidentical @lysnikolaou WDYT?

[terryjreedy]

import ast

code1="""f"{something:'}" """
code2="""f'{something:\\'}' """
print(code1)
print(code2)

tree1=ast.parse(code1)
tree2=ast.parse(code2)
print("Trees equal: ", tree1 == tree2)

dump1 = ast.dump(tree1, indent=2)
dump2 = ast.dump(tree2, indent=2)
print("Dumps equal: ", dump1 == dump2)
print("Unparses equal: ", ast.unparse(tree1) == ast.unparse(tree2))  # IE both same bug.

prints

f"{something:'}" 
f'{something:\'}' 
Trees equal:  False
Dumps equal:  True
Unparses equal:  True

Assuming that the tree difference is as unconsequential for compiles as it is for the dumps, the escape looks fine to me. ast.unparse(tree2) == astunparse(

[pablogsal]

Assuming that the tree difference is as unconsequential for compiles as it is for the dumps,

Looks like it is:

>>> class A:
...   def __format__(self, *args):
...      print(args)
...      return "A"
...
>>> f"{A():'}"
("'",)
'A'
>>> f"{A():\'}"
("'",)
'A'

and

>>> dis.dis(compile("""f\"{A():\'}\"""", "<string>", "exec"))
  0           0 RESUME                   0

  1           2 PUSH_NULL
              4 LOAD_NAME                0 (A)
              6 PRECALL                  0
             10 CALL                     0
             20 LOAD_CONST               0 ("'")
             22 FORMAT_VALUE             4 (with format)
             24 POP_TOP
             26 LOAD_CONST               1 (None)
             28 RETURN_VALUE
>>> dis.dis(compile("""f\"{A():'}\"""", "<string>", "exec"))
  0           0 RESUME                   0

  1           2 PUSH_NULL
              4 LOAD_NAME                0 (A)
              6 PRECALL                  0
             10 CALL                     0
             20 LOAD_CONST               0 ("'")
             22 FORMAT_VALUE             4 (with format)
             24 POP_TOP
             26 LOAD_CONST               1 (None)

[15r10nk]

@pablogsal do you want to fix this issue? I could fix this bug if you want. That would be my first cpython Bugfix.

[pablogsal]

@pablogsal do you want to fix this issue? I could fix this bug if you want. That would be my first cpython Bugfix.

Yes, let's fix it. Can you make a PR and CC me into it? Let's get your first CPython bug fix in 🚀