Auto dedent -c arguments

[Erotemic]

Feature or enhancement

The Python CLI should automatically dedent the argument given to "-c".

I raised this issue on the Python-Ideas mailing list and it got some positive feedback, so I'm moving forward with it here and in a proof-of-concept PR.

Pitch

I have what I think is a fairly low impact quality of life improvement to suggest for the python CLI.

When I'm not working in Python I tend to be working in bash. But often I want to break out and do something quick in Python. I find the python -c CLI very useful for this. For one liners it's perfect. E.g.

NEW_VAR=$(python -c "import pathlib; print(pathlib.Path('$MYVAR').parent.parent)")

And even if I want to do something multi-line it's pretty easy

NEW_VAR=$(python -c "
import pathlib
for _ in range(10):
    print('this is a demo, bear with me')
")

But the problem is when I'm writing bash inside a function or some other nested code, I would like to have nice indentation in my bash file, but if I write something like this:

mybashfunc(){
    python -c "
    import pathlib
    for _ in range(10):
        print('this is a demo, bear with me')
    "
}

I get IndentationError: unexpected indent.

This means I have to write the function ugly like this:

mybashfunc(){
    python -c "
import pathlib
for _ in range(10):
    print('this is a demo, bear with me')
"
}

Or use a helper function like this:

codeblock()
{
    __doc__='
    copy-pastable implementation
    Prevents indentation errors in bash
    '
    echo "$1" | python -c "import sys; from textwrap import dedent; print(dedent(sys.stdin.read()).strip('\n'))"
}

mybashfunc(){
    python -c $(codeblock "
    import pathlib
    for _ in range(10):
        print('this is a demo, bear with me')
    ")
}

Or more recently I found that this is a low-impact workaround:

mybashfunc(){
    python -c "if 1:
    import pathlib
    for _ in range(10):
        print('this is a demo, bear with me')
    "
}

But as a certain Python dev may say: "There must be a better way."

Would there be any downside to the Python CLI automatically dedenting the input string given to -c? I can't think of any case off the top of my head where it would make a previously valid program invalid. Unless I'm missing something this would strictly make previously invalid strings valid.

Thoughts?

Previous discussion

On the mailing list there were these responses:

Lucas Wiman said:

Very strong +1 to this. That would be useful and it doesn't seem like there's a downside. I often make bash functions that pipe files or database queries to Python for post-processing. I also sometimes resort to Ruby because it's easy to write one-liners in Ruby and annoying to write one-liners in python/bash.

I suppose there's some ambiguity in the contents of multi-line """strings""". Should indentation be stripped at all in that case? E.g.
python -c "
'''
some text
''''
"

But it seems simpler and easier to understand/document if you pre-process the input like using an algorithm like this:

• If the first nonempty line has indentation, and all subsequent lines either start with the same indentation characters or are empty, then remove that prefix from those lines.

I think that handles cases where editors strip trailing spaces or the first line is blank. So e.g.:
python -c "
some_code_here()
"
Then python receives something like "\n some_code_here\n"

python -c "
some_code here()

if some_some_other_code():
    still_more_code()

"
Then python receives something like "\n some_code_here\n\n if ..."

This wouldn't handle cases where indentation is mixed and there is a first line, e.g.:
python -c "first_thing()
if second_thing():
third_thing()
"
That seems simple enough to avoid, and raising a syntax error is reasonable in that case.

Best wishes,
Lucas

There was also positive feedback from @cameron-simpson and @barry-scott suggested a PR with an implementation might move this forward. I have a proof-of-concept PR and this is the corresponding issue for it.

Linked PRs

• gh-103997: Automatically dedent the argument to "-c" #103998
• gh-103997: Fix and test _PyUnicode_Dedent #138620
• gh-103997: Remove incorrect statements about -c dedenting #138624

[hauntsaninja]

You may be interested in something like https://github.com/hauntsaninja/pyp

[sunmy2019]

I am currently working to help this out.

The clock is ticking. We need more reviewers.

[hauntsaninja]

Thanks for the feature suggestion and tagging me on the PR!

I think it's unlikely that this makes 3.12. Looking at the ideas thread: https://mail.python.org/archives/list/python-ideas@python.org/thread/6SYN2MQIP5DTF6INZ4SP2YKLF6P4VJOW/#6SYN2MQIP5DTF6INZ4SP2YKLF6P4VJOW I don't see any participation from core devs, so probably needs a little more consensus.

My unorganised personal thoughts:

• First, I'm a new core dev and risk averse, so while I'm happy to review the PR, I will not be hitting merge in 3.12.
• The use case seems reasonable to me and I don't spot a backward compatibility issue. I actually wouldn't mind dedenting in the shell either; IPython does this and it's quite convenient when copy pasting code.
• That said, I'm not sure there exists a behaviour for multiline strings that I like.
• Overall, dedenting or the if 1: trick are reasonable workarounds. Very broadly speaking, Python shouldn't try to optimise too much for shell use cases; this can impose undesirable constraints on readability. That's why I wrote my magical shell tool!
• If you can't find core developers willing to hit merge, or if there's some disagreement amongst core devs on this issue, it might be worth going through the PEP process.

Also, a little off topic, but I've been appreciating your contributions sunmy2019! Thanks for helping make Python better!

[sunmy2019]

Let's target this to 3.13. This indeed might be worth going through the PEP process.

https://peps.python.org/pep-0001/

We will need a core-dev sponsor then.

[Erotemic]

Does this need a PEP? If so, should I start on a draft? If not, what if anything is on my plate to move this forward?

@hauntsaninja pyp is a nifty tool, and something I'll likely add to my tool stack, but it would be nice to write readable Python scripts in instances where installing extra packages isn't an option. And unfortunately I found an issue with the if 1 trick. It only works if you have indented code. In the case where your code is pushed all the way to the left it breaks because the if requires indentation. It would be much simpler if using python -c worked neatly out of the box. I also do like the idea of auto-dedenting inputs to the REPL interpreter like IPython does, and that could certainly reuse some of this work, but it makes sense to keep the scope small for now.

[sunmy2019]

Does this need a PEP?

It seems we cannot find a core dev willing to merge this.

If so, should I start on a draft?

We need to find a core dev as required by PEP 1.

I think we should find one first, or we may end up with nothing.

[merwok]

This doesn’t seem to be PEP-sized, but definitely discuss it on the forum.

Compiler dedent was added recently in #106411, str.dedent is being discussed, this should all be discussed together.

[encukou]

Sorry for chiming in late, but reading the comments about handling tabs vs. spaces, I realized there's another possible way to achieve this: add a new top-level grammar rule rather than preprocessing the string.

We already have:

file: [statements] ENDMARKER

and could add:

script: [statements] ENDMARKER | INDENT statements DEDENT ENDMARKER

I haven't tried this. I assume it won't be trivial, but the complexity is likely to be in adding this to compile, exec etc., rather than wrangling strings in C.

[Erotemic]

I like the idea of a new grammar rule. It makes sense to me that a Python program should "just work" as long as indentation is consistent. In my mind there is no reason it has to start at level 0. However, that is a much bigger change and I'm sure it would require a PEP.

I would be inclined to merge this first as it provides immediate benefit for an existing use case. While a grammar rule does seem more elegant to me, I'm not sure if the use-case is there. I don't see .py scripts benefiting from non-zero indentation.

[sunmy2019]

Sorry for chiming in late, but reading the comments about handling tabs vs. spaces, I realized there's another possible way to achieve this: add a new top-level grammar rule rather than preprocessing the string.

We already have:

file: [statements] ENDMARKER

and could add:

script: [statements] ENDMARKER | INDENT statements DEDENT ENDMARKER

I haven't tried this. I assume it won't be trivial, but the complexity is likely to be in adding this to compile, exec etc., rather than wrangling strings in C.

After a discussion with friends, we'd say the main usage would still be embedding codes inside other codes.

just like

def foo():
   x = """
      xxx
      xxx
   """
   exec(x)

Also in other languages, such as https://pyo3.rs/v0.20.2/python_from_rust#want-to-run-statements-then-use-run

[merwok]

Python ideas discussions happen on Discuss, not Github – please post there about the feature request, its rationale and use cases, examples in other languages, etc.

[encukou]

Indeed. And for the discussion, you'll need to write a very PEP-like document anyway, so I recommend using the PEP structure.

If you can't find the time to start the discussion, let me know. I like the feature :)
And if it turns out to need a PEP, I can be your sponsor.