Preserve command-line history after interpreter crash

[skirpichev]

When the CPython interpreter crashes - recent history entries are lost. An easy reproducer (on any Linux):

$ ulimit -v $((1024*256))
$ pip install -q gmpy2
$ python  # started with empty history
Python 3.14.0a2+ (heads/main:e2713409cf, Dec  2 2024, 07:50:36) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from gmpy2 import mpz
>>> mpz(2222222222222211111111122222222222222)**33322222
GNU MP: Cannot allocate memory (size=503998640)
Aborted
$ python  # history is empty, again :(
...

That "works" both in the old repl and in the new repl. Sometimes you even can't recover all session from the terminal screen.

IMO, if it's a feature - it's a misfeature. I would expect, that all entered input will be preserved in the history. Or such behavior can be optionally turned on. BTW, this feature "doesn't work" in the IPython shell.

Linked PRs

• gh-127495: Append to history file after every statement in PyREPL #132294

[rruuaanng]

I'm wondering if this has something to do with bash. I'm not sure what processing the python-shell does when it is interrupted, but the handling of history is in bash.

[skirpichev]

No, it's not about bash at all, it's history is fine in the given example. It's all about python repl (>>> ) history.

The shell session shown here just to demonstrate a close-to-real-life example (extension doesn't handle well out of memory condition and abort the python interpreter process). But you could just do kill -SIGABRT <python interactive interpreter> (or it's analog in Windows).

[rruuaanng]

I seem to understand that you may need a file similar to .history to store the history of the previous pyrepl. Even if the interpreter crashes, it can be restored through the file.
Or similar methods to recover the loss of records caused by crashes.

[skirpichev]

you may need a file similar to .history to store the history of the previous pyrepl.

Perhaps. I was planning to work on the issue, but I've no review-ready patch yet. Let me know if you would like to work on this.

[rruuaanng]

you may need a file similar to .history to store the history of the previous pyrepl.

Perhaps. I was planning to work on the issue, but I've no review-ready patch yet. Let me know if you would like to work on this.

I'm not going to deal with that, feel free to do so. But I will take this opportunity to learn pyrepl :)

[devdanzin]

I seem to understand that you may need a file similar to .history to store the history of the previous pyrepl.

There's .python_history already, which normally stores the history. The issue is that it's written at interpreter exit, which doesn't happen in case of a crash (or os._exit()):

cpython/Lib/site.py

Lines 578 to 586 in 53908bd

	def write_history():
	try:
	readline_module.write_history_file(history)
	except (FileNotFoundError, PermissionError):
	# home directory does not exist or is not writable
	# https://bugs.python.org/issue19891
	pass
	atexit.register(write_history)

[skirpichev]

This patch works for me:

diff --git a/Lib/_pyrepl/simple_interact.py b/Lib/_pyrepl/simple_interact.py
index a08546a931..b6e6023f09 100644
--- a/Lib/_pyrepl/simple_interact.py
+++ b/Lib/_pyrepl/simple_interact.py
@@ -31,7 +31,7 @@
 import sys
 import code

-from .readline import _get_reader, multiline_input
+from .readline import _get_reader, multiline_input, write_history_file


 _error: tuple[type[Exception], ...] | type[Exception]
@@ -143,6 +143,10 @@ def maybe_run_command(statement: str) -> bool:

             input_name = f"<python-input-{input_n}>"
             more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single")  # type: ignore[call-arg]
+            try:
+                write_history_file()
+            except (FileNotFoundError, PermissionError):
+                pass
             assert not more
             input_n += 1
         except KeyboardInterrupt:

@pablogsal, I'm on the right way? BTW, as noted in the description: I think it's rather a bug than the feature request. The IPython shell e.g. keeps history well.

[devdanzin]

This patch works for me:

Can you do a quick check of whether it impacts performance by pasting a long block of e.g. a = "a" * 100 lines? If it's writing to the history file once per line (not sure it is), it might slow things down (which probably isn't a big issue for the REPL use case).

If there's a significant impact, it might be better to write history when a new history entry is created, as IIUC it would only write once for the whole long block.

[skirpichev]

it might be better to write history when a new history entry is created

It already on that way: writing is not per-line, but per history entry.

On another hand, using something like readline.append_history_file() would be better, of course.

Edit:

Version with append_history_file() to play with

diff --git a/Lib/_pyrepl/readline.py b/Lib/_pyrepl/readline.py
index be229488e5..deb2bfb618 100644
--- a/Lib/_pyrepl/readline.py
+++ b/Lib/_pyrepl/readline.py
@@ -89,6 +89,7 @@
     # "set_pre_input_hook",
     "set_startup_hook",
     "write_history_file",
+    "append_history_file",
     # ---- multiline extensions ----
     "multiline_input",
 ]
@@ -446,6 +447,7 @@ def read_history_file(self, filename: str = gethistoryfile()) -> None:
                         del buffer[:]
                     if line:
                         history.append(line)
+        self.set_history_length(self.get_current_history_length())
 
     def write_history_file(self, filename: str = gethistoryfile()) -> None:
         maxlength = self.saved_history_length
@@ -457,6 +459,19 @@ def write_history_file(self, filename: str = gethistoryfile()) -> None:
                 entry = entry.replace("\n", "\r\n")  # multiline history support
                 f.write(entry + "\n")
 
+    def append_history_file(self, filename: str = gethistoryfile()) -> None:
+        reader = self.get_reader()
+        saved_length = self.get_history_length()
+        length = self.get_current_history_length() - saved_length
+        history = self.get_reader().get_trimmed_history(length)
+        f = open(os.path.expanduser(filename), "a",
+                 encoding="utf-8", newline="\n")
+        with f:
+            for entry in history:
+                entry = entry.replace("\n", "\r\n")  # multiline history support
+                f.write(entry + "\n")
+        self.set_history_length(saved_length + len(history))
+
     def clear_history(self) -> None:
         del self.get_reader().history[:]
 
@@ -526,6 +541,7 @@ def insert_text(self, text: str) -> None:
 get_current_history_length = _wrapper.get_current_history_length
 read_history_file = _wrapper.read_history_file
 write_history_file = _wrapper.write_history_file
+append_history_file = _wrapper.append_history_file
 clear_history = _wrapper.clear_history
 get_history_item = _wrapper.get_history_item
 remove_history_item = _wrapper.remove_history_item
diff --git a/Lib/_pyrepl/simple_interact.py b/Lib/_pyrepl/simple_interact.py
index a08546a931..c168549bec 100644
--- a/Lib/_pyrepl/simple_interact.py
+++ b/Lib/_pyrepl/simple_interact.py
@@ -31,7 +31,7 @@
 import sys
 import code
 
-from .readline import _get_reader, multiline_input
+from .readline import _get_reader, multiline_input, append_history_file
 
 
 _error: tuple[type[Exception], ...] | type[Exception]
@@ -143,6 +143,10 @@ def maybe_run_command(statement: str) -> bool:
 
             input_name = f"<python-input-{input_n}>"
             more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single")  # type: ignore[call-arg]
+            try:
+                append_history_file()
+            except (FileNotFoundError, PermissionError):
+                pass
             assert not more
             input_n += 1
         except KeyboardInterrupt:

[skirpichev]

PR is ready to review: #132294

[skirpichev]

Related d.p.o thread: https://discuss.python.org/t/better-pyrepl-history-file-format/87918