Skip to content

gh-99813: Start using SSL_sendfile when available #99907

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 46 commits into
base: main
Choose a base branch
from

Conversation

illia-v
Copy link
Contributor

@illia-v illia-v commented Nov 30, 2022

New _ssl__SSLSocket_sendfile_impl function is very similar to _ssl__SSLSocket_write_impl with a few adjustments for the SSL_sendfile syscall.

I am glad there was non-SSL socket.socket.sendfile and underlying socket.socket._sendfile_use_sendfile implemented. I reused that code for new ssl.SSLSocket._sendfile_use_ssl_sendfile by moving shared logic to socket.socket._sendfile_zerocopy.

And test.test_ssl.ThreadedTests.test_sendfile needed to have a socket bound before an SSL context wrapped it because this seemed to affect kTLS availability.

@illia-v illia-v changed the title gh-99813: Start using SSL_sendfile when available gh-99813: Start using SSL_sendfile when available Nov 30, 2022
@illia-v illia-v marked this pull request as ready for review March 13, 2023 21:29
@illia-v
Copy link
Contributor Author

illia-v commented Mar 13, 2023

@gpshead could you please review this when you have a chance?

@AlexWaygood AlexWaygood requested a review from gpshead July 26, 2023 17:49
@illia-v
Copy link
Contributor Author

illia-v commented Jul 26, 2023

I've recently pushed some changes after testing the new functionality manually by sending small and large files from Linux with the tls module loaded, it worked well

Copy link
Member

@gpshead gpshead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I won't have time to look at this in detail for a while. I'm just dropping a note about one issue seen while quickly skimming it.

@illia-v illia-v requested a review from gpshead August 2, 2023 19:54
@illia-v
Copy link
Contributor Author

illia-v commented Mar 27, 2024

I'd appreciate it if this can make it in 3.13

@illia-v illia-v requested a review from picnixz as a code owner April 10, 2025 23:17
Copy link
Member

@picnixz picnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments. The if-else if blocks need to follow PEP-7 but you can also use a switch if you want (it can be clearer to read).

@illia-v illia-v requested a review from picnixz April 12, 2025 14:22
Copy link
Member

@picnixz picnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few final comments

with client_context.wrap_socket(socket.socket(),
server_hostname=hostname) as s:
s.connect((HOST, server.port))
# kTLS seems to work only with a connection created before
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you check whether it "seems" to work or not? (for a personal project I would accept this but I'm interested in knowing whether this is an issue with Python SSL or OpenSSL)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I'll post more details about this soon

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can reproduce the same issue with C code using OpenSSL (without Python). Please check my findings here. I mentioned a possible patch, let me know if we can apply it or there can be undesirable consequences.

FYI, ssl.OP_ENABLE_KTLS to enable kTLS has been available since Python 3.12 (and it could be added to options as a simple integer even before). And the issue is not specific to SSL_sendfile, it affects kTLS availability for simple reads and writes.

I added my reproducer to an existing OpenSSL issue openssl/openssl#19676 (comment).

@illia-v
Copy link
Contributor Author

illia-v commented Apr 19, 2025

@picnixz OpenSSL provides SSL_OP_ENABLE_KTLS_TX_ZEROCOPY_SENDFILE which will be good to have in Python together with the change. Should I add it in this PR or separately after this one is merged?

@illia-v illia-v requested a review from picnixz April 19, 2025 15:33
Copy link
Member

@picnixz picnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel we need explicit functions for common error messages just to always use the same ones (but it's a separate issue)

}
else if (sockstate == SOCKET_HAS_BEEN_CLOSED) {
PyErr_SetString(get_state_sock(self)->PySSLErrorObject,
"Underlying socket has been closed.");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checking this:

  • if the error message already exists, use the same text (namely capitalized text ending with a period)
  • if not, try to see if error messages are using periods or not and/or are capitalized and pick the one that is the most common

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The exact error message exists in four other places in the file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the message "The sendfile operation timed out" above without the period, it's siblings don't end with a period too:

  • "The handshake operation timed out"
  • "The write operation timed out"
  • "The read operation timed out"

@illia-v illia-v requested a review from picnixz April 28, 2025 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants