gh-148292: Update SSLSocket.read() for OpenSSL 4

[vstinner]

Add _got_eof attribute to avoid calling SSL_read_ex() again after SSL_ERROR_EOF.

• Issue: 3.15.0a8 fails test_urllib2_localnet with OpenSSL 4.0.0beta1 #148292

[vstinner]

@picnixz @gpshead: Would you mind to review this (draft) change?

I marked the PR as a draft since I'm not sure if the fix makes sense and is correct.

See #148600 (comment) to reproduce the issue and build Python 3.15 with OpenSSL 4.0.0.

Note: #148601 (Add Modules/_ssl_data_40.h data) has no effect on test_urllib2_localnet (it does still fail).

[vstinner]

See #146217 (comment) for differences between OpenSSL 3 and OpenSSL 4.

In short:

• On OpenSSL 3, the 3rd read fails with SSLEOFError("UNEXPECTED_EOF_WHILE_READING"), and the following 4th read fails with SSLEOFError("EOF occurred in violation of protocol").
• On OpenSSL 4, the 3rd read fails with SSLEOFError("UNEXPECTED_EOF_WHILE_READING"), and the following 4th read fails with SSLError("A failure in the SSL library occurred").

[picnixz]

I don't think it's the correct change because I need to investigate. The reason why I don't think it's correct is because the code path being taken to trigger "A failure in the SSL library occurred" means that the last OpenSSL error code was not set (either we cleared it accidently or they didn't set it correctly) and this is something that can happen elsewhere.

[picnixz]

FTR, this may be related #148594.

[vstinner]

FTR, this may be related #148594.

It's not related. test_urllib2_localnet still fails with this change:

diff --git a/Modules/_ssl.c b/Modules/_ssl.c
index 4e563379098..d17cd308628 100644
--- a/Modules/_ssl.c
+++ b/Modules/_ssl.c
@@ -2938,6 +2938,7 @@ _ssl__SSLSocket_read_impl(PySSLSocket *self, Py_ssize_t len,
 
     do {
         Py_BEGIN_ALLOW_THREADS;
+        ERR_clear_error();
         retval = SSL_read_ex(self->ssl, mem, (size_t)len, &count);
         err = _PySSL_errno(retval == 0, self->ssl, retval);
         Py_END_ALLOW_THREADS;

[vstinner]

On OpenSSL 4, the 3rd read fails with SSLEOFError("UNEXPECTED_EOF_WHILE_READING"), and the following 4th read fails with SSLError("A failure in the SSL library occurred").

The 3rd read fails with SSL_R_UNEXPECTED_EOF_WHILE_READING:

• SSL_read_ex(): 0
• errno: 0
• SSL_get_error(ssl, retcode): SSL_ERROR_SSL (1)
• ERR_peek_last_error(): 167772454
• ERR_GET_LIB(e): ERR_LIB_SSL (20)
• ERR_GET_REASON(e): SSL_R_UNEXPECTED_EOF_WHILE_READING (294)

Related code in Modules/_ssl.c:

#if defined(SSL_R_UNEXPECTED_EOF_WHILE_READING)
            /* OpenSSL 3.0 changed transport EOF from SSL_ERROR_SYSCALL with
             * zero return value to SSL_ERROR_SSL with a special error code. */
            if (ERR_GET_LIB(e) == ERR_LIB_SSL &&
                    ERR_GET_REASON(e) == SSL_R_UNEXPECTED_EOF_WHILE_READING) {
                p = PY_SSL_ERROR_EOF;
                type = state->PySSLEOFErrorObject;
                errstr = "EOF occurred in violation of protocol";
            }
#endif

The 4th read doesn't seem to set any specific error :-( It only says that it's an "SSL error".

• SSL_read_ex(): 0
• errno: 0
• SSL_get_error(ssl, retcode): SSL_ERROR_SSL (1)
• ERR_peek_last_error(): 0
• ERR_GET_LIB(e): 0
• ERR_GET_REASON(e): 0

[vstinner]

@picnixz:

(...) the code path being taken to trigger "A failure in the SSL library occurred" means that the last OpenSSL error code was not set (either we cleared it accidently or they didn't set it correctly) and this is something that can happen elsewhere.

How can I debug such issue? According to my previous comment, OpenSSL doesn't set any error. It only says that it's a "SSL error". Even if I check ERR_peek_last_error() just after SSL_read_ex() it's still 0.

It seems like OpenSSL 4 changed SSL_read_ex() error when called after SSL_R_UNEXPECTED_EOF_WHILE_READING.

[picnixz]

This may be a bug in OpenSSL 4. Something changed in their errors:

SSL_get_error() no longer depends on the state of the error stack, so it is no longer necessary to empty the error queue before the TLS/SSL I/O operations.

So.... maybe we may ourselves be doing bad things elsewhere. I honestly do not know if the problem is on our side or their side. It may be on our side because we maybe assume that the stack entry is still there? (idk)

[vstinner]

Ok, here is a simpler Python reproducer script only using socket and ssl modules:

import os.path
import socket
import ssl
import threading

CERT = os.path.join('Lib', 'test', 'certdata', 'keycert.pem')
HOST = '127.0.0.1'
HOSTNAME = 'localhost'

class Server(threading.Thread):
    def __init__(self):
        super().__init__()
        self.listening = threading.Event()
        self.address = None

    def run(self):
        context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
        context.load_cert_chain(CERT)
        server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        #server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        #server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
        server_sock.bind((HOST, 0))
        server_sock.listen(5)

        self.address = server_sock.getsockname()
        self.listening.set()

        sock, addr = server_sock.accept()
        sslconn = context.wrap_socket(sock, server_side=True)

        request = b''
        while True:
            chunk = sslconn.recv(65537)
            request += chunk
            if b'\r\n\r\n' in request:
                break
        print(f"server got request: {request!r}")

        print("server sendall")
        sslconn.sendall(
            b'HTTP/1.0 200 OK\r\n'
            b'Server: TestHTTP/ Python/3.15.0a8+\r\n'
            b'Date: Thu, 16 Apr 2026 12:42:37 GMT\r\n'
            b'Content-type: text/plain\r\n\r\n')
        sslconn.sendall(b'we care a bit')

        print("server shutdown write")
        sslconn.shutdown(socket.SHUT_WR)
        print("server close socket")
        sslconn.close()

        server_sock.close()

def main():
    server = Server()
    server.start()
    server.listening.wait()
    port = server.address[1]

    context = ssl.create_default_context(cafile=CERT)
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((HOST, port))
    sslsock = context.wrap_socket(sock, server_hostname=HOSTNAME)

    sslsock.sendall(b'GET /bizarre HTTP/1.0\r\n\r\n')
    sslobj = sslsock._sslobj

    def read(prefix, sslobj):
        try:
            data = sslobj.read(1024)
            result = repr(data)
        except ssl.SSLError as exc:
            result = f'<{exc!r}>'
        print(prefix, result)

    for i in range(1, 5):
        read(f"client read #{i}:", sslobj)
    sslsock.close()

    server.join()

if __name__ == "__main__":
    main()

Output with OpenSSL 3:

server got request: b'GET /bizarre HTTP/1.0\r\n\r\n'
server sendall
server shutdown write
server close socket
client read #1: b'HTTP/1.0 200 OK\r\nServer: TestHTTP/ Python/3.15.0a8+\r\nDate: Thu, 16 Apr 2026 12:42:37 GMT\r\nContent-type: text/plain\r\n\r\n'
client read #2: b'we care a bit'
client read #3: <SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:2970)')>
client read #4: <SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2970)')>

Output with OpenSSL 4:

server got request: b'GET /bizarre HTTP/1.0\r\n\r\n'
server sendall
server shutdown write
server close socket
client read #1: b'HTTP/1.0 200 OK\r\nServer: TestHTTP/ Python/3.15.0a8+\r\nDate: Thu, 16 Apr 2026 12:42:37 GMT\r\nContent-type: text/plain\r\n\r\n'
client read #2: b'we care a bit'
client read #3: <SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:2984)')>
client read #4: <SSLError(1, 'A failure in the SSL library occurred (_ssl.c:2984)')>

OpenSSL 4 behaves differently on the last (4th) read (after UNEXPECTED_EOF_WHILE_READING): it fails with a generic A failure in the SSL library occurred error instead of EOF occurred in violation of protocol (OpenSSL 3).