fix: private_encrypt/public_decrypt broken on OpenSSL 3.x

[toddr-bot]

What

Fixes private_encrypt() and public_decrypt() being broken on OpenSSL 3.x with any padding except NO_PADDING.

Why

rsa_crypt() is shared by four operations: encrypt, decrypt, private_encrypt, and public_decrypt. On OpenSSL 3.x, it unconditionally forced OAEP padding for all non-NO_PADDING modes. But private_encrypt/public_decrypt use EVP_PKEY_sign/EVP_PKEY_verify_recover under the hood, where OAEP is not a valid padding mode. This caused an OpenSSL error ("illegal or unsupported padding mode") for any call to private_encrypt() or public_decrypt() with the default padding.

This is the same family of issues as #61 — the v0.35 Marvin attack mitigation broke signature-related operations. This fix complements #103 (which fixes sign()/verify()).

How

Replaced the unused int public parameter in rsa_crypt() with int is_encrypt to distinguish encryption from sign/verify_recover operations:

• Encryption path (encrypt/decrypt): forces OAEP, rejects PSS (existing behavior)
• Sign path (private_encrypt/public_decrypt): passes through the user's padding choice; rejects OAEP and PSS with clear error messages pointing to the correct API

Testing

• 375 tests pass (364 existing + 10 new in t/private_encrypt.t)
• Verified private_encrypt/public_decrypt roundtrip with NO_PADDING
• Verified clear croak messages for OAEP and PSS on the sign path
• Verified encrypt/decrypt behavior unchanged

🤖 Generated with Claude Code

---

Quality Report

Changes: 4 files changed, 131 insertions(+), 15 deletions(-)

Code scan: clean

Tests: passed (OK)

Branch hygiene: clean

Generated by Kōan post-mission quality pipeline

[atoomic]

view errors from smokers

t/private_encrypt.t ......... 
1..10
ok 1 - private_encrypt with no_padding succeeds
ok 2 - public_decrypt(private_encrypt(data)) round-trips with no_padding
ok 3 - private_encrypt with OAEP behavior on pre-3.x (skipped)
ok 4 - public_decrypt with OAEP behavior on pre-3.x (skipped)
ok 5 - private_encrypt with PSS behavior on pre-3.x (skipped)
ok 6 - public_decrypt with PSS behavior on pre-3.x (skipped)
ok 7 - encrypt with OAEP still works
ok 8 - decrypt with OAEP round-trips
not ok 9 - encrypt with PSS still croaks

#   Failed test 'encrypt with PSS still croaks'
#   at t/private_encrypt.t line 83.
#                   'RSA.xs:389: OpenSSL error: unknown padding type at t/private_encrypt.t line 82.
# '
#     doesn't match '(?^:RSA-PSS cannot be used for encryption)'
ok 10 - public key private_encrypt croaks
# Looks like you failed 1 test of 10.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/10 subtests 

@toddr-bot rebase

[timlegge]

Assigning to myself . I need to look at this closer

[toddr-bot]

Rebase: fix: private_encrypt/public_decrypt broken on OpenSSL 3.x

Branch koan.toddr.bot/fix-private-encrypt-padding rebased onto main and force-pushed.

Diff: 4 files changed, 135 insertions(+), 18 deletions(-)

Review feedback was analyzed and applied.

Actions

• Resolved merge conflicts (1 round(s))
• Rebased koan.toddr.bot/fix-private-encrypt-padding onto upstream/main
• Applied review feedback
• Force-pushed koan.toddr.bot/fix-private-encrypt-padding to origin
• CI passed

CI

CI passed.

---

Automated by Kōan

[atoomic]

@toddr-bot rebase

[toddr-bot]

Rebase: fix: private_encrypt/public_decrypt broken on OpenSSL 3.x

Branch koan.toddr.bot/fix-private-encrypt-padding rebased onto main and force-pushed.

Diff: 4 files changed, 127 insertions(+), 16 deletions(-)

Review feedback was analyzed and applied.

Changes

• Moved PSS-for-encrypt guard before #if OPENSSL_VERSION_NUMBER >= 0x30000000L so it fires on all OpenSSL versions, not just 3.x. On pre-3.x smokers, encrypt() with PSS was falling through to the legacy RSA code path which produced a generic "unknown padding type" error instead of the friendly croak message. This matches how the Marvin attack (PKCS#1 v1.5) guard is already placed before the ifdef.
• Simplified test 9 in t/private_encrypt.t to use like() unconditionally instead of branching on OpenSSL version, since the PSS croak now fires on all versions.

Actions

• Resolved merge conflicts (1 round(s))
• Rebased koan.toddr.bot/fix-private-encrypt-padding onto upstream/main
• Applied review feedback
• Force-pushed koan.toddr.bot/fix-private-encrypt-padding to origin
• CI passed

CI

CI passed.

---

Automated by Kōan

[timlegge]

@toddr-bot rebase this

[toddr-bot]

Rebase: fix: private_encrypt/public_decrypt broken on OpenSSL 3.x

Branch koan.toddr.bot/fix-private-encrypt-padding rebased onto main and force-pushed.

Diff: 4 files changed, 127 insertions(+), 16 deletions(-)

Review feedback was analyzed and applied.

Actions

• Resolved merge conflicts (1 round(s))
• Rebased koan.toddr.bot/fix-private-encrypt-padding onto upstream/main
• Force-pushed koan.toddr.bot/fix-private-encrypt-padding to origin
• CI passed

CI

CI passed.

---

Automated by Kōan

[timlegge]

@toddr-bot rebase from main

[toddr-bot]

Rebase: fix: private_encrypt/public_decrypt broken on OpenSSL 3.x

Branch koan.toddr.bot/fix-private-encrypt-padding rebased onto main and force-pushed.

Diff: 4 files changed, 122 insertions(+), 11 deletions(-)

Review feedback was analyzed and applied.

Actions

• Already-solved check: skipped (Claude call failed)
• Resolved merge conflicts (1 round(s))
• Rebased koan.toddr.bot/fix-private-encrypt-padding onto upstream/main
• Force-pushed koan.toddr.bot/fix-private-encrypt-padding to origin
• CI check enqueued (async)

CI

CI will be checked asynchronously.

---

Automated by Kōan

[timlegge]

looks good