Skip to content

Bug: Importing AWSEncryptionSDKProvider in the AWSLambdaPowertoolsPythonV2 layer results in ModuleNotFoundError: No module named '_cffi_backend' #3852

Closed

Description

@jaidisido

Expected Behaviour

from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import (
    AWSEncryptionSDKProvider,
)

should run without issue on a py3.11 Lambda function with the latest powertools layer attached.

Current Behaviour

Instead, the above import is raising the exception ModuleNotFoundError: No module named '_cffi_backend'.

There seems to be an issue with the way cffi (a dependency of cryptography and aws-sdk-encryption) was packaged in the powertools layer. From the little research I have done, it looks like powertools uses the same base image to package their layers but cffi is sensitive to the base python version on which it's installed:
https://stackoverflow.com/questions/76443834/aws-lambda-python-3-10-no-module-named-cffi-backend

Please note that this is indirectly affecting the awswrangler layers:
aws/aws-sdk-pandas#2688 (comment)

Code snippet

import json

from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import (
    AWSEncryptionSDKProvider,
)

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Possible Solution

Packaging each layer on its dedicated python version might solve this issue

Steps to Reproduce

Run the above code snippet in a Lambda (py311) with the powertools layer:
arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:64

Python 3.12 is also impacted. Haven't tested with other runtimes.

This bug was introduced with the addition of cryptography as a dependency in the powertools layer

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.11

Packaging format used

Lambda Layers

Debugging logs

Function Logs
_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import AWSEncryptionSDKProvider
File "/opt/python/aws_lambda_powertools/utilities/data_masking/provider/kms/aws_encryption_sdk.py", line 10, in <module>
from aws_encryption_sdk import (
File "/opt/python/aws_encryption_sdk/__init__.py", line 19, in <module>
from aws_encryption_sdk.caches.local import LocalCryptoMaterialsCache  # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/aws_encryption_sdk/caches/__init__.py", line 22, in <module>
from cryptography.hazmat.primitives import hashes
File "/opt/python/cryptography/hazmat/primitives/hashes.py", line 9, in <module>
from cryptography.hazmat.bindings._rust import openssl as rust_openssl
pyo3_runtime.PanicException: Python API call failed
INIT_REPORT Init Duration: 375.98 ms	Phase: init	Status: error	Error Type: Runtime.ExitError
ModuleNotFoundError: No module named '_cffi_backend'
thread '<unnamed>' panicked at /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.20.2/src/err/mod.rs:788:5:
Python API call failed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "/var/runtime/bootstrap.py", line 63, in <module>
main()
File "/var/runtime/bootstrap.py", line 60, in main
awslambdaricmain.main([os.environ["LAMBDA_TASK_ROOT"], os.environ["_HANDLER"]])
File "/var/lang/lib/python3.11/site-packages/awslambdaric/__main__.py", line 21, in main
bootstrap.run(app_root, handler, lambda_runtime_api_addr)
File "/var/lang/lib/python3.11/site-packages/awslambdaric/bootstrap.py", line 472, in run
request_handler = _get_handler(handler)
^^^^^^^^^^^^^^^^^^^^^
File "/var/lang/lib/python3.11/site-packages/awslambdaric/bootstrap.py", line 53, in _get_handler
m = importlib.import_module(modname.replace("/", "."))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lang/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/var/task/lambda_function.py", line 3, in <module>
from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import (
File "/opt/python/aws_lambda_powertools/utilities/data_masking/provider/kms/__init__.py", line 1, in <module>
from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import AWSEncryptionSDKProvider
File "/opt/python/aws_lambda_powertools/utilities/data_masking/provider/kms/aws_encryption_sdk.py", line 10, in <module>
from aws_encryption_sdk import (
File "/opt/python/aws_encryption_sdk/__init__.py", line 19, in <module>
from aws_encryption_sdk.caches.local import LocalCryptoMaterialsCache  # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/aws_encryption_sdk/caches/__init__.py", line 22, in <module>
from cryptography.hazmat.primitives import hashes
File "/opt/python/cryptography/hazmat/primitives/hashes.py", line 9, in <module>
from cryptography.hazmat.bindings._rust import openssl as rust_openssl
pyo3_runtime.PanicException: Python API call failed
INIT_REPORT Init Duration: 1047.40 ms	Phase: invoke	Status: error	Error Type: Runtime.ExitError
START RequestId: c824ff77-923f-4972-96f4-edc9f8a3c011 Version: $LATEST
RequestId: c824ff77-923f-4972-96f4-edc9f8a3c011 Error: Runtime exited with error: exit status 1
Runtime.ExitError
END RequestId: c824ff77-923f-4972-96f4-edc9f8a3c011

Activity

  1. added
    bugSomething isn't working
    triagePending triage from maintainers
    on Feb 26, 2024
  2. boring-cyborg commented on Feb 26, 2024

    @boring-cyborg

    Thanks for opening your first issue here! We'll come back to you as soon as we can.
    In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

  3. heitorlessa commented on Feb 26, 2024

    @heitorlessa
    Contributor

    looking...

  4. moved this from Triage to Working on it in Powertools for AWS Lambda (Python)on Feb 26, 2024
  5. heitorlessa commented on Feb 26, 2024

    @heitorlessa
    Contributor

    confirmed. @leandrodamascena is gonna send a patch to remove Data Masking from this current layer (regression), and we will look into building a new layer for every Python version due to CFFI.

    We'll update Data Masking docs in the meantime too. We'll make a patch release today.

  6. github-actions commented on Feb 26, 2024

    @github-actions
    Contributor

    ⚠️COMMENT VISIBILITY WARNING⚠️

    This issue is now closed. Please be mindful that future comments are hard for our team to see.

    If you need more assistance, please either tag a team member or open a new issue that references this one.

    If you wish to keep having a conversation with other community members under this issue feel free to do so.

  7. heitorlessa commented on Feb 26, 2024

    @heitorlessa
    Contributor

    making a patch release now where the V2 Layer doesn't have Data masking dependency (Encryption SDK which brings cffi).

  8. moved this from Coming soon to Shipped in Powertools for AWS Lambda (Python)on Feb 26, 2024
  9. github-actions commented on Feb 26, 2024

    @github-actions
    Contributor

    This is now released under 2.34.2 version!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions