Repository navigation
Description
Expected Behaviour
from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import (
AWSEncryptionSDKProvider,
)should run without issue on a py3.11 Lambda function with the latest powertools layer attached.
Current Behaviour
Instead, the above import is raising the exception ModuleNotFoundError: No module named '_cffi_backend'.
There seems to be an issue with the way cffi (a dependency of cryptography and aws-sdk-encryption) was packaged in the powertools layer. From the little research I have done, it looks like powertools uses the same base image to package their layers but cffi is sensitive to the base python version on which it's installed:
https://stackoverflow.com/questions/76443834/aws-lambda-python-3-10-no-module-named-cffi-backend
Please note that this is indirectly affecting the awswrangler layers:
aws/aws-sdk-pandas#2688 (comment)
Code snippet
import json
from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import (
AWSEncryptionSDKProvider,
)
def lambda_handler(event, context):
# TODO implement
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}Possible Solution
Packaging each layer on its dedicated python version might solve this issue
Steps to Reproduce
Run the above code snippet in a Lambda (py311) with the powertools layer:
arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:64
Python 3.12 is also impacted. Haven't tested with other runtimes.
This bug was introduced with the addition of cryptography as a dependency in the powertools layer
Powertools for AWS Lambda (Python) version
latest
AWS Lambda function runtime
3.11
Packaging format used
Lambda Layers
Debugging logs
Function Logs
_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import AWSEncryptionSDKProvider
File "/opt/python/aws_lambda_powertools/utilities/data_masking/provider/kms/aws_encryption_sdk.py", line 10, in <module>
from aws_encryption_sdk import (
File "/opt/python/aws_encryption_sdk/__init__.py", line 19, in <module>
from aws_encryption_sdk.caches.local import LocalCryptoMaterialsCache # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/aws_encryption_sdk/caches/__init__.py", line 22, in <module>
from cryptography.hazmat.primitives import hashes
File "/opt/python/cryptography/hazmat/primitives/hashes.py", line 9, in <module>
from cryptography.hazmat.bindings._rust import openssl as rust_openssl
pyo3_runtime.PanicException: Python API call failed
INIT_REPORT Init Duration: 375.98 ms Phase: init Status: error Error Type: Runtime.ExitError
ModuleNotFoundError: No module named '_cffi_backend'
thread '<unnamed>' panicked at /github/home/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.20.2/src/err/mod.rs:788:5:
Python API call failed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
File "/var/runtime/bootstrap.py", line 63, in <module>
main()
File "/var/runtime/bootstrap.py", line 60, in main
awslambdaricmain.main([os.environ["LAMBDA_TASK_ROOT"], os.environ["_HANDLER"]])
File "/var/lang/lib/python3.11/site-packages/awslambdaric/__main__.py", line 21, in main
bootstrap.run(app_root, handler, lambda_runtime_api_addr)
File "/var/lang/lib/python3.11/site-packages/awslambdaric/bootstrap.py", line 472, in run
request_handler = _get_handler(handler)
^^^^^^^^^^^^^^^^^^^^^
File "/var/lang/lib/python3.11/site-packages/awslambdaric/bootstrap.py", line 53, in _get_handler
m = importlib.import_module(modname.replace("/", "."))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lang/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/var/task/lambda_function.py", line 3, in <module>
from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import (
File "/opt/python/aws_lambda_powertools/utilities/data_masking/provider/kms/__init__.py", line 1, in <module>
from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import AWSEncryptionSDKProvider
File "/opt/python/aws_lambda_powertools/utilities/data_masking/provider/kms/aws_encryption_sdk.py", line 10, in <module>
from aws_encryption_sdk import (
File "/opt/python/aws_encryption_sdk/__init__.py", line 19, in <module>
from aws_encryption_sdk.caches.local import LocalCryptoMaterialsCache # noqa
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/python/aws_encryption_sdk/caches/__init__.py", line 22, in <module>
from cryptography.hazmat.primitives import hashes
File "/opt/python/cryptography/hazmat/primitives/hashes.py", line 9, in <module>
from cryptography.hazmat.bindings._rust import openssl as rust_openssl
pyo3_runtime.PanicException: Python API call failed
INIT_REPORT Init Duration: 1047.40 ms Phase: invoke Status: error Error Type: Runtime.ExitError
START RequestId: c824ff77-923f-4972-96f4-edc9f8a3c011 Version: $LATEST
RequestId: c824ff77-923f-4972-96f4-edc9f8a3c011 Error: Runtime exited with error: exit status 1
Runtime.ExitError
END RequestId: c824ff77-923f-4972-96f4-edc9f8a3c011Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsShipped
Activity
boring-cyborg commentedon Feb 26, 2024
Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link
heitorlessa commentedon Feb 26, 2024
looking...
heitorlessa commentedon Feb 26, 2024
confirmed. @leandrodamascena is gonna send a patch to remove Data Masking from this current layer (regression), and we will look into building a new layer for every Python version due to CFFI.
We'll update Data Masking docs in the meantime too. We'll make a patch release today.
github-actions commentedon Feb 26, 2024
This issue is now closed. Please be mindful that future comments are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.
heitorlessa commentedon Feb 26, 2024
making a patch release now where the V2 Layer doesn't have Data masking dependency (Encryption SDK which brings cffi).
github-actions commentedon Feb 26, 2024
This is now released under 2.34.2 version!