-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RecursionError: Maximum recursion depth exceeded #4061
Comments
I am facing this maximum recursion depth issue suddenly as well when trying to check if the object exists in the s3 bucket using
It used to work before but not sure if something changed suddenly.
|
Hi @MustaphaU, thanks for reaching out. If you limit the script to only be initializing a client (no actual operations), do you still have this behavior? In other words, what is the minimum reproducible code snippet that produces this recursion depth error? Thanks! |
Hi @RyanFitzSimmonsAK
Thank you. Edit: The error persists. Apologies for the back and forth. Yes, |
I am also getting the same error. It was working fine a few weeks ago. |
So, s3.Bucket(settings.S3_BUCKET).put_object(Key=key, Body=file_data) works, but the following code doesn't. This is a nightmare :) res = self.s3.put_object(Bucket=settings.S3_BUCKET,
Key=key,
Body=file_data) |
Probably same thing goes to get_object |
Given that you're only seeing this behavior in Sagemaker inference scripts, it's likely not purely a Boto3 problem. I've reached out to the Sagemaker team for more information about this issue, and will update this issue whenever I have more information. Ticket # for internal use : P133939124 |
Neither I nor the service team were able to reproduce this issue. Could you provide the following information?
|
@RyanFitzSimmonsAK |
seeing this issue as well, except with creating clients for the boto3 secrets_manager |
Hi, just an update. The service team was able to reproduce this behavior, and is working on determining the root cause. |
this is fantastic news! thank you team! :) just for external planning and orientation, are there any ideas roughly if this is a high-priority issue or some other level? appreciate the help, and very happy you can reproduce the issue :) |
I was facing the same issue when trying to build a sagemaker serving tenserflow image.
this was suggested in stackoverflow thread here: https://stackoverflow.com/questions/45425236/gunicorn-recursionerror-with-gevent-and-requests-in-python-3-6-2 |
Thanks for the suggestion. I had tried this fix but it didn't resolve the issue. I mentioned it here on stackoverflow |
You don't clarify it, but have you added it to your model inference code or you built the sagemaker image with it? didn't work for me when I tried it on the inference code. It has to happen prior any other python import happens. |
@deepblue-phoenix Would you mind sharing your workaround for this issue? Does anyone have a solution or a timeline on this? |
You could try the suggestions by @shresthapradip or this workaround by @pmaoui if it applies to your case. |
Hello, I am having the same issue on a sagemaker custom inference.py script (attached). I tried both using gevent.monkey.patch_all() and gevent.monkey.patch_all(ssl=False), but the issue persists. I hope there will be a solution soon. My inference.py :
|
For those using gevent, there is an issue here being tracked on their side for that: gevent/gevent#1826. This issue appears to be specific to RHEL-based systems. Please note that we do not provide or officially support gevent with our networking setup. Any issues related to gevent will need to be addressed by the gevent team. |
I also faced same issue ,But this can be fix using
Thanks, Everyone |
This thread was helpful for debugging this issue, so I'm posting my team's context and solution to this problem. We encountered this issue after updating packages in a flask application that uses gunicorn to launch "gevent workers" on python 3.10. The issue appears to have been caused by gevent monkey patching occurring too late after the application python process was started. Gunicorn itself has a built in warning log for this that looks like
We'd seen this warning in the past without it causing problems, but with newly updated packages we ran into this issue when downloads files from s3 using boto3. Two ways to fix this. One was to follow the advice in this close gunicorn github issue and NOT use a import gevent.monkey
gevent.monkey.patch_all()
# Monkey patching need to happen here before anything else.
# Gunicorn automatically monkey patches the worker processes when using gevent workers.
# But the way it does this does not strongly guarantee that the monkey patching will
# happen before this file loads, which can cause issues with core libraries like SSL.
import multiprocessing # noqa: E402 Outside of the gunicorn I think there are two paths to try to debug this: Path 1) You know you're already using gevent to monkey patch
Path 2) You don't think you're using gevent at all.
from gevent.monkey import is_module_patched
...
# Place this where it makes sense for your application
if is_module_patched("socket"): # Socket will VERY LIKELY be patched by any lib using gevent
raise RuntimeError("Gevent was already monkey patched")
else:
logging.info("Gevent was NOT monkeypatched") |
Describe the bug
I need help with this recursion error
maximum recursion depth exceeded
from boto3. This occurs when I initialize an s3 client in my inference script to allow me read s3 objects. Your insights will be deeply appreciated! Similar issue was posted on stackoverflow 2 months ago here: https://stackoverflow.com/questions/77786275/aws-sagemaker-endpoint-maximum-recursion-depth-exceeded-error-when-calling-botoHere is the relevant code block responsible for the error:
Expected Behavior
The s3 client created to enable access to the s3 objects
Current Behavior
Here is the full error log:
Reproduction Steps
simply initializing an s3 client within an inference script like so:
Possible Solution
No response
Additional Information/Context
No response
SDK version used
1.34.55
Environment details (OS name and version, etc.)
Sagemaker endpoint for Tensorflow serving
The text was updated successfully, but these errors were encountered: