Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ROS node: GPU memory leak #2

Open
tomsal opened this issue Apr 8, 2021 · 0 comments
Open

ROS node: GPU memory leak #2

tomsal opened this issue Apr 8, 2021 · 0 comments

Comments

@tomsal
Copy link
Member

tomsal commented Apr 8, 2021

We have recently discovered a GPU memory leak when using the ROS node. I haven't found the time yet to fix this.

Symptoms:
GPU memory allocated by ROS node increases with the number of messages processed and crashes after a while (depending on GPU memory size) with a cudnn allocation error:

...
Error message:
[E] [TRT] ../rtSafe/safeContext.cpp (105) - Cudnn Error in initializeCommonContext: 4 (Could not initialize cudnn, please check cudnn installation.)
[E] [TRT] FAILED_ALLOCATION: std::exception

When playing 10 fps rosbags we experienced an increase in memory from ~1.3GB over 15 minutes.

Possible problem:
I suspect that this is a problem that arises somewhere at the interface between NVIDIA TensorRT and Instance Stixels. We haven't experienced this with the PyTorch based Cityscapes evaluation script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant