Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minimize module loading #2102

Open
petdance opened this issue Sep 12, 2023 · 5 comments
Open

Minimize module loading #2102

petdance opened this issue Sep 12, 2023 · 5 comments

Comments

@petdance
Copy link
Contributor

I'm loading Mojo::UserAgent in my web app to go fetch pages behind the scenes. Unfortunately, this brings in many unnecessary modules that have to get loaded in my app, such as Pod::Usage and Getopt::Long.

It would be nice if a module like Mojo::UserAgent that uses something like monkey_patch and term_escape wouldn't also bring in all the other unnecessary modules as well.

Maybe a Mojo::Util::CLI for things that are only used by CLI tools, for example?

@jberger
Copy link
Member

jberger commented Sep 12, 2023

While I don't generally feel we need to micro-optimize module loading across the stack, I guess I can see that there are a few dependencies in Mojo::Util that could be easily late-loaded when needed and which are rarely needed unless in specific contexts. I would consider supporting a PR (good for first timers?) that late-loads some of those rarely used dependencies specifically in Mojo::Util.

[Edit: changed from implied promise to support to consider supporting]

@kraih
Copy link
Member

kraih commented Sep 12, 2023

Late loading seems like it would be bad for prefork memory usage. I'd be very sceptical of any PRs in that direction and would want to see data on memory usage for various deployment scenarios.

@jberger
Copy link
Member

jberger commented Sep 12, 2023

In general that would be true, however, at least Pod::Usage would rarely need to be early-loaded for preforking. I guess I could argue GetOpt::Long either way, though in a non-webapp/Mojo::UserAgent context it would certainly seem to be a win.

On the other hand, it might make things a bit harder for say fat-packing deployments. Missing dependencies because they weren't detected at load-time. Though on the other-other hand, core modules like Pod::Usage and GetOpt::Long don't have to be fat-packed anyway.

I've been debating whether say the Socket dependencies in network_contains would be worth late-loading, but yes, that would suffer from a preforking perspective. All of this is very interesting. @petdance do you have numbers on what the memory usage penalty is for those?

@petdance
Copy link
Contributor Author

I don't have any memory usage penalty numbers, because I wasn't even at that point yet. So far, it's just a "Hmm, why is that here?"

Every so often I do a survey of modules that are loaded in our preloaded web app. Currently, we have 983 from Algorithm::Verhoeff to XML::Xerces. The goal is to make sure that everything gets early loaded (through a startup script if necessary) vs. late loaded. (I also make sure that there are no Test::* modules that get loaded somehow.)

While perusing the list, I was surprised to see the modules I mentioned above: Getopt::Long and the six Pod::*, since nothing in the app would every do anything with either of them. I tracked down what was bringing them in (since it wasn't in our code tree). (Thanks to Devel::TraceLoad!). And then I wrote this issue up.

So, no, I'm not saying "OMG your preloading is ruining my app!!11!1!!!". On the other hand, I could see it becoming a problem in the future pretty easily just by having something added to Mojo::Util. That's why my first thought, as someone who knows not much about Mojo, was to split CLI-only tools into Mojo::Util::CLI.

@petdance
Copy link
Contributor Author

FYI, I found that it was about 13MB less memory used in my Apache children if I don't use Mojo::UserAgent. Here are the modules that stopped getting loaded if I remove Mojo::UserAgent from my web app's modules:

  • Class::Struct
  • Cpanel::JSON::XS
  • File::stat
  • Getopt::Long
  • IO::Poll
  • Pod::Escapes
  • Pod::Simple
  • Pod::Simple::BlackBox
  • Pod::Simple::LinkSection
  • Pod::Text
  • Pod::Usage
  • and of course Mojo::*

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants