Download models in the background

[nkoenig]

Signed-off-by: Nate Koenig [email protected]

🎉 New feature

Closes #1260.

This is an alternative to #1484.

Summary

The PR changes the Server load to:

1. Disable Fuel downloads.
2. Parse the SDF file/string.
3. Remove all models, actors, and lights from each world.
4. Create the worlds (runners).
5. Enable Fuel downloads and force simulation to be paused. User cannot unpause.
6. Start a thread that parses the SDF file again. This time models are downloaded.
7. Add the models back into the worlds.
8. Allow the pause state in simulation to change.

The above will be followed even if nothing needs to be downloaded.

I think the biggest downside to this approach is the double parsing of SDF.

Test it

I haven't added tests yet. Looking for general feedback first.

Checklist

• Signed all commits for DCO
• Added tests
• Added example and/or tutorial
• Updated documentation (as needed)
• Updated migration guide (as needed)
• Consider updating Python bindings (if the library has them)
• codecheck passed (See contributing)
• All tests passed (See test coverage)
• While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

[ahcorde]

WorldByName is not defined, is there any other related PR ?

[ahcorde]

The resource spawner and the user command can be improved downloading models in the background.

Should I open a PR for each use case ? or can we included this changes here too ?

[ahcorde]

When I restart the simulation some models disappear (even if it's not a fuel model)

[ahcorde]

When I try to run with the flags I got some crashes and some error messages:

[GUI] [Err] [SceneManager.cc:207] Visual: [ground_plane] already exists
[GUI] [Err] [SceneManager.cc:207] Visual: [box] already exists
[GUI] [Err] [SceneManager.cc:207] Visual: [cylinder] already exists
[GUI] [Err] [SceneManager.cc:207] Visual: [sphere] already exists
[GUI] [Err] [SceneManager.cc:207] Visual: [capsule] already exists
[GUI] [Err] [SceneManager.cc:207] Visual: [ellipsoid] already exists
[GUI] [Err] [BaseStorage.hh:927] Another item already exists with name: sun

[ahcorde]

I'm getting this error

Traceback (most recent call last):
	2: from /home/ahcorde/gz_garden/install/bin/gz:308:in `<main>'
	1: from /home/ahcorde/gz_garden/install/lib/ruby/gz/cmdsim7.rb:343:in `execute'
/home/ahcorde/gz_garden/install/lib/ruby/gz/cmdsim7.rb:329:in `parse': invalid option: --parallel-download (OptionParser::InvalidOption)

Suggested change

	" --parallel-download Download assets in a background thread. \n"\
	" --wait-for-assets Download assets in a background thread. \n"\

[nkoenig]

WorldByName is not defined, is there any other related PR ?

Here it is: gazebosim/sdformat#1121

[nkoenig]

When I restart the simulation some models disappear (even if it's not a fuel model)

Does this still happen to you? I've made some changes. If so, what command did you run on the command line?

[ahcorde]

When I restart the simulation some models disappear (even if it's not a fuel model)

    [
      
        ![background_reset](https://user-images.githubusercontent.com/1933907/187023007-57048dbe-2450-4031-ad96-c364ba218578.gif)
      
    ](https://user-images.githubusercontent.com/1933907/187023007-57048dbe-2450-4031-ad96-c364ba218578.gif)
    
    
      
        
          
        
        
          
          
        
      
      [
        
          
        
      ](https://user-images.githubusercontent.com/1933907/187023007-57048dbe-2450-4031-ad96-c364ba218578.gif)
    
   [ ![background_reset](https://user-images.githubusercontent.com/1933907/187023007-57048dbe-2450-4031-ad96-c364ba218578.gif) ](https://user-images.githubusercontent.com/1933907/187023007-57048dbe-2450-4031-ad96-c364ba218578.gif)
  
    [
      
        ![background_reset](https://user-images.githubusercontent.com/1933907/187023007-57048dbe-2450-4031-ad96-c364ba218578.gif)
      
    ](https://user-images.githubusercontent.com/1933907/187023007-57048dbe-2450-4031-ad96-c364ba218578.gif)
    
    
      
        
          
        
        
          
          
        
      
      [
        
          
        
      ](https://user-images.githubusercontent.com/1933907/187023007-57048dbe-2450-4031-ad96-c364ba218578.gif)
    
   [ ](https://user-images.githubusercontent.com/1933907/187023007-57048dbe-2450-4031-ad96-c364ba218578.gif)

Does this still happen to you? I've made some changes. If so, what command did you run on the command line?

@nkoenig I'm getting a crash, I can see the follwoing messages:

Msg [NameManager::issueNewName] (World::Skeleton | shapes) The name [ground_plane] is a duplicate, so it has been renamed to [ground_plane(1)]
Msg [NameManager::issueNewName] (World::Skeleton | shapes) The name [box] is a duplicate, so it has been renamed to [box(1)]
Msg [NameManager::issueNewName] (World::Skeleton | shapes) The name [cylinder] is a duplicate, so it has been renamed to [cylinder(1)]
Msg [NameManager::issueNewName] (World::Skeleton | shapes) The name [sphere] is a duplicate, so it has been renamed to [sphere(1)]
Msg [NameManager::issueNewName] (World::Skeleton | shapes) The name [capsule] is a duplicate, so it has been renamed to [capsule(1)]
Msg [NameManager::issueNewName] (World::Skeleton | shapes) The name [ellipsoid] is a duplicate, so it has been renamed to [ellipsoid(1)]

...

[GUI] [Err] [SceneManager.cc:207] Visual: [ground_plane] already exists
[GUI] [Err] [SceneManager.cc:207] Visual: [box] already exists
[GUI] [Err] [SceneManager.cc:207] Visual: [cylinder] already exists
[GUI] [Err] [SceneManager.cc:207] Visual: [sphere] already exists
[GUI] [Err] [SceneManager.cc:207] Visual: [capsule] already exists
[GUI] [Err] [SceneManager.cc:207] Visual: [ellipsoid] already exists
[GUI] [Err] [BaseStorage.hh:927] Another item already exists with name: sun

and this is the backtrace

Stack trace (most recent call last) in thread 230483:
#26   Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in 
#25   Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fec81f5c132, in clone
#24   Object "/lib/x86_64-linux-gnu/libpthread.so.0", at 0x7fec81e22608, in 
#23   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fec7ad069d1, in 
#22   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fec7ad05784, in QThread::exec()
#21   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fec7aecd3aa, in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)
#20   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fec7af26434, in QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)
#19   Object "/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0", at 0x7fec78d0d4a2, in g_main_context_iteration
#18   Object "/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0", at 0x7fec78d0d3ff, in 
#17   Object "/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0", at 0x7fec78d0d17c, in g_main_context_dispatch
#16   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fec7af26e36, in 
#15   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fec7aed1487, in QCoreApplicationPrivate::sendPostedEvents(QObject*, int, QThreadData*)
#14   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fec7aece809, in QCoreApplication::notifyInternal2(QObject*, QEvent*)
#13   Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5", at 0x7fec7a66a0ef, in QApplication::notify(QObject*, QEvent*)
#12   Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5", at 0x7fec7a660a65, in QApplicationPrivate::notify_helper(QObject*, QEvent*)
#11   Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fec7aefac29, in QObject::event(QEvent*)
#10   Object "/home/ahcorde/gz_garden/install/lib/gz-gui-7/plugins/libMinimalScene.so", at 0x7fec45688348, in gz::gui::plugins::RenderThread::RenderNext(gz::gui::plugins::RenderSync*)
#9    Object "/home/ahcorde/gz_garden/install/lib/gz-gui-7/plugins/libMinimalScene.so", at 0x7fec45695a2f, in gz::gui::plugins::RenderThreadRhiOpenGL::RenderNext(gz::gui::plugins::RenderSync*)
#8    Object "/home/ahcorde/gz_garden/install/lib/gz-gui-7/plugins/libMinimalScene.so", at 0x7fec4568b459, in gz::gui::plugins::GzRenderer::Render(gz::gui::plugins::RenderSync*)
#7    Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fec7aece809, in QCoreApplication::notifyInternal2(QObject*, QEvent*)
#6    Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5", at 0x7fec7a66a0ef, in QApplication::notify(QObject*, QEvent*)
#5    Object "/usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5", at 0x7fec7a660a54, in QApplicationPrivate::notify_helper(QObject*, QEvent*)
#4    Object "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", at 0x7fec7aece51a, in QCoreApplicationPrivate::sendThroughObjectEventFilters(QObject*, QEvent*)
#3    Object "/home/ahcorde/gz_garden/install/lib/gz-sim-7/plugins/gui/libGzSceneManager.so", at 0x7fec42fc1d6f, in gz::sim::v7::GzSceneManager::eventFilter(QObject*, QEvent*)
#2    Object "/home/ahcorde/gz_garden/install/lib/gz-sim-7/plugins/gui/libGzSceneManager.so", at 0x7fec42fc2116, in gz::sim::v7::GzSceneManagerPrivate::OnRender()
#1    Object "/home/ahcorde/gz_garden/install/lib/libgz-sim7-rendering.so.7", at 0x7fec42d3f7d4, in gz::sim::v7::RenderUtil::Update()
#0    Object "/home/ahcorde/gz_garden/install/lib/libgz-sim7-rendering.so.7", at 0x7fec42e33661, in gz::sim::v7::SceneManager::CreateLight(unsigned long, sdf::v13::Light const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)

[codecov]

Codecov Report

Attention: Patch coverage is 87.43590% with 49 lines in your changes missing coverage. Please review.

Project coverage is 64.74%. Comparing base (5a76b28) to head (cf5f36a).
Report is 19 commits behind head on gz-sim7.

❗ Current head cf5f36a differs from pull request most recent head 656f6d6

Please upload reports for the commit 656f6d6 to get more accurate results.

Files	Patch %	Lines
src/ServerPrivate.cc	76.92%	21 Missing ⚠️
src/SimulationRunner.cc	83.46%	21 Missing ⚠️
src/LevelManager.cc	94.93%	4 Missing ⚠️
src/SdfEntityCreator.cc	96.82%	2 Missing ⚠️
src/Conversions.cc	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           gz-sim7    #1669      +/-   ##
===========================================
+ Coverage    64.72%   64.74%   +0.01%     
===========================================
  Files          357      357              
  Lines        29160    29192      +32     
===========================================
+ Hits         18874    18900      +26     
- Misses       10286    10292       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ahcorde]

GUI is blocked when the models are being downloaded

[ahcorde]

conflicts, did you review my last comment?

[nkoenig]

GUI is blocked when the models are being downloaded

Shouldn't the GUI block if you're waiting for assets?

[iche033]

There seems to be a GUI race condition. Sometimes I see Gazebo loads the GUI configuration from the world file correctly but sometimes it uses the default GUI config.

This is from running the fuel_textured_mesh.sdf world file:

GUI uses config from world file:

GUI uses default configurations:

[azeey]

@nkoenig since Garden will be EOL in about a month and this is a nontrivial PR, I suggest retargetting this PR to either main or gz-sim8. Also, do we still want to get this into the Ionic release?

[nkoenig]

@nkoenig since Garden will be EOL in about a month and this is a nontrivial PR, I suggest retargetting this PR to either main or gz-sim8. Also, do we still want to get this into the Ionic release?

I'll retarget this to gz-sim8.