Skip to content.

plope

Personal tools
You are here: Home » Plope Software » Plope Software Collector » reload not working reliably in supervisor 3.0a3
 
 
Views
Issue 253 of Plope Software Collector [Plope]
Title: reload not working reliably in supervisor 3.0a3
Status: Resolved Security related: No
Description: William Dode reports: Hi, When i type reload in supervisorctl it generaly (not every time) doesn't restart supervisord : supervisor> status crieu...
From: chrism on: Dec 22, 07 10:49
chrism Last update: Dec 22, 07 10:49
Topic/class: supervisor/bug Importance: medium
Version info:
Assigned: chrism
Issue 253 Transcript
3 entries
= Resolve - Entry #3 by chrism on Jan 19, 2008 1:39 pm

 Status: Accepted => Resolved

Somehow during reload a channel related to an http server stays open beyond the lifetime of the server itself. I fixed this by ensuring that all channels related to servers in the asyncore socket_map are closed when the servers are closed, e.g.:

Index: options.py
===================================================================
--- options.py (revision 678)
+++ options.py (working copy)
@@ -902,6 +902,22 @@
     def close_httpservers(self):
         for config, server in self.httpservers:
             server.close()
+ map = self.get_socket_map()
+ # server._map is a reference to the asyncore socket_map
+ for dispatcher in map.values():
+ # For unknown reasons, sometimes an http_channel
+ # dispatcher in the socket map related to servers
+ # remains open *during a reload*. If one of these
+ # exists at this point, we need to close it by hand
+ # (thus removing it from the asyncore.socket_map). If
+ # we don't do this, 'cleanup_fds' will cause its file
+ # descriptor to be closed, but it will still remain in
+ # the socket_map, and eventually its file descriptor
+ # will be passed to # select(), which will bomb. See
+ # also http://www.plope.com/software/collector/253
+ dispatcher_server = getattr(dispatcher, 'server', None)
+ if dispatcher_server is server:
+ dispatcher.close()
 
     def close_logger(self):
         self.logger.close()


________________________________________
= Comment - Entry #2 by chrism on Jan 19, 2008 12:29 pm

Traceback (most recent call last):
  File "bin/supervisord", line 7, in ?
    sys.exit(
  File "/Users/chrism/projects/supervisor/supervisor2/src/supervisor/supervisord.py", line 319, in main
    go(options)
  File "/Users/chrism/projects/supervisor/supervisor2/src/supervisor/supervisord.py", line 329, in go
    d.main()
  File "/Users/chrism/projects/supervisor/supervisor2/src/supervisor/supervisord.py", line 92, in main
    self.run()
  File "/Users/chrism/projects/supervisor/supervisor2/src/supervisor/supervisord.py", line 110, in run
    self.runforever()
  File "/Users/chrism/projects/supervisor/supervisor2/src/supervisor/supervisord.py", line 198, in runforever
    r, w, x = self.options.select(r, w, x, timeout)
  File "/Users/chrism/projects/supervisor/supervisor2/src/supervisor/options.py", line 973, in select
    return select.select(r, w, x, timeout)
select.error: (9, 'Bad file descriptor')


________________________________________
= Request - Entry #1 by chrism on Dec 22, 2007 10:49 am

 Status: Pending => Accepted

 Supporters added: chrism

William Dode reports:

Hi,

When i type reload in supervisorctl it generaly (not every time) doesn't
restart supervisord :

supervisor> status
crieur RUNNING pid 4623, uptime 0:14:27
webgen RUNNING pid 4681, uptime 0:07:37
supervisor> reload
Really restart the remote supervisord process y/N? y
Restarted supervisord
supervisor> status
error: socket.error, (2, 'No such file or directory'): file: <string> line: 1

supervisord as died (in ps)

in the supervisord.log the last lines :

2007-12-21 18:28:36,545 INFO waiting for crieur, webgen to die
2007-12-21 18:28:36,552 INFO stopped: webgen (terminated by SIGTERM)
2007-12-21 18:28:36,554 INFO stopped: crieur (terminated by SIGTERM)
2007-12-21 18:28:36,585 INFO RPC interface 'supervisor' initialized
2007-12-21 18:28:36,586 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2007-12-21 18:28:36,587 INFO daemonizing the supervisord process