start & stop & start temboard-agent (9.0.1) quickly does not work

[l00ptr]

Hello,

Since temboard-agent 9.0.1 i have a few issues when trying to stop & start quickly (eg using some automated tests in CI) the agent, here are a few steps to reproduce what i observe on my side (outside of my tests suite / CI):

• Install temboard-agent 9.0.1
• Run temboard-agent in backround: temboard-agent --config /tmp/temboard-agent-17-test.conf &
• Get the PID of the temboard-agent (from previous step)
• kill the first temboard-agent and launch (quickly) a new one temboard-agent with: kill <pid first agent> && temboard-agent --config /tmp/temboard-agent-17-test.conf
• After the last operation, we can observe this error:

INFO:  app: Starting temboard-agent. version=9.0.1
INFO:  app: Using config file /tmp/temboard-agent-17-test.conf.
WARNING:  agent: Hostname dontbecurious is not a FQDN.
INFO:  serve: Resetting monitoring data.
CRITICAL:  app: Failed to start HTTPS server: [Errno 98] Address already in use.

When I run top, pstree or any other tool to watch the processes, i can see a few temboard processes (and the especially the web worker) taking a few seconds before dying. So i guess killing all the process take some time the HTTP(s) port is kept open / busy by a nearly dead process.

[pgiraud]

I made some more tests, I came to the conclusion that it's not because of child processes that it takes some time to get the process actually terminated.

The kill command doesn't actually wait for the process to be stopped. It returns immediately. Unless passed the -9 (KILL signal).

With an agent already running, the following command force stops all the processes and temboard agent can be started immediately:

$ pkill $PID -9 && temboard-agent --config /tmp/temboard-agent-17-test.conf

It looks like temboard tries to stop gracefully and it can take some time.

Considering that the agent runs in a terminal with the following (first) command, using wait can help making sure that the process is effectively stopped before running temboard again with no error.

$ temboard-agent --config /tmp/temboard-agent-17-test.conf 2> &
$ kill $PID && wait $PID && temboard-agent --config /tmp/temboard-agent-17-test.conf