Hi Steve,
Thanks for acknowledging the issue.
In my last email I missed mentioning the fix that I have implemented.
Issue 1) As you said, I have fixed it by replacing the ev_signal by ev_child as below.
struct ev_child sigchld_watcher;
ev_child_init (&sigchld_watcher, child_handler, 0, 0);
ev_child_start (EV_DEFAULT_ &sigchld_watcher);
static void child_handler(EV_P_ ev_child *w, int revents)
{
int pid;
if (w->rpid == dispatcher_pid()) {
dispatcher_reaped();
}
}
Issue 2) In auditd.c main(), child_handler is registered not immediately after
init_dispatcher() is called. I have modified the audit to register ev_child immediately
after init_dispatcher() as below. Or maybe before calling init_dispatcher(). This fixed
issue 2 for me. Below extract is from documentation of libev for ev_child:
" It is permissible to install a child watcher after the child has been forked (which
implies it might have already exited), as long as the event loop isn't entered (or is
continued from a watcher), i.e., forking and then immediately registering a watcher for
the child is fine, but forking and registering a watcher a few event loop iterations later
or in the next callback invocation is not."
if (init_dispatcher(&config)) {
if (pidfile)
unlink(pidfile);
tell_parent(FAILURE);
return 1;
}
ev_child_init (&sigchld_watcher, child_handler, 0, 0);
ev_child_start (EV_DEFAULT_ &sigchld_watcher);
Issue 3) With the above fix for issue 2, I did not see the issue 3 getting occurred for
me. This could be because shutdown_dispatcher() is called from dispatcher_reaped() where
the status on the pipe is not checked.
Regards,
Ketan
-----Original Message-----
From: Steve Grubb [mailto:sgrubb@redhat.com]
Sent: Friday, January 06, 2017 4:12 AM
To: linux-audit(a)redhat.com
Cc: Bhagwat, Shriniketan Manjunath <shriniketan.bhagwat(a)hpe.com>
Subject: Re: Auditd reconfigure using SIGHUP
On Thursday, January 5, 2017 11:04:55 AM EST Bhagwat, Shriniketan Manjunath
wrote:
Greetings,
I was trying to reconfigure the Audit using SIGHUP and came across
with below behavior of Audit. The audit version I am using is 2.3.6.
I have been able to reproduce the issue. I am still tracing through the problem. As best I
can tell, there are at least 3 problems. 1) libev eating the signal - this will be fixed
by defining EV_CHILD_ENABLE to 0. This will make all child interference go away; 2) on
startup audispd detects no plugins and exits before auditd can register a handler in
scenario 2 - this might be fixed with a sigaction handler until libev starts processing
events; 3) pipe descriptors being invalid in audispd sometimes. This one is a mystery.
Still investigating. Thanks for reporting this.
-Steve
Scenario 1: Starting auditd with active audispd plugin.
1. Activate the audispd plugin by setting active = yes in
configuration file
(example: /etc/audisp/plugins.d/syslog.conf) 2. Start the Auditd.
Starting auditd will create the both Auditd and Audispd process. 3.
Deactivate the audispd plugin by setting active = no in configuration
file. 4. Reconfigure the Auditd by sending SIGHUP to Auditd (>> kill
-SIGHUP <auditd_pid>). This reconfiguration stops the audispd process.
5. Activate the audispd plugin by setting active = yes in
configuration file. 6. Reconfigure the Auditd by sending SIGHUP to
Auditd. This should start the audispd process, however audispd process will not be
started in this reconfiguration.
Scenario2: Starting auditd with no active audispd plugin.
1. Start the Auditd. Starting auditd will create the only Auditd process.
2. Activate the audispd plugin by setting active = yes in
configuration file. 3. Reconfigure the Auditd by sending SIGHUP to
Auditd. This should start the audispd process, however in few cases
audispd process will not be started in this reconfiguration. 4.
Deactivate the audispd plugin by setting active = no in configuration
file. 5. Reconfigure the Auditd by sending SIGHUP to Auditd. This reconfiguration stops
the audispd process.
6. Activate the audispd plugin by setting active = yes in
configuration file. 7. Reconfigure the Auditd by sending SIGHUP to
Auditd. This should start the audispd process, however audispd process
will not be started in this reconfiguration.
As per the change log of audit version 2.3.2, below fix was made.
Fix: In auditd, restart dispatcher on SIGHUP if it had previously
exited
I have analyzed Auditd code of version 2.3.6 and below is my observation.
When the Auditd is started, it calls init_dispatcher() to start the Audispd.
init_dispatcher() starts the Audispd and maintains its pid value in a
global variable (auditd-dispatch.c). When the audispd is terminated,
SIGCHLD handler of Auditd i.e child_handler() does the waitpid() to
remove the child process from zombie state and calls
dispatcher_reaped(), where this pid is set to zero. During reconfigure
using SIGHUP,
reconfigure_dispatcher() checks for this pid value. If pid is valid
then, SIGHUP is sent to audispd otherwise init_dispatcher() is called
to start the audispd.
Auditd uses event loop ev_signal to track the child process (SIGCHLD).
The event loop from libev also has child handler childcb() (in ev.c).
This handler childcb() also does the waitpid(). When audispd
terminates, libev's child handler is getting called first, waitpid()
removes the audispd from zombie state. Then the control is passed to
Auditd's child handler child_handler(). Since the audispd has been
already removed from process stable, waitpid() call done in
child_handler() will return ECHILD. Hence the dispatcher_reaped() is
not getting called to make the internally maintained pid variable to
zero. This results in subsequent SIGHUP set to Auditd end up in sending SIGHUP to
non-existing audispd process.
In case of step 3 of scenario 2: When the auditd is started without
any active audispd plugin(step1), Auditd starts the audispd. However
since there are no active plugins Audispd will get terminated. In this
case
shutdown_dispatcher() was called from dispatch_event() where pid value
was set to zero. Hence the reconfiguration in step 3 started audispd
process in my case.
I have put traces in auditd code to validate the above behavior. This
behavior is applicable for Audit version 2.6.4 as well. Is this
behavior a known issue?
Regards,
Ketan