5:25 p.m.
Hello,
On Thursday, July 7, 2022 12:05:28 AM EDT Ken Hornstein wrote:
So we've been struggling with getting audisp-remote working in a
reliable manner. In summary, it works but the networking seems fragile.
We are using Kerberos authentication with audisp-remote, but that
doesn't seem to be related to the fragility (sadly the Kerberos support
does make it trivial to completely hang the server, but that's another
issue).
2 Weeks ago I wrote a model to go looking for certain kinds of problems in
kerberos. The results were that it's probably leaking memory. And on the
client side, I don't think it was fully resetting all the kerberos variables
on failure - which may be contributing to the problems.
This is on RHEL 7 which ships with audit-2.8.5, but as far as
I can tell the relevant code hasn't changed much from there to what
is on GitHub.
There are differences. I'd trust the current code in github more than the old
code.
After staring at the code a lot and doing some experiments,
here's what
I believe to be true. I'll gladly take corrections for anything I get
wrong.
- If a connection has _never_ been made successfully by audisp-remote,
it will retry the connection (in theory there's a limit to retries,
but that seems to be per-message; it will retry on every new message).
Fine, that seems reasonable.
In github, the first connection should do unlimited retries.
- If the connection is lost for almost any reason (see below), the
connection is never retried using the default configuration. There might
be some corner cases where a retry can happen, but in my experience that
is rare. Once it's gone, it never gets retried, and audit messages build
up until the queue overflows.
The behavior for what to do became a configuration item around 3.0.
- In theory if a graceful shutdown is received by audisp-remote
(either
a zero-length read or a "ENDING" audit message), then retries can
happen; this is indicated by the "remote_ended" flag in the code.
This would happen if, for example, the aggregating server needed to reboot.
But
in my experience that is rare; during my experiments when I rebooted
our audit server that message was never sent (I guess the audit server
stop was received after the interfaces were shut down). If the audit
server crashes or you have a network failure, you end up getting an
error on a write and then the network is marked down and you get into
never-retry state.
- If you turn on heartbeats via heartbeat_timeout, the network connection
_will_ retry when a heartbeat is sent. However, the subtle issue here
is that a heartbeat is only sent when there are no incoming audit
messages within the heartbeat timeout.
It is advisable to use the heartbeat option. This way each end can detect the
other "disappeared" for some reason.
The key issue seems to be in this part of the loop in main() (this
section
is entered when audisp-remote receives an audit record):
// See if input fd is also set
if (FD_ISSET(ifd, &rfd)) {
do {
if (remote_fgets(event, sizeof(event),
ifd)) { if (!transport_ok && remote_ended &&
(config.remote_ending_action
== FA_RECONNECT || !connected_once)) { quiet = 1;
if (init_transport() ==
ET_SUCCESS)
{ remote_ended = 0; connected_once = 1; }
quiet = 0;
}
In short, when a new audit record is received, init_transport()
(which tries to connect to the audit server) is only called _IF_ the
connection is down (transport_ok == 0) _and_ remote_ended is true _and_
remote_ending_action is set to FA_RECONNECT (the default) _or_ there
hasn't been at least one successful connection (connected_once == 0).
The problem with that is at least in our environment remote_ended is
never set to 1, so when the connection drops it is never retried, and
there aren't any other entry points in the normal event loop that would
ever cause the connection to retry.
I want to think this has been fixed in the current code. It is one of the
subtle changes since 2.8.5.
The heartbeat code calls relay_event() directly (code that sends
audit
events normally calls send_one() which returns if transport_ok is false)
and relay_event() calls either relay_sock_ascii() or relay_sock_managed()
and those two functions will call init_transport() if the network
connection is down. But as mentioned above, you need to make sure that
you try to send a heartbeat every so often; if you have a server generating
audit messages constantly then there won't be a heartbeat if you set the
heartbeat timeout too high.
You _can_ get a network connection retry if you encounter an error
inside of relay_sock_ascii() or relay_sock_managed(); I can't say
that didn't happen with us, but it sure seemed like it wasn't sufficient
and having the transport marked as failed was inevitible.
So, I guess my questions are:
- Is this all accurate?
It's been a long time since I did anything with 2.8.5. I'll take your word
for it.
- Is this how it's SUPPOSED to be? At least for us, network
glitches
happen enough that most of our hosts ended up with overflowing
audisp-remote queues. Setting the heartbeat timeout seems to have
resolved that (but it took a little experimentation to figure out
the right value). It just seems surprising that it was easy to get
into a situation where you'd never retry a connection.
I know there are people on this list that are using it reliably in
production. But, the problems were worked out mostly in the 3.0 release. The
kerberos code is donated code. I have not personally tested it myself due to
the problems in setting up the infrastructure. But from my review 2 weeks
ago, it looks like it would have problems in any error situation. I committed
some updates today which should make krb5 support better.
The non-kerberos code has been heavily tested. You might try that to see if
it works better. But if you are on the old code, there were problems fixed in
the 3.0 release. I think people using it are not using the krb5 code and
create a vpn or ssh tunnel for encryption.
Best Regards,
-Steve