Re: Maximum Value for q_depth
Hello,
On Wednesday, December 8, 2021 11:00:50 PM EST Amjad Gabbar wrote:
Got it. Makes sense to me. Thanks for the explanation Steve.
One last thing though based on the discussion we had, if the kernel is able
to offload events even during bursts, wouldn’t setting q_depth
=backlog_limit be enough?
Depends on the client attached to the af_unix socket. The kernel could have a
burst and then another. If the client isn't reading the af_unix socket fast
enough, there could still be overflows. That said, it is likely to be enough.
But I'd look into the client app that reads the af_unix socket to see why
it's taking so long.
The reason being if there was an overflow on the kernel side, a
different
message would be printed in the logs but because it is all dispatch errors,
I assume the kernel is able to handle the burst which is why the reasoning
of increasing q_depth to backlog_limit.
The q_depth being 90 is your likely problem. Realistically, that only allows
20 - 30 events to be enqueued.
-Steve
On Wed, Dec 8, 2021 at 4:38 PM Steve Grubb <sgrubb(a)redhat.com>
wrote:
> On Wednesday, December 8, 2021 4:54:18 PM EST Amjad Gabbar wrote:
> > 1. The version of auditd is 1:2.8.4-3 and the plugins are af_unix.conf
>
> and
>
> > syslog.conf for audisp. The q_depth is currently set to 80 and I think
> > it
> > calls for an increase but not sure if there is a way to figure out what
>
> the
>
> > proper number would be?
>
> There is no good calculation that I can give you. It depends on the
> average
> rate of incoming events and the rate that they can be offloaded to the
> plugins
> + some margin in case there is a burst. Looking at the 2.8.5 code, the
> default is 250.
>
> https://github.com/linux-audit/audit-userspace/blob/2.8_maintenance/init.
> d/ audispd.conf
>
> So, you should at least set it that high. Maybe a bit higher.
>
> > 2. Another thing I would like to follow up on is the difference between
> > q_depth and backlog_limit. My assumption was if there is any drop due
> > to
>
> a
>
> > burst of events it would be addressed by the backlog limit. Just would
>
> like
>
> > some clarification on this and how this is an event dispatcher issue?
>
> The backlog limit is inside the kernel. This is the buffer that holds
> events
> that are waiting for the audit daemon to offload them. Once the audit
> daemon
> has them, it sends it to the dispatcher which also buffers events because
> not
> all plugins are able to receive the events as soon as they arrive at the
> dispatcher.
>
> So, for brief bursts, the kernel backlog will handle the load. But once
> they
> are pulled out of the kernel, the q_depth controls how much to hold
> waiting
> for plugins. If this number needs to increase much, then the plugins are
> having problems. The syslog plugin should be fine. I'd look more at the
> af_unix plugin. The client that attaches to it needs to unload events
> quickly. I'd investigate the af_unix client to see if it's the problem.
>
> Cheers,
> -Steve