Got it. Makes sense to me. Thanks for the explanation Steve.

One last thing though based on the discussion we had, if the kernel is able to offload events even during bursts, wouldn’t setting q_depth =backlog_limit be enough?

The reason being if there was an overflow on the kernel side, a different message would be printed in the logs but because it is all dispatch errors, I assume the kernel is able to handle the burst which is why the reasoning of increasing q_depth to backlog_limit.

Thanks 
Amjad

On Wed, Dec 8, 2021 at 4:38 PM Steve Grubb <sgrubb@redhat.com> wrote:
On Wednesday, December 8, 2021 4:54:18 PM EST Amjad Gabbar wrote:
> 1. The version of auditd is 1:2.8.4-3 and the plugins are af_unix.conf and
> syslog.conf for audisp. The q_depth is currently set to 80 and I think it
> calls for an increase but not sure if there is a way to figure out what the
> proper number would be?

There is no good calculation that I can give you. It depends on the average
rate of incoming events and the rate that they can be offloaded to the plugins
+ some margin in case there is a burst. Looking at the 2.8.5 code, the
default is 250.

https://github.com/linux-audit/audit-userspace/blob/2.8_maintenance/init.d/
audispd.conf

So, you should at least set it that high. Maybe a bit higher.


> 2. Another thing I would like to follow up on is the difference between
> q_depth and backlog_limit. My assumption was if there is any drop due to a
> burst of events it would be addressed by the backlog limit. Just would like
> some clarification on this and how this is an event dispatcher issue?

The backlog limit is inside the kernel. This is the buffer that holds events
that are waiting for the audit daemon to offload them. Once the audit daemon
has them, it sends it to the dispatcher which also buffers events because not
all plugins are able to receive the events as soon as they arrive at the
dispatcher.

So, for brief bursts, the kernel backlog will handle the load. But once they
are pulled out of the kernel, the q_depth controls how much to hold waiting
for plugins. If this number needs to increase much, then the plugins are
having problems. The syslog plugin should be fine. I'd look more at the
af_unix plugin. The client that attaches to it needs to unload events
quickly. I'd investigate the af_unix client to see if it's the problem.

Cheers,
-Steve