Hi Steve,

Based on our discussion above, I performed some analysis as to why we were seeing so many events. The reason seems to be due to the default rules being triggered every time a cron job runs. We have numerous cron jobs running per minute as a result of which multiple different events(LOGIN, USER_END,CRED_DISP etc) are generated each time a cron job runs. As we do not enable SELinux, disabling these thing use subj_type=crond_t is not a viable option.

1. I have tried the following way to exclude using msg_type and exe together and it seems to work.

-a exclude,always -F msgtype=MAC_IPSEC_EVENT -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_AUTH -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_ACCT -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=CRED_REFR -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=CRED_DISP -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=CRED_ACQ -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_START -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=USER_END -F exe=/usr/sbin/cron
-a exclude,always -F msgtype=SERVICE_START -F exe=/usr/sbin/cron

Just want to make sure there is nothing I am missing here and that this only excludes the msg types for the cron executable.

2. Apart from these messages, there is a LOGIN message that gets generated each time a cron runs. Eventhough, the LOGIN message in auditd does not have an exe field, the following statement surprisingly seems to be working.

-a exclude,always -F msgtype=LOGIN -F exe=/usr/sbin/cron

I can still see LOGIN messages for other users but the cron LOGIN messages seem to be suppressed. Could you provide some detail as to how this is happening and is the expected result.

3. Is there a better way to suppress these cron messages that I am not considering apart from the SELinux option mentioned.

Thank You.

On Thu, Dec 9, 2021 at 8:19 AM Steve Grubb <sgrubb@redhat.com> wrote:

Hello,

On Wednesday, December 8, 2021 11:00:50 PM EST Amjad Gabbar wrote:
> Got it. Makes sense to me. Thanks for the explanation Steve.
>
> One last thing though based on the discussion we had, if the kernel is able
> to offload events even during bursts, wouldn’t setting q_depth
> =backlog_limit be enough?

Depends on the client attached to the af_unix socket. The kernel could have a
burst and then another. If the client isn't reading the af_unix socket fast
enough, there could still be overflows. That said, it is likely to be enough.
But I'd look into the client app that reads the af_unix socket to see why
it's taking so long.

> The reason being if there was an overflow on the kernel side, a different
> message would be printed in the logs but because it is all dispatch errors,
> I assume the kernel is able to handle the burst which is why the reasoning
> of increasing q_depth to backlog_limit.

The q_depth being 90 is your likely problem. Realistically, that only allows
20 - 30 events to be enqueued.

-Steve

> On Wed, Dec 8, 2021 at 4:38 PM Steve Grubb <sgrubb@redhat.com> wrote:
> > On Wednesday, December 8, 2021 4:54:18 PM EST Amjad Gabbar wrote:
> > > 1. The version of auditd is 1:2.8.4-3 and the plugins are af_unix.conf
> >
> > and
> >
> > > syslog.conf for audisp. The q_depth is currently set to 80 and I think
> > > it
> > > calls for an increase but not sure if there is a way to figure out what
> >
> > the
> >
> > > proper number would be?
> >
> > There is no good calculation that I can give you. It depends on the
> > average
> > rate of incoming events and the rate that they can be offloaded to the
> > plugins
> > + some margin in case there is a burst. Looking at the 2.8.5 code, the
> > default is 250.
> >
> > https://github.com/linux-audit/audit-userspace/blob/2.8_maintenance/init.
> > d/ audispd.conf
> >
> > So, you should at least set it that high. Maybe a bit higher.
> >
> > > 2. Another thing I would like to follow up on is the difference between
> > > q_depth and backlog_limit. My assumption was if there is any drop due
> > > to
> >
> > a
> >
> > > burst of events it would be addressed by the backlog limit. Just would
> >
> > like
> >
> > > some clarification on this and how this is an event dispatcher issue?
> >
> > The backlog limit is inside the kernel. This is the buffer that holds
> > events
> > that are waiting for the audit daemon to offload them. Once the audit
> > daemon
> > has them, it sends it to the dispatcher which also buffers events because
> > not
> > all plugins are able to receive the events as soon as they arrive at the
> > dispatcher.
> >
> > So, for brief bursts, the kernel backlog will handle the load. But once
> > they
> > are pulled out of the kernel, the q_depth controls how much to hold
> > waiting
> > for plugins. If this number needs to increase much, then the plugins are
> > having problems. The syslog plugin should be fine. I'd look more at the
> > af_unix plugin. The client that attaches to it needs to unload events
> > quickly. I'd investigate the af_unix client to see if it's the problem.
> >
> > Cheers,
> > -Steve