Re: Auditd errors on busy hosts when rolling over log files
On Mon, 2013-11-04 at 08:24 -0500, Steve Grubb wrote:
Thanks Steve.
I did a little experimentation today.
On a system that generates around 7500 audit events every five minutes I
changed, without success, the following:
In auditd.conf
- changed num_logs from 9 to 5 although I didn't expect a change as I
move out the rolled over (audit.log.?) log files as part of the
processing so there shouldn't be a big file rename impost
- changed priority_boost from 4 to 8
In audit.rules
- changed backlog from 32K to 64K to 96K to 128K
- changed rules to reduce the recorded events per 5 minute interval from
7500 to 500-600 for the same period.
This particular system is running audit-1.8.2-el5 but I see a similar
problem on a RHEL 6.4 box which I believe is running audit-2.2-2.el6.
I did note that if I executed the sync(1) command before signaling
auditd to roll over (ie execute /bin/kill -s USR1 pid) the error
SOMETIMES did not appear.
So I am a little bit lost.
I believe that the actual effect is just
- the cost of two additional lines in /var/log/messages
- the loss a few logs
My actual process is to
a. roll over the log file
b. run an ausearch --interpret like command
Perhaps my alternative is to modify my ausearch-like command to be state
full and have it process only new events as per a patch I made to
ausearch some time back
Subject: [PATCH] ausearch: Add checkpoint capability and have
incomplete logs carry forward when processing multiple audit.log
files
Date: 05/11/2013 03:59:34 PM
Am open to any suggestions ... I think the key issue is that I reduced
the generated commends into audit.log from 7500 to 600 per five minute
interval but I still see the error.
Rgds
On Monday, November 04, 2013 07:46:18 PM Burn Alting wrote:
> Hi,
>
> I have some quite busy hosts, that emit the following errors when I
> request the audit log file is rolled over (via a kill -s USR1
> auditdpid).
>
> Error receiving audit netlink packet(No buffer space available)
> Error sending signal_info request (No buffer space available)
>
> >From reading earlier posts (circa 2009) it would appear my options are
>
> a. Increase backlog buffer (currently 32768)
> b. Increase priority_boost (currently 4)
> c. Reduce the number of log files (currently 9)
Another corollary to this is that you can increase the file size and decrease
the total files which would help on rotation.
> Does anyone have a feel for which of the above should offer the best
> return?
There are 2 more options:
1) Review the rules to make sure you are not getting events that you really do
not need. If you have a lot of false positives, then you might add some
arguments that better narrow the results. For example, perhaps you have this
rule:
-a always,exit -F arch=b64 -S clock_settime -k time-change
This can give a lot of false positives. The one that really matters is when a
program sets CLOCK_REALTIME (the wall clock). So, the rule can be re-written
as:
-a always,exit -F arch=b64 -S clock_settime -F a0=0 -k time-change
which narrows its scope.
2) You might experiment with cgroups.
> Are their other configuration parameters I could adjust (aside from
> changing my ruleset in audit.rules)?
There might be general disk tuning parameters in sysctl that could help as
well. Choice of file system also has performance impacts. I haven't done any
experimenting on the performance side, but I know there are people here that
also have very busy systems.
-Steve