RE: Odd memory usage in auditd
Doh, originally just replied to Steve rather than to the list!
Ross
-----Original Message-----
From: linux-audit-bounces(a)redhat.com [mailto:linux-audit-
bounces(a)redhat.com] On Behalf Of Steve Grubb
Sent: 07 October 2010 20:51
To: linux-audit(a)redhat.com
Subject: Re: Odd memory usage in auditd
On Thursday, October 07, 2010 05:52:49 am Ross Kirk wrote:
> Has anybody got any advice for the following problem? As I'm seeing
some
> very odd behaviour with the auditd daemon in RHEL5.2 where under
heavy
> system load the auditd process doesn't free any resources until all
memory
> is consumed and the kernel kills the process with an Out Of Memory
error.
I seem to recall something about disk flushing causing auditd to look
like its the culprit. Do you have barriers enabled on ext3? You might
also try setting the flushing to something else like none and see if
that does anything.
Currently barriers are not enabled as it wasn't something I was aware of. However it
does sound like something I may well want to be enabled so I will give the various flush
settings a try and see if the barrier option affects anything.
So if auditd is not the culprit what do you suspect the problem is?
> The system I have is a heavily customised RHEL5.2 with some
fairly
> stringent auditing rules specified, the config is attached. In
addition to
> these rules there will be various SELinux AVCs being raised as well
as
> events from my own software so the auditing system is kept quite
busy, see
> the attached report.txt for the aureport summary .
I don't see anything terribly unusual. The audit rules didn't make it,
but the backlog setting is the only thing I would be interested in
seeing.
-b 16384
> I can reproduce this behaviour consistently by generating a
heavy
system
> load (CPU 100% usage) while also generating a significant number of
audit
> events. After about 20 minutes the auditd process will have grown
from 8Mb
> of memory to around 1Gb;
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 3037 root 16 -3 2763m 921m 16 S 3.7 91.2 0:26.49 auditd
>
> If the system is kept busy eventually auditd will consume all the
memory
> available on the system and the process be killed by the kernel with
an
> Out Of Memory error.
Try playing with the disk flushing and let us know how that works out.
There
are no known memory leaks in recent version of auditd. I try to keep
malloc down to a minimum to prevent this and memory fragmentation to
creep in.
With;
* flush = none Completely different behaviour from previous, memory usage never changed
for my entire test run
* flush = data No increase in memory usage at all
* flush = sync No increase in memory usage at all
Changing the ext3 barrier setting didn't make any changes to the above results nor to
the behaviour I saw when "incremental" was set.
Well as the other settings are better behaved I can change to using "data"
without any problems I believe. Arguably this is probably a better choice for me anyway!
Thanks for your help!
Ross