Re: Odd memory usage in auditd
On Thursday, October 07, 2010 05:52:49 am Ross Kirk wrote:
Has anybody got any advice for the following problem? As I'm
seeing some
very odd behaviour with the auditd daemon in RHEL5.2 where under heavy
system load the auditd process doesn't free any resources until all memory
is consumed and the kernel kills the process with an Out Of Memory error.
I seem to recall something about disk flushing causing auditd to look like its
the culprit. Do you have barriers enabled on ext3? You might also try setting
the flushing to something else like none and see if that does anything.
The system I have is a heavily customised RHEL5.2 with some fairly
stringent auditing rules specified, the config is attached. In addition to
these rules there will be various SELinux AVCs being raised as well as
events from my own software so the auditing system is kept quite busy, see
the attached report.txt for the aureport summary .
I don't see anything terribly unusual. The audit rules didn't make it, but the
backlog setting is the only thing I would be interested in seeing.
I can reproduce this behaviour consistently by generating a heavy
system
load (CPU 100% usage) while also generating a significant number of audit
events. After about 20 minutes the auditd process will have grown from 8Mb
of memory to around 1Gb;
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3037 root 16 -3 2763m 921m 16 S 3.7 91.2 0:26.49 auditd
If the system is kept busy eventually auditd will consume all the memory
available on the system and the process be killed by the kernel with an
Out Of Memory error.
Try playing with the disk flushing and let us know how that works out. There
are no known memory leaks in recent version of auditd. I try to keep malloc
down to a minimum to prevent this and memory fragmentation to creep in.
-Steve