4:15 p.m.
Hello All,
I have some systems that have just been updated to from RHEL 5.2 to RHEL
5.3. The version of auditd is 1.7.7 and SnareLinux is 1.5.0.
Some time after the update ran, I noticed that the amount of free memory
on the systems had dramatically gone down. Running top, I saw that
auditd had sucked up lots of memory:
top - 09:25:59 up 2 days, 2 min, 1 user, load average: 0.00, 0.03,
0.00
Tasks: 111 total, 1 running, 110 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.1%si,
0.0%st
Mem: 4043368k total, 3994488k used, 48880k free, 64656k buffers
Swap: 8385920k total, 352k used, 8385568k free, 449884k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31136 root 12 -3 6568m 3.1g 568 S 0.0 81.0 0:08.84 auditd
3861 root 15 0 26152 16m 364 S 0.0 0.4 0:00.01 restorecond
29474 root 16 0 97.9m 14m 2852 S 0.0 0.4 0:00.16 sshd
29382 root 17 0 124m 13m 1464 S 0.0 0.3 0:00.09 crond
29478 gsmith 15 0 97.9m 12m 976 S 0.0 0.3 0:00.01 sshd
5611 root 15 0 58912 9048 4208 S 0.0 0.2 3:50.38 dsmc
3937 root 15 0 96556 7588 916 S 0.0 0.2 3:54.00 sec.pl
When I first started looking at the problem, I figured the easiest fix
was reboot since a new version of auditd was installed along with a new
kernel and there was a mismatch between the auditd daemon and the
running kernel. After rebooting, I watched a 'top' display and did not
see the auditd daemon percolate up. It turns out that it does percolate
up to the top of the memory usage; I didn't watch it long enough.
I wondered if there was a difference between 32 bit and 64 bit systems.
As it turns out, there isn't one; it's seems to be slower to happen on
32 bit systems probably due to slower CPU's
Next, I wondered if this is due to the version of Snare running on the
systems. Just to be sure, I rebuilt Snare for the 64 bit system,
de-installed Snare, installed the newly rebuilt Snare, and started
auditd. Again, the auditd daemon started to percolate to the top of the
memory usage.
So, does this have anything to do with the number of rules that auditd
has to deal with? I started trimming back the rules one at a time,
restarting auditd and watching a top display. The number of rules
doesn't have any affect until you get to zero rules and then auditd
doesn't move up.
Next, I ran a valgrind on auditd; one on a RHEL 5.3 system with no rules
and another one with the standard set of rules put on our systems.
criticality=0 event=execve uid=*,root
criticality=1 event=(mount,umount,umount2,settimeofday,swapon,swapoff,
reboot,setdomainname,create_module,delete_module,quotactl)
uid=*,root
criticality=2 event=execve exe=*passwd*
criticality=3 event=(login_auth,login_start,logout)
criticality=4 event=acct_mgmt
After 2 hours of running the former had 0 malloc/free problems with the
leak summary. The latter on the other hand:
==30021==
==30021== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 1)
==30021== malloc/free: in use at exit: 147,024,501 bytes in 16,262
blocks.
==30021== malloc/free: 27,612 allocs, 11,350 frees, 249,669,818 bytes
allocated.
==30021== For counts of detected errors, rerun with: -v ==30021==
searching for pointers to 16,262 not-freed blocks.
==30021== checked 10,651,128 bytes.
==30021==
==30021== LEAK SUMMARY:
==30021== definitely lost: 146,976,704 bytes in 16,246 blocks.
==30021== possibly lost: 36,464 bytes in 5 blocks.
==30021== still reachable: 11,333 bytes in 11 blocks.
==30021== suppressed: 0 bytes in 0 blocks.
Next, I ran a strace of auditd to see if I could spot where it was doing
all the malloc's. Starting up a strace of fresh instance of auditd,
eventually auditd settled into this loop:
clock_gettime(CLOCK_MONOTONIC, {432334, 891462998}) = 0
epoll_wait(0, {{EPOLLIN, {u32=3, u64=3}}}, 64, 59743) = 1
clock_gettime(CLOCK_MONOTONIC, {432334, 891681998}) = 0
recvfrom(3, "&\0\0\0\33\5\0\0\0\0\0\0\0\0\0\0audit(1233595754"..., 8988,
MSG_DONTWAIT, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 54
writev(5, [{"\0\0\0\0\20\0\0\0\33\5\0\0&\0\0\0", 16},
{"audit(1233595754.282:836040): c"..., 38}], 2) = 54
And so it goes. Nothing leaps out at me from this tight code sequence
that it's glomming memory.
I wondered if this odd behavior was exhibited back in RHEL 5.2 and we
just never noticed. So, I installed RHEL 5.2 in a virtual machine,
installed all the patches, and installed Snare. After several hours of
going at it, auditd has not moved up the chart at all.
So, it appears to be an auditd/Snare interaction on RHEL 5.3. I have a
RHEL 5.3 system on which I'm experimenting with auditd, rulesets and
prelude (no Snare). The ruleset has a lot of rules in it by comparison
(> 200) and everybody is well-behaved.
Any thoughts on what may be happening?
Best Regards,
Gary Smith