Problem with auditd/SnareLinux on RHEL 5.3 - auditd glomming memory
by Smith, Gary R
Hello All,
I have some systems that have just been updated to from RHEL 5.2 to RHEL
5.3. The version of auditd is 1.7.7 and SnareLinux is 1.5.0.
Some time after the update ran, I noticed that the amount of free memory
on the systems had dramatically gone down. Running top, I saw that
auditd had sucked up lots of memory:
top - 09:25:59 up 2 days, 2 min, 1 user, load average: 0.00, 0.03,
0.00
Tasks: 111 total, 1 running, 110 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.1%si,
0.0%st
Mem: 4043368k total, 3994488k used, 48880k free, 64656k buffers
Swap: 8385920k total, 352k used, 8385568k free, 449884k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31136 root 12 -3 6568m 3.1g 568 S 0.0 81.0 0:08.84 auditd
3861 root 15 0 26152 16m 364 S 0.0 0.4 0:00.01 restorecond
29474 root 16 0 97.9m 14m 2852 S 0.0 0.4 0:00.16 sshd
29382 root 17 0 124m 13m 1464 S 0.0 0.3 0:00.09 crond
29478 gsmith 15 0 97.9m 12m 976 S 0.0 0.3 0:00.01 sshd
5611 root 15 0 58912 9048 4208 S 0.0 0.2 3:50.38 dsmc
3937 root 15 0 96556 7588 916 S 0.0 0.2 3:54.00 sec.pl
When I first started looking at the problem, I figured the easiest fix
was reboot since a new version of auditd was installed along with a new
kernel and there was a mismatch between the auditd daemon and the
running kernel. After rebooting, I watched a 'top' display and did not
see the auditd daemon percolate up. It turns out that it does percolate
up to the top of the memory usage; I didn't watch it long enough.
I wondered if there was a difference between 32 bit and 64 bit systems.
As it turns out, there isn't one; it's seems to be slower to happen on
32 bit systems probably due to slower CPU's
Next, I wondered if this is due to the version of Snare running on the
systems. Just to be sure, I rebuilt Snare for the 64 bit system,
de-installed Snare, installed the newly rebuilt Snare, and started
auditd. Again, the auditd daemon started to percolate to the top of the
memory usage.
So, does this have anything to do with the number of rules that auditd
has to deal with? I started trimming back the rules one at a time,
restarting auditd and watching a top display. The number of rules
doesn't have any affect until you get to zero rules and then auditd
doesn't move up.
Next, I ran a valgrind on auditd; one on a RHEL 5.3 system with no rules
and another one with the standard set of rules put on our systems.
criticality=0 event=execve uid=*,root
criticality=1 event=(mount,umount,umount2,settimeofday,swapon,swapoff,
reboot,setdomainname,create_module,delete_module,quotactl)
uid=*,root
criticality=2 event=execve exe=*passwd*
criticality=3 event=(login_auth,login_start,logout)
criticality=4 event=acct_mgmt
After 2 hours of running the former had 0 malloc/free problems with the
leak summary. The latter on the other hand:
==30021==
==30021== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 1)
==30021== malloc/free: in use at exit: 147,024,501 bytes in 16,262
blocks.
==30021== malloc/free: 27,612 allocs, 11,350 frees, 249,669,818 bytes
allocated.
==30021== For counts of detected errors, rerun with: -v ==30021==
searching for pointers to 16,262 not-freed blocks.
==30021== checked 10,651,128 bytes.
==30021==
==30021== LEAK SUMMARY:
==30021== definitely lost: 146,976,704 bytes in 16,246 blocks.
==30021== possibly lost: 36,464 bytes in 5 blocks.
==30021== still reachable: 11,333 bytes in 11 blocks.
==30021== suppressed: 0 bytes in 0 blocks.
Next, I ran a strace of auditd to see if I could spot where it was doing
all the malloc's. Starting up a strace of fresh instance of auditd,
eventually auditd settled into this loop:
clock_gettime(CLOCK_MONOTONIC, {432334, 891462998}) = 0
epoll_wait(0, {{EPOLLIN, {u32=3, u64=3}}}, 64, 59743) = 1
clock_gettime(CLOCK_MONOTONIC, {432334, 891681998}) = 0
recvfrom(3, "&\0\0\0\33\5\0\0\0\0\0\0\0\0\0\0audit(1233595754"..., 8988,
MSG_DONTWAIT, {sa_family=AF_NETLINK, pid=0, groups=00000000}, [12]) = 54
writev(5, [{"\0\0\0\0\20\0\0\0\33\5\0\0&\0\0\0", 16},
{"audit(1233595754.282:836040): c"..., 38}], 2) = 54
And so it goes. Nothing leaps out at me from this tight code sequence
that it's glomming memory.
I wondered if this odd behavior was exhibited back in RHEL 5.2 and we
just never noticed. So, I installed RHEL 5.2 in a virtual machine,
installed all the patches, and installed Snare. After several hours of
going at it, auditd has not moved up the chart at all.
So, it appears to be an auditd/Snare interaction on RHEL 5.3. I have a
RHEL 5.3 system on which I'm experimenting with auditd, rulesets and
prelude (no Snare). The ruleset has a lot of rules in it by comparison
(> 200) and everybody is well-behaved.
Any thoughts on what may be happening?
Best Regards,
Gary Smith
16 years
[Patch] Fix the bug of "exec /path-to-script" in audisp-remote.conf cann't be recognized
by Chu Li
Hi Steve,
I set "exec /path-to-script" in /etc/audisp/audisp-remote.conf for
"network_failure_action". the syslog said that it's an invalid option.
It seems a bug in audisp-remote.conf.
Here is my patch for the latest codes. What's your opinion about such
modification?
Signed-off-by: Chu Li<chul(a)cn.fujitsu.com>
---
--- audisp/plugins/remote/remote-config.orig.c 2009-02-16 22:19:36.000000000
+0800
+++ audisp/plugins/remote/remote-config.c 2009-02-16 22:54:25.000000000 +0800
@@ -118,13 +118,13 @@ static const struct kw_pair keywords[] =
{"krb5_principal", krb5_principal_parser, 0 },
{"krb5_client_name", krb5_client_name_parser, 0 },
{"krb5_key_file", krb5_key_file_parser, 0 },
- {"network_failure_action", network_failure_action_parser, 0 },
- {"disk_low_action", disk_low_action_parser, 0 },
- {"disk_full_action", disk_full_action_parser, 0 },
- {"disk_error_action", disk_error_action_parser, 0 },
- {"remote_ending_action", remote_ending_action_parser, 0 },
- {"generic_error_action", generic_error_action_parser, 0 },
- {"generic_warning_action", generic_warning_action_parser, 0 },
+ {"network_failure_action", network_failure_action_parser, 1 },
+ {"disk_low_action", disk_low_action_parser, 1 },
+ {"disk_full_action", disk_full_action_parser, 1 },
+ {"disk_error_action", disk_error_action_parser, 1 },
+ {"remote_ending_action", remote_ending_action_parser, 1 },
+ {"generic_error_action", generic_error_action_parser, 1 },
+ {"generic_warning_action", generic_warning_action_parser, 1 },
{ NULL, NULL, 0 }
};
@@ -541,17 +541,13 @@ static int action_parser(struct nv_pair
int i;
for (i=0; fail_action_words[i].name != NULL; i++) {
if (strcasecmp(nv->value, fail_action_words[i].name) == 0) {
- *actp = fail_action_words[i].option;
- return 0;
- } else if (i == FA_EXEC) {
- if (strncasecmp(fail_action_words[i].name,
- nv->value, 4) == 0) {
+ if (i == FA_EXEC) {
if (check_exe_name(nv->option, line))
return 1;
*exep = strdup(nv->option);
- *actp = FA_EXEC;
- return 0;
}
+ *actp = fail_action_words[i].option;
+ return 0;
}
}
syslog(LOG_ERR, "Option %s not found - line %d", nv->value, line);
Regards
Chu Li
16 years
audit-viewer
by Dan Gruhn
Seeing the recommendation of audit-viewer I downloaded it to give it a try.
I have audit-viewer-0.4 and get the following error from "make install"
Byte-compiling python modules...
client.py dialog_base.py event_dialog.py event_source.py filters.py
format_versions.py list_properties.py list_tab.py File
"/usr/local/share/audit-viewer/list_tab.py", line 558
store_data[column + 1] = l.pop(0) if l else ''
^
SyntaxError: invalid syntax
Is it just me or should I try the 0.3 version?
Dan
16 years
Audit Prelude Logout Tracking
by Dan Gruhn
I''m working on an X86_64 RHEL 5.2 system and for NISPOM Chapt. 8 I'm
looking to modify the audisp-prelude plugin so that I can get logout
events displayed.
I see the information in the audit.log as USER_END and have done a small
mod in the handle_event routine in audisp-prelude.c so that it looks for
AUDIT_USER_END but I've run across the following things:
1) sshd goes through a login/logout cycle ending in USER_END and all is
good.
node=node01 type=USER_END msg=audit(1234979707.894:203): user pid=7422
uid=0 auid=0 subj=system_u:system_r:unconfined_t:s0-s0:c0.c1023
msg='PAM: session close acct="root" : exe="/usr/sbin/sshd"
(hostname=master, addr=10.1.4.100, terminal=ssh res=success)'
2) gdm-binary goes through the same login/logout cycle, but on the
USER_END audit message it is missing some information, in particular the
source hostname:
node=master type=USER_END msg=audit(1234988646.589:364): user pid=6868
uid=0 auid=0 subj=system_u:system_r:xdm_t:s0-s0:c0.c1023 msg='PAM:
session close acct="root" : exe="/usr/sbin/gdm-binary" (hostname=?,
addr=?, terminal=:0 res=success)'
3) When crond runs, it goes through a similar cycle (but without the
USER_LOGIN step) ending with USER_END
node=master type=USER_END msg=audit(1234989001.710:371): user pid=9517
uid=0 auid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='PAM:
session close acct="root" : exe="/usr/sbin/crond" (hostname=?, addr=?,
terminal=cron res=success)'
I want to ignore the crond operations and be able to fill in the
information from gdm-binary. Has any one done this prelude logout
tracking before or have any ideas how I can proceed.
As always, a pointer to more information is quite acceptable.
Dan
16 years
Exclusion with recursive watch
by Ameel Kamboh
I am currently using audit-1.6.5-9.el5.
I have watch rules that look for modifications to directory content,
example:
-w /etc -p aw
I would like to add an exception not to watch "/etc/mydir".
I know that audit 1.6 will watch /etc and all subdirs within that.
Is there a way I can add this exception?
Thanks
Ameel
16 years
auditctl -1 User ID
by Dan Gruhn
I'm getting an auditctl startup SELinux violation that is showing up
with a user ID of -1 (4294967295 in my case). I can fix the violation,
but before I do I thought I saw something a while back about setting a
parameter or defining a variable on power-up so that one didn't get the
-1 for something that came up in the wrong order.
I've search blog.gmane.org/gmane.linux.redhat.security.audit and can't
seem to find it. Am I all wet or can someone help me with this?
Thanks,
Dan
16 years
Re: Remote audit clients on RHEL 5.2
by Dan Gruhn
I talked with the folks on the fedora-selinux-list and here is the
result of that discussion:
The text file (auditd.te) for the proper SELinux policy module is as
follows:
-----------
module auditd 0.0.3;
require {
class tcp_socket accept;
type auditd_t;
attribute reserved_port_type;
class tcp_socket { name_bind };
}
type audit_port_t;
typeattribute audit_port_t reserved_port_type;
allow auditd_t audit_port_t:tcp_socket { name_bind };
allow auditd_t self:tcp_socket accept;
---------------
Once you have the auditd.te file, you can compile it and check it for
any errors:
checkmodule -M -m -o auditd.mod auditd.te
If you have no errors, you can then package the .mod file:
semodule_package -o auditd.pp -m auditd.pp
After packaging, insert it into SELinux:
semodule -i auditd.pp
You should now be able to find the policy module using the
system-config-selinux GUI.
After all of that, the port can be enable under SELinux as per the RHEL
5.3 release notes:
semanage port -a -t audit_port_t -p tcp 60
My systems are now running clean. Thanks for the help.
Dan
Steve Grubb wrote:
> On Thursday 12 February 2009 12:48:47 pm Dan Gruhn wrote:
>
>> My system is a stand-alone in a secure environment so I can't just run a
>> piece of software and get an update, and it is currently locked into 5.2
>> as we're working to get it approved by various powers. Is there any way
>> to get the SE Linux policy from the 5.3 update as a separate piece?
>>
>
> I was hoping Dan Walsh would answer...its possible, but I don't know if the
> selinux people pull it with a bunch of other changes into the reference
> policy or not. You might be able to just get the 5.3 policy and look for the
> audit files and transplant them into 5.2 policy and diff against original 52
> policy to make a patch. You might need to ask on the Fedora-selinux mail list
> or the NSA selinux policy mail list if no one answers soon.
>
> -Steve
>
--
Dan Gruhn
Group W Inc.
8315 Lee Hwy, Suite 303
Fairfax, VA, 22031
PH: (703) 752-5831
FX: (703) 752-5851
16 years
[patch 3/3] kernel/auditsc.c: fix warning
by akpm＠linux-foundation.org
From: Andrew Morton <akpm(a)linux-foundation.org>
kernel/auditsc.c:745: warning: 'audit_set_auditable' defined but not used
Reported-by: Rakib Mullick <rakib.mullick(a)gmail.com>
Cc: Valdis.Kletnieks(a)vt.edu
Cc: Ingo Molnar <mingo(a)elte.hu>
Cc: Al Viro <viro(a)zeniv.linux.org.uk>
Cc: Eric Paris <eparis(a)redhat.com>
Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org>
---
kernel/auditsc.c | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
diff -puN kernel/auditsc.c~kernel-auditscc-fix-warning kernel/auditsc.c
--- a/kernel/auditsc.c~kernel-auditscc-fix-warning
+++ a/kernel/auditsc.c
@@ -364,6 +364,15 @@ static int grow_tree_refs(struct audit_c
ctx->tree_count = 31;
return 1;
}
+
+static void audit_set_auditable(struct audit_context *ctx)
+{
+ if (!ctx->prio) {
+ ctx->prio = 1;
+ ctx->current_state = AUDIT_RECORD_CONTEXT;
+ }
+}
+
#endif
static void unroll_tree_refs(struct audit_context *ctx,
@@ -741,14 +750,6 @@ void audit_filter_inodes(struct task_str
rcu_read_unlock();
}
-static void audit_set_auditable(struct audit_context *ctx)
-{
- if (!ctx->prio) {
- ctx->prio = 1;
- ctx->current_state = AUDIT_RECORD_CONTEXT;
- }
-}
-
static inline struct audit_context *audit_get_context(struct task_struct *tsk,
int return_valid,
int return_code)
_
16 years, 1 month
[patch 2/3] audit: EXECVE record: remove bogus newline
by akpm＠linux-foundation.org
From: Jiri Pirko <jpirko(a)redhat.com>
EXECVE records contain a newline after every argument. auditd converts
"\n" to " " so you cannot see newlines even in raw logs, but they're there
nevertheless. If you're not using auditd, you need to work round them.
These '\n' chars are can be easily replaced by spaces when creating record
in kernel. Note there is no need for trailing '\n' in an audit record.
record before this patch:
"type=EXECVE msg=audit(1231421801.566:31): argc=4 a0=\"./test\"\na1=\"a\"\na2=\"b\"\na3=\"c\"\n"
record after this patch:
"type=EXECVE msg=audit(1231421801.566:31): argc=4 a0=\"./test\" a1=\"a\" a2=\"b\" a3=\"c\""
Signed-off-by: Jiri Pirko <jpirko(a)redhat.com>
Acked-by: Eric Paris <eparis(a)redhat.com>
Cc: Al Viro <viro(a)zeniv.linux.org.uk>
Signed-off-by: Andrew Morton <akpm(a)linux-foundation.org>
---
kernel/auditsc.c | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
diff -puN kernel/auditsc.c~audit-execve-record-remove-bogus-newline kernel/auditsc.c
--- a/kernel/auditsc.c~audit-execve-record-remove-bogus-newline
+++ a/kernel/auditsc.c
@@ -1023,7 +1023,7 @@ static int audit_log_single_execve_arg(s
{
char arg_num_len_buf[12];
const char __user *tmp_p = p;
- /* how many digits are in arg_num? 3 is the length of a=\n */
+ /* how many digits are in arg_num? 3 is the length of " a=" */
size_t arg_num_len = snprintf(arg_num_len_buf, 12, "%d", arg_num) + 3;
size_t len, len_left, to_send;
size_t max_execve_audit_len = MAX_EXECVE_AUDIT_LEN;
@@ -1109,7 +1109,7 @@ static int audit_log_single_execve_arg(s
* so we can be sure nothing was lost.
*/
if ((i == 0) && (too_long))
- audit_log_format(*ab, "a%d_len=%zu ", arg_num,
+ audit_log_format(*ab, " a%d_len=%zu", arg_num,
has_cntl ? 2*len : len);
/*
@@ -1129,7 +1129,7 @@ static int audit_log_single_execve_arg(s
buf[to_send] = '\0';
/* actually log it */
- audit_log_format(*ab, "a%d", arg_num);
+ audit_log_format(*ab, " a%d", arg_num);
if (too_long)
audit_log_format(*ab, "[%d]", i);
audit_log_format(*ab, "=");
@@ -1137,7 +1137,6 @@ static int audit_log_single_execve_arg(s
audit_log_n_hex(*ab, buf, to_send);
else
audit_log_format(*ab, "\"%s\"", buf);
- audit_log_format(*ab, "\n");
p += to_send;
len_left -= to_send;
@@ -1165,7 +1164,7 @@ static void audit_log_execve_info(struct
p = (const char __user *)axi->mm->arg_start;
- audit_log_format(*ab, "argc=%d ", axi->argc);
+ audit_log_format(*ab, "argc=%d", axi->argc);
/*
* we need some kernel buffer to hold the userspace args. Just
_
16 years, 1 month
