RE: Linux audit performance impact
Hi Steve,
Thanks for the quick reply.
Please look in-line for my replies.
Regards,
Logeswari.
-----Original Message-----
From: Steve Grubb [mailto:sgrubb@redhat.com]
Sent: Wednesday, January 28, 2015 8:46 PM
To: linux-audit(a)redhat.com
Cc: Viswanath, Logeswari P (MCOU OSTL)
Subject: Re: Linux audit performance impact
Hello,
On Wednesday, January 28, 2015 02:57:58 PM Viswanath, Logeswari P wrote:
We want to know audit performance impact on RHEL and Suse linux to
help us evaluate linux audit as data source for our host based IDS.
When we ran our own performance test with a test audispd plugin, we
found if a system can perform 200000 open/close system calls per
second without auditing, system can perform only 3000 open/close
system calls auditing is enabled for open/close system call which is a
HUGE impact on the system performance. It would be great if anyone can help us answering
the following questions.
1) Is this performance impact expected? If yes, what is the reason
behind it and can we fix it?
I'll leave this for the kernel guys to answer. That said, I think more detailed
information might be helpful.
If auditd is not started and events go to syslog, does the performance change?
To do this audit=1 on boot line and auditctl -R /etc/rules.d/your.rules
Logeswari=>System can perform 15000 open/close system calls per second which is better
than earlier results.
what rules do you have loaded?
Logeswari=> # auditctl -l
LIST_RULES: exit,always syscall=open,close
What do you get when audit is enabled and no rules loaded?
Logeswari=> Impact is there but not major.
If you have other syscall rules loaded that are not open and openat or close, does the
performance change? I suspect that if you trigger a rule, you are thrown onto the slow
path. Open is perhaps the most lengthy because of multiple auxiliary records and path
resolution. But we need data to tell.
Logeswari=> Yes, there is an major impact. I enabled write system call and this rule is
first in the set of rules along with open/close.
That said, I know that the kernel audit path changed a couple years ago so it might be
worthwhile to test against an old kernel to see if the change has affected performance.
Logeswari=> We tested with kernel 2.6.32. Should we test with old/new kernel?
-Steve
2) Have anyone done any benchmarking for performance impact? If
yes,
can you please share the numbers and also the steps/programs used the
run the same.
3) Help us validating the performance test we have done in our test
setup using the steps mentioned along with the results attached.
Attached test program (loader.c) to invoke open and close system calls.
Attached idskerndsp is the audispd plugin program.
We used time command to determine how much time the system took to
complete
50000 open/close system calls without (results attached
Without-auditing) and with auditing enabled on the system
(With-auditing-NOLOG-audispd-plugin
and With-auditing-RAW)
System details:
1 CPU machine
OS Version
RHEL 6.5
Kernel Version
uname -r
2.6.32-431.el6.x86_64
Note: auditd was occupying 35% of CPU and was sleeping for most of the
time whereas kauditd was occupying 20% of the CPU.
Thanks & Regards,
Logeswari.