Re: Linux audit performance impact

Hi Steve, Thanks for the quick reply. Please look in-line for my replies. Regards, Logeswari. -----Original Message----- From: Steve Grubb [mailto:sgrubb@redhat.com] Sent: Wednesday, January 28, 2015 8:46 PM To: linux-audit(a)redhat.com Cc: Viswanath, Logeswari P (MCOU OSTL) Subject: Re: Linux audit performance impact Hello, On Wednesday, January 28, 2015 02:57:58 PM Viswanath, Logeswari P wrote: > We want to know audit performance impact on RHEL and Suse linux to > help us evaluate linux audit as data source for our host based IDS. > When we ran our own performance test with a test audispd plugin, we > found if a system can perform 200000 open/close system calls per > second without auditing, system can perform only 3000 open/close > system calls auditing is enabled for open/close system call which is a > HUGE impact on the system performance. It would be great if anyone can help us answering the following questions. > > > 1) Is this performance impact expected? If yes, what is the reason > behind it and can we fix it? I'll leave this for the kernel guys to answer. That said, I think more detailed information might be helpful. If auditd is not started and events go to syslog, does the performance change? To do this audit=1 on boot line and auditctl -R /etc/rules.d/your.rules Logeswari=>System can perform 15000 open/close system calls per second which is better than earlier results. what rules do you have loaded? Logeswari=> # auditctl -l LIST_RULES: exit,always syscall=open,close What do you get when audit is enabled and no rules loaded? Logeswari=> Impact is there but not major. If you have other syscall rules loaded that are not open and openat or close, does the performance change? I suspect that if you trigger a rule, you are thrown onto the slow path. Open is perhaps the most lengthy because of multiple auxiliary records and path resolution. But we need data to tell. Logeswari=> Yes, there is an major impact. I enabled write system call and this rule is first in the set of rules along with open/close. That said, I know that the kernel audit path changed a couple years ago so it might be worthwhile to test against an old kernel to see if the change has affected performance. Logeswari=> We tested with kernel 2.6.32. Should we test with old/new kernel? -Steve > 2) Have anyone done any benchmarking for performance impact? If yes, > can you please share the numbers and also the steps/programs used the > run the same. > > 3) Help us validating the performance test we have done in our test > setup using the steps mentioned along with the results attached. > > Attached test program (loader.c) to invoke open and close system calls. > Attached idskerndsp is the audispd plugin program. > We used time command to determine how much time the system took to > complete > 50000 open/close system calls without (results attached > Without-auditing) and with auditing enabled on the system > (With-auditing-NOLOG-audispd-plugin > and With-auditing-RAW) > > System details: > > 1 CPU machine > > OS Version > RHEL 6.5 > > Kernel Version > uname -r > 2.6.32-431.el6.x86_64 > > Note: auditd was occupying 35% of CPU and was sleeping for most of the > time whereas kauditd was occupying 20% of the CPU. > > Thanks & Regards, > Logeswari. -- Linux-audit mailing list Linux-audit(a)redhat.com https://www.redhat.com/mailman/listinfo/linux-audit