Hi, we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74 CPUs and >= 400G RAM. When the system is busy with large SAP jobs, it goes onto its knees with cpu %system up to 80%, thus making the SAP jobs run twice as long. As soon as you stop auditd everything returns to normal... Facts: RHEL6 instances on RHEL7 hosts. the rule set (see below) runs fine on any other system with less cpus (<64, maybe this is the cut off?). We have smaller systems with this rule set that rotate the audit file nearly every minute without any noticable performance hit, these SAP systems rotate once every 20-24hours....

Agree with Steve's suggestion re: "-S all". Also might help if you sort your rules to put all the ones with '-F auid>=400' below a single line rule like this: -a never,exit -F auid<400 and remove the '-F auid>=400' from all of the rules below it. Like so: -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F key=USER_EXEC -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=4294967295 -F key=USER_EXEC -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid>=5000000 -F auid!=4294967295 -F key=S3DATA -a never,exit -F auid<400 -a always,exit -F path=/etc/environment -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/login.defs -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/rsyslog.conf -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/ssh/sshd_config -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/cron.allow -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/cron.deny -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/cron.d -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/cron.daily -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/cron.hourly -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/cron.monthly -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/cron.weekly -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/aliases -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/alternatives -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/at.allow -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/at.deny -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/audisp/plugins.d/syslog.conf -F perm=wa -F key=CRIT_AUDIT -a always,exit -F path=/etc/audisp/audispd.conf -F perm=wa -F key=CRIT_AUDIT -a always,exit -F path=/etc/audit/auditd.conf -F perm=wa -F key=CRIT_AUDIT -a always,exit -F path=/etc/bashrc -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/crontab -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/shells -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/default -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/depmod.conf -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/etc/depmod.d -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/exports -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/group -F perm=wa -F key=USER_MGMT -a always,exit -F path=/etc/passwd -F perm=wa -F key=USER_MGMT -a always,exit -F path=/etc/shadow -F perm=wa -F key=USER_MGMT -a always,exit -F path=/etc/inittab -F perm=wa -F key=CRIT_CONF -a always,exit -F dir=/bin -F perm=wa -F key=CRIT_PROG -a always,exit -F dir=/sbin -F perm=wa -F key=CRIT_PROG -a always,exit -F dir=/usr/bin -F perm=wa -F key=CRIT_PROG -a always,exit -F dir=/usr/sbin -F perm=wa -F key=CRIT_PROG -a always,exit -F dir=/etc/init.d -F perm=wa -F key=CRIT_PROG -a always,exit -F path=/etc/nsswitch.conf -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/ldap.conf -F perm=wa -F key=USER_MGMT -a always,exit -F path=/etc/sssd/sssd.conf -F perm=wa -F key=USER_MGMT -a always,exit -F dir=/var/spool/cron -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/var/spool/atjobs -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/usr/bin/sudo -F perm=x -F key=USER_MGMT -a always,exit -F path=/etc/sudoers -F perm=wa -F key=USER_MGMT -a always,exit -F dir=/etc/sudoers.d -F perm=wa -F key=USER_MGMT -a always,exit -F dir=/etc/pam.d -F perm=wa -F key=CRIT_PAM -a always,exit -F dir=/etc/security -F perm=wa -F key=CRIT_CONF -a always,exit -F path=/etc/libaudit.conf -F perm=wa -F key=CRIT_AUDIT -a always,exit -F path=/etc/init.d/auditd -F perm=wa -F key=CRIT_AUDIT -a always,exit -F dir=/appdata/daten/S3_audit -F perm=rwa -F auid<10000 -F auid!=4294967295 -F key=S3DATA On Fri, May 19, 2017 at 4:52 PM Klaus Lichtenwalder <klic(a)mnet-online.de&gt; wrote: > Hi, > > we have a few SAP systems on RHEV (so virtualized on KVM) with >= 74 > CPUs and >= 400G RAM. > When the system is busy with large SAP jobs, it goes onto its knees with > cpu %system up to 80%, thus making the SAP jobs run twice as long. As > soon as you stop auditd everything returns to normal... > > Facts: > RHEL6 instances on RHEL7 hosts. > the rule set (see below) runs fine on any other system with less cpus > (<64, maybe this is the cut off?). We have smaller systems with this > rule set that rotate the audit file nearly every minute without any > noticable performance hit, these SAP systems rotate once every > 20-24hours.... > > Anyone has an idea? > > Here's an excerpt from "perf top": > with auditd running: > > > Samples: 28M of event 'cpu-clock', Event count (approx.): 236747914918 > > Overhead Shared Object Symbol > > 23.13% [kernel] [k] get_task_cred > > 10.05% [kernel] [k] audit_filter_rules > > 4.21% [kernel] [k] _spin_unlock_irqrestore > > 3.30% libdb2e.so.1 [.] sqlbfix > > 2.92% [kernel] [k] finish_task_switch > > 1.69% disp+work [.] rrol_in > > 1.69% disp+work [.] rrol_out > > 0.98% [kernel] [k] run_timer_softirq > > 0.96% [kernel] [k] rcu_process_gp_end > > > > auditd stopped: > > > Samples: 3M of event 'cpu-clock', Event count (approx.): 526535382557 > > Overhead Shared Object Symbol > > 2.41% disp+work [.] memcmpU16 > > 2.32% disp+work [.] MmxMalloc2 > > 2.25% disp+work [.] ab_Rudi > > 2.07% disp+work [.] rrol_out > > 1.98% disp+work [.] rrol_in > > 1.95% disp+work [.] ab_CompByCmpCntx > > 1.88% libdb2e.so.1 [.] sqlbfix > > 1.73% disp+work [.] MmxFree2 > > 1.62% [kernel] [k] run_timer_softirq > > 1.56% [kernel] [k] __do_softirq > > 1.39% disp+work [.] ab_InitRcDecompress > > > > These are the audit rules: > > auditctl -l > > -a always,exit -S all -F path=/etc/environment -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F path=/etc/login.defs -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/rsyslog.conf -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F path=/etc/ssh/sshd_config -F perm=wa -F > auid>=400 -F key=CRIT_CONF > > -a always,exit -S all -F path=/etc/cron.allow -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/cron.deny -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/cron.d -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/cron.daily -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/cron.hourly -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/cron.monthly -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/cron.weekly -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/aliases -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/alternatives -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F path=/etc/at.allow -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/at.deny -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/audisp/plugins.d/syslog.conf -F > perm=wa -F auid>=400 -F key=CRIT_AUDIT > > -a always,exit -S all -F path=/etc/audisp/audispd.conf -F perm=wa -F > auid>=400 -F key=CRIT_AUDIT > > -a always,exit -S all -F path=/etc/audit/auditd.conf -F perm=wa -F > auid>=400 -F key=CRIT_AUDIT > > -a always,exit -S all -F path=/etc/bashrc -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/crontab -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/shells -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/default -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/depmod.conf -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F dir=/etc/depmod.d -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/exports -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/group -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F path=/etc/passwd -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F path=/etc/shadow -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F path=/etc/inittab -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F dir=/bin -F perm=wa -F auid>=400 -F > key=CRIT_PROG > > -a always,exit -S all -F dir=/sbin -F perm=wa -F auid>=400 -F > key=CRIT_PROG > > -a always,exit -S all -F dir=/usr/bin -F perm=wa -F auid>=400 -F > key=CRIT_PROG > > -a always,exit -S all -F dir=/usr/sbin -F perm=wa -F auid>=400 -F > key=CRIT_PROG > > -a always,exit -S all -F dir=/etc/init.d -F perm=wa -F auid>=400 -F > key=CRIT_PROG > > -a always,exit -S all -F path=/etc/nsswitch.conf -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F path=/etc/ldap.conf -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F path=/etc/sssd/sssd.conf -F perm=wa -F > auid>=400 -F key=USER_MGMT > > -a always,exit -S all -F dir=/var/spool/cron -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/var/spool/atjobs -F perm=wa -F auid>=400 > -F key=CRIT_CONF > > -a always,exit -S all -F path=/usr/bin/sudo -F perm=x -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F path=/etc/sudoers -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -S all -F dir=/etc/sudoers.d -F perm=wa -F auid>=400 -F > key=USER_MGMT > > -a always,exit -F arch=b64 -S execve -F auid>=500 -F auid<10000 -F > key=USER_EXEC > > -a always,exit -F arch=b64 -S execve -F auid>=5000000 -F auid!=-1 -F > key=USER_EXEC > > -a always,exit -S all -F dir=/etc/pam.d -F perm=wa -F auid>=400 -F > key=CRIT_PAM > > -a always,exit -S all -F dir=/etc/security -F perm=wa -F auid>=400 -F > key=CRIT_CONF > > -a always,exit -S all -F path=/etc/libaudit.conf -F perm=wa -F auid>=400 > -F key=CRIT_AUDIT > > -a always,exit -S all -F path=/etc/init.d/auditd -F perm=wa -F auid>=400 > -F key=CRIT_AUDIT > > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F > auid>=400 -F auid<10000 -F auid!=-1 -F key=S3DATA > > -a always,exit -S all -F dir=/appdata/daten/S3_audit -F perm=rwa -F > auid>=5000000 -F auid!=-1 -F key=S3DATA > > > > -- > ------------------------------------------------------------------------ > Klaus Lichtenwalder, Dipl. Inform., http://www.lichtenwalder.name/ > PGP Key fingerprint: 3AE6 044D 1161 1ABF AC2D 23B3 4C15 7232 FDCA 0980 > > -- > Linux-audit mailing list > Linux-audit(a)redhat.com > https://www.redhat.com/mailman/listinfo/linux-audit >

Am 19. Mai 2017 23:41:58 MESZ schrieb Stephen Buchanan <stephenwb(a)gmail.com&gt;: >Agree with Steve's suggestion re: "-S all". Also might help if you >sort I now know where -S all stems from... Some watches add a -S all by themselves... Probably created an audit.rules file by textually working from there and duplicating rules

>your rules to put all the ones with '-F auid>=400' below a single >line rule >like this: >-a never,exit -F auid<400 > >and remove the '-F auid>=400' from all of the rules below it. > ... I did this, and verified it, but there was absolutely no difference to unsorted rules having​ -S all also specified Still cpu %system up to 50% and run time of jobs 100% longer. This was on a vm with 72 cpus Klaus

>> your rules to put all the ones with '-F auid>=400' below a single >> line rule >> like this: >> -a never,exit -F auid<400 >> >> and remove the '-F auid>=400' from all of the rules below it. >> > ... > > I did this, and verified it, but there was absolutely no difference > to unsorted rules having​ -S all also specified > > Still cpu %system up to 50% and run time of jobs 100% longer. > This was on a vm with 72 cpus >