Hello, Over the last day or two, I re-worked the user space audit code to be able to control the new file system audit subsystem. As I was doing the work, I became concerned about the performance impact since it appears to be using the syscall exit filter. The syscall exit filter (and entry filter) is expensive to use except in cases where you need to use it. This is because each rule in it must be examined during each syscall to see if the current syscall is of interest. The current lspp configuration has 10 syscall audit rules. I became curious what the measured impact would be with the current file system audit implementation. I decide to run the same performance test that I tested the audit system with a couple weeks ago when inode and IPC problems were noticed. I used the lspp.16 kernel with profile=2 boot param. The following table shows the results: rules seconds 0 49 10 56 25 75 50 115 75 143 90 185

0 rules had this for function usage: 1284 __d_lookup 4.7380 1170 __link_path_walk 0.3098 1065 avc_has_perm_noaudit 1.2144 706 _atomic_dec_and_lock 8.4048 612 do_path_lookup 0.8204 561 dput 1.2986 509 _raw_spin_lock 2.1477 10 rules had this: 1295 __d_lookup 4.7786 1089 audit_filter_syscall 6.3684 1081 __link_path_walk 0.2862 889 avc_has_perm_noaudit 1.0137 676 audit_getname 2.6000 658 do_path_lookup 0.8820 596 _atomic_dec_and_lock 7.0952 25 rules had this: 3193 audit_filter_rules 3.0009 2178 audit_filter_syscall 12.7368 1280 __d_lookup 4.7232 1131 __link_path_walk 0.2994 956 avc_has_perm_noaudit 1.0901 652 _atomic_dec_and_lock 7.7619 530 dput 1.2269 50 rules had this: 11213 audit_filter_rules 10.5385 4654 audit_filter_syscall 27.2164 4100 selinux_task_ctxid 141.3793 1212 __d_lookup 4.4723 1103 __link_path_walk 0.2920 1012 avc_has_perm_noaudit 1.1539 788 _atomic_dec_and_lock 9.3810 75 had this: 15351 audit_filter_rules 14.4276 6032 audit_filter_syscall 35.2749 2066 selinux_task_ctxid 71.2414 1237 __d_lookup 4.5646 1184 __link_path_walk 0.3135 1014 avc_has_perm_noaudit 1.1562 592 _atomic_dec_and_lock 7.0476 and 90 rules had this: 18287 audit_filter_rules 17.1870 9173 audit_filter_syscall 53.6433 4346 selinux_task_ctxid 149.8621 1314 __link_path_walk 0.3479 1218 __d_lookup 4.4945 1070 avc_has_perm_noaudit 1.2201 682 _atomic_dec_and_lock 8.1190 As you can see, the audit_filter_rules and audit_filter_syscall overwhelmed the profile quickly. It would not be unreasonable for a system to have 40 watches. The lspp rules have 56 of them. With 10 syscall rules added, the performance of a correctly configured lspp machine will be similar to the 75 rules test. This represents a 186% performance hit compared to no audit rules. I do not believe optimizing the audit_filter_rules function will solve the problem. I think the file system audit algorithm needs to be re-thought. It simply cannot penalize every syscall. There are several ways to solve the problem. Maybe what we need to do is use the watch list to store watches on and add a new field to the context. If a watch is triggered it sets the flag in the context. When syscall exit is done, it checks the flag and if set, does both the watch list and the exit list. Otherwise, it skips the watch list. I don't know if this is feasible, or a preferred solution, but we need to start looking at how to decouple the exit list and watches. -Steve -- Linux-audit mailing list Linux-audit(a)redhat.com https://www.redhat.com/mailman/listinfo/linux-audit

On Monday 10 April 2006 23:51, Amy Griffis wrote: > 1) what audit rules did you use? I used the lspp rules to get the 1st 10, and the rest were against files in /etc/test.

-w /etc/sysconfig/console -w /etc/sysconfig/pm -w /etc/sysconfig/system-config-users -w /etc/sysconfig/init -w /etc/sysconfig/hwconf -w /etc/sysconfig/netdump -w /etc/sysconfig/selinux -w /etc/sysconfig/hsqldb -w /etc/sysconfig/system-config-securitylevel -w /etc/sysconfig/mouse -w /etc/sysconfig/saslauthd.rpmnew -w /etc/sysconfig/netdump_id_dsa.pub -w /etc/sysconfig/clock -w /etc/sysconfig/grub -w /etc/sysconfig/wpa_supplicant ...

-a entry,possible -S chmod -S fchmod -S chown -S fchown -S lchown -a entry,possible -S creat -S open -S truncate -S ftruncate -a entry,possible -S mkdir -S rmdir -a entry,possible -S unlink -S rename -S link -S symlink

-a entry,always -S setxattr -S lsetxattr -S fsetxattr -S removexattr -S lremovexattr -S fremovexattr -a entry,always -S mknod -a entry,always -S mount -a entry,always -S adjtimex -S settimeofday

> 2) what system call(s) did you measure? access("/usr/include", 0); The watch rules were never triggered because I wanted to measure the overhead where no audit events occur. The syscall exercises the file system without doing any IO, which would complicate things, too. -Steve

I also don't like the idea of handling this by all those syscalls or using "all" because user space tools could get out of sync with the kernel. On any kernel upgrade, there could be a new syscall that allows file system access. The user space tools wouldn't know about it and wouldn't provide automatic coverage.

Steve Grubb wrote: [Tue Apr 11 2006, 05:01:23PM EDT] > On Tuesday 11 April 2006 12:11, Amy Griffis wrote: > > -a exit,always -S chmod -S fchmod -S chown -S fchown -S lchown > > -S creat -S open -S truncate -S ftruncate -S mkdir -S rmdir -S unlink > > -S rename -S link -S symlink -F watch=/etc/sysconfig/console > > > > Now you don't have any rules for access(), so using it as the test > > case is much more interesting. > > OK, I re-worked auditctl to use these syscalls instead of "all". I then re-ran > the tests on the same kernel as I was testing on since lspp.17 has slab debug > stuff turned on again. > > rules seconds loss > 0 50 0% > 10 52 4% > 25 56 12% > 50 69 38% > 75 81 62% > 90 87 74% Hmm, that's interesting, thanks. > The 75 rule performance hit is now 62%. So there is some improvement in > performance. RHEL4 has a 6% hit for 90 rules. Do you mean 10 rules + 80 watches? Or 90 rules? > We've narrowed the difference, but I don't consider this solved. I think there are three syscall groups for which we want to consider the performance impact of having a large number of rules. 1. syscalls for which there are no rules In most use cases, the majority of syscalls will fall into this group. That makes this group the most important because it likely has the most effect on general system performance. Based on your numbers above, it looks like the impact here is unacceptable. For this group, there should be no filtering overhead at all. To achieve this, we must have some indication per-list whether there are any rules for a given syscall. If there are no rules for the syscall, don't even walk the list. 2. syscalls for which there are a small number of rules For this group we must walk the list, and some filtering overhead is acceptable. How much overhead is acceptable is a factor of how many syscalls we would typically expect to be in this group, and how often we would expect those syscalls to be used. If we need to optimize for this case, we have a couple of options. a) Provide features which reduce the number of rules for a heavy offender. E.g. for filesystem auditing, - allow multiple watches per rule, akin to multiple inodes per rule - allow a single watch on a directory to apply to many files b) Separate the rules for a heavy offender, e.g. by putting them in a separate list 3. syscalls for which there are a large number of rules For this group, the filtering overhead is the most significant and optimization is more difficult. For some use cases, having a rules tree instead of a rules list might help. For filesystem auditing, when you want to audit a large number of inodes or watches, being able to audit an entire sub-tree with a single rule would help that particular use case. However, if you want to audit specific inodes/watches that are spread throughout a filesystem, the syscall-exit-based filtering is always going to exact a penalty. The only way I can think of to mitigate this is to hang the rule data off of the inodes themselves, and receive a callback for each filesystem event. You can do the filtering from the callback, and eliminate the list traversal on syscall exit. This is basically the RHEL4 implementation, and there are a few reasons why we can't do it this way right now. The first reason is that inotify is doing something similar, and we need to attempt to consolidate similar pieces of kernel functionality. Inotify, however, does not support all of the filesystem events we care about auditing. Additionally for those events that inotify does support, its hooks are placed in such a way that events are not produced for failed operations (which we care about). To use this type of implementation in audit, we must be able to either significantly extend inotify, or justify our need for having our own implementation to kernel.org. I think the former is more preferable than the latter. > I also don't like the idea of handling this by all those syscalls Yes, it makes the rules long and ugly. Auditctl could support keywords on the command line that map to various groups of system calls. That would be more user-friendly. > or using "all" because user space tools could get out of sync with > the kernel. On any kernel upgrade, there could be a new syscall that > allows file system access. The user space tools wouldn't know about > it and wouldn't provide automatic coverage. True, we would have to keep an eye on new syscalls. Hope this helps, Amy

On Mon, Apr 17, 2006 at 10:27:34AM -0500, Timothy R. Chavez wrote: > Maybe this is a completely stupid thought, but what about the option of > adding a per-syscall filter list table, indexed by system-call number. That's how LAuS worked... You'd need to support multiple lists to handle multiple personalities (ie 32bit code running on x86_64). The amount of space used isn't too bad; it would also be possible to use reference counting to share entries for identical rules.

Al, proposed a different solution. You might want to check with him for details. It was discussed at the Monday Telecon.

On Fri, Apr 21, 2006 at 11:10:21AM -0400, Linda Knippers wrote: > > > Al, proposed a different solution. You might want to check with him for > > details. It was discussed at the Monday Telecon. > > Maybe Al could post something? With the buzz on the phone line some > of the discussion was hard to follow. Basically, add 3 families of rule lists. Rule that has one AUDIT_INODE or AUDIT_WATCH field and would currently sit in audit_filter_list[n] would be moved to audit_filter_list[AUDIT_NR_FILTERS + n * 31 + ino % 31] where ino is inode number from the AUDIT_INODE/AUDIT_WATCH field of that rule. Everything else would remain where it is now. If ->ino changes during the lifetime, rule would have to be moved between these lists. When we are trying to match context with rules on (current) list #n, we _know_ that many of them won't match just on the grounds of ->ino mismatch. With that splitting of lists we can skip most of those - rules from the current list #n will be on list #n and 31 lists starting with AUDIR_NR_FILTERS + 31*n. We only need to scan n (that's where non-watch rules remain) AUDIT_NR_FILTERS + 31*n + ctx->names[i].ino % 31 for each i less than ctx->name_count. Everything else is not going to match and doesn't have to be looked at.

On Tuesday 11 April 2006 17:01, Steve Grubb wrote: > OK, I re-worked auditctl to use these syscalls instead of "all". I then > re-ran the tests on the same kernel as I was testing on since lspp.17 has > slab debug stuff turned on again. > > rules ?seconds ? ?loss > 0 ? ? ? ?50 ? ? ? ? ? ?0% > 10 ? ? ?52 ? ? ? ? ? ?4% > 25 ? ? ?56 ? ? ? ? ? ?12% > 50 ? ? ?69 ? ? ? ? ? ?38% > 75 ? ? ?81 ? ? ? ? ? ?62% > 90 ? ? ?87 ? ? ? ? ? ?74% I re-ran the analysis with the lspp.24 kernel. This is the results: rules seconds loss 0 57.4 0.0% 10 57.8 0.7% 25 56.7 +1.2% 50 58.6 2.1% 75 59.7 4.0% 90 59.1 3.0% The results look good for this test case. Thanks to everyone that helped solve this problem! Good job.

I think do_path_lookup & _raw_spin_lock jump out as the biggest changes.