On Tuesday, October 07, 2014 03:39:51 PM Richard Guy Briggs wrote: A bit of an aside from the patch, but in my opinion the parser should be made a bit more robust so that it can handle fields in any particular order. I agree that having fields in a "canonical ordering" is helpful, both for tools and people, but the tools shouldn't require it in my opinion. Steve, why exactly can't the userspace parser handle fields in any order? How difficult would it be to fix? -- paul moore security and virtualization @ redhat

On Saturday, October 11, 2014 11:42:06 AM Steve Grubb wrote: I understand you are catering to the "power user" here, but I don't see that as an excuse for not being able to parse well formed name/value audit record string if the order isn't exactly what you expect. I believe this will only become more and more of a problem as things move forward. I think this is something we need to fix soon. Steve, would you be willing to fix the audit userspace parser so it can handle fields in an arbitrary order? If not, would you be willing to accept patches for the userspace that would accomplish this? -- paul moore security and virtualization @ redhat

On Tuesday, October 21, 2014 05:08:22 PM Richard Guy Briggs wrote: gid is important for things that might allow access by file permissions. But the syscall logging is going to have that and much more. In this case, access is granted by having a posix capability. So, all we want is what's the process, who's the user, which session/tty is this coming from to find all events that might be related. That will cause problems at this point. Its hard to say if that is sufficient for all cases. When access is granted by posix capability, sure. This is probably good for most kernel originating events. But there are times extra info is needed. I don't understand what you are getting at. -Steve

On Tuesday, October 21, 2014 03:56:10 PM Steve Grubb wrote: This is getting back to my earlier concerns/questions about field ordering, or at the very least I'm going to hijack this conversation and steer it towards field ordering ;) Before we go to much farther, I'd really like us to agree that ordering is not important, can we do that? As a follow up, what do we need to do to make that happen in the userspace tools? -- paul moore security and virtualization @ redhat

[NOTE: Since we are wandering off the original topic I took the liberty of trimming the To/CC line to just the audit list folks, feel free to add back as necessary.] On Tuesday, October 21, 2014 09:24:05 PM Richard Guy Briggs wrote: Exactly. I'm starting to think that we need to resolve this issue before we introduce any new features into the kernel. No, we're already seeing that a single fixed ordering is bad, adding an alternate fixed ordering just kicks the can down the road a bit further and sets a bad precedence for future development. It is time to get rid of the fixed ordering in the audit records. -- paul moore security and virtualization @ redhat

On Tuesday, October 21, 2014 09:24:05 PM Richard Guy Briggs wrote: There really are worse problems to solve. Also, all this changing things will keep me from adding new analytical capabilities to the audit tools because I continue to have to re-do the bottom layers instead of progress towards making things better for admins. I think we can create the audit_log_task_info_lite function and use it for your new events. -Steve

On Wed, 2014-10-22 at 10:25 -0400, Steve Grubb wrote: Incorrect. The length of the message makes it perfectly clear how much data the kernel sent, and thus if that data includes the version or backlog_wait_time. I thought this had been discussed before... The answer is 'check how much data you got from the kernel'

On Wed, 2014-10-22 at 10:36 -0400, Steve Grubb wrote: completely irrelevant... The code in audit_get_reply() seems like poor practice... You aren't doing any sanity checks on the data coming back. I guess that's why userspace has so much trouble handling things. It's just completely ignore the length the kernel gave you...

On 10/22/2014 10:12 AM, Eric Paris wrote: IIUC we've got to have some assurance that the path is legit for forensics. Technically I believe I understand and concur with what you are saying Eric, but as a guy on the far end of the process I know I need to be able to reference a complete path to a FD. One which we believe did exist at the time the mod occurred. To me, sometimes isn't really good enough. But A path probably is. ... No? Thx, LCB -- LC (Lenny) Bruzenak lenny(a)magitekltd.com

On Wed, 2014-10-22 at 10:51 -0500, LC Bruzenak wrote: path. That I agree with. But imagine I clone a new mount namespace and don't share my changes with the parent namespace. Now I mount something new in that child namespace. What is A path for a file in the new mount? From the parent namespace there is NO path, ABSOLUTELY NO PATH. (guess which mount namespace auditd lives in, by the way). From the PoV of the processes in the child mount namespace there is A path, but it's possibly/probably completely meaningless to the admin. /etc/shadow != /etc/shadow the admin cares about... readlink() doesn't work in this case either. Sometimes there just plain is no path. So yeah, I'm betting MOST of the time we can come up with A path, but that's not exactly what you want either :-( Say I set a watch for failure to open /path/to/my/file. If someone comes along and says open(/path/to/my/file) but they do not have execute permissions on /path/to/ their request will be denied. Not because they didn't have permission to open /path/to/my/file, but because they didn't have permission to open /path/to. Watches do not, and can not, emit a rule for that. The rule you requested (failure to open /path/to/my/file) was not violated. The kernel did not try to open /path/to/my/file. It tried to open /path/to/ and died right there. If you care about things being unable to open /path/to/, put a watch on /path/to (although I'm not 100% such watches actually work, but at least the theory is right and maybe that could be fixed)

On Wednesday, October 22, 2014 02:18:00 PM Eric Paris wrote: All of these are privileged ops. Meaning you would have to be tracking the actions of an admin who could do all kinds of bad things to a system. Does openat allow an fd in one name space and a file name in another? Without getting into trick situations like this, how about we get it for the normal run-in-the-mill-nothing-tricky-going-on-here system? Even that would be immensely helpful. Especially since glibc has switched over to openat and newer platforms like aarch64 don't have open(2). If I have this rule -a always,exit -S openat -F exit=-EPERM The path is sitting in arg2. It not lost or inaccessible. We can't see it in current audit records because we get a pointer to the string as arg2. I would think if path resolution couldn't happen, we could get a substitute record that identifies the string passed as arg2. -Steve

On Wednesday, October 22, 2014 11:28:46 AM Paul Moore wrote: But it illustrates the point. There are tools that depend on an ordering and format. There are more programs that just ausearch that needs to be considered if the fields change. For example, Someone could do things like this: retval = auparse_find_field(au, "auid"); retval = auparse_next_field(au); retval = auparse_next_field(au); retval = auparse_find_field(au, res"); Where, if the field ordering can't be guaranteed, the code becomes: retval = auparse_find_field(au, "auid"); retval = auparse_first_field(au); retval = auparse_find_field(au, "pid"); retval = auparse_first_field(au); retval = auparse_find_field(au, "uid"); retval = auparse_first_field(au); retval = auparse_find_field(au, res"); Which of the two is likely to be faster? Especially when doing realtime analysis as a plugin and needing to finish because another event is coming in? Just like a binary struct has to maintain an order of data members if written to disk, the sequence of fields need to be maintained in a text record. Its worked well for the 10 years I've been working on the audit code. There has to be a review cycle when any new events are created. People generally make up field names without looking at the catalogue, they use an already assigned name for something different, they put them in the wrong order, they have dangling words instead of following name=value, they use the wrong event type, they might not put enough information in the event, or they put fields in the wrong order. All these issues are caught and fixed by review. There are more pressing needs on the user space side of this. reordering fields means I have to drop all my current plans and redo something that's been working fine. This is why it takes so long to get audit utilities and reports that are fast, understandable, and the right information. The problem at hand is Richard wants to make 2 new events. He wants to know what goes in them. We can make a function that pulls the right information together and add it to his new event. The immediate problem is solved. We shouldn't be doing much modifying of records. They really should be pretty static unless requirements change. For new records, there is no guarantee that the function created before is the right information for the new event. It just depends on what it is as to what needs to be collected. Then due to all the misused fields, we still need to have review to make sure there's no typo. Speaking of which, I just found a typo in AUDIT_FEATURE_CHANGE events. That is just some kernel issues off the top of my head. Things come up all the time. Most of the time things are found because someone is asking questions or I am trying to make sense of the audit trail. As for user space tools, yes there is a TODO file in the audit tarball. I don't know if the kernel maintainers have a TODO list published anywhere. -Steve

On 10/22/2014 03:06 PM, Paul Moore wrote: OK; I swear if you change this I'm going to parse EVERY field straight into a SQLite file first, since I'd have to go change code anyway. :-) I have code which is based on the examples, from years back, which believe there is order. It can be changed if needed; rather not but could. I suspect there are others... LCB -- LC (Lenny) Bruzenak lenny(a)magitekltd.com

On 10/22/2014 03:44 PM, Paul Moore wrote: I have no doubt my old code looks like Steve's first example, not the second. But as I said, code can be changed if the assumptions about ordering are thrown out. You're making a pretty big splash over here Paul! Very impressive... :-) LCB -- LC (Lenny) Bruzenak lenny(a)magitekltd.com

On 10/22/2014 04:29 PM, Paul Moore wrote: I hear you. But there's working and there's working well. As long as we don't suffer a search response degradation by changing the assumptive order, as I said, I'm OK with going back and reworking code. If it makes searching real data unusable, it's now broken some operational stuff. As it is, I already have to move the last 24 hours off to a different area and make searching go day-by-day only. Otherwise the logs are too big. I chose a 100MB file size a while back as a sweet spot based on the observation that each file could be parsed in some time acceptable to security managers. Just some info on the amount of data I've seen in real world examples: On a normally-busy site, each day, we will generate ~1-4GB of events. It used to be much more, but our team has managed to keep it down. I'd say that often more time is spent reducing the events than analyzing them. If not, they grow to a size you simply cannot parse in people-time. I did have way more, but we've become better adept at spotting and preventing event storms. Here is a current test machine's audit log directory. I have to go look to see what went wrong, obviously something did, but this kind of thing can and does happen in the real world. Note the time stamps. 100MB every minute or two is moving right along. We push all audit data off to a collector machine so it doesn't dominate the disks or otherwise hog the business server resources. BTW - funny that I just happened to login to this machine to check its status. Whatever was running amok stopped soon after 00:57. This sometimes just happens. I've seen this go on for hours more in fielded systems. Almost always something is really wrong, but the system is supposed to maintain security and be usable through these times as well. [root@audit audit]# ls -alt total 4179016 -rw-------. 1 root root 84835735 Oct 23 02:49 audit.log drwxr-x---. 2 root root 4096 Oct 23 00:57 . -r--------. 1 root root 104857789 Oct 23 00:57 audit.log.1 -r--------. 1 root root 104857663 Oct 23 00:56 audit.log.2 -r--------. 1 root root 104857639 Oct 23 00:55 audit.log.3 -r--------. 1 root root 104857608 Oct 23 00:54 audit.log.4 -r--------. 1 root root 104857874 Oct 23 00:52 audit.log.5 -r--------. 1 root root 104857864 Oct 23 00:51 audit.log.6 -r--------. 1 root root 104857670 Oct 23 00:50 audit.log.7 -r--------. 1 root root 104857740 Oct 23 00:49 audit.log.8 -r--------. 1 root root 104857644 Oct 23 00:48 audit.log.9 -r--------. 1 root root 104857871 Oct 23 00:47 audit.log.10 -r--------. 1 root root 104857602 Oct 23 00:46 audit.log.11 -r--------. 1 root root 104857755 Oct 23 00:45 audit.log.12 -r--------. 1 root root 104857907 Oct 23 00:44 audit.log.13 -r--------. 1 root root 104857973 Oct 23 00:43 audit.log.14 -r--------. 1 root root 104857632 Oct 23 00:42 audit.log.15 -r--------. 1 root root 104857843 Oct 23 00:42 audit.log.16 -r--------. 1 root root 104857769 Oct 23 00:41 audit.log.17 -r--------. 1 root root 104857864 Oct 23 00:40 audit.log.18 -r--------. 1 root root 104857862 Oct 23 00:39 audit.log.19 -r--------. 1 root root 104857757 Oct 23 00:38 audit.log.20 -r--------. 1 root root 104857601 Oct 23 00:36 audit.log.21 -r--------. 1 root root 104857663 Oct 23 00:35 audit.log.22 -r--------. 1 root root 104857711 Oct 23 00:34 audit.log.23 -r--------. 1 root root 104857806 Oct 23 00:33 audit.log.24 -r--------. 1 root root 104857722 Oct 23 00:32 audit.log.25 -r--------. 1 root root 104857684 Oct 23 00:31 audit.log.26 -r--------. 1 root root 104857761 Oct 23 00:30 audit.log.27 -r--------. 1 root root 104857975 Oct 23 00:29 audit.log.28 -r--------. 1 root root 104857702 Oct 23 00:28 audit.log.29 -r--------. 1 root root 104857771 Oct 23 00:27 audit.log.30 -r--------. 1 root root 104857990 Oct 23 00:25 audit.log.31 -r--------. 1 root root 104857629 Oct 23 00:24 audit.log.32 -r--------. 1 root root 104857663 Oct 23 00:23 audit.log.33 -r--------. 1 root root 104857852 Oct 23 00:22 audit.log.34 -r--------. 1 root root 104857841 Oct 23 00:21 audit.log.35 -r--------. 1 root root 104857634 Oct 23 00:20 audit.log.36 -r--------. 1 root root 104857690 Oct 23 00:19 audit.log.37 -r--------. 1 root root 104857623 Oct 23 00:18 audit.log.38 -r--------. 1 root root 104857711 Oct 23 00:17 audit.log.39 -r--------. 1 root root 104857867 Oct 23 00:02 audit.log.40 I started an "aureport -i --summary". I realize my aureport and ausearch don't use the auparse library, but they will soon, if not already, in the newer code. I just wanted to give you a feel for data amounts in your considerations. Also I have designs for some new tools I want to deploy which definitely will use libauparse, if it's usable. At 4GB+ of audit data, this report isn't returning yet. The nice thing is it isn't leaking memory, or if it is, it isn't gushing (from "top"): 31485 root 20 0 217m 104m 89m R 99.7 0.4 22:29.91 aureport So I guess from a real user perspective I'm just worried that fixing an order dependency will have adverse effects on retrieval. So my vote would be to promote anything which makes searching/parsing faster. Or we need a different paradigm entirely. I've been doing some other stuff while writing this. The aureport isn't done after 37+ minutes: 31485 root 20 0 220m 96m 78m R 100.0 0.4 37:22.38 aureport I realize if I were sitting inside a google infrastructure this would be different, but I'm not. So when guys try to understand what happened during, in this case, the first hour of the day, the time it takes to return the answer can make them think it's broken. This is why to me, speed of parse/retrieval is important. The aureport is still cranking: 31485 root 20 0 221m 76m 57m R 100.0 0.3 44:21.15 aureport Now if a few queries happen at the same time, and they can, we start seeing some real work on this machine. It's a reasonable machine - dual quad core Xeon with 24GB of RAM - dedicated to storage and retrieval of system audit data (from "cat /proc/cpuinfo"): ... processor : 7 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU X5677 @ 3.47GHz stepping : 2 cpu MHz : 1600.000 cache size : 12288 KB So hopefully this gives you a sense of the way the audit data can stack up; to me every millisecond/event saved is potentially hours earned. Report is still running btw: 31485 root 20 0 222m 103m 83m R 99.9 0.4 56:18.64 aureport From what I've read on the list, there are other users who have similar loads. I'm not sure if this makes any real difference in your thought process or not, and you probably knew most of this anyway. Just wanted to throw out some data points for consideration in case they matter. I waited until this morning to send this so I could see how long the aureport took. [root@audit audit]# time aureport -i --summary Summary Report ====================== Range of time in logs: 10/23/2014 00:01:02.305 - 10/23/2014 09:58:40.745 Selected time for report: 10/23/2014 00:01:02 - 10/23/2014 09:58:40.745 Number of changes in configuration: 2892 Number of changes to accounts, groups, or roles: 0 Number of logins: 47 Number of failed logins: 2 Number of authentications: 2712 Number of failed authentications: 355 Number of users: 8 Number of terminals: 41 Number of host names: 7 Number of executables: 112 Number of files: 536772 Number of AVC's: 5791 Number of MAC events: 3045 Number of failed syscalls: 2502700 Number of anomaly events: 525 Number of responses to anomaly events: 0 Number of crypto events: 14 Number of keys: 36 Number of process IDs: 30494 Number of events: 8526658 real 428m11.405s user 427m21.833s sys 0m4.014s Thank you! LCB -- LC (Lenny) Bruzenak lenny(a)magitekltd.com

On Wednesday, October 22, 2014 04:06:47 PM Paul Moore wrote: Except you can have problems when the event is like this auid= pid= old uid= new uid= res= and yes there are places like that. The performance really is the main issue. kprintf is the same speed no matter what the order. There just isn't a need to be altering events. We've had to add a few things here and there, but its only been a couple changes. What events do people need to change and why? There's not been any discussion that I know of saying we need to add fields or change them around. Yes. Many of the utilities. ausyscall & autrace might be the only ones not affected. People have been asking for 1) compressed logs 2) auparse needs to untangle interlaced events 3) people want to suppress certain records or events to save disk space 4) we need to handle federated IDs 5) we need to enrich events on the fly so that uids are preserved 6) we need a validation suite to ensure that user space apps are generating correct events (lightdm for example is not audit aware meaning many desktops can't be analyzed) 7) we need to get better at handling vast quantities of logs 8) we need some plugin for logstash 9) we need to allow people to format events their way 10) we need easier to understand reports 11) we need to be able to compare in kernel and on disk audit rules 12) auvirt is very broken and in need of rewrite 13) bash tab completions might be helpful 14) eventually combine audispd and auditd into 1 process. There's more than that. But it would be nice of people using audit and its tools also say what they are needing. What field ordering problem is preventing kernel work? That would be nice. -Steve

On Wednesday, October 22, 2014 05:00:03 PM Paul Moore wrote: You'll never get new uid which is really the one you want. I thought it was a question of what to put in an event. As for ordering all you have to do is check the event with ausearch-test to see if its well formed. He's making a new event. Its not changing things around. Exactly the opposite, those are about the only ones clean because they are the only ones not parsing logs. Not the daemon or library directly for this. But if you want to look into this, you'll need some really big logs for testing. You'll need at least 100MB to see performance variations. If we can keep performance reasonably close, I'd take patches. I know it will be slower. Well, its this TODO list that makes it a bad time for me to even consider re- doing something that is working fine. We can't have nice things because we keep mucking in the bottom layers. I thought there was some agreement to write a function and use it. User space doesn't need re-writing for that. -Steve

On 14/10/22, Steve Grubb wrote: And this is the type of thing that needs to be cleaned up, disambiguating which field you actually want. Both "old" and "new" are orphaned keywords that you have indicated are ignored by the parser, so should be cleaned up to old_uid= and uid=, according to the rules you have set out. - RGB -- Richard Guy Briggs <rbriggs(a)redhat.com&gt; Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat Remote, Ottawa, Canada Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545

On Thursday, October 30, 2014 10:48:28 AM Richard Guy Briggs wrote: You need to start feature= with a space. For example, see how it gets appended to subj=: time->Mon Oct 27 16:11:21 2014 type=FEATURE_CHANGE msg=audit(1414440681.713:17710): ppid=13599 pid=13618 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="auditctl" exe="/usr/sbin/auditctl" subj=system_u:system_r:auditctl_t:s0feature=loginuid_immutable old=0 new=1 old_lock=0 new_lock=1 res=1 -Steve

On 14/10/22, Steve Grubb wrote: Good. Where do we start? Alphabetical order seems like an obvious but not very useful order. How about an order based on classes of fields and importance or length of data within them? Start with who (subject), did what (verb) to what (object) with what result. Within each of those, have a standard order. Can that go in the "Audit Event Parsing Library Specifications"? There is also the standardization of field keywords that has already had some action/correction. - RGB -- Richard Guy Briggs <rbriggs(a)redhat.com&gt; Senior Software Engineer, Kernel Security, AMER ENG Base Operating Systems, Red Hat Remote, Ottawa, Canada Voice: +1.647.777.2635, Internal: (81) 32635, Alt: +1.613.693.0684x3545