Hi, In working on the IDS piece of the audit system, there are 3 detections that I need to solve: watched files, watched execution, and making an executable. In this particular case, an admin may have watches and several things for archival purposes, but may want an alert if something changes in specific files. In order for the IDS system to be able to distinguish an open of a watched file from an open of a *special* watched file that an alert should be sent for, I'd like to propose a standard way of alerting the IDS that this record needs additional scrutiny. The key field has been in all syscall records for at least 8 kernel releases, so I think it is well established that it could be used by this point. It is 32 char's long. This is longer than what most people will ever need. What I'd like to do is to use this field for 2 purposes. One is to continue using it as is. The other could be to tell the IDS that this record needs review.

If a key field starts with "ids-" then we could parse it for a specific set of uses. We need to be able to distinguish: file, exec, and mkexe. Each of these would alert the IDS as to which alert category this fits under. Next, IDMEF has 4 severity ratings: info, low, medium, and high. These could be shortened to: inf, low, med, hi. This means you could tell the IDS that this is an execution attempt that means medium severity by this, "ids-exec-med". To continue to be able to use the key field for searching, I would propose just adding a comma to separate the ids part from the searchable part. So, you could do something like this: "ids-file-med,shadow-password". auditctl can be fixed so that when it spots "ids-" in the key field, it will check that only valid types and severities are being used. This would not affect what auditd does in any way. Thoughts? Comments?

-Steve -- Linux-audit mailing list Linux-audit(a)redhat.com https://www.redhat.com/mailman/listinfo/linux-audit

On Wednesday 19 March 2008 13:12:22 Linda Knippers wrote: > Rather than using the key for two purposes and introducing special key > words, couldn't an admin just tell the IDS which he's are of interest? > And what the priority of each one is? The problem is that you can tell the IDS that you want any reads of /opt/my-secrets, but unless you have a matching audit rule you will not get any records. This allows you to make sure you have a watch paired with its meaning.

However, *no* amount of special tagging will allow the IDS to disambiguate these two cases: 1) An audit rule was set, but no events generated because no activity matched.

2) An audit rule wasn't set at all.

"unless you have a matching audit rule you will not get any records" means exactly that - so tagging the records you don't receive isn't useful.

There *is* the more general case of "I had a generic rule and a special watch and *both* fired" - but that problem is in no way IDS specific,

files. In order for the IDS system to be able to distinguish an open of a watched file from an open of a *special* watched file that an alert should be sent for, I'd like to propose a standard way of alerting the IDS that this record needs additional scrutiny.

On Wednesday 19 March 2008 14:05:42 Valdis.Kletnieks(a)vt.edu wrote: > On Wed, 19 Mar 2008 13:02:48 EDT, Steve Grubb said: >> files. In order for the IDS system to be able to distinguish an open of a >> watched file from an open of a *special* watched file that an alert >> should be sent for, I'd like to propose a standard way of alerting the >> IDS that this record needs additional scrutiny. > Why do we need special handling for something the IDS should be able to do > for itself? Something has to tell the IDS what to do for these 3 particular detections. It could come from a configuration file that it reads, or it could come from the event stream that it reads. Its just a matter of *where* the instructions come from. > If your IDS system doesn't already have a copy of the list of > "special" watched files, you have *bigger* problems. Not really, just think about the advantages of this approach. o You do not need to express host:file relationships if you are checking aggregate logs o You always have a matching audit rule to make sure you get the data you need o The event when recorded to disk has the meaning associated with it in case you wonder why something didn't get classified the way you thought it should o This technique is higher performance since you do not need iterate across a list of all files for each incoming event. o If the target file has a hardlink that is manipulated, the IDS won't be fooled because a different path showed up in the event stream. (This might even come into play for mount tricks that alter paths.) These are the easy ones that I can think up easily. While we are at it, the disadvantages to using the IDS configuration file approach: o The config file will become a mess when you get to this one entry that contains all file names one after another. Or you will have 2 config files one for the options and one for the files. You will of course need 2 more files for the other 2 detections, so now you have 4 config files to setup. o No guarantee that any audit rules exist to give you the events you need. o Lower performance o Uses more system memory o Susceptible to path name tricks o Code will be far more complicated and harder to read or understand due to size. o You have to be very careful to keep aggregate server rules in sync with remote system.

-Steve -- Linux-audit mailing list Linux-audit(a)redhat.com https://www.redhat.com/mailman/listinfo/linux-audit

On Wednesday 19 March 2008 15:04:57 Linda Knippers wrote: > I'm not sure why all of the above apply. Because this IDS is part of the audit system. > If an IDS has a dependency on audit and specific audit rules to get the > information it needs, it can use the information in its config file to > construct the audit rules it needs. Then you surely have duplicate rules controlled by 2 systems. The first rule in the audit.rules file is -D which would delete not only the audit event rules for archival purposes, but any IDS placed rules. There is not a simple way of deleting the rules placed by auditctl vs the ones placed by the IDS. The IDS system would also need to be prodded to reload its set of rules again.

On Wednesday 19 March 2008 15:04:57 Linda Knippers wrote: > I'm not sure why all of the above apply. Because this IDS is part of the audit system.

> If an IDS has a dependency on audit and specific audit rules to get the > information it needs, it can use the information in its config file to > construct the audit rules it needs. Then you surely have duplicate rules controlled by 2 systems. The first rule in the audit.rules file is -D which would delete not only the audit event rules for archival purposes, but any IDS placed rules. There is not a simple way of deleting the rules placed by auditctl vs the ones placed by the IDS. The IDS system would also need to be prodded to reload its set of rules again.

> I don't think an IDS config file needs to be any more complicated than an > audit rules, and in fact should be simpler. I think it would be more complicated going down this path for a number of reasons.

-Steve

On Wednesday 19 March 2008 15:55:23 Linda Knippers wrote: >> Because this IDS is part of the audit system. > Is there something that describes what you're building so we can > have the right context to comment on this? I presented this: http://people.redhat.com/sgrubb/audit/summit07_audit_ids.odp at last year's Red Hat Summit. The idea is roughly the same, but the configuration is slightly different.

> I assumed you were building something that would be a dispatcher plug-in or > something rather than building something new into the core audit subsystem. It is. Its tightly integrated.

>>> If an IDS has a dependency on audit and specific audit rules to get the >>> information it needs, it can use the information in its config file to >>> construct the audit rules it needs. >> Then you surely have duplicate rules controlled by 2 systems. The first >> rule in the audit.rules file is -D which would delete not only the audit >> event rules for archival purposes, but any IDS placed rules. There is not >> a simple way of deleting the rules placed by auditctl vs the ones placed >> by the IDS. The IDS system would also need to be prodded to reload its >> set of rules again. > An IDS should be able to be prodded to reload its rules. Sure, but you have the audit system loading CAPP rules with all those watches and then maybe the admin wants any write to shadow to be a high alert since he's the only user and won't be changing his password. We would either need to analyze the rules and make sure they are simplistic enough for the IDS to be guaranteed an event, or just add more rules to make sure we got 'em. In this case, we may windup creating records that the admin did not want on the disk.

i just see this as progressing into a mess. We have 2 things that have different ideas about what needs to be tracked. Neither understanding why the other is doing something and not happy because either too much data is going to disk or not enough events to trap something important.

By using the key field, its in plain sight and done with purpose. Not enough events to trap something important, widen it in audit.rules and you also know that this will send more to disk. No suprises there.

-Steve

So maybe all we need is for the ids config file needs to be of the form key type priority

so I can set up my audit rule however I want say -a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F dir=/etc -k 500EPERM -a always,exit -F perms=wa -F subj_role=webadmin_r -F exit=-EPERM -k webadminEPERM And my ids config file would look like: 500EPERM        file    med webadminEPERM   exec    high

And on startup the ids can easily look to see if 500EPERM and webadminEPERM are actually keys to real rules just for sanity sake.  

Is the reverse mapping from key to ids action really so expensive that this is unreasonable?

I tend to also agree with the part of the discussion which says that it isn't audit's place to decide that some rules are meant for disk and some rules aren't.

Steve Grubb wrote: > On Wednesday 19 March 2008 17:41:07 Eric Paris wrote: >> So maybe all we need is for the ids config file needs to be of the form >> >> key type priority > > And hostname. Remember that this could be run from an aggregator. > >> so I can set up my audit rule however I want say >> >> -a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F dir=/etc -k >> 500EPERM -a always,exit -F perms=wa -F subj_role=webadmin_r -F >> exit=-EPERM -k webadminEPERM >> >> And my ids config file would look like: >> >> 500EPERM file med >> webadminEPERM exec high > > This is pretty close to the idea that I started with. Then I thought, how > do I make this engine run faster? How do I reduce memory consumption > (since the keys have to be stored in memory)? How does overloading the key field help either of those?

> How do I make sure that the keys are there and correct? That can be a startup check.

>> Is the reverse mapping from key to ids action really so expensive that >> this is unreasonable? > > Consider a datacenter with many hosts that may want to run this off of > the aggregator. There will be a high rate of incoming events and a bit of > comparing to figure out if each event something we care about. I think if someone is using audit as an IDS, they're going to care about everything they're auditing or why are they auditing it?

> With my proposal, I can tell with strncmp(key, "ids-", 4) if this is > anything we need to pay attention to. So, inspection of 4 bytes let me > decide yes/no. Who is doing this, the auditd, the dispatcher or the plugin?

Couldn't you hash the key?

> Its a finite amount of time and doesn't linearly slow down the system as > more hosts and files of interest are configured. It scales well. > >> I tend to also agree with the part of the discussion which says that it >> isn't audit's place to decide that some rules are meant for disk and >> some rules aren't. > > I agree and never proposed that. I guess Eric and I were both confused then by your comment about the admin ending up with more audit events on disk than intended.

Steve Grubb wrote: > I wouldn't call this overloading the key field. The key field's purpose > is to allow searches for groups of audit events. You've traditionally > used this from the command prompt with ausearch. Now with the auparse > library, it can be used programmatically for searching by more utilities. > > The way that it helps is that I can tell yes/no in 4 byte compares and > without storing any linked lists, hashes, or binary trees. This optimization for a user-space component doesn't seem worth the hack. You are overloading the field

> How do you do that remotely? Same way you would check that audit is running remotely. You can either check the state of the target system or not,

>> I guess Eric and I were both confused then by your comment about the >> admin ending up with more audit events on disk than intended. > > -a always,exit -F perms=wa -F auid>=500 -F exit=-EPERM -F > path=/etc/password > > and > > -w /etc/password > > collect entirely different amounts of data. So the admin would create the rule he wants, create a unique key and tell the IDS what the key is and how it should react when it sees a record with that key.

> > So the admin would create the rule he wants, create a unique key and > > tell the IDS what the key is and how it should react when it sees a > > record with that key. > > Yes. What I want to do is also say that if you create your unique key a > certain way, you get this additional benefit of realtime alerting or > correlation. if you choose not to, then its treated as any other event I agree with Steve's compromise with speed, but maybe there's no way out of using hashes or linked lists in the general case. What if a message is important not only for the IDS plugin but also to a fictional 'real-time compliance reporting' plugin - both wanting to use the key field to carry special things, and in the same event: type=USER_ACCT msg=... key=ids-file-high,sox-fault-med,actual_key

The plug-ins would need to check if their specific identifier is present not only in the first 4 bytes, but after every comma.

If it's desirable to support the general case, instead of putting everything in one single 'key' field, maybe having an index just like execve arguments: type=USER_ACCT msg=... key[0]=ids-file-high key[1]=sox-fault-med key[2]=actual_key

Still need to iterate through all keys in the worst case, but the plugins could individually chose between having the rules hardcoded (for speed) or configurable.

On Fri, 2008-03-21 at 08:50 -0400, Steve Grubb wrote: > On Friday 21 March 2008 06:28:42 Klaus Heinrich Kiwi wrote: > > If it's desirable to support the general case, instead of putting > > everything in one single 'key' field, maybe having an index just like > > execve arguments: > > type=USER_ACCT msg=... key[0]=ids-file-high key[1]=sox-fault-med > > key[2]=actual_key > > That's a thought. In a way, I'd rather stay in one key field so that we > have a path forward right now. It won't complicate any existing kernel > code. Besides, doing any kernel change means waiting about 3-4 months for > 2.6.26 to be released. If we work out a standard that is backwards > compatible, it should work all the way back to about 2.6.16. I prefer this approach - NOT putting more than 1 key item in the key field. I see order issues.

My first thought was to overload the key field based on the event. For IDS events one would specify "-K" (for example) and the IDS triple Steve proposed as appropriate in the 31-byte text area. For another plugin need, choose a different constant - "-I" - or whatever.

But the important part to me is that the auditctl take care of any ordering issues, rather than faulty people.

> My first thought was to overload the key field based on the > event. For IDS events one would specify "-K" (for example) and the IDS > triple Steve proposed as appropriate in the 31-byte text area. For > another plugin need, choose a different constant - "-I" - or whatever. I'd rather treat this like the -S option where it can be given multiple times if we go this route. Given the code in the kernel, having multiple key fields will require some significant patching.

> But the important part to me is that the auditctl take care of any > ordering issues, rather than faulty people. I could even fix auditctl to allow multiple -k fields, but glue them together with commas if that were helpful. I could event fix auditctl to split them back out for rule listing purposes. We could also fix auparse to be able to do the splitting in the key field too so that this paradigm is supported and enforced by the whole toolchain. So, I could give the illusion of multiple key fields but without making any drastic kernel changes. Would this be acceptable?