1:22 p.m.
Steve G wrote:
> If you have to include code for parsing the current format, why
the rush
> to change the kernel output?
>
I was thinking that it should be done in near future so its not forgotten. But
that is the only reason. It could be delayed for a while.
But back to the original question, any preference for non-conflicting names? :)
CC'ing linux-audit.
Some comments regarding userspace object managers and the userspace AVC:
in general userspace object managers will introduce new fields to the
AVC messages. For example the AVC's generated by the X server have
fields such as window=, property=, and extension= for X-specific things
which do not appear in the kernel AVC's. So it should be relatively
easy to add new keywords to the dictionary, or even have the audit
system gracefully accept keywords that are not in its dictionary.
If all of these keywords in the data dictionary have to be unique, I'm
wondering if it might be useful to use a 3-tuple instead of a
(name,value) pair. The 3-tuple would consist of (namespace,name,value)
with namespace coming from a defined list of subsystems. So for example
there would be an "SELinux" namespace encompassing all of the selinux
keywords, so that the "result" and "perms" keywords from the previous
example would not conflict with the "other" ones which would presumably
be in a different namespace. Or just prefix the names with "selinux-",
"syscall-", etc.
Another request I have is that if this scheme moves forward, that we do
away with the audit_log_user_avc_message(3) family of functions and
instead introduce an interface that takes an array of name/value pairs
making up the audit message, or a single string which it could parse as
name/value pairs. This would be one-size-fits-all and would avoid the
10+ parameter situation with the current functions.
--
Eamon Walsh <ewalsh(a)tycho.nsa.gov>
National Security Agency
1:43 p.m.
New subject: missing avc message field names
Eamon Walsh wrote:
Steve G wrote:
>> If you have to include code for parsing the current format, why the
>> rush to change the kernel output?
>>
>
> I was thinking that it should be done in near future so its not
> forgotten. But
> that is the only reason. It could be delayed for a while.
>
> But back to the original question, any preference for non-conflicting
> names? :)
>
>
>
CC'ing linux-audit.
Some comments regarding userspace object managers and the userspace AVC:
in general userspace object managers will introduce new fields to the
AVC messages. For example the AVC's generated by the X server have
fields such as window=, property=, and extension= for X-specific things
which do not appear in the kernel AVC's. So it should be relatively
easy to add new keywords to the dictionary, or even have the audit
system gracefully accept keywords that are not in its dictionary.
Good point - as long as there are userspace generated audit messages it
will be hard to enforce uniqueness.
If all of these keywords in the data dictionary have to be unique,
I'm
wondering if it might be useful to use a 3-tuple instead of a
(name,value) pair. The 3-tuple would consist of (namespace,name,value)
with namespace coming from a defined list of subsystems. So for example
there would be an "SELinux" namespace encompassing all of the selinux
keywords, so that the "result" and "perms" keywords from the previous
example would not conflict with the "other" ones which would presumably
be in a different namespace. Or just prefix the names with "selinux-",
"syscall-", etc.
The prefixes seems simpler and match with the current audit messages
more closely.
Karl
2:07 p.m.
New subject: missing avc message field names
Karl MacMillan wrote:
Eamon Walsh wrote:
> Steve G wrote:
>
>>> If you have to include code for parsing the current format, why the
>>> rush to change the kernel output?
>>>
>>>
>> I was thinking that it should be done in near future so its not
>> forgotten. But
>> that is the only reason. It could be delayed for a while.
>>
>> But back to the original question, any preference for non-conflicting
>> names? :)
>>
>>
>>
>>
> CC'ing linux-audit.
>
> Some comments regarding userspace object managers and the userspace AVC:
> in general userspace object managers will introduce new fields to the
> AVC messages. For example the AVC's generated by the X server have
> fields such as window=, property=, and extension= for X-specific things
> which do not appear in the kernel AVC's. So it should be relatively
> easy to add new keywords to the dictionary, or even have the audit
> system gracefully accept keywords that are not in its dictionary.
>
>
Good point - as long as there are userspace generated audit messages it
will be hard to enforce uniqueness.
> If all of these keywords in the data dictionary have to be unique, I'm
> wondering if it might be useful to use a 3-tuple instead of a
> (name,value) pair. The 3-tuple would consist of (namespace,name,value)
> with namespace coming from a defined list of subsystems. So for example
> there would be an "SELinux" namespace encompassing all of the selinux
> keywords, so that the "result" and "perms" keywords from the
previous
> example would not conflict with the "other" ones which would presumably
> be in a different namespace. Or just prefix the names with "selinux-",
> "syscall-", etc.
>
>
The prefixes seems simpler and match with the current audit messages
more closely.
How about "prefix.keyword=value", selinux.result=denied,
selinux.perms=foo,bar,baz.
--
Eamon Walsh <ewalsh(a)tycho.nsa.gov>
National Security Agency
2:56 p.m.
New subject: missing avc message field names
On Monday 29 January 2007 14:22, Eamon Walsh wrote:
Steve G wrote:
> But back to the original question, any preference for non-conflicting
> names? :)
CC'ing linux-audit.
Some comments regarding userspace object managers and the userspace AVC:
in general userspace object managers will introduce new fields to the
AVC messages.
Which brings up the question of does everything have to be an AVC message
type? If we ever want to have binary audit records, we need to think of these
audit messages as if they were database tables - each type normalized. This
is, I think, where we are ultimately heading.
For example the AVC's generated by the X server have fields such
as window=,
property=, and extension= for X-specific things which do not appear in the
kernel AVC's. So it should be relatively easy to add new keywords to the
dictionary, or even have the audit system gracefully accept keywords that
are not in its dictionary.
The parsing API has a iterator approach and makes no assumptions about what is
supposed to be there or not. So, it would accept new name/value pairs without
any problem.
If all of these keywords in the data dictionary have to be unique,
I'm
wondering if it might be useful to use a 3-tuple instead of a
(name,value) pair.
I just don't see audit growing that much bigger. Of course, I may be wrong.
The 3-tuple would consist of (namespace,name,value) with namespace
coming
from a defined list of subsystems. So for example there would be
an "SELinux" namespace encompassing all of the selinux keywords, so that
the "result" and "perms" keywords from the previous example would
not
conflict with the "other" ones which would presumably be in a different
namespace. Or just prefix the names with "selinux-", "syscall-",
etc.
That would add a lot to disk space consumption in the current logging format.
The real problem that we will have if there is a common name share among
different records but with different meaning, is when people do a search. You
should be able to say "I want all audit records with the perm field" and get
records that are of the right type. Or another example, "I want records with
perm == 0". The perm field in AVC's with get run through strtol which will
yield 0 and then you have a match for an AVC when it should have been file
related. We need unambiguous names to make searching work as expected.
Another request I have is that if this scheme moves forward, that we
do
away with the audit_log_user_avc_message(3) family of functions and
instead introduce an interface that takes an array of name/value pairs
making up the audit message, or a single string which it could parse as
name/value pairs. This would be one-size-fits-all and would avoid the
10+ parameter situation with the current functions.
The one function you speak of was designed specifically for a python program.
But otherwise the API is designed to enforce common format across a variety
of applications. I do not see this family of functions growing much bigger at
all. I think we have auditing in nearly all the trusted apps right now. But I
also want to point out that if we ever want to have binary audit records, we
would have to have a standard API that takes certain parameters and converts
them to their binary equivalents. To allow a free style API would mean that
it cannot be converted to binary format.
-Steve
3:16 p.m.
New subject: missing avc message field names
Steve Grubb wrote:
On Monday 29 January 2007 14:22, Eamon Walsh wrote:
> Steve G wrote:
>> But back to the original question, any preference for non-conflicting
>> names? :)
> CC'ing linux-audit.
>
> Some comments regarding userspace object managers and the userspace AVC:
> in general userspace object managers will introduce new fields to the
> AVC messages.
Which brings up the question of does everything have to be an AVC message
type?
I don't know that it is necessary, but it is certainly preferable. All
of these messages serve exactly the same purpose - splitting them would
be a pain for any app that wants to process them (not to mention dealing
with backwards compatibility).
For example, the policy generation tools that I am working on could
process these X messages because they are AVC messages and I only
require the core fields to be present (i.e., source / target types,
granted / denied, class, and permissions). If they were split out I
would a) have to process more message types and b) track which versions
of the audit subsystem contain which message types.
If we ever want to have binary audit records, we need to think of these
audit messages as if they were database tables - each type
normalized. This
is, I think, where we are ultimately heading.
What problems do binary audit records solve? I think that binary records
introduce significant problems and would need to have significant benefits.
The biggest issue, of course, is that it would prevent the use of any
tools that process the files as text (grep, tail, awk, seaudit,
setroubleshoot, etc., etc.). Some can be moved over to the audit
library, but it is still useful to use grep, tail, and other shell tools
to search logs. I know that you discourage that practice, but I believe
that it is (and will remain) useful.
A related problem is that rigidly enforcing the formatting forces
updates to the audit system for any slight change to the output.
Tracking those changes is going to be incredibly difficult for tools
that process audit data. The audit library will help, but won't
alleviate all problems (like the introduction of new record types).
Regular breakage of tools is just going to discourage adoption of the
audit system.
> For example the AVC's generated by the X server have fields
such as window=,
> property=, and extension= for X-specific things which do not appear in the
> kernel AVC's. So it should be relatively easy to add new keywords to the
> dictionary, or even have the audit system gracefully accept keywords that
> are not in its dictionary.
The parsing API has a iterator approach and makes no assumptions about what is
supposed to be there or not. So, it would accept new name/value pairs without
any problem.
Great.
> If all of these keywords in the data dictionary have to be
unique, I'm
> wondering if it might be useful to use a 3-tuple instead of a
> (name,value) pair.
I just don't see audit growing that much bigger. Of course, I may be wrong.
Why? I hope there will be a growing list of selinux object managers. And
what about third-party tools that are security critical?
> The 3-tuple would consist of (namespace,name,value) with
namespace coming
> from a defined list of subsystems. So for example there would be
> an "SELinux" namespace encompassing all of the selinux keywords, so that
> the "result" and "perms" keywords from the previous example would
not
> conflict with the "other" ones which would presumably be in a different
> namespace. Or just prefix the names with "selinux-", "syscall-",
etc.
That would add a lot to disk space consumption in the current logging format.
The real problem that we will have if there is a common name share among
different records but with different meaning, is when people do a search. You
should be able to say "I want all audit records with the perm field" and get
records that are of the right type. Or another example, "I want records with
perm == 0". The perm field in AVC's with get run through strtol which will
yield 0 and then you have a match for an AVC when it should have been file
related. We need unambiguous names to make searching work as expected.
I don't understand - Eamon is suggesting creating unique names across
all message types by introducing a prefix, which seems like what you are
saying is necessary. And prefixes don't have to take much room (e.g.,
sel instead of selinux-). The advantage to me is that it would be
natural when there are conflicts (e.g., the sel prefix on selperms would
let me know it would work for the selinux related message types).
Karl
4:49 p.m.
New subject: missing avc message field names
On Monday 29 January 2007 16:16, Karl MacMillan wrote:
Steve Grubb wrote:
> On Monday 29 January 2007 14:22, Eamon Walsh wrote:
>> Steve G wrote:
>>> But back to the original question, any preference for non-conflicting
>>> names? :)
>>
>> CC'ing linux-audit.
>>
>> Some comments regarding userspace object managers and the userspace AVC:
>> in general userspace object managers will introduce new fields to the
>> AVC messages.
>
> Which brings up the question of does everything have to be an AVC message
> type?
I don't know that it is necessary, but it is certainly preferable.
I'd also like to make sure that all AVCs really are AVCs. I think there are
some error messages that come out as AVCs when they should be SE_ERR or
USER_SE_ERR.
All of these messages serve exactly the same purpose - splitting them
would
be a pain for any app that wants to process them (not to mention dealing
with backwards compatibility).
You already have that problem. AUDIT_AVC and AUDIT_USER_AVC.
For example, the policy generation tools that I am working on could
process these X messages because they are AVC messages and I only
require the core fields to be present (i.e., source / target types,
granted / denied, class, and permissions). If they were split out I
would a) have to process more message types and b) track which versions
of the audit subsystem contain which message types.
You already have them split out...or did you totally skip AUDIT_USER_AVC? So,
you already have to solve the problem so why not generalize the solution? The
parsing API should be able to help with this. If you tell it to get all the
different types and they don't exist, you don't get a hit except the ones
that do exist.
If we ever want to have binary audit records, we need to think of
these
> audit messages as if they were database tables - each type normalized.
> This is, I think, where we are ultimately heading.
What problems do binary audit records solve?
Speed and compression. People doing serious auditing have very large files.
Compression is extremely important. Also when dealing with large files, any
performance trick you can find helps immensely.
The biggest issue, of course, is that it would prevent the use of
any
tools that process the files as text (grep, tail, awk, seaudit,
setroubleshoot, etc., etc.).
ausearch -m all --raw | grep anything you want
Some can be moved over to the audit library, but it is still useful
to use
grep, tail, and other shell tools to search logs. I know that you discourage
that practice, but I believe that it is (and will remain) useful.
What happens when the file format changes - zlib compression for example?
>> If all of these keywords in the data dictionary have to be
unique, I'm
>> wondering if it might be useful to use a 3-tuple instead of a
>> (name,value) pair.
>
> I just don't see audit growing that much bigger. Of course, I may be
> wrong.
Why? I hope there will be a growing list of selinux object managers
I doubt there very many security related pieces of information that is not
already in the dictionary. There are a finite number of security objects and
their attributes.
And what about third-party tools that are security critical?
They have the TRUSTED_APP message type and they can put anything in that they
want. I consider that one completely freestyle.
I don't understand - Eamon is suggesting creating unique names
across
all message types by introducing a prefix, which seems like what you are
saying is necessary.
Nearly all names are unique (and the problem should be solved when I code up
seperm and seresults). Adding more letters to the name eats the diskspace. We
only do it when necessary. That is what drives the choice of "res" instead
of "results" for the audit system.
-Steve
5:48 p.m.
New subject: missing avc message field names
>>> Steve G wrote:
>>>
I'd also like to make sure that all AVCs really are AVCs. I think there are
some error messages that come out as AVCs when they should be SE_ERR or
USER_SE_ERR.
This is a separate issue, this results from the userspace AVC's logging
callback not having a type field. I'm aware of this problem and willing
to fix it but I'd like to get a clear picture of what the final auditing
format will be first, so I'm glad this discussion is taking place.
You already have them split out...or did you totally skip
AUDIT_USER_AVC? So,
you already have to solve the problem so why not generalize the solution? The
parsing API should be able to help with this. If you tell it to get all the
different types and they don't exist, you don't get a hit except the ones
that do exist.
My point here was simply to warn in advance that userspace object
managers will in general introduce widely varying fields to their AVC
messages and that the auditing record format should be equipped to deal
with that. Who knows what types of objects will appear in these
messages in the future: there might be SELinux-enhanced e-mail clients,
office applications, file managers in the future. Making the dictionary
easily extensible now will likely pay off later and the prefix idea is a
way to do that.
As far as binary records go, if that is the future direction then I
don't see why the disk space argument is being made for not having a
prefix. Presumably in binary form the keywords will be assigned
numbers that are fixed size, so that the length of the string
representation shouldn't matter. And even in text form if the logs are
compressed then the length issue will be mediated.
>
I doubt there very many security related pieces of information that is not
already in the dictionary. There are a finite number of security objects and
their attributes.
Again I think that the audit system should support future expansion if
it's going to be the new way for handling AVC's and other security events.
> And what about third-party tools that are security critical?
>
They have the TRUSTED_APP message type and they can put anything in that they
want. I consider that one completely freestyle.
How does this work with binary records? How will these messages be
searched?
Nearly all names are unique (and the problem should be solved when I
code up
seperm and seresults). Adding more letters to the name eats the diskspace. We
only do it when necessary. That is what drives the choice of "res" instead
of "results" for the audit system.
I do think it should be selinux.perms, selinux.result and the other AVC
stuff should be selinux.foo. These are long names I concede but they
are clearer and log size is already an issue anyway, moving to
compressed or binary will solve the problem.
6:25 a.m.
New subject: missing avc message field names
On Tuesday 30 January 2007 10:48, Eamon Walsh <ewalsh(a)tycho.nsa.gov> wrote:
there might be SELinux-enhanced e-mail clients,
office applications, file managers in the future
Yes, we need all that.
There are some people interested in SE enhanced MUAs.
One issue is that SE-X is required for full functionality in this regard
(let's assume for the sake of discussion that almost everyone who matters
uses a GUI MUA). Another issue is that the design of MUAs is tending towards
greater integration with the desktop environment and larger more complex code
bases.
I'm thinking of starting to attack this by developing a password sequestration
system for MUAs. The idea being that the MUA would run a SETGID program and
request a POP connection, it would be returned a file handle for an
authenticated connection but have no way of obtaining the password that was
used. This will offer significant security benefits in a non-SE environment
and even better protection with SE Linux. A compromised MUA would not be
able to obtain a password list and send it to a hostile party (it would be
able to proxy access to the POP server and to send copies of all stored
messages). Given the incidence of passwords being used for multiple
functions this would significantly mitigate the risks of MUA based attacks.
The current situation is tending towards having an ever increasing amount of
the practical system integrity dependant on the integrity of a single user
account (in which all programs run with the same security context).
Getting upstream support for labelled email is going to be very difficult with
the current client side security situation.
Now if we could just get web browsers to have their functionality split into
multiple programs with different security contexts...
--
russell(a)coker.com.au
http://etbe.blogspot.com/ My Blog
http://www.coker.com.au/sponsorship.html Sponsoring Free Software development
8:49 a.m.
New subject: missing avc message field names
Steve Grubb wrote:
On Monday 29 January 2007 16:16, Karl MacMillan wrote:
> Steve Grubb wrote:
<snip>
> All of these messages serve exactly the same purpose - splitting
them would
> be a pain for any app that wants to process them (not to mention dealing
> with backwards compatibility).
You already have that problem. AUDIT_AVC and AUDIT_USER_AVC.
Yes, but adding even more message types makes the problem worse. More
importantly, what does splitting AVC messages even more accomplish? In
fact, I would rather that AUDIT_AVC and AUDIT_USER_AVC were not split.
> For example, the policy generation tools that I am working on
could
> process these X messages because they are AVC messages and I only
> require the core fields to be present (i.e., source / target types,
> granted / denied, class, and permissions). If they were split out I
> would a) have to process more message types and b) track which versions
> of the audit subsystem contain which message types.
You already have them split out...or did you totally skip AUDIT_USER_AVC? So,
you already have to solve the problem so why not generalize the solution? The
parsing API should be able to help with this. If you tell it to get all the
different types and they don't exist, you don't get a hit except the ones
that do exist.
You keep talking about the future - which doesn't yet exist.
So today, if you add a message type I would have to somehow determine
which audit version I was dealing with so that I didn't pass an invalid
record type to ausearch (because it bails). Definitely problematic.
Using the audit library would be better if it ignores invalid record
types, but it still requires tool updates.
> If we ever want to have binary audit records, we need to think
of these
>
>> audit messages as if they were database tables - each type normalized.
>> This is, I think, where we are ultimately heading.
> What problems do binary audit records solve?
Speed and compression. People doing serious auditing have very large files.
Compression is extremely important. Also when dealing with large files, any
performance trick you can find helps immensely.
gzip or bzip2 compression is not sufficient? Is audit searching
performance the most important consideration?
Look, I'm not against binary logs, I'm just trying to point out that I
think that the transition will not be as easy as you seem to suggest.
> The biggest issue, of course, is that it would prevent the use of
any
> tools that process the files as text (grep, tail, awk, seaudit,
> setroubleshoot, etc., etc.).
ausearch -m all --raw | grep anything you want
tail -f happens to be my favorite counter example, but I am certain
there are other useful tricks for monitoring logs that will break. Not
to mention the number of log monitoring and aggregation tools that
assume text logs.
> Some can be moved over to the audit library, but it is still
useful to use
> grep, tail, and other shell tools to search logs. I know that you discourage
> that practice, but I believe that it is (and will remain) useful.
What happens when the file format changes - zlib compression for example?
zcat :) Seriously, what you seem to be suggesting is that the presence
of the audit library will make it possible to arbitrarily change the
audit log format. I believe that the library is not sufficient -
changing the format will continue to cause breakage and frustration. I'm
concerned about that as someone that writes applications that are
consumers of the audit system.
>>> If all of these keywords in the data dictionary have to
be unique, I'm
>>> wondering if it might be useful to use a 3-tuple instead of a
>>> (name,value) pair.
>> I just don't see audit growing that much bigger. Of course, I may be
>> wrong.
> Why? I hope there will be a growing list of selinux object managers
I doubt there very many security related pieces of information that is not
already in the dictionary. There are a finite number of security objects and
their attributes.
Almost by definition a userspace object manager adds new security
critical objects. See Eamon's example.
> And what about third-party tools that are security critical?
They have the TRUSTED_APP message type and they can put anything in that they
want. I consider that one completely freestyle.
Why is it acceptable to have arbitrary name/value pairs in that message
type and not in others?
Karl
11:06 a.m.
New subject: missing avc message field names
Karl MacMillan wrote:
>> The biggest issue, of course, is that it would prevent the
use of any
>> tools that process the files as text (grep, tail, awk, seaudit,
>> setroubleshoot, etc., etc.).
>
> ausearch -m all --raw | grep anything you want
>
tail -f happens to be my favorite counter example, but I am certain
there are other useful tricks for monitoring logs that will break. Not
to mention the number of log monitoring and aggregation tools that
assume text logs.
This is fairly off topic here (selinux list) but I agree with Karl. As a
recovering admin I think I can say that admins expect to be able to use
various unix utilities to inspect log files, particularly tail -f. While
I'm all for applications putting their data in private data formats and
using tools and libraries to inspect them I think it is generally
considered that everything in /var/log is fair game to inspect with
anything available on systems (including perl, python, sed, awk, tail,
grep, etc).
You will certainly be rubbing most admins the wrong way by forcing them
through a different interface that won't support some common commands
like tail -f.
There are probably hundreds of utilities that look through these files
as well, what is going to happen when people try to add audit.log to a
log watcher that emails logs to them? Huge binary dumps in email are
going to make people turn off the audit daemon, not modify their apps to
use different tools/libraries.
11:28 a.m.
New subject: missing avc message field names
On Tue, 30 Jan 2007 12:06:06 EST, Joshua Brindle said:
This is fairly off topic here (selinux list) but I agree with Karl. As a
recovering admin I think I can say that admins expect to be able to use
various unix utilities to inspect log files, particularly tail -f.
As a counter-example - lastcomm and last.
If you want to use tail -f, don't run auditd, and use syslog-ng(*) or similar
to send the msgs you're interested in to a file that you can tail -f.
Or you *can* tail -f the file, just be ready to deal with the fact that it
contains binary data, same as the process accounting file and the last-login
file.
(*) syslog-ng can route to logfiles based on a regexp, so you don't have to
send all kernel output to the same file...
12:45 p.m.
New subject: missing avc message field names
--- Joshua Brindle <jbrindle(a)tresys.com> wrote:
This is fairly off topic here (selinux list) but I
agree with Karl. As a
recovering admin I think I can say that admins
expect to be able to use
various unix utilities to inspect log files,
particularly tail -f. While
I'm all for applications putting their data in
private data formats and
using tools and libraries to inspect them I think it
is generally
considered that everything in /var/log is fair game
to inspect with
anything available on systems (including perl,
python, sed, awk, tail,
grep, etc).
You will certainly be rubbing most admins the wrong
way by forcing them
through a different interface that won't support
some common commands
like tail -f.
There are probably hundreds of utilities that look
through these files
as well, what is going to happen when people try to
add audit.log to a
log watcher that emails logs to them? Huge binary
dumps in email are
going to make people turn off the audit daemon, not
modify their apps to
use different tools/libraries.
Based on the Unix experience I find myself
agreeing with this assessment. Binary (or
compressed) audit logs don't get read very
often. A mechanism like audit_filters(5) from
Irix makes the problem more manageable, but
the truth is that humans like their information
human readable. Disk space used to be a major
problem, and I/O bandwidth still is (you can
overwhelm any system with too much audit no
matter how optimal your audit data) but the
cost of translation-on-read is going to stop
most humans from ever doing it.
Casey Schaufler
casey(a)schaufler-ca.com
11:42 a.m.
New subject: missing avc message field names
On Tuesday 30 January 2007 09:49, Karl MacMillan wrote:
> You already have that problem. AUDIT_AVC and AUDIT_USER_AVC.
Yes, but adding even more message types makes the problem worse. More
importantly, what does splitting AVC messages even more accomplish?
It allows users to tune the audit system to the kinds of messages they want to
keep in their logs, makes analysis of the kinds of problems being reported
easier, and allows normalization of records for binary use.
In fact, I would rather that AUDIT_AVC and AUDIT_USER_AVC were not
split.
They have to be, ironically, because of SE Linux. :)
> Speed and compression. People doing serious auditing have very
large
> files. Compression is extremely important. Also when dealing with large
> files, any performance trick you can find helps immensely.
gzip or bzip2 compression is not sufficient?
No, its an intermediate solution on the way to a better more difficult
solution.
Is audit searching performance the most important consideration?
Storage size is the more important followed by speed. As soon as you allow
more records, they will have more to search.
Look, I'm not against binary logs, I'm just trying to point
out that I
think that the transition will not be as easy as you seem to suggest.
No, I'm sure it will be time consuming and done in phases. The first step is
providing the insulating layer so that apps do not have to care.
>> The biggest issue, of course, is that it would prevent the
use of any
>> tools that process the files as text (grep, tail, awk, seaudit,
>> setroubleshoot, etc., etc.).
>
> ausearch -m all --raw | grep anything you want
tail -f happens to be my favorite counter example, but I am certain
there are other useful tricks for monitoring logs that will break.
You can still use this on your system. What will be happening in the audit
system is that there will be choices as to how the information will be
stored. Text will still be an option. However, the portable way will be
ausearch based.
>> And what about third-party tools that are security
critical?
>
> They have the TRUSTED_APP message type and they can put anything in that
> they want. I consider that one completely freestyle.
Why is it acceptable to have arbitrary name/value pairs in that message
type and not in others?
Because by definition third party means we don't have the source to review or
fix.
-Steve
4:53 p.m.
New subject: missing avc message field names
On Tue, 2007-01-30 at 09:49 -0500, Karl MacMillan wrote:
Steve Grubb wrote:
> ausearch -m all --raw | grep anything you want
tail -f happens to be my favorite counter example, but I am certain
there are other useful tricks for monitoring logs that will break. Not
to mention the number of log monitoring and aggregation tools that
assume text logs.
To be fair the new audit dispatcher already has a plugin that does the
same thing as "tail -f" without needing to call stat(), and that'll be
released before auditd has binary logs ... although one could certainly
argue that it's not as obvious, it seems like a small price.
--
James Antill <jantill(a)redhat.com>
6:50 p.m.
New subject: missing avc message field names
James Antill wrote:
On Tue, 2007-01-30 at 09:49 -0500, Karl MacMillan wrote:
> Steve Grubb wrote:
>> ausearch -m all --raw | grep anything you want
> tail -f happens to be my favorite counter example, but I am certain
> there are other useful tricks for monitoring logs that will break. Not
> to mention the number of log monitoring and aggregation tools that
> assume text logs.
To be fair the new audit dispatcher already has a plugin that does the
same thing as "tail -f" without needing to call stat(), and that'll be
released before auditd has binary logs ... although one could certainly
argue that it's not as obvious, it seems like a small price.
So you will have that wheel reinvented soon - that still leaves many,
many more that you have no control over.
Karl
11:29 p.m.
New subject: missing avc message field names
Karl MacMillan wrote:
James Antill wrote:
> On Tue, 2007-01-30 at 09:49 -0500, Karl MacMillan wrote:
>> Steve Grubb wrote:
>>> ausearch -m all --raw | grep anything you want
>> tail -f happens to be my favorite counter example, but I am certain
>> there are other useful tricks for monitoring logs that will break.
>> Not to mention the number of log monitoring and aggregation tools
>> that assume text logs.
>
> To be fair the new audit dispatcher already has a plugin that does the
> same thing as "tail -f" without needing to call stat(), and that'll be
> released before auditd has binary logs ... although one could certainly
> argue that it's not as obvious, it seems like a small price.
>
So you will have that wheel reinvented soon - that still leaves many,
many more that you have no control over.
Even with a tail replacement there has to be thousands of internally
written and maintained log monitoring and reporting apps that will
break, this is a fundamental change in how logging works on linux, not
something that can or should be changed on a whim (or otherwise).
4:59 p.m.
New subject: missing avc message field names
On Wednesday 31 January 2007 16:29, Joshua Brindle <jbrindle(a)tresys.com>
wrote:
Even with a tail replacement there has to be thousands of internally
written and maintained log monitoring and reporting apps that will
break, this is a fundamental change in how logging works on linux, not
something that can or should be changed on a whim (or otherwise).
Most such programs assume that log files keep the same name until a cron job
renames them. The current practice of auditd rotating it's log files has
probably broken the majority of such programs already.
Also Steve Grubb suggested having a configuration option for plain-text files
which will avoid the problems with binary files.
If we work with the assumption that indexed log files are required for sites
with significant audit requirements due to the volume of logs and the need to
get responses in a reasonable amount of time then we have two options. One
is a binary format, the other is to have index files along-side the text
files.
Having separate index files introduces complications for renaming and other
file management (complexity is bad for reliability), even without the issue
of the sys-admin wanting to rename their own log files.
So it seems that the option of a binary log file is required.
Maybe there should be an option to have auditd write a binary log file as well
as either a text log file or logging via syslog? That way the admin could
have the index benefits of a binary log as well as having text files. If
there were two log files then the second copy wouldn't need to be written
synchronously so the IO load would not double.
--
russell(a)coker.com.au
http://etbe.blogspot.com/ My Blog
http://www.coker.com.au/sponsorship.html Sponsoring Free Software development
5:40 a.m.
New subject: missing avc message field names
On Wednesday 31 January 2007 17:59, Russell Coker wrote:
Maybe there should be an option to have auditd write a binary log
file as
well as either a text log file or logging via syslog?
This should be possible. The audit event dispatcher typically has had a
plugin that relays audit events to syslog. It will likely be a little while
before there are binary formatted logs. I guess my message is really that you
might not want to assume that the site will have text based logs for user
support. Text logs are not being deprecated. Its that there will be more
options soon.
-Steve
6533
days inactive
6536
days old
linux-audit@lists.linux-audit.osci.io
17 comments
9 participants
participants (9)
-
Casey Schaufler -
Eamon Walsh -
James Antill -
Joshua Brindle -
Karl MacMillan -
Karl MacMillan -
Russell Coker -
Steve Grubb -
Valdis.Kletnieks＠vt.edu